See http://math.nist.gov/~DPorter/tcltk/trofs/
Tcl package that provides commands to create, mount, and unmount archives containing read-only filesystems.
File format (version 1, as found in trofs-0.4.3)
(reconstructed from trofs-0.4.3/library/procs.tcl by PS)
The trofs archive file format is a concatenation of files bundled in directories, where each directory contains file, link and directory TOC entries which point to an absolute location in the archive file and specify the length of the item in the file. All offsets are from the start of the archive file. Each toc is a utf-8 encoded dict
<arbitrary header (optional)> 0x1a <file1 data> <file2 data> <file3 data> <dir1/file1 data> <dir1/file2 data> <dir1/dirA/file1 data> <dir1/dirA TOC> <dir1 TOC> <dir2/file1 data> <dir2 TOC> <root TOC> 0x1a trofs01 (literal string - trofs file signature) <size of root TOC (big endian 32-bit int)>
TOC:
file1 {F <length> <offset>}
file2 {F 15 32}
dir1 {D <size> <offset>}
dir2 {D 203 12542}
link1 {L <target>}
link2 {L ../file1}What is the reason for inventing Yet Another Archive Format And Making It Tcl Specific, and not using - say - ZIP? -jcw
PWQ 27 Mar 05, I would go one further. Since the FS is to be read only, why don't you adopt ext2-fs or dos-fs, or even cram-fs. The advantages of these is that you can mount these under linux and create/modify them easily. When you re-invent the wheel, it should be a better wheel.
Would makes no sense on Windows -jcw Jcw: Hence the dos-fs option. I would rephrase Windows makes no sense PWQ.
PS 27Mar05 The short answer I got from dgp was that it is open for discussion. The main reason for me writing this format down here is to familiarize myself with the trofs code. As far as I can tell, a lot of effort went into making the trofs code thread safe. The next step will be adding compression to trofs, and after that I am likely to hack it to read (a subset of) zipfiles. The significant difference between the zip file format and trofs file format is the directory structure. In zipfiles, a main TOC holds all files, trofs has a toc for each directory. I think the simplest trick will be to turn the zip TOC into a nested dict, that would preserve most of the trofs internals.
SEH 28Mar05 -- If you were to use the Debian .deb format, then you would get the benefit of re-use, and to boot every existing Debian package could be trivially converted to a Tcl module. Then Tcl could be used as a general-purpose software/install management tool, rather than simply a Tcl package tool. I dream of using Debian-style archives on Windows as well as Unix. That would be another opportunity to widen the appeal and use of Tcl in the computer-user community.
30Mar05 -- The tar virtual filesystem included in the Tclvfs package has pure-Tcl code for reading from standard tar files. So why not simply use the tar format for your filesystem and adapt the existing code, or better yet use the tar vfs?
DGP One of the main motivations was to write code that provided a Tcl_Filesystem that did not make use of the tclvfs package. This was in part to prove that the Tcl_Filesystem interface was usable by someone other than the person who wrote it, and in part to avoid some (perceived?) limitations imposed by the tclvfs package itself. In particular, I believe that the scheme tclvfs uses to determine what paths belong to what filesystem is entirely prefix based, which prevents nested mounts. trofs does not have that limitation. Also, because tclvfs defines filesystems in terms of callback scripts, there must be an interp in order to use tclvfs filesystems. trofs filesystems can remain mounted even after all interps have been destroyed. This suggests that with further work, trofs filesystems will be able to be mounted before any interp is created, and this will offer a solution to some of the tricky startup problems experience by programs like Tclkit that wish to access data from a virtual filesystem as part of interp initialization.
Regarding the archive format suggestions, I've never thought the archive format was the point. I actively don't care what the format is. Completely open to changing it, so long as doing so actually provides some benefit.
jcw - I appreciate the fact that trofs addresses startup issues. It's a most unfortunate that this was done using Yet More C Code, since compiled code prevents tinkering/evolution (see my a critical mindset about policy rant). None of this is likely to be a performance bottleneck if it were to be coded in Tcl. Is there no way to decouple the startup logic further so interps can be inited without depending on a file system, and then "injecting" some Tcl scripting logic to start things off?
As for the trofs format - isn't this turning the argument around a bit? There is a perfectly valid (ubiquitous, in fact) ZIP archive format and zipvfs driver, it is not clear to me how trofs improves on it (other than w.r.t. the current startup issues).
TclVFS could be improved further if it has limitations - AFAICT it is merely a thin reflect-to-tcl wrapper.
DGP Further decoupling and other improvements to Tcl's startup are better addressed elsewhere. Yes, that's in progress on the HEAD of Tcl. That said, if you want to evaluate Tcl code, you need a Tcl interp, so anything that must be done as part of creating a Tcl interp and preparing it to evaluate scripts must be written in C.
Turning the argument around -- yes, I'd agree if I already knew the zip format, and if I actively rejected it in favor of the trofs format instead. That's not the case. I needed an archive, and I lacked the patience to learn about any existing ones. The one that's in place now is pretty much a trivial copy of data structures to disk file. I did leave the door open to change it later, once I or someone else had the patience to make a better choice. Note that I didn't even bother to document the format; Pascal did that as an aid to his own modifications, and then (unwisely?) posted it here. Then the piling on began.
At one level TclVFS is a thin wrapper, but where I believe it acquires its limitations is in its C-coded part that implements the Tcl_Filesystem it provides. VfsPathInFilesystem imposes the "no nested mounts" constraint. No matter how clever each Tcl-coded client filesystem of tclvfs may be, the innards of tclvfs get in the way.
pcam 24/07/2007 I have just tried with ActiveState 8.5b8
package require trofs
Both tclsh and wish exit without any message. Surely this is a bug.
LV 2007 July 26 Because of this discussion, I reported a bug with the problem, and worked with the Activestate folks to track down the problem. ActiveTcl 8.4.15.x was fine, but the ActiveTcl 8.5.x with trofs 0.4.4 was broken. A coding change has been made and I'm testing now.
jevinskie - 2022-07-15 17:56:09
Howdy! I found trofs is used by Questa simulator and needed to modify some files to enable HiDPI support. One thing I noticed is that ToC offsets don’t appear to be from the start of the file but instead from the start of the parent ToC. I’ve ported trofs extraction and creation to Python if anyone is looking for a non-C version.