As of Tcl 8.4a3 (and in Tcl 8.4a4 heavily debugged), Tcl's filesystem is completely 'virtual filesystem aware'. What this means is that in principle you can divert ''all desired'' filesystem activity away from the native operating system and to something else. In practice what this means is that, given appropriate extensions, ordinary Tcl code can operate on remote files (on ftp sites or over a http connection) or inside archives (zip files, for instance), without realising that is what it is doing. (Historical note: the VFS idea for Tcl was originally built in pure Tcl by [Matt Newman] for use in [TclKit], and later re-introduced at the C level by [Vince Darley], as described in TIP #17). The nice thing about it being at the C level is that ''all'' Tcl extensions (including Tk) are automatically vfs-aware (or, more precisely, completely unaware that they are operating on non-native files). There are two known extensions which make use of this API (in addition to the native filesystem which itself uses the same API): ''testreporting filesystem'' This is part of Tcl's test code/suite. When registered it allows all filesystem activity to be reported upon (a brief message describing each filesystem access is printed to stderr). ---- ''tclvfs extension'' This is at http://sourceforge.net/projects/tclvfs/ and should compile easily on win/unix/macos. Most of the code is in fact in Tcl (there is one C file vfs.c) so you can easily extend it, add new vfs types, fix existing ones, etc; the C part simply exposes the vfs support in the core to Tcl). The tclvfs extensions is used by [tclkit]. The latest version of tclvfs contains a number of performance enhancements. By default it comes with vfs support for zip, ftp, http, mk4, ns filesystems. If you would like to add/use new filesystem types, you can do that in one of two ways: use 'tclvfs' and write the new filesystem in Tcl; or write a new filesystem in C (or indeed you could write/modify something like tclvfs which exposes the vfs to tcl in a different way). Examples: Take a look at the [One-line web browser in Tcl]. Using the tclvfs extension, you can now do things like this (after a ''package require vfs''): vfs::urltype::Mount ftp file copy ftp://foo.bar.com/pub/Readme . file copy ftp://user:password@foo.bar.com/private.txt . or vfs::urltype::Mount ftp set image [image create photo -file ftp://ftp.ucsd.edu/pub/alpha/tcl/alphatk/alphatk.gif] pack [label .b -image $image] or vfs::zip::Mount foo.zip foo.zip cd foo.zip ; glob * or vfs::urltype::Mount ftp set listing [glob -dir ftp://ftp.scriptics.com/pub *] or vfs::urltype::Mount http set fd [open http://sourceforge.net/projects/tcl] set contents [read $fd] ; close $fd or nested mounts: vfs::ftp::Mount ftp://ftp.ucsd.edu/pub pub vfs::zip::Mount pub/archive.zip pub/archive.zip % load a .dll from inside a .zip which sits on a remote ftp site. load pub/archive.zip/foo.dll (Caveat: I haven't actually tried this last one....) ---- Notice that you need to ''Mount'' non-native filesystems before you can use them. There are two kinds of mounts that tclvfs supports. The first kind is a particular mount point such as an archive 'foo.zip' or 'ftp://ftp.scriptics/com/pub' which can be mapped onto any point in the local filesystem. The contents of those mounts (i.e. the contents of the zip archive or the contents of the remote ftp site) are then accessible as normal, local files. The second kind of mount is a 'protocol' (somewhat clumsily called a 'urltype' in the current vfs library), which is illustrated in 3 of the examples above. Here we are effectively creating a new ''drive'' called 'ftp://' or 'http://', and any access within that ''drive'' causes the tclvfs library to attempt to perform a mount of the first kind so that the contents can be accessed. Please help develop the tclvfs package further! ---- '''Future Vfs Thoughts (i.e. not implemented in Tcl 8.4)!''' ''asynchronicity'' Asynchronous filesystem access would be useful, primarily for copying files, but also potentially other purposes: file copy -command progress http://a.b.c/foo . set fd [open http://a.b.c/foo w] puts $fd -command progress $megabytes load -command progress http://a.b.c/tk84.dll Note that both 'file rename', and 'load' may require cross-filesystem copies. ''WebDAV'' [[future VFS thoughts: BXXP? WebDAV?]] WebDAV would be great! It should be possible to write a WebDAV implementation entirely in Tcl using the tclvfs extension. Anyone have the knowledge/expertise to try this? I (''Vince'') can provide help and testing on the vfs side of things (i.e. explain what the 5-10 procedures are that need writing), but I don't know enough about http and its extensions. ''file locking'' Many asynchronous protocols or operations would benefit from some kind of locking mechanism. It might be good to extend the interface in the future with this. (Note that any vfs can provide its own 'file attributes' both readable and writable so in principle any other features can be supported through attributes using the existing vfs code). ''exec'' This Tcl command effectively boils down to forking off a variety of processes and hooking their input/output/errors up appropriately. Most of this code is quite generic, and ends up in 'TclpCreateProcess' for the actual forking and execution of another process (whose name is given by 'argv[[0]]' in TclpCreateProcess). Would it be possible to make a Tcl_FSCreateProcess which can pass the command on either to the native filesystem or to virtual filesystems? The simpler answer is "yes", given that we can simply examine 'argv[[0]]' and see if it is it is a path in a virtual filesystem and then hand it off appropriately, but could a vfs actually implement anything sensible? The kind of thing I'm thinking of is this: we mount an ftp site and would then like to execute various ftp commands directly. Now, we could use 'ftp::Quote' (from the ftp package) to send commands directly, but why not 'exec' them? If my ftp site is mounted at /tcl/ftppub, why couldn't "exec /tcl/ftppub FOO arg1 arg2" attempt a verbatim "FOO arg1 arg2" command on the ftp connection? (Or would perhaps "exec /tcl/ftppub/FOO arg1 arg2" be the command?). Similarly a Tcl 'namespace' filesystem could use 'exec' to evaluate code in the relevant namespace (of course you could just use 'namespace eval' directly, but then you couldn't hook the code up to input/output pipes). If anyone wants to have a look at a patch which does most of this on Windows: ftp://ftp.ucsd.edu/pub/alpha/tcl/chanExec.patch should do the trick. 4 tests fail, however, so any help fixing that would be great. It should be pretty easy to move the patch over to Unix as well. ''tclvfs syntax'' All tclvfs commands exposed to Tcl go through a 'subcommand' syntax. So the mount command you give (say 'foo') is called with: "eval $foo deletefile ...." for example. It might be nice to add the option of calling a namespace command instead (something like 'namespace $ns eval {foo deletefile ...}). Pretty much all the vfs code actually does this anyway, using a very simple 'foo' to divert to namespace commands. ---- '''TclVfs building, compilation and bugs''' There are a variety of fixes to tclvfs compilation and running in the latest release. In particular, various TEA files have been added, and the code works with multiple interpreters very well. ---- Note: tclvfs is known to build and run ok on: * windows * linux * solaris (please add your platform when it works). There are preview stand-alone builds of [TclKit] for Windows and Linux, which use the new VFS core and include support for the above examples (see [http://www.equi4.com/previews/]). I have not yet packaged the adjusted sources, which are essentially off-the-shelf Tcl, Tk, MetaKit, and TclVfs releases... -- [JCW] ---- [LV]: SPARC/Solaris 2.6 , Tcl 8.4a4 , tclvfs from Nov 11, 2001. After copying copies of config.sub, config.guess, and tcl.m4 from tcl's CVS distribution, tclvfs will configure and compile as well as passes most of the test suite. However, there was a problem with the ftp test where, on my machine, the ftp takes 60+ seconds to connect. In this case, tclvfs times out and the test fails. ---- [JCW] VFS in the core offers an interesting option: taking all file system calls out of the core, and bringing them back in as an extension. This is one of the areas where I see a quite unique opportunity: with FS calls out (and using a wrapped exe such as [TclKit] to provide ''just'' access to runtime files stored at the end of the executable), one has a system which is provably incapable of accessing (let alone modifying) the local disk. Such a setup, with sockets still included, would offer a formidable alternative to ''client-side Java'' [deployment], with all the power of scripting and Tk at its disposal. There are a few extra steps to take (such as disabling exec, and load, and piped open)*, but that's about it. The 8.4a4 core has all the logic in place to start modularizing file system calls. TclKit already fully relies on memory-mapped access to the executable for all runtime files, so ''we're pretty close to such a setup, IMO''... (*) There's no need to disable 'load' since it will be disabled anyway if a native filesystem isn't present. Both 'exec' and piped open end up at TclpCreateProcess (see 'exec' discussion above). This and related procedures (in tclPipe.c and tclWin|UnixPipe.c) in turn call 'TclpOpenFile, TclpCloseFile, TclpMakeFile, etc'. The question is whether we can suitably abstract all of this away into the filesystem table. The easiest would perhaps be if the generic code tclPipe.c used channels everywhere instead of TclFile file descriptors. Then there would only need to be a single filesystem-specific call 'TclpCreateProcess' which could go in the lookup table. This would then allow us (very easily) to create a version of Tcl which is incapable of accessing the local disk, since all native filesystem support could actually be in a separate library. Does anyone understand enough about the unix/win pipeline code (remember there's no 'exec' on macos) to know whether a shift from TclFile to Channels would be a problem? See above, for a patch which does most of this: ftp://ftp.ucsd.edu/pub/alpha/tcl/chanExec.patch ---- See also critlib [http://www.equi4.com/critlib/] - the [CriTcl] library of useful extensions to Tcl. CriTlib includes a wrapper for VFS. ---- [Category Package]