A virtual file system, or VFS, provides access to its own resources via a standard file system interface.
As of version Tcl 8.4, Tcl provides an API for creating and registering virtual file systems. Virtually any system can use this API as an interface to its own resources. This makes it possible to navigate and manage those resources using standard file system routines such as cd, pwd, glob, file, or load.
Virtual file systems exist, among other things, for files on remote sites, in-memory file systems, archive files such as zip, and prebuilt software collections. Since VFS support is implemented at the C level, all Tcl extensions have access to files on virtual file systems. To work with a file, an application, e.g. Tk, BLT, Img, etc., does just what it's always done: pass the name of the file to the appropriate routine. The application doesn't need any particular knowledge of virtual file systems, which is the entire point of them.
Note from the editor: Try to keep tclvfs-specific material on that page, and general discussions of Tcl's VFS capabilities here.
The first VFS routines for Tcl were writeen in pure Tcl by Matt Newman for use in Tclkit, and later re-introduced at the C level by Vince Darley, as described in TIP 17 .
In Tcl itself there is a virtual file system registered that provides access to the native file system. This virtual file system is also the default, or fall-through file system that claims ownership of any file that no other registered virtual file system claims. The source code for this system is in tclIOUtil.c and also in the the various platform-specific files, e.g. tclWinFile.c and tclWinFCmd.c.
An updated version of zvfs.c is available at http://sourceforge.net/projects/tclpro/ . Checkout the Patches link. The direct link is http://pdqi.com/browsex/download/v2.0/zvfs.c
There is a provisional patch for prowrap which attempts to make it use these APIs. Unfortunately, prowrap's build system is so unfriendly that this patch has not been tested. With a little effort, this patch would allow prowrap to support glob (as well as generally have a more robust virtual filesystem).
See also its successor, TclApp.
This is part of Tcl's test code/suite. When registered it allows all file system activity to be reported upon (a brief message describing each file system access is printed to stderr by default).
Asynchronous file system access would be useful, primarily for copying files, but also potentially other purposes:
file copy -command progress http://a.b.c/foo . set fd [open http://a.b.c/foo w] puts $fd -command progress $megabytes load -command progress http://a.b.c/tk84.dll
Both file rename, and load may require cross-file-system copies.
Strictly speaking, the second of the three examples above (puts ... -command) doesn't have anything to do with the new vfs API. It is 100% channel based. In fact the real need here is for something which bridges the current gap between the channel api and the vfs api. This gap is structurally very clean, but, from the examples above, needs to be plugged!
Allow a single directory's contents to come from a merge of a variety of sources.
[future VFS thoughts: BXXP? WebDAV?]
WebDAV would be great! It should be possible to write a WebDAV implementation entirely in Tcl using the tclvfs extension. See tclvfs for a simple webdav implementation (needs work).
Allow a proxy for a CD-ROM that passes reads through but stores files that are written in a proxy directory structure. The program would have no simple way of knowing that the data being read comes from a read-only medium.
escargo: See http://wks.uts.ohio-state.edu/sysadm_course/html/sysadm-67.html
escargo 2004-07-13: See A collate/broadcast virtual filesystem for an implementation!
Closely related is the idea of mounting several directories on top of each other, with things like glob returning the union (or the concatenation) of all underlying items. Some of this is possible now in VFS (but there is no implementation), but I'm sure some of this will require further Tcl core enhancements.
Many asynchronous protocols or operations would benefit from some kind of locking mechanism. It might be good to extend the interface in the future with this. (Note that any vfs can provide its own 'file attributes' both readable and writable so in principle any other features can be supported through attributes using the existing vfs code).
exec boils down to forking off a variety of processes and hooking their input/output/errors up appropriately. Most of this code is quite generic, and ends up in 'TclpCreateProcess' for the actual forking and execution of another process (whose name is given by 'argv[0]' in TclpCreateProcess). Would it be possible to make a Tcl_FSCreateProcess which can pass the command on either to the native file system or to virtual file systems? The simpler answer is "yes", given that we can simply examine 'argv[0]' and see if it is it is a path in a virtual file system and then hand it off appropriately, but could a vfs actually implement anything sensible? The kind of thing I'm thinking of is this: we mount an ftp site and would then like to execute various ftp commands directly. Now, we could use 'ftp::Quote' (from the ftp package) to send commands directly, but why not 'exec' them? If my ftp site is mounted at /tcl/ftppub, why couldn't "exec /tcl/ftppub FOO arg1 arg2" attempt a verbatim "FOO arg1 arg2" command on the ftp connection? (Or would perhaps "exec /tcl/ftppub/FOO arg1 arg2" be the command?). Similarly a Tcl 'namespace' file system could use 'exec' to evaluate code in the relevant namespace (of course you could just use 'namespace eval' directly, but then you couldn't hook the code up to input/output pipes).
If anyone wants to have a look at a patch which does most of this on Windows: ftp://ftp.ucsd.edu/pub/alpha/tcl/chanExec.patch should do the trick. 4 tests fail, however, so any help fixing that would be great. It should be pretty easy to move the patch over to Unix as well.
See VFS, exec and command pipelines
JCW: VFS in the core offers an interesting option: taking all file system calls out of the core, and bringing them back in as an extension. This is one of the areas where I see a quite unique opportunity: with FS calls out (and using a wrapped exe such as Tclkit to provide just access to runtime files stored at the end of the executable), one has a system which is provably incapable of accessing (let alone modifying) the local file system. Such a setup, with sockets still included, would offer a formidable alternative to client-side Java deployment, with all the power of scripting and Tk at its disposal. There are a few extra steps to take, such as disabling exec, and load, and piped open, but that's about it. The 8.4a4 core has all the logic in place to start modularizing file system calls. Tclkit already fully relies on memory-mapped access to the executable for all runtime files, so we're pretty close to such a setup, IMO...
Regarding exec, load, and piped open, there's no need to disable load since it is disabled anyway if a native file system isn't present. Both exec and piped open end up at TclpCreateProcess (see exec discussion above). This and related procedures (in tclPipe.c and tclWin|UnixPipe.c) in turn call TclpOpenFile, TclpCloseFile, TclpMakeFile, etc,. The question is whether we can suitably abstract all of this away into the file system table. The easiest would perhaps be if the generic code tclPipe.c used channels everywhere instead of TclFile file descriptors. Then there would only need to be a single file-system-specific call,TclpCreateProcess which could go in the lookup table. This would then allow us (very easily) to create a version of Tcl which is incapable of accessing the local file system, since all native file system support could actually be in a separate library.
Does anyone understand enough about the unix/win pipeline code (remember there's no exec on macos) to know whether a shift from TclFile to channels would be a problem? See above, for a patch which does most of this: ftp://ftp.ucsd.edu/pub/alpha/tcl/chanExec.patch
Such a rewritten exec core could allow diversion of pipes to or from ordinary Tcl code as well (as in Streams). This would make a 'stream' a real, robust, part of Tcl.
jcw 2002-10-21: Idea for yet another VFS driver: a readonly file system which does a script-indirection. In other words, when opening file abc, the driver reads in abc, evals the contents as a Tcl script, and returns the output. A refinement would be to call a read procedure when reading, a stat procedure when doing a stat on the file, etc.
The uses? Well, endless really - imagine a set of scripts, each of which does a build of an extension. Then doing a full build consists of nothing more than doing "file copy ...". In fact, one could do "load blah.so", which the extension built on-the-fly.
Or more traditionally: storing scripts which generate HTML pages, and using such a set to re-generate a website. Or scripts which take a certain datafile, apply some transformations, and return the result.
Endless. The possibilities of such, eh, "active scripting files" :) really are endless.
RS tried vfs::ftp, but ran on the following error (*** for anonymized parts):
% package require vfs::ftp
1.0
% vfs::ftp::Mount suchenwi:***@sux000 /sux
0
% glob /sux/*
Unknown state "list_sent"
while executing
"error "Unknown state \"$ftp(State)\"""
("default" arm line 2)
invoked from within
"switch -- $ftp(State) {
list {
fileevent $sock readable [list [namespace current]::HandleList $s $sock]
set ftp(Source..."
(procedure "::ftp::InitDataConn" line 26)
invoked from within
"::ftp::InitDataConn 0 sock1748 ***.***.124.202 20"Any ideas?
Vince: Sure! The ftp package is throwing an error, so you should be able to debug that... Looks like either an ftp package bug or a problem on your server.
Aud 2011-02-28: I'm curious if it's possible to replace Tcl's own internal handling for normal file system paths (as in not URIs) in this way. It would be interesting if you could provide total integration in Tcl scripts with another VFS layer (Gnome's gvfs, for example) with just the use of some glue C code.