A '''virtual [file] system''', or '''VFS''',  provides access to its own
resources via a standard file system interface. 



** See Also **

   [Caching VFS], by [CMcC]:   Along an additive VFS.

   [Core VFS support]:   

   [with_mounted_dir]:   

   [vfs filesystem configuration]:   

   [http://www.ibm.com/developerworks/library/l-sc12.html?ca=dnt-417%|%Putting virtual file systems to work], by [Cameron Laird], 2003-04-29:   An introduction to virtual file system.

   [tclvfs]:   For writing virtual file systems in pure Tcl.  It should compile easily on various platforms, as there is only one file, `vfs.c`, to compile.  It includes a number of virtual file systems for mounting [zip] archives, [ftp] sites, [http] sites, [webdav] remote file systems, [metakit] archives ( which is what [tclkit] does), and even commands in Tcl namespaces.


   [One-line web browser in Tcl]:   



** Description **

As of version [Changes in Tcl/Tk 8.4%|%Tcl 8.4], Tcl
[https://www.tcl-lang.org/man/tcl/TclLib/FileSystem.htm%|%provides an API] for
creating and registering virtual file systems.  Virtually any system can use
this API as an interface to its own resources.  This makes it possible to
navigate and manage those resources using standard file system routines such as
`[cd]`, `[pwd]`, `[glob]`, `[file]`, or `[load]`.

Virtual file systems exist, among other things, for files on remote sites,
in-memory file systems, archive files such as [zip], and prebuilt software
collections.  Since VFS support is implemented at the [C] level, all Tcl
extensions have access to files on virtual file systems.  To work with a file,
an application, e.g. Tk, BLT, Img, etc., does just what it's always done:  pass
the name of the file to the appropriate routine.  The application doesn't need
any particular knowledge of virtual file systems, which is the entire point of
them.

'''Note from the editor''': Try to keep [tclvfs]-specific material on that page,
and general discussions of Tcl's [VFS] capabilities here.



** History of VFS Development in Tcl **

The first VFS routines for Tcl were writeen in pure Tcl by [Matt Newman]
for use in [Tclkit], and later re-introduced at the [C] level by [Vince
Darley], as described in [TIP]
[http://www.tcl.tk/cgi-bin/tct/tip/17.html%|%17].



** Native File system **

In Tcl itself there is a virtual file system registered that provides access to
the native file system.  This virtual file system is also the default, or
fall-through file system that claims ownership of any file that no other
registered virtual file system claims.  The source code for this system is in
`tclIOUtil.c` and also in the the various platform-specific files, e.g.
`tclWinFile.c` and `tclWinFCmd.c`.



** [prowrap] **

An updated version of `zvfs.c` is available at http://sourceforge.net/projects/tclpro/.
Checkout the Patches link.  The direct link is http://pdqi.com/browsex/download/v2.0/zvfs.c

There is a [https://sourceforge.net/tracker/?func=detail&aid=555411&group_id=13969&atid=313969%|%provisional patch] for prowrap which attempts to make it use these APIs.
Unfortunately, prowrap's build system is so unfriendly that this patch has not been tested.  With a little effort, this patch would allow prowrap to support `[glob]` (as well as generally have a more robust virtual filesystem).

See also its successor, [TclApp].



** testreporting file system **

This is part of Tcl's test code/suite.  When registered it allows all file system activity to be reported upon (a brief message describing each file system access is printed to stderr by default).



** Feature Request:  Asynchronicity **

Asynchronous file system access would be useful, primarily for copying files, but also potentially other purposes:

====== 
file copy -command progress http://a.b.c/foo .   

set fd [open http://a.b.c/foo w]
puts $fd -command progress $megabytes

load -command progress http://a.b.c/tk84.dll
====== 

Both `[file rename]`, and [load] may require cross-file-system copies.

Strictly speaking, the second of the three examples above (puts ... -command) doesn't have anything to do with the new vfs API.  It is 100% ''channel'' based.  In fact the real need here is for something which bridges the current gap between the channel api and the vfs api.  This gap is structurally very clean, but, from the examples above, needs to be plugged!


** Feature Request: Overlapping Mounts **

Allow a single directory's contents to come from a merge of a variety of sources.



** Feature Request: [WebDAV] **

[[future VFS thoughts:  BXXP?  WebDAV?]]

[WebDAV] would be great!  It should be possible to write a WebDAV
implementation entirely in Tcl using the tclvfs extension.  See
[tclvfs] for a simple webdav implementation (needs work).



** Feature Request: Translucent File System (Writeable CD-ROMs) **

Allow a proxy for a CD-ROM that passes reads through but stores files that
are written in a proxy directory structure. The program would have no simple
way of knowing that the data being read comes from a read-only medium.

[escargo]:  See http://wks.uts.ohio-state.edu/sysadm_course/html/sysadm-67.html

[escargo] 2004-07-13: See [A collate/broadcast virtual filesystem] for an implementation!



** Feature Request: File Overlays **

Closely related is the idea of mounting several directories on top of each other, with things like `[glob]` returning the union (or the concatenation) of all underlying items.  Some of this is possible now in VFS (but there is no implementation), but I'm sure some of this will require further Tcl core enhancements.



** File Locking **

Many asynchronous protocols or operations would benefit from some kind of locking mechanism.  It might be good to extend the interface in the future with this.  (Note that any vfs can provide its own 'file attributes' both readable and writable so in principle any other features can be supported through attributes using the existing vfs code).



** `[exec]` **

`[exec]` boils down to forking off a variety of processes and hooking their input/output/errors up appropriately.  Most of this code is quite generic, and ends up in 'TclpCreateProcess' for the actual forking and execution of another process (whose name is given by 'argv[[0]]' in TclpCreateProcess).  Would it be possible to make a Tcl_FSCreateProcess which can pass the command on either to the native file system or to virtual file systems?  The simpler answer is "yes", given that we can simply examine 'argv[[0]]' and see if it is it is a path in a
virtual file system and then hand it off appropriately, but could a vfs
actually implement anything sensible?  The kind of thing I'm thinking of is this: we mount an ftp site and would then like to execute various ftp
commands directly.  Now, we could use 'ftp::Quote' (from the ftp package) to send commands directly, but why not 'exec' them?  If my ftp site is mounted at /tcl/ftppub, why couldn't "exec /tcl/ftppub FOO arg1 arg2" attempt a verbatim "FOO arg1 arg2" command on the ftp connection?  (Or would perhaps "exec /tcl/ftppub/FOO arg1 arg2" be the command?).  Similarly a Tcl 'namespace' file system could use 'exec' to evaluate code in the relevant namespace (of course you could just use 'namespace eval' directly, but then you couldn't hook the code up to input/output pipes).

If anyone wants to have a look at a patch which does most of this on Windows: ftp://ftp.ucsd.edu/pub/alpha/tcl/chanExec.patch should do the trick.  4 tests fail, however, so any help fixing that would be great.  It should be pretty easy to move the patch over to Unix as well.

See [VFS, exec and command pipelines]



** Move the File System Out of the Core **

[JCW]: VFS in the core offers an interesting option: taking all file system calls out of the core, and bringing them back in as an extension.  This is one of the areas where I see a quite unique opportunity: with FS calls out (and using a wrapped exe such as [Tclkit] to provide ''just'' access to runtime files stored at the end of the executable), one has a system which is provably incapable of accessing (let alone modifying) the local file system.  Such a setup, with sockets still included, would offer a
formidable alternative to ''client-side Java'' [deployment], with all the power of scripting and Tk at its disposal.  There are a few extra steps to take, such as disabling `[exec]`, and `[load]`, and piped `[open]`, but that's about it.  The 8.4a4 core has all the logic in place to start modularizing file system calls.  Tclkit already fully relies on memory-mapped access to the executable for all runtime files, so
''we're pretty close to such a setup, IMO''...

Regarding `[exec]`, `[load]`, and piped `[open]`, there's no need to disable `[load]` since it is disabled anyway if a native file system isn't present.  Both `[exec]` and piped `[open]` end up at `TclpCreateProcess` (see `[exec]` discussion above).  This and related procedures (in `tclPipe.c` and `tclWin`|`UnixPipe.c`) in turn call `TclpOpenFile`, `TclpCloseFile`, `TclpMakeFile`, etc,.  The question is whether we can suitably abstract all of this away into the file system table.  The easiest would perhaps be if the generic code `tclPipe.c` used channels everywhere instead of `TclFile` file descriptors.  Then there would only need to be a single file-system-specific call,`TclpCreateProcess` which could go in the lookup table.  This would then allow us (very easily) to create a version of Tcl which is incapable of accessing the local file system, since all native file system support could actually be in a separate library.

Does anyone understand enough about the unix/win pipeline code (remember there's no `[exec]` on macos) to know whether a shift from `TclFile` to channels would be a problem?  See above, for a patch which does most of this: ftp://ftp.ucsd.edu/pub/alpha/tcl/chanExec.patch

Such a rewritten `[exec]` core could allow diversion of pipes to or from ordinary Tcl code as well (as in [Streams]).  This would make a 'stream' a real, robust, part of Tcl.



** Scripted Files ** 

[jcw] 2002-10-21: Idea for yet another VFS driver: a readonly file system which does a '''script-indirection'''.  In other words, when opening file `abc`, the driver reads in `abc`, evals the contents as a Tcl script, and returns the output.  A refinement would be to call a `read` procedure when reading, a `stat` procedure when doing a stat on the file, etc.

The uses?  Well, endless really - imagine a set of scripts, each of which does a build of an extension. Then doing a full build consists of nothing more than doing "file copy ...".  In fact, one could do "load blah.so", which the extension built on-the-fly.

Or more traditionally: storing scripts which generate HTML pages, and using such a set to re-generate a website.  Or scripts which take a certain datafile, apply some transformations, and return the result.

''Endless.  The possibilities of such, eh, "active scripting files" :) really are endless.''


** vfs::ftp Error **

[RS] tried vfs::ftp, but ran on the following error (*** for anonymized parts):

======none
% package require vfs::ftp
1.0
% vfs::ftp::Mount suchenwi:***@sux000 /sux
0
% glob /sux/*
Unknown state "list_sent"
   while executing
"error "Unknown state \"$ftp(State)\"""
   ("default" arm line 2)
   invoked from within
"switch -- $ftp(State) {
       list {
           fileevent $sock readable [list [namespace current]::HandleList $s $sock]
           set ftp(Source..."
   (procedure "::ftp::InitDataConn" line 26)
   invoked from within
"::ftp::InitDataConn 0 sock1748 ***.***.124.202 20"
====== 

Any ideas?

[Vince]: Sure!  The ftp package is throwing an error, so you should be able to debug that... Looks like either an ftp package bug or a problem on your server.



** Intercepting Native File System Operations **

[Aud] 2011-02-28:  I'm curious if it's possible to replace Tcl's own internal handling for normal file system paths (as in not URIs) in this way. It would be interesting if you could provide total integration in Tcl scripts with another VFS layer (Gnome's gvfs, for example) with just the use of some glue C code.


<<categories>> Package | Glossary | VFS