VFS, exec and command pipelines

With Tcl 8.4's vfs, all filesystem access except TclpCreateProcess (and a few related helper functions) is hooked into a lookup table. This means that all filesystem operations visible at the Tcl or C level can operate on a vfs, with the exception of 'exec' and 'open |cmd'.

It would be nice to be able to provide a more general mechanism (*) to such command pipelines, especially given that stdout/stderr/stdin are already nicely abstracted as Tcl_Channels.

The stumbling block appears to be that all the 'pipe' related code doesn't actually use Tcl_Channels. Instead it uses a new 'TclFile' structure which is a lightweight wrapper around a native file-descriptor (and is slightly different on Windows vs Unix). Then there are a whole suite of new functions TclpOpenFile, TclpCloseFile, TclpMakeFile, TclpReleaseFile which are there just to operate on those structures. There is significant code-overlap between these TclFile-specific functions and the more general Tcl_Channel code.

Does anyone know any reason why all those 'TclFile' structures couldn't simply be removed and replaced with Tcl_Channels? If this were done, all of the 'exec' baggage and associated APIs could be made rather simpler and potentially exposed to the Tcl level through the tclvfs extension.

* Obvious benefits are:

• easier to write things like BLT's 'bgexec' and generally interact with pipelines from within Tcl
• allow implementation of pipeline components from within Tcl (e.g. pipe output from one application through a Tcl proc and into another application).
• force all native filesystem access through the 'Tcl_FS' lookup tables, and thereby make it very easy to compile a version of Tcl which can't access the native filesystem (by removing the 'nativeFilesystem' lookup table).

AK: I don't know the true reason, but speculate that the implementation of command pipelines came before channels and there just left as they initially implemented. A problem might be how to implement the OS unnamed pipes between the processes in the pipeline without burdening tcl with doing this.

Vince: with a little effort, I've now got the basic stuff all working with Tcl_Channels on Windows (i.e. TclFile, WinFile don't appear anywhere). 'exec' and pipes still work in most cases, but somes uses of 'exec tclsh foo.tcl' seem to fail. This means some of exec.test fails (5 tests now). The problem seems to be a crash on exit inside TclFinalizeIOSubsystem (tclIO.c) when a garbage statePtr is used -- am trying to track down where that garbage pointer comes from. The unnamed pipes bit operates just the same as before, except we have to get the OS handle from a Tcl_Channel instead of a TclFile.

AK: Regarding easiness of ripping out the whole FS code note that this has impact on the interp startup as well (intialization of auto paths, package paths, library path, encoding path, ...) and on the encoding subsystem (as it cannot load any external encoding anymore).

Vince: Yes, I imagine that there are bugs in Tcl which might not let it startup properly without a filesystem. But really, Tcl shouldn't care if all the paths you mention aren't available. Certainly it won't be able to autoload new code, use encodings, etc, but you should still be able to get a '%' prompt out of Tcl. This is one of the areas where modularising Tcl will require a bit of work, I think.

AK: Oh, I get a prompt with the FS ripped out of 8.3.4 (see mod-8-3-4-branch of the Tcl CVS), after the message that application initialization failed. I'll have to hunt the location down and disable this too.

A Windows patch (win eols too) is available here: ftp://ftp.ucsd.edu/pub/alpha/tcl/chanExec.patch , 4 tests fail, however, so any help fixing that would be great. It should be pretty easy to move the patch over to Unix as well.