Version 3 of VFS, exec and command pipelines

Updated 2001-11-27 12:23:05

With Tcl 8.4's vfs, all filesystem access except TclpCreateProcess (and a few related helper functions) is hooked into a lookup table. This means that all filesystem operations visible at the Tcl or C level can operate on a vfs, with the exception of 'exec' and 'open |cmd'.

It would be nice to be able to provide a more general mechanism (*) to such command pipelines, especially given that stdout/stderr/stdin are already nicely abstracted as Tcl_Channels.

The stumbling block appears to be that all the 'pipe' related code doesn't actually use Tcl_Channels. Instead it uses a new 'TclFile' structure which is a lightweight wrapper around a native file-descriptor (and is slightly different on Windows vs Unix). Then there are a whole suite of new functions TclpOpenFile, TclpCloseFile, TclpMakeFile, TclpReleaseFile which are there just to operate on those structures. There is significant code-overlap between these TclFile-specific functions and the more general Tcl_Channel code.

Does anyone know any reason why all those 'TclFile' structures couldn't simply be removed and replaced with Tcl_Channels? If this were done, all of the 'exec' baggage and associated APIs could be made rather simpler and potentially exposed to the Tcl level through the tclvfs extension.


* Obvious benefits are:

  • easier to write things like BLT's 'bgexec' and generally interact with pipelines from within Tcl
  • allow implementation of pipeline components from within Tcl (e.g. pipe output from one application through a Tcl proc and into another application).
  • force all native filesystem access through the 'Tcl_FS' lookup tables, and thereby make it very easy to compile a version of Tcl which can't access the native filesystem (by removing the 'nativeFilesystem' lookup table).

AK: I don't know the true reason, but speculate that the implementation of command pipelines came before channels and there just left as they initially implemented. A problem might be how to implement the OS unnamed pipes between the processes in the pipeline without burdening tcl with doing this.


Vince: with a little effort, I've now got the basic stuff all working with Tcl_Channels on Windows (i.e. TclFile, WinFile don't appear anywhere). 'exec' still works, _unless_ Tcl is itself the spawned off process (so 'exec ls' or 'ls' work fine, but 'exec tclsh foo.tcl' fails). This means most of exec.test fails. The problem seems to be a crash on exit inside TclFinalizeIOSubsystem (tclIO.c) when a garbage statePtr is used. The unnamed pipes bit operates just the same as before, except we have to get the OS handle from a Tcl_Channel instead of a TclFile.