Are there other useful open examples?

BR - Access mode "wb" doesn't work with any version of Tcl with which I tried it.

It isn't really needed either as we have fconfigure for that.

Vince -- it would only take a few lines of code added to Tcl's core to automatically interpret "b" for binary.

kennykb reports in the chat room that he recalls it is from Tcl 7.3 days or earlier.

schlenk -- The "b" flag is included in TIP #183, so it is available in Tcl 8.5.

LV -- Don't forget to add an example of using -mode (which is now in open rather than fconfigure) for configuring baud rate, etc. on a serial port...

MJ - Above remark may be a bit vague. What's meant is that fconfigure options that are specific for a certain channel type are documented on the man page for the command creating that type of channel. Hence the -mode option is documented in man open.

One nice feature of Tcl is that one can not only open files, but at least on Unix and Windows, one can also open pipelines to other processes.

(does "pipeline" mean "I can write to this and the other process will read it", or does it also mean "I can read from this what the other process writes" ?) Both. The exec and open commands make a remarkable pair. No other common language has anything approaching their portability, maturity, and applicability.

One variation on the fd8 example above which can be useful for debugging is

 set fd8 [open "| tee tmp/in | simple2 | tee tmp/out 2>tmp/err" r+]

In addition to running simple2 as an asynchronous child process, this also creates three files tmp/in, tmp/out, and tmp/err to which will be written everything that passes over simple2's stdin, stdout, and stderr respectively. tee here is a Unix utility that copies everything from stdin to stdout, but optionally also writes all of it to the files given as arguments.

Arjen Markus I have experimented a bit with plain Tcl driving another program. As this needs to work on Windows 95 (NT, ...) as well as UNIX in four or five flavours, I wanted to use plain Tcl, not Expect (however much I would appreciate the chance to do something really useful with Expect - apart from Android :-).

I think it is worth a page of its own, but here is a summary:

• Open the pipeline and make sure buffering is minimal via

        set inout [open "|myprog" "r+"]
        fconfigure $inout -buffering line

Buffering might be out of your hands, though, for "real 16-bit commandline applications", which apparently don't have the ability to flush (reliably) except on close.

• Set up fileevent handlers for reading from the process and reading from stdin.
• Make sure the process ("myprog" above) does not buffer its output to stdout!

Neil Madden - here is a real-life example of interacting with a program through a pipe. The program in question is ispell - a UNIX spell-checking utility. I use it to spell-check the contents of a text widget containing LaTeX markup. There are a number of issues to deal with:

• Keeping track of the position of the word in the widget.
• Filtering out useless (blank) lines from ispell
• Filtering out the version info that my version of ispell dumps out.
• When passing in a word which is a TeX command (e.g. \maketitle), ispell returns nothing at all.
• Careful handling of blocking.

The example does not use fileevent, as this would complicate this particular example. The options passed to ispell are -a (which makes it non-interactive) and -t (which makes it recognize TeX input).

 set contents [split [$text get 1.0 end] \n]
 set pipe [open "|ispell -a -t" r+]
 fconfigure $pipe -blocking 0 -buffering line
 set ver [gets $pipe] ;# Ignore the initial version line
 set linenum 1
 foreach line $contents {
     set wordnum 1
     foreach word [split $line] {
         puts $pipe $word   ;# Feed word to ispell
         while {1} {
             set len [gets $pipe res]
             if {$len > 0} {
                 # A valid result
                 # do stuff
                 continue
             } else {
                 if {[fblocked $pipe]} {
                     # No output
                     break
                 } elseif {[eof $pipe]} {
                     # Pipe closed
                     catch {close $pipe}
                     return
                 }
                 # A blank line - skip
             }
         }
         incr wordnum
     }
     incr linenum
 }

Thanks to Kevin Kenny for helping me figure this out.

For another real-life example of open |, see Driving ExifTool from Tcl.

Note that specifying "a" for the open argument need NOT be the optimal way of opening a file for appending (a violation of Tcl's normal least surprise policy). The "a" works by seeking to the end of file, whereas in some other languages the vanilla append utilizes a special service provided by the OS. The respecive behaviors can be quite different.

DKF - Here's how to tell whether you've got a problem. Open a file in append mode (with "a" or "a+") several times and write a single line to the file each time (I'd suggest writing digits here; that's easy to understand!) Do not explicitly flush or close those file handles. Instead, exit the program; the language's shut-down code should handle the flushing and closing for you. Then look at the contents of the file produced

How to know if the expected behavior happened? Simple. Check if all the values you wrote actually made it to the file. Order does not matter here. If you got all the lines written, that is good. Your language is doing proper (thread- and process-safe) appending, and this is one of the things that must be supportable on platforms that do POSIX. It's also one of the things that tends to get supported early because it is useful for not losing data from system logs...

But what if not all the data makes it? Well, that's really bad because it means that if you had two programs writing to a log file, you might lose some of the log messages. It happens in Tcl because Tcl tries to do a seek() to the end of the file when it is first opened and then presumes that's good enough. Instead, we get a race condition. Using the APPEND flag (which translates into the low-level O_APPEND flag to the PrivoxyWindowOpen() syscall) alters the semantics of writes so that a seek-to-end happens as an atomic part of the write itself, and that ensures that no data is ever lost (though of course apps still need to take care to only ever write complete records/lines to the file.)

Thanks to dkf for this tip. So, instead of doing "a" or "a+", you currently need to

  set f [open $theLogFile {WRONLY CREAT APPEND}] ;# instead of "a"

or

  set f [open $theLogFile {RDWR CREAT APPEND}]   ;# instead of "a+"

to avoid race conditions.

Does Tcl's current (8.4.1.1 and earlier) behavior of "a" have some benefit over the behavior of APPEND? Or should this difference be considered a bug that is fixed? FW seconds this question.

LES on 11 Apr 2005: This page's last update was stamped "1 Sep 2003" when I came here now and changed it. Has that question been answered yet?

Lars H: I'll hazard an answer. Perhaps the benefit of "a" over APPEND is that it is less surprising?! Imagine a program that appends some data to e.g. a list of changes and then seeks to the start to write an updated timestamp. If someone went in and replaced that "a" with {WRONLY CREAT APPEND} then the surprising result would be that all timestamps ended up at the end of the file, because every puts would seek to the end, would it not?

DKF: If you need random access to a file, you shouldn't use "a". You certainly wouldn't be doing this if you were using stdio.

Some additional discussion of the stderr gimmick above is probably appropriate.

In Alpha, there are many real-life examples of communicating with external processes using pipes. The examples include

• aspell
• tex, bibtex, dvips, gs (in fact it is altogether a very fancy Tcl frontend to tex, featuring many Tcl tricks like regexping log files to figure out what to do next, etc.)
• maxima (cf. described on this Wiki at Maxima through a pipe)
• coq

as well as standard shells. There are several pages dedicated to these issues on the AlphaTcl Wiki at http://alphatcl.sourceforge.net/wikit/ , see notably http://alphatcl.sourceforge.net/wikit/180 which is an overview page. The problems discussed include how to capture stderr, how to filter out prompts and transform an asynchronous interaction into a synchronous one with timeout mechanisms using vwait, how to link a process to a window, how to provide active cells in an Alpha document, a sort of worksheet interface to an external process, how to provide step functionality (stepping through a list of commands, at each step seeing the result in a console), and a check-as-you-type spellchecker (using aspell).

Recall that Alpha is a text editor, just as powerful and configurable as Emacs, but having Tcl as extension language instead of lisp! The AlphaTcl library (http://alphatcl.sf.net ) is nearly 20000 lines of Tcl code, full of gems and useful tricks. AlphaX runs on OSX, AlphaTk (written entirely in TclTk) runs on any platform.

MG just found out (on July 27, 2005) that Tcl 8.4.9 doesn't seem to be able to open hidden files, at least on Windows XP. For example...

 2 % set fid [open test w]
 file9fa4c0
 3 % close $fid
 4 % file attributes test -hidden 1
 5 % set fid [open test w]
 couldn't open "test": permission denied
 6 % file attributes test -hidden 0
 7 % set fid [open test w]
 file9fda20
 8 % close $fid

Anyone know if this is intentional behaviour, or whether it's a bug?

WangFan I use "catch {open "|plink -ssh $account@$host" w+} input}", in which "plink" is download from http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html . When pipe is readable, I read the pipe, but I always get NULL. Does anybody know why? The tcl script is shown below:

        package require Tk

        frame .t
        text .t.log
        pack .t.log
        pack .t
        set account user
        set host 10.1.1.2
        if [catch {open "|plink -ssh $account@$host" w+} input] {
                .t.log insert end $input\n
                return 0
        } else {
                fconfigure $input -blocking 0 -buffering line -translation crlf -eofchar {}
                fileevent $input readable Log
        }
        proc Log {} {
              global input

              if [eof $input] {
                    Stop
              } else {
                    gets $input line
                    .t.log insert end $line\n
                    .t.log see end
              }
              return
        }
        proc Stop {} {
              global input
              catch {close $input}
              return
        }

Ken: I need to run 2 scripts located in 2 different files. Both scripts have some common input and output variables that are needed. I intend to use 'open' command but realise it is only for files. Is there any way i can script it so that i can use fileevent handle to receive input and output to the other script? Thanks.

Answer: yes. Details are probably better discussed in comp.lang.tcl or the Chat.

mariuszgrad - 2009-08-17 15:06:22

I tried code like this:

proc uncompress_tgz { bindata } {
  set cmd "|tar -C -O -xzf -"
  set fid  [ open $cmd "r+" ]
  fconfigure $fid -translation binary
  fconfigure $fid -buffering none
  fconfigure $fid -blocking  false
  puts -nonewline $fid $bindata
  puts [ read $fid ]
  close $fid
}

Unfortunately it didn't work out. Problem was that there was no output from tar command.

The solution for it was simple:

  set fid  [ open $cmd "w" ]

It looks that when $fid was opened only for writing (it was my first access to it) and therefor it stuck when i tried to pull out data from it (with read $fid) command. Greetings

AMG: This doesn't make sense. How can you [read] from $fid if it's open only for writing? As for your original problem, I suspect the tar command doesn't close its stdout until its stdin is closed. You will need to use the new, optional direction argument to [close] [L1 ] to close the write side of the $fid channel before reading from $fid.

AMG: Here's a one-liner for creating an empty file:

close [open $filename w]

See also:

• gets
• read
• file
• puts
• catch
• an "Example of reading and writing to a piped command"
• and this page [L2 ] from the standard tutorial.