Reading and writing to a piped command
#! /bin/env tclsh
proc communicate {chan msg pump respond eof } {
fileevent $chan readable [list apply [list {chan respond eof} {
set res [gets $chan]
if {$res ne {}} {
fileevent $chan readable {}
{*}$respond $res
}
if {[eof $chan]} {
fileevent $chan readable {}
{*}$eof
}
}] $chan $respond $eof]
fileevent $chan writable [list apply [list {chan msg pump} {
catch {puts $chan $msg}
#something like this may be needed if the child program can be configured
#with reasonable output buffering
#fileevent $chan writable [list apply [list {chan pump} {
# puts $chan $pump
#}] $chan $pump]
fileevent $chan writable {}
}] $chan $msg $pump]
}
proc process {chan count args} {
puts "sed output: $args"
if {$count < 2} {
communicate $chan hello \n [namespace code [list process $chan [incr count]]] \
[namespace code [list closed $chan]]
} else {
catch {close $chan}
set ::done 0
}
}
proc closed {chan} {
#flush first to catch any pipe error, getting it out of the way in order to grab
#the exist status with [close]
if {[catch {flush $chan} eres eopts]} {
set status [catch {close $chan} eres eopts]
set ::done $status
}
set ::done 0
}
#try this line, which causes a sed error, to see how that's handled
#set chan [open {|sed -l {s/hello/goodbye} 2>@stderr} r+]
set chan [open {|sed -l s/hello/goodbye/ 2>@stderr} r+]
fconfigure $chan -buffering none
communicate $chan hello \n [namespace code [list process $chan 1]] \
[namespace code [list closed $chan]]
vwait ::done
return -code $::doneIt's up to the individual program when to print out its results. The -l option to BSD sed configures sed to print output whenever at least one line of output is ready. -u accomplishes the same for some other versions of sed. The pump feature of this example can be used pump some program-specifc value through the channel until the desired output is collected. It's a hack, but depending on the program, might be the only way to accomplish the task. Expect is the fully-featured tool for this type of task. Another approach would be to fork the current process into a producer and a consumer.
Arjen Markus: I have experimented a bit with plain Tcl driving another program. As this needs to work on Windows 95 (NT, ...) as well as UNIX in four or five flavours, I wanted to use plain Tcl, not Expect (however much I would appreciate the chance to do something really useful with Expect - apart from Android :-).
I think it is worth a page of its own, but here is a summary:
set inout [open |[list myprog] r+] fconfigure $inout -buffering line
Buffering might be out of your hands, though, for "real 16-bit commandline applications", which apparently don't have the ability to flush (reliably), except on close.
Neil Madden - here is a real-life example of interacting with a program through a pipe. The program in question is ispell - a UNIX spell-checking utility. I use it to spell-check the contents of a text widget containing LaTeX markup. There are a number of issues to deal with:
The example does not use [fileevent], as this would complicate this particular example. The options passed to ispell are -a (which makes it non-interactive) and -t (which makes it recognize TeX input).
set contents [split [$text get 1.0 end] \n]
set pipe [open [list | ispell -a -t] r+]
fconfigure $pipe -blocking 0 -buffering line
set ver [gets $pipe] ;# Ignore the initial version line
set linenum 1
foreach line $contents {
set wordnum 1
foreach word [split $line] {
puts $pipe $word ;# Feed word to ispell
while 1 {
set len [gets $pipe res]
if {$len > 0} {
# A valid result
# do stuff
continue
} else {
if {[fblocked $pipe]} {
# No output
break
} elseif {[eof $pipe]} {
# Pipe closed
catch {close $pipe}
return
}
# A blank line - skip
}
}
incr wordnum
}
incr linenum
}Thanks to Kevin Kenny for helping me figure this out.
Recently on comp.lang.tcl someone was trying to get the following code to work:
proc gzip {buf} {
set fd [open "|gzip -c" r+]
fconfigure $fd -translation binary -encoding binary
puts $fd $buf
flush $fd
set buf [read $fd]
close $fd
return $buf
}Here's an altered version in an attempt to get it to work.
#! /usr/tcl84/bin/tclsh
proc gzip {buf} {
set fd [open "|gzip -c" "r+"]
fconfigure $fd -translation binary -encoding binary
puts -nonewline $fd $buf
puts "output finished to gzip"
flush $fd
puts "flush finished to gzip"
set buf [read $fd]
puts "read finished from gzip"
close $fd
return $buf
}
proc gunzip {buf} {
set fd [open "|gzip -d" "r+"]
fconfigure $fd -translation binary -encoding binary
puts -nonewline $fd $buf
flush $fd
set buf [read $fd]
close $fd
return $buf
}
set a [gzip "This is a test"]
puts "finish compression"
set b [gunzip $a]
puts "finish uncompression"
puts $bAlas, it still doesn't work. The output and flush debug statements appear. But the message after the read doesn't appear.
Now, an alternative version of the command was proposed:
proc gzip {buf} {
return [exec gzip -c << $buf]
}However, that version doesn't demonstrate the method to read and write from a piped command. So I'm hoping that someone comes along with a fix for the initial code.
Lars H: Is the problem that gzip won't finish until its input has been read to end? The only way to be sure there won't be more data is to close the input end of the gzip pipe, and that can't be done without closing the output end as well. Tricky. Mind you, I've always felt the idea that one uses the same channel both for reading and writing (but of two distinct data streams) rather odd.
LV: The end of file on input might be an issue . Frankly, I'm uncertain that the notation _should_ work. That is to say, I don't know that both stdin and stdout are being associated with that one returned file handle. I have seen, and probably written, code that used a normal open of a file and did both read and write type operations. However, I don't recall whether I've seen a pipe example of that.
RM: The underlying system buffers the output. You need to use the "unbuffer" command like:
set fd [open "|unbuffer gzip -c" "r+"]
lexfiend Note that unbuffer is part of Expect, and may thus require additional work on "Some/All Batteries Not Included" Tcl setups.
AMG: I don't see how unbuffer would help, since I suspect the problem is with buffering inside gzip, and to get it to emit the last block of output you need to close its input. Can anyone confirm whether it does or not? I just apt-get installed expect yet for some reason it didn't install unbuffer.
AMG: I have several comments. Let's discuss.