'''Reading and writing to a piped command''' <> ** See Also ** [open]: [pipe]: [gzip]: [unbuffer]: [How can I run data through an external filter?]: ** Description ** The methods for accomplishing some task involving writing to and then reading from a piped command depend primarily on whether the communication will be interactive or non-interactive. The non-interactive case is fairly straight-forward and robust. The interactive case can be problematic, since external programs may buffer incoming and outgoing data in arbitrary ways. [Expect] is the best all-purpose tool for interactive communication with another program. ** Example: Non-Interactive: gzip ** ====== proc gzip {buf} { set chan [open "|gzip -c" r+] fconfigure $chan -translation binary -encoding binary puts $chan $buf flush $chan chan close $chan write set buf [read $chan] close $chan return $buf } ====== The `write` side of `$chan` must be closed so that gzip reads any remaining data in the buffers processes it, and sends the result to its stdtout. For versions of Tcl that don't have [chan close ?direction?], try the following workaround: ====== proc gzip {buf} { return [exec gzip -c << $buf] } ====== ** Example: Interactive Conversation with another Program ** ====== #! /bin/env tclsh proc communicate {chan msg pump respond eof } { fileevent $chan readable [list apply [list {chan respond eof} { set res [gets $chan] if {$res ne {}} { fileevent $chan readable {} {*}$respond $res } if {[eof $chan]} { fileevent $chan readable {} {*}$eof } }] $chan $respond $eof] fileevent $chan writable [list apply [list {chan msg pump} { catch {puts $chan $msg} #something like this may be needed if the child program can be configured #with reasonable output buffering #fileevent $chan writable [list apply [list {chan pump} { # puts $chan $pump #}] $chan $pump] fileevent $chan writable {} }] $chan $msg $pump] } proc process {chan count args} { puts "sed output: $args" if {$count < 2} { communicate $chan hello \n [namespace code [list process $chan [incr count]]] \ [namespace code [list closed $chan]] } else { catch {close $chan} set ::done 0 } } proc closed {chan} { #flush first to catch any pipe error, getting it out of the way in order to grab #the exist status with [close] if {[catch {flush $chan} eres eopts]} { set status [catch {close $chan} eres eopts] set ::done $status } set ::done 0 } #try this line, which causes a sed error, to see how that's handled #set chan [open {|sed -l {s/hello/goodbye} 2>@stderr} r+] set chan [open {|sed -l s/hello/goodbye/ 2>@stderr} r+] fconfigure $chan -buffering none communicate $chan hello \n [namespace code [list process $chan 1]] \ [namespace code [list closed $chan]] vwait ::done return -code $::done ====== It's up to the individual program when to print out its results. The `-l` option to [BSD] sed configures sed to print output whenever at least one line of output is ready. `-u` accomplishes the same for some other versions of sed. The `pump` feature of this example can be used pump some program-specific value through the channel until the desired output is collected. It's a hack, but depending on the program, might be the only way to accomplish the task. [Expect] is the fully-featured tool for this type of task. Another approach would be to [fork] the current process into a producer and a consumer. ** Example: ispell for a text widget ** [Neil Madden] - here is a real-life example of interacting with a program through a pipe. The program in question is ''ispell'' - a UNIX spell-checking utility. I use it to spell-check the contents of a text widget containing [LaTeX] markup. There are a number of issues to deal with: * Keeping track of the position of the word in the widget. * Filtering out useless (blank) lines from ispell * Filtering out the version info that my version of ispell dumps out. * When passing in a word which is a TeX command (e.g. \maketitle), ispell returns nothing at all. * Careful handling of blocking. The example does not use [[`[fileevent]`], as this would complicate this particular example. The options passed to ispell are -a (which makes it non-interactive) and -t (which makes it recognize TeX input). ====== set contents [split [$text get 1.0 end] \n] set pipe [open [list | ispell -a -t] r+] fconfigure $pipe -blocking 0 -buffering line set ver [gets $pipe] ;# Ignore the initial version line set linenum 1 foreach line $contents { set wordnum 1 foreach word [split $line] { puts $pipe $word ;# Feed word to ispell while 1 { set len [gets $pipe res] if {$len > 0} { # A valid result # do stuff continue } else { if {[fblocked $pipe]} { # No output break } elseif {[eof $pipe]} { # Pipe closed catch {close $pipe} return } # A blank line - skip } } incr wordnum } incr linenum } ====== Thanks to [Kevin Kenny] for helping me figure this out. [DDG] - The wiki page [Spellcheck Widget using Aspell] contains an example on how to use aspell or hunspell to check a given text string and how to use the suggestions by aspell/hunspell to replace the wrong words. ** To Sort: [Arjen Markus] ** [Arjen Markus]: I have experimented a bit with plain Tcl driving another program. As this needs to work on Windows 95 (NT, ...) as well as UNIX in four or five flavours, I wanted to use plain Tcl, not Expect (however much I would appreciate the chance to do something really useful with Expect - apart from [Android] :-). I think it is worth a page of its own, but here is a summary: * Open the pipeline and make sure buffering is minimal via ====== set inout [open |[list myprog] r+] fconfigure $inout -buffering line ====== Buffering might be out of your hands, though, for "real 16-bit commandline applications", which apparently don't have the ability to flush (reliably), except on close. * Set up [[`[fileevent]`] handlers for reading from the process and reading from [stdin]. * Make sure the process ("myprog" above) does not buffer its output to [stdout]! ** Example: Python 3 pipe ** [DDG] - 2022-01-12: To start a python3 pipe you have to select the command line arguments *-iu* for interactive mode and unbuffered output. Here an example to simulate the terminal output of the python3 interpreter ====== set code "x=1 print(x) y=x+1 print(y)" proc piperead {pipe args} { if {![eof $pipe]} { #puts "read $pipe : $args" set got [gets $pipe] puts "$got" } } set pipe [open "|python3 -ui" r+] fconfigure $pipe -buffering none -blocking false fileevent $pipe readable [list piperead $pipe] set pywait [list] foreach pyline [split $code \n] { puts stdout ">>> $pyline" puts $pipe $pyline # need some delay for the pipe after 100 [list append pywait ""] vwait pywait } close $pipe puts "Finished python3 pipe!" exit 0 ====== And this is the output: ====== >>> x=1 >>> print(x) 1 >>> y=x+1 >>> print(y) 2 ====== ** Example: R pipe ** [DDG] - 2022-01-12: To start a R pipe again you have to use the interactive argument *--interactive*, further you can suppress the R welcome message by using the *-q* option. Here an example to mimic the R terminal: ====== set code "x=1 print(x) y=x+1 print(y) print(R.version.string)" proc piperead {pipe args} { if {![eof $pipe]} { #puts "read $pipe : $args" set got [gets $pipe] if {$got ne ""} { puts "$got" } } } set pipe [open "|R -q --interactive" r+] fconfigure $pipe -buffering none -blocking false fileevent $pipe readable [list piperead $pipe] set rwait [list] foreach rline [split $code \n] { puts $pipe $rline # need some delay for the pipe after 100 [list append rwait ""] vwait rwait } close $pipe puts "Finished R pipe!" exit 0 ====== And here the output: ====== > x=1 > print(x) [1] 1 > y=x+1 > print(y) [1] 2 > print(R.version.string) [1] "R version 4.0.5 (2021-03-31)" Finished R pipe! ====== <> Example | Channel | Interprocess Communication