Reading and writing to a piped command
The methods for accomplishing some task involving writing to and then reading from a piped command depend primarily on whether the communication will be interactive or non-interactive. The non-interactive case is fairly straight-forward and robust. The interactive case can be problematic, since external programs may buffer incoming and outgoing data in arbitrary ways. Expect is the best all-purpose tool for interactive communication with another program.
proc gzip {buf} {
set chan [open "|gzip -c" r+]
fconfigure $chan -translation binary -encoding binary
puts $chan $buf
flush $chan
chan close $chan write
set buf [read $chan]
close $chan
return $buf
}The write side of $chan must be closed so that gzip reads any remaining data in the buffers processes it, and sends the result to its stdtout.
For versions of Tcl that don't have chan close ?direction?, try the following workaround:
proc gzip {buf} {
return [exec gzip -c << $buf]
}#! /bin/env tclsh
proc communicate {chan msg pump respond eof } {
fileevent $chan readable [list apply [list {chan respond eof} {
set res [gets $chan]
if {$res ne {}} {
fileevent $chan readable {}
{*}$respond $res
}
if {[eof $chan]} {
fileevent $chan readable {}
{*}$eof
}
}] $chan $respond $eof]
fileevent $chan writable [list apply [list {chan msg pump} {
catch {puts $chan $msg}
#something like this may be needed if the child program can be configured
#with reasonable output buffering
#fileevent $chan writable [list apply [list {chan pump} {
# puts $chan $pump
#}] $chan $pump]
fileevent $chan writable {}
}] $chan $msg $pump]
}
proc process {chan count args} {
puts "sed output: $args"
if {$count < 2} {
communicate $chan hello \n [namespace code [list process $chan [incr count]]] \
[namespace code [list closed $chan]]
} else {
catch {close $chan}
set ::done 0
}
}
proc closed {chan} {
#flush first to catch any pipe error, getting it out of the way in order to grab
#the exist status with [close]
if {[catch {flush $chan} eres eopts]} {
set status [catch {close $chan} eres eopts]
set ::done $status
}
set ::done 0
}
#try this line, which causes a sed error, to see how that's handled
#set chan [open {|sed -l {s/hello/goodbye} 2>@stderr} r+]
set chan [open {|sed -l s/hello/goodbye/ 2>@stderr} r+]
fconfigure $chan -buffering none
communicate $chan hello \n [namespace code [list process $chan 1]] \
[namespace code [list closed $chan]]
vwait ::done
return -code $::doneIt's up to the individual program when to print out its results. The -l option to BSD sed configures sed to print output whenever at least one line of output is ready. -u accomplishes the same for some other versions of sed. The pump feature of this example can be used pump some program-specific value through the channel until the desired output is collected. It's a hack, but depending on the program, might be the only way to accomplish the task. Expect is the fully-featured tool for this type of task. Another approach would be to fork the current process into a producer and a consumer.
Neil Madden - here is a real-life example of interacting with a program through a pipe. The program in question is ispell - a UNIX spell-checking utility. I use it to spell-check the contents of a text widget containing LaTeX markup. There are a number of issues to deal with:
The example does not use [fileevent], as this would complicate this particular example. The options passed to ispell are -a (which makes it non-interactive) and -t (which makes it recognize TeX input).
set contents [split [$text get 1.0 end] \n]
set pipe [open [list | ispell -a -t] r+]
fconfigure $pipe -blocking 0 -buffering line
set ver [gets $pipe] ;# Ignore the initial version line
set linenum 1
foreach line $contents {
set wordnum 1
foreach word [split $line] {
puts $pipe $word ;# Feed word to ispell
while 1 {
set len [gets $pipe res]
if {$len > 0} {
# A valid result
# do stuff
continue
} else {
if {[fblocked $pipe]} {
# No output
break
} elseif {[eof $pipe]} {
# Pipe closed
catch {close $pipe}
return
}
# A blank line - skip
}
}
incr wordnum
}
incr linenum
}Thanks to Kevin Kenny for helping me figure this out.
DDG - The wiki page Spellcheck Widget using Aspell contains an example on how to use aspell or hunspell to check a given text string and how to use the suggestions by aspell/hunspell to replace the wrong words.
Arjen Markus: I have experimented a bit with plain Tcl driving another program. As this needs to work on Windows 95 (NT, ...) as well as UNIX in four or five flavours, I wanted to use plain Tcl, not Expect (however much I would appreciate the chance to do something really useful with Expect - apart from Android :-).
I think it is worth a page of its own, but here is a summary:
set inout [open |[list myprog] r+] fconfigure $inout -buffering line
Buffering might be out of your hands, though, for "real 16-bit commandline applications", which apparently don't have the ability to flush (reliably), except on close.
DDG - 2022-01-12: To start a python3 pipe you have to select the command line arguments *-iu* for interactive mode and unbuffered output. Here an example to simulate the terminal output of the python3 interpreter
DDG - 2022-01-31: Added error catching and -q option for quiet start.
set code "x=1
print(x)
y=x+1
print(y)
# error checking
print(z)
"
proc piperead {pipe args} {
if {![eof $pipe]} {
#puts "read $pipe : $args"
set got [regsub {.+> } [gets $pipe] ""]
puts "$got"
}
}
# 2022-01-31: adding stderr redirection
set pipe [open "|python3 -uiq 2>@1" r+]
fconfigure $pipe -buffering none -blocking false
fileevent $pipe readable [list piperead $pipe]
set pywait [list]
foreach pyline [split $code \n] {
puts stdout ">>> $pyline"
puts $pipe $pyline
# need some delay for the pipe
after 200 [list append pywait ""]
vwait pywait
}
close $pipe
puts "Finished python3 pipe!"
exit 0And this is the output:
>>> x=1 >>> print(x) 1 >>> y=x+1 >>> print(y) 2 >>> # error checking >>> print(z) Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'z' is not defined
DDG - 2022-01-12: To start a R pipe again you have to use the interactive argument *--interactive*, further you can suppress the R welcome message by using the *-q* option. Here an example to mimic the R terminal:
DDG - 2022-01-31: Redirected stderr to stdout to see errors of R-code.
set code "x=1
print(x)
y=x+1
print(y)
print(R.version.string)
print(z)
x=x+2
print(x)
"
proc piperead {pipe args} {
if {![eof $pipe]} {
#puts "read $pipe : $args"
set got [gets $pipe]
if {$got ne ""} {
puts "$got"
}
}
}
set pipe [open "|R -q --interactive 2>@1" r+]
fconfigure $pipe -buffering none -blocking false
fileevent $pipe readable [list piperead $pipe]
set rwait [list]
foreach rline [split $code \n] {
puts $pipe $rline
# need some delay for the pipe
after 100 [list append rwait ""]
vwait rwait
}
close $pipe
puts "Finished R pipe!"
exit 0And here the output:
> x=1 > print(x) [1] 1 > y=x+1 > print(y) [1] 2 > print(R.version.string) [1] "R version 4.1.2 (2021-11-01)" > print(z) Error in print(z) : object 'z' not found > x=x+2 > print(x) [1] 3 Finished R pipe!
DDG - 2022-01-31: Here as well an pipe example for Gnu-Octave (there are problems with buffering and that Octave echos all commands):
set code "x=1
disp(x)
y=x+1
disp(y)
# error ?
disp(z)
x=x+2
disp(x)
"
proc piperead {pipe args} {
if {![eof $pipe]} {
set got [gets $pipe]
if {$got ne "" && $got ne "ans = 0"} {
puts [regsub {.*oct> } $got "> "]
}
}
}
set pipe [open "|octave --interactive --no-gui --norc --silent 2>@1" r+]
fconfigure $pipe -buffering none -blocking false
fileevent $pipe readable [list piperead $pipe]
puts $pipe {PS1("oct> ")}
puts $pipe "page_screen_output(1);"
#puts $pipe "page_output_immediately(1);"
flush $pipe
set rwait [list]
foreach rline [split $code \n] {
flush $pipe
puts stdout "> $rline"
puts $pipe $rline
flush $pipe
after 200 [list append rwait ""]
vwait rwait
}
close $pipe
puts "Finished Octave pipe!"
exit 0And here the output (not perfect, but a starter):
> x=1 > disp(x) > x = 1 > 1 > y=x+1 > y = 2 > disp(y) > 2 > # error ? > disp(z) > error: 'z' undefined near line 1, column 1 > x=x+2 > x = 3 > disp(x) > 3
DDG - 2023-05-24: Here as well an pipe example for Julia-Octave (there are problems with buffering, Julia is not fast echoing to the terminal and even flushing every line does not help, plotting is very slow, so we give one second for every line, so that Julia has time to evaluate the code:
set code "x=1
println(x)
y=x+1
println(y)
println(z)
x=x+2
println(x)
using PyPlot ;# fast than using Plots, but still slow!
ioff();
x = range(0, 1, length=20)
y = x .^ 2
plot(x, y) ;# figure plotting is sooo slow!!
savefig(\"jpipe.png\")
println(\"Done!\")
"
proc piperead {pipe args} {
if {![eof $pipe]} {
#puts "read $pipe : $args"
set got [gets $pipe]
if {$got ne ""} {
puts "$got"
}
} elseif {[fblocked $pipe]} {
# No output
break
} else {
close $pipe
}
}
set pipe [open "|julia -i --threads 1 2>@1" r+]
fconfigure $pipe -buffering line -blocking true
fileevent $pipe readable [list piperead $pipe]
set rwait [list]
foreach rline [split $code \n] {
puts $pipe $rline
puts $pipe "flush(stdout)"
flush $pipe
if {[regexp {^[^s]} $rline]} {
puts "julia> $rline"
}
# need some delay for the pipe
after 1000 [list append rwait ""] ; # wait for Julia to evaluate this line of code
vwait rwait
}And here the output:
[bariuke]$ tclsh julia-pipe.tcl
julia> x=1
1
julia> println(x)
1
julia> y=x+1
2
julia> println(y)
2
julia> println(z)
ERROR: UndefVarError: z not defined
Stacktrace:
[1] top-level scope
@ none:1
julia> x=x+2
3
julia> println(x)
3
julia> using PyPlot
julia> ioff();
julia> x = range(0, 1, length=20)
julia> y = x .^ 2
PyObject <matplotlib.pyplot._IoffContext object at 0x7fb7cf0351b0>
0.0:0.05263157894736842:1.0
julia> plot(x, y)
20-element Vector{Float64}:
0.0
0.0027700831024930744
0.011080332409972297
0.02493074792243767
0.04432132963988919
0.06925207756232686
0.09972299168975068
0.13573407202216065
0.17728531855955676
0.22437673130193903
0.27700831024930744
0.33518005540166207
0.39889196675900274
0.46814404432132967
0.5429362880886426
0.6232686980609419
0.709141274238227
0.8005540166204986
0.8975069252077561
1.0
julia> println("Done!")
1-element Vector{PyCall.PyObject}:
PyObject <matplotlib.lines.Line2D object at 0x7fb7cc92c1f0>
Done!DDG: 2023-05-24 - Anyone an idea to speedup things in Julia?