Version 24 of Example of reading and writing to a piped command

Updated 2022-01-31 08:29:17 by DDG

Reading and writing to a piped command

See Also

open
pipe
gzip
unbuffer
How can I run data through an external filter?

Description

The methods for accomplishing some task involving writing to and then reading from a piped command depend primarily on whether the communication will be interactive or non-interactive. The non-interactive case is fairly straight-forward and robust. The interactive case can be problematic, since external programs may buffer incoming and outgoing data in arbitrary ways. Expect is the best all-purpose tool for interactive communication with another program.

Example: Non-Interactive: gzip

proc gzip {buf} {
    set chan [open "|gzip -c" r+]
    fconfigure $chan -translation binary -encoding binary
    puts $chan $buf
    flush $chan
    chan close $chan write
    set buf [read $chan]
    close $chan
    return $buf
}

The write side of $chan must be closed so that gzip reads any remaining data in the buffers processes it, and sends the result to its stdtout.

For versions of Tcl that don't have chan close ?direction?, try the following workaround:

proc gzip {buf} {
    return [exec gzip -c << $buf]
}

Example: Interactive Conversation with another Program

#! /bin/env tclsh

proc communicate {chan msg pump respond eof } {
    fileevent $chan readable [list apply [list {chan respond eof} {
        set res [gets $chan]
        if {$res ne {}} {
            fileevent $chan readable {}
            {*}$respond $res
        }
        if {[eof $chan]} {
            fileevent $chan readable {}
            {*}$eof
        }
    }] $chan $respond $eof]

    fileevent $chan writable [list apply [list {chan msg pump} {
        catch {puts $chan $msg}
        #something like this may be needed if the child program can be configured
        #with reasonable output buffering
        #fileevent $chan writable [list apply [list {chan pump} {
        #       puts $chan $pump
        #}] $chan $pump]
        fileevent $chan writable {}
    }] $chan $msg $pump]
}

proc process {chan count args} {
    puts "sed output: $args"
    if {$count < 2} {
        communicate $chan hello \n [namespace code [list process $chan [incr count]]] \
            [namespace code [list closed $chan]]
    } else {
        catch {close $chan}
        set ::done 0
    }
}

proc closed {chan} {
    #flush first to catch any pipe error, getting it out of the way in order to grab
    #the exist status with [close]
    if {[catch {flush $chan} eres eopts]} {
        set status [catch {close $chan} eres eopts]
        set ::done $status
    }
    set ::done 0
}

#try this line, which causes a sed error, to see how that's handled
#set chan [open {|sed -l {s/hello/goodbye} 2>@stderr} r+]

set chan [open {|sed -l s/hello/goodbye/ 2>@stderr} r+]
fconfigure $chan -buffering none

communicate $chan hello \n [namespace code [list process $chan 1]] \
    [namespace code [list closed $chan]]


vwait ::done
return -code $::done

It's up to the individual program when to print out its results. The -l option to BSD sed configures sed to print output whenever at least one line of output is ready. -u accomplishes the same for some other versions of sed. The pump feature of this example can be used pump some program-specific value through the channel until the desired output is collected. It's a hack, but depending on the program, might be the only way to accomplish the task. Expect is the fully-featured tool for this type of task. Another approach would be to fork the current process into a producer and a consumer.

Example: ispell for a text widget

Neil Madden - here is a real-life example of interacting with a program through a pipe. The program in question is ispell - a UNIX spell-checking utility. I use it to spell-check the contents of a text widget containing LaTeX markup. There are a number of issues to deal with:

  • Keeping track of the position of the word in the widget.
  • Filtering out useless (blank) lines from ispell
  • Filtering out the version info that my version of ispell dumps out.
  • When passing in a word which is a TeX command (e.g. \maketitle), ispell returns nothing at all.
  • Careful handling of blocking.

The example does not use [fileevent], as this would complicate this particular example. The options passed to ispell are -a (which makes it non-interactive) and -t (which makes it recognize TeX input).

set contents [split [$text get 1.0 end] \n]
set pipe [open [list | ispell -a -t] r+]
fconfigure $pipe -blocking 0 -buffering line
set ver [gets $pipe] ;# Ignore the initial version line
set linenum 1
foreach line $contents {
    set wordnum 1
    foreach word [split $line] {
        puts $pipe $word   ;# Feed word to ispell
        while 1 {
            set len [gets $pipe res]
            if {$len > 0} {
                # A valid result
                # do stuff
                continue
            } else {
                if {[fblocked $pipe]} {
                    # No output
                    break
                } elseif {[eof $pipe]} {
                    # Pipe closed
                    catch {close $pipe}
                    return
                }
                # A blank line - skip
            }
        }
        incr wordnum
    }
    incr linenum
}

Thanks to Kevin Kenny for helping me figure this out.

DDG - The wiki page Spellcheck Widget using Aspell contains an example on how to use aspell or hunspell to check a given text string and how to use the suggestions by aspell/hunspell to replace the wrong words.

To Sort: Arjen Markus

Arjen Markus: I have experimented a bit with plain Tcl driving another program. As this needs to work on Windows 95 (NT, ...) as well as UNIX in four or five flavours, I wanted to use plain Tcl, not Expect (however much I would appreciate the chance to do something really useful with Expect - apart from Android :-).

I think it is worth a page of its own, but here is a summary:

  • Open the pipeline and make sure buffering is minimal via
set inout [open |[list myprog] r+]
fconfigure $inout -buffering line

Buffering might be out of your hands, though, for "real 16-bit commandline applications", which apparently don't have the ability to flush (reliably), except on close.

  • Set up [fileevent] handlers for reading from the process and reading from stdin.
  • Make sure the process ("myprog" above) does not buffer its output to stdout!

Example: Python 3 pipe

DDG - 2022-01-12: To start a python3 pipe you have to select the command line arguments *-iu* for interactive mode and unbuffered output. Here an example to simulate the terminal output of the python3 interpreter

DDG - 2022-01-31: Added error catching and -q option for quiet start.

set code "x=1
print(x)
y=x+1
print(y)
# error checking
print(z)
"

proc piperead {pipe args} {
    if {![eof $pipe]} {
        #puts "read $pipe : $args" 
        set got [regsub {.+> } [gets $pipe] ""]
        puts "$got"
    }   
}
# 2022-01-31: adding stderr redirection
set pipe [open "|python3 -uiq 2>@1" r+]
fconfigure $pipe -buffering none -blocking false
fileevent $pipe readable [list piperead $pipe]
set pywait [list]
foreach pyline [split $code \n] {
    puts stdout ">>> $pyline"
    puts $pipe $pyline
    # need some delay for the pipe
    after 200 [list append pywait ""]
    vwait pywait
}
close $pipe
puts "Finished python3 pipe!"
exit 0

And this is the output:

>>> x=1
>>> print(x)
1

>>> y=x+1

>>> print(y)
2

>>> # error checking

>>> print(z)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'z' is not defined

Example: R pipe

DDG - 2022-01-12: To start a R pipe again you have to use the interactive argument *--interactive*, further you can suppress the R welcome message by using the *-q* option. Here an example to mimic the R terminal:

DDG - 2022-01-31: Redirected stderr to stdout to see errors of R-code.

set code "x=1
print(x)
y=x+1
print(y)
print(R.version.string)
print(z)
x=x+2
print(x)
"

proc piperead {pipe args} {
    if {![eof $pipe]} {
        #puts "read $pipe : $args" 
        set got [gets $pipe]
        if {$got ne ""} {
            puts "$got"
        }
    }   
}

set pipe [open "|R -q --interactive 2>@1" r+]
fconfigure $pipe -buffering none -blocking false
fileevent $pipe readable [list piperead $pipe]
set rwait [list]
foreach rline [split $code \n] {
    puts $pipe $rline
    # need some delay for the pipe
    after 100 [list append rwait ""]
    vwait rwait
}
close $pipe
puts "Finished R pipe!"
exit 0

And here the output:

> x=1
> print(x)
[1] 1
> y=x+1
> print(y)
[1] 2
> print(R.version.string)
[1] "R version 4.1.2 (2021-11-01)"
> print(z)
Error in print(z) : object 'z' not found
> x=x+2
> print(x)
[1] 3
Finished R pipe!