'''Reading and writing to a piped command'''

<<toc>>

** See Also **

   * [open]:   
   * [pipe]:   
   * [gzip]:   
   * [unbuffer]:   
   * [How can I run data through an external filter?]:   

** Description **

The methods for accomplishing some task involving writing to and then reading
from a piped command depend primarily on whether the communication will be
interactive or non-interactive.  The non-interactive case is fairly
straight-forward and robust.  The interactive case can be problematic, since
external programs may buffer incoming and outgoing data in arbitrary ways.
[Expect] is the best all-purpose tool for interactive communication with
another program.


** Example: Non-Interactive: gzip **

======
proc gzip {buf} {
    set chan [open "|gzip -c" r+]
    fconfigure $chan -translation binary -encoding binary
    puts $chan $buf
    flush $chan
    chan close $chan write
    set buf [read $chan]
    close $chan
    return $buf
}
======

The `write` side of `$chan` must be closed so that gzip reads any remaining
data in the buffers processes it, and sends the result to its stdtout. 

For versions of Tcl that don't have [chan close ?direction?], try the following
workaround:

======
proc gzip {buf} {
    return [exec gzip -c << $buf]
}
======




** Example: Interactive Conversation with another Program **

======
#! /bin/env tclsh

proc communicate {chan msg pump respond eof } {
    fileevent $chan readable [list apply [list {chan respond eof} {
        set res [gets $chan]
        if {$res ne {}} {
            fileevent $chan readable {}
            {*}$respond $res
        }
        if {[eof $chan]} {
            fileevent $chan readable {}
            {*}$eof
        }
    }] $chan $respond $eof]

    fileevent $chan writable [list apply [list {chan msg pump} {
        catch {puts $chan $msg}
        #something like this may be needed if the child program can be configured
        #with reasonable output buffering
        #fileevent $chan writable [list apply [list {chan pump} {
        #       puts $chan $pump
        #}] $chan $pump]
        fileevent $chan writable {}
    }] $chan $msg $pump]
}

proc process {chan count args} {
    puts "sed output: $args"
    if {$count < 2} {
        communicate $chan hello \n [namespace code [list process $chan [incr count]]] \
            [namespace code [list closed $chan]]
    } else {
        catch {close $chan}
        set ::done 0
    }
}

proc closed {chan} {
    #flush first to catch any pipe error, getting it out of the way in order to grab
    #the exist status with [close]
    if {[catch {flush $chan} eres eopts]} {
        set status [catch {close $chan} eres eopts]
        set ::done $status
    }
    set ::done 0
}

#try this line, which causes a sed error, to see how that's handled
#set chan [open {|sed -l {s/hello/goodbye} 2>@stderr} r+]

set chan [open {|sed -l s/hello/goodbye/ 2>@stderr} r+]
fconfigure $chan -buffering none

communicate $chan hello \n [namespace code [list process $chan 1]] \
    [namespace code [list closed $chan]]


vwait ::done
return -code $::done
======

It's up to the individual program when to print out its results.  The `-l`
option to [BSD] sed configures sed to print output whenever at least one line
of output is ready.  `-u` accomplishes the same for some other versions of sed.
The `pump` feature of this example can be used pump some program-specific value
through the channel until the desired output is collected.  It's a hack, but
depending on the program, might be the only way to accomplish the task.
[Expect] is the fully-featured tool for this type of task.  Another approach
would be to [fork] the current process into a producer and a consumer.



** Example: ispell for a text widget **

[Neil Madden] - here is a real-life example of interacting with a program
through a pipe. The program in question is ''ispell'' - a UNIX spell-checking
utility. I use it to spell-check the contents of a text widget containing
[LaTeX] markup. There are a number of issues to deal with:

   * Keeping track of the position of the word in the widget.
   * Filtering out useless (blank) lines from ispell
   * Filtering out the version info that my version of ispell dumps out.
   * When passing in a word which is a TeX command (e.g. \maketitle), ispell returns nothing at all.
   * Careful handling of blocking.

The example does not use [[`[fileevent]`], as this would complicate this
particular example. The options passed to ispell are -a (which makes it
non-interactive) and -t (which makes it recognize TeX input).

======
set contents [split [$text get 1.0 end] \n]
set pipe [open [list | ispell -a -t] r+]
fconfigure $pipe -blocking 0 -buffering line
set ver [gets $pipe] ;# Ignore the initial version line
set linenum 1
foreach line $contents {
    set wordnum 1
    foreach word [split $line] {
        puts $pipe $word   ;# Feed word to ispell
        while 1 {
            set len [gets $pipe res]
            if {$len > 0} {
                # A valid result
                # do stuff
                continue
            } else {
                if {[fblocked $pipe]} {
                    # No output
                    break
                } elseif {[eof $pipe]} {
                    # Pipe closed
                    catch {close $pipe}
                    return
                }
                # A blank line - skip
            }
        }
        incr wordnum
    }
    incr linenum
}
======

Thanks to [Kevin Kenny] for helping me figure this out.

[DDG] - The wiki page [Spellcheck Widget using Aspell] contains an example on how to use aspell or hunspell to check a given text string and how to use the suggestions by aspell/hunspell to replace the wrong words.

** To Sort: [Arjen Markus] **

[Arjen Markus]: I have experimented a bit with plain Tcl driving another
program. As this needs to work on Windows 95 (NT, ...) as well as UNIX in four
or five flavours, I wanted to use plain Tcl, not Expect (however much I would
appreciate the chance to do something really useful with Expect - apart from
[Android] :-).

I think it is worth a page of its own, but here is a summary:

   * Open the pipeline and make sure buffering is minimal via

======
set inout [open |[list myprog] r+]
fconfigure $inout -buffering line
======

Buffering might be out of your hands, though, for "real 16-bit commandline
applications", which apparently don't have the ability to flush (reliably),
except on close.

   * Set up [[`[fileevent]`] handlers for reading from the process and reading from [stdin].

   * Make sure the process ("myprog" above) does not buffer its output to [stdout]!

** Example: Python 3 pipe **

[DDG] - 2022-01-12: To start a python3 pipe you have to select the command line arguments *-iu* for interactive mode and unbuffered output. Here an example to simulate the terminal output of the python3 interpreter

[DDG] - 2022-01-31: Added error catching and ''-q'' option for quiet start.

======
set code "x=1
print(x)
y=x+1
print(y)
# error checking
print(z)
"

proc piperead {pipe args} {
    if {![eof $pipe]} {
        #puts "read $pipe : $args" 
        set got [regsub {.+> } [gets $pipe] ""]
        puts "$got"
    }   
}
# 2022-01-31: adding stderr redirection
set pipe [open "|python3 -uiq 2>@1" r+]
fconfigure $pipe -buffering none -blocking false
fileevent $pipe readable [list piperead $pipe]
set pywait [list]
foreach pyline [split $code \n] {
    puts stdout ">>> $pyline"
    puts $pipe $pyline
    # need some delay for the pipe
    after 200 [list append pywait ""]
    vwait pywait
}
close $pipe
puts "Finished python3 pipe!"
exit 0
======

And this is the output:

======
>>> x=1
>>> print(x)
1

>>> y=x+1

>>> print(y)
2

>>> # error checking

>>> print(z)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'z' is not defined
======

** Example: R pipe **

[DDG] - 2022-01-12: To start a R pipe again you have to use the interactive argument *--interactive*, further you can suppress the R welcome message by using the *-q* option. Here an example to mimic the R terminal:

[DDG] - 2022-01-31: Redirected stderr to stdout to see errors of R-code.

======
set code "x=1
print(x)
y=x+1
print(y)
print(R.version.string)
print(z)
x=x+2
print(x)
"

proc piperead {pipe args} {
    if {![eof $pipe]} {
        #puts "read $pipe : $args" 
        set got [gets $pipe]
        if {$got ne ""} {
            puts "$got"
        }
    }   
}

set pipe [open "|R -q --interactive 2>@1" r+]
fconfigure $pipe -buffering none -blocking false
fileevent $pipe readable [list piperead $pipe]
set rwait [list]
foreach rline [split $code \n] {
    puts $pipe $rline
    # need some delay for the pipe
    after 100 [list append rwait ""]
    vwait rwait
}
close $pipe
puts "Finished R pipe!"
exit 0
======

And here the output:

======
> x=1
> print(x)
[1] 1
> y=x+1
> print(y)
[1] 2
> print(R.version.string)
[1] "R version 4.1.2 (2021-11-01)"
> print(z)
Error in print(z) : object 'z' not found
> x=x+2
> print(x)
[1] 3
Finished R pipe!
======

** Example: Octave pipe **

[DDG] - 2022-01-31: Here as well an pipe example for Gnu-Octave (there are problems with buffering and that Octave echos all commands):

======
set code "x=1
disp(x)
y=x+1
disp(y)
# error ?
disp(z)
x=x+2
disp(x)
"

proc piperead {pipe args} {
    if {![eof $pipe]} {
        set got [gets $pipe]
        if {$got ne "" && $got ne "ans = 0"} {
            puts [regsub {.*oct> } $got "> "]
        }
    }   
}

set pipe [open "|octave --interactive --no-gui --norc --silent 2>@1" r+]
fconfigure $pipe -buffering none -blocking false
fileevent $pipe readable [list piperead $pipe]
puts $pipe {PS1("oct> ")}
puts $pipe "page_screen_output(1);"
#puts $pipe "page_output_immediately(1);"
flush $pipe 

set rwait [list]
foreach rline [split $code \n] {
    flush $pipe 
    puts stdout "> $rline"
    puts $pipe $rline
    flush $pipe
    after 200 [list append rwait ""]
    vwait rwait
}
close $pipe
puts "Finished Octave pipe!"
exit 0
======

And here the output (not perfect, but a starter):

======
> x=1
> disp(x)
> x = 1
> 1
> y=x+1
> y = 2
> disp(y)
> 2
> # error ?
> disp(z)
> error: 'z' undefined near line 1, column 1
> x=x+2
> x = 3
> disp(x)
> 3
====== 



<<categories>> Example | Channel | Interprocess Communication