'''Reading and writing to a piped command'''

<<toc>>

** See Also **

   * [open]:   
   * [pipe]:   
   * [gzip]:   
   * [unbuffer]:   
   * [How can I run data through an external filter?]:   

** Description **

The methods for accomplishing some task involving writing to and then reading
from a piped command depend primarily on whether the communication will be
interactive or non-interactive.  The non-interactive case is fairly
straight-forward and robust.  The interactive case can be problematic, since
external programs may buffer incoming and outgoing data in arbitrary ways.
[Expect] is the best all-purpose tool for interactive communication with
another program.


** Example: Non-Interactive: gzip **

======
proc gzip {buf} {
    set chan [open "|gzip -c" r+]
    fconfigure $chan -translation binary -encoding binary
    puts $chan $buf
    flush $chan
    chan close $chan write
    set buf [read $chan]
    close $chan
    return $buf
}
======

The `write` side of `$chan` must be closed so that gzip reads any remaining
data in the buffers processes it, and sends the result to its stdtout. 

For versions of Tcl that don't have [chan close ?direction?], try the following
workaround:

======
proc gzip {buf} {
    return [exec gzip -c << $buf]
}
======




** Example: Interactive Conversation with another Program **

======
#! /bin/env tclsh

proc communicate {chan msg pump respond eof } {
    fileevent $chan readable [list apply [list {chan respond eof} {
        set res [gets $chan]
        if {$res ne {}} {
            fileevent $chan readable {}
            {*}$respond $res
        }
        if {[eof $chan]} {
            fileevent $chan readable {}
            {*}$eof
        }
    }] $chan $respond $eof]

    fileevent $chan writable [list apply [list {chan msg pump} {
        catch {puts $chan $msg}
        #something like this may be needed if the child program can be configured
        #with reasonable output buffering
        #fileevent $chan writable [list apply [list {chan pump} {
        #       puts $chan $pump
        #}] $chan $pump]
        fileevent $chan writable {}
    }] $chan $msg $pump]
}

proc process {chan count args} {
    puts "sed output: $args"
    if {$count < 2} {
        communicate $chan hello \n [namespace code [list process $chan [incr count]]] \
            [namespace code [list closed $chan]]
    } else {
        catch {close $chan}
        set ::done 0
    }
}

proc closed {chan} {
    #flush first to catch any pipe error, getting it out of the way in order to grab
    #the exist status with [close]
    if {[catch {flush $chan} eres eopts]} {
        set status [catch {close $chan} eres eopts]
        set ::done $status
    }
    set ::done 0
}

#try this line, which causes a sed error, to see how that's handled
#set chan [open {|sed -l {s/hello/goodbye} 2>@stderr} r+]

set chan [open {|sed -l s/hello/goodbye/ 2>@stderr} r+]
fconfigure $chan -buffering none

communicate $chan hello \n [namespace code [list process $chan 1]] \
    [namespace code [list closed $chan]]


vwait ::done
return -code $::done
======

It's up to the individual program when to print out its results.  The `-l`
option to [BSD] sed configures sed to print output whenever at least one line
of output is ready.  `-u` accomplishes the same for some other versions of sed.
The `pump` feature of this example can be used pump some program-specific value
through the channel until the desired output is collected.  It's a hack, but
depending on the program, might be the only way to accomplish the task.
[Expect] is the fully-featured tool for this type of task.  Another approach
would be to [fork] the current process into a producer and a consumer.



** Example: ispell for a text widget **

[Neil Madden] - here is a real-life example of interacting with a program
through a pipe. The program in question is ''ispell'' - a UNIX spell-checking
utility. I use it to spell-check the contents of a text widget containing
[LaTeX] markup. There are a number of issues to deal with:

   * Keeping track of the position of the word in the widget.
   * Filtering out useless (blank) lines from ispell
   * Filtering out the version info that my version of ispell dumps out.
   * When passing in a word which is a TeX command (e.g. \maketitle), ispell returns nothing at all.
   * Careful handling of blocking.

The example does not use [[`[fileevent]`], as this would complicate this
particular example. The options passed to ispell are -a (which makes it
non-interactive) and -t (which makes it recognize TeX input).

======
set contents [split [$text get 1.0 end] \n]
set pipe [open [list | ispell -a -t] r+]
fconfigure $pipe -blocking 0 -buffering line
set ver [gets $pipe] ;# Ignore the initial version line
set linenum 1
foreach line $contents {
    set wordnum 1
    foreach word [split $line] {
        puts $pipe $word   ;# Feed word to ispell
        while 1 {
            set len [gets $pipe res]
            if {$len > 0} {
                # A valid result
                # do stuff
                continue
            } else {
                if {[fblocked $pipe]} {
                    # No output
                    break
                } elseif {[eof $pipe]} {
                    # Pipe closed
                    catch {close $pipe}
                    return
                }
                # A blank line - skip
            }
        }
        incr wordnum
    }
    incr linenum
}
======

Thanks to [Kevin Kenny] for helping me figure this out.

[DDG] - The wiki page [Spellcheck Widget using Aspell] contains an example on how to use aspell or hunspell to check a given text string and how to use the suggestions by aspell/hunspell to replace the wrong words.

** To Sort: [Arjen Markus] **

[Arjen Markus]: I have experimented a bit with plain Tcl driving another
program. As this needs to work on Windows 95 (NT, ...) as well as UNIX in four
or five flavours, I wanted to use plain Tcl, not Expect (however much I would
appreciate the chance to do something really useful with Expect - apart from
[Android] :-).

I think it is worth a page of its own, but here is a summary:

   * Open the pipeline and make sure buffering is minimal via

======
set inout [open |[list myprog] r+]
fconfigure $inout -buffering line
======

Buffering might be out of your hands, though, for "real 16-bit commandline
applications", which apparently don't have the ability to flush (reliably),
except on close.

   * Set up [[`[fileevent]`] handlers for reading from the process and reading from [stdin].

   * Make sure the process ("myprog" above) does not buffer its output to [stdout]!

** Example: Python 3 pipe **

[DDG] - 2022-01-12: To start a python3 pipe you have to select the command line arguments *-iu* for interactive mode and unbuffered output. Here an example to simulate the terminal output of the python3 interpreter

[DDG] - 2022-01-31: Added error catching and ''-q'' option for quiet start.

======
set code "x=1
print(x)
y=x+1
print(y)
# error checking
print(z)
"

proc piperead {pipe args} {
    if {![eof $pipe]} {
        #puts "read $pipe : $args" 
        set got [regsub {.+> } [gets $pipe] ""]
        puts "$got"
    }   
}
# 2022-01-31: adding stderr redirection
set pipe [open "|python3 -uiq 2>@1" r+]
fconfigure $pipe -buffering none -blocking false
fileevent $pipe readable [list piperead $pipe]
set pywait [list]
foreach pyline [split $code \n] {
    puts stdout ">>> $pyline"
    puts $pipe $pyline
    # need some delay for the pipe
    after 200 [list append pywait ""]
    vwait pywait
}
close $pipe
puts "Finished python3 pipe!"
exit 0
======

And this is the output:

======
>>> x=1
>>> print(x)
1

>>> y=x+1

>>> print(y)
2

>>> # error checking

>>> print(z)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'z' is not defined
======

** Example: R pipe **

[DDG] - 2022-01-12: To start a R pipe again you have to use the interactive argument *--interactive*, further you can suppress the R welcome message by using the *-q* option. Here an example to mimic the R terminal:

[DDG] - 2022-01-31: Redirected stderr to stdout to see errors of R-code.

======
set code "x=1
print(x)
y=x+1
print(y)
print(R.version.string)
print(z)
x=x+2
print(x)
"

proc piperead {pipe args} {
    if {![eof $pipe]} {
        #puts "read $pipe : $args" 
        set got [gets $pipe]
        if {$got ne ""} {
            puts "$got"
        }
    }   
}

set pipe [open "|R -q --interactive 2>@1" r+]
fconfigure $pipe -buffering none -blocking false
fileevent $pipe readable [list piperead $pipe]
set rwait [list]
foreach rline [split $code \n] {
    puts $pipe $rline
    # need some delay for the pipe
    after 100 [list append rwait ""]
    vwait rwait
}
close $pipe
puts "Finished R pipe!"
exit 0
======

And here the output:

======
> x=1
> print(x)
[1] 1
> y=x+1
> print(y)
[1] 2
> print(R.version.string)
[1] "R version 4.1.2 (2021-11-01)"
> print(z)
Error in print(z) : object 'z' not found
> x=x+2
> print(x)
[1] 3
Finished R pipe!
======

** Example: Octave pipe **

[DDG] - 2022-01-31: Here as well an pipe example for Gnu-[Octave] (there are problems with buffering and that Octave echos all commands):

======
set code "x=1
disp(x)
y=x+1
disp(y)
# error ?
disp(z)
x=x+2
disp(x)
"

proc piperead {pipe args} {
    if {![eof $pipe]} {
        set got [gets $pipe]
        if {$got ne "" && $got ne "ans = 0"} {
            puts [regsub {.*oct> } $got "> "]
        }
    }   
}

set pipe [open "|octave --interactive --no-gui --norc --silent 2>@1" r+]
fconfigure $pipe -buffering none -blocking false
fileevent $pipe readable [list piperead $pipe]
puts $pipe {PS1("oct> ")}
puts $pipe "page_screen_output(1);"
#puts $pipe "page_output_immediately(1);"
flush $pipe 

set rwait [list]
foreach rline [split $code \n] {
    flush $pipe 
    puts stdout "> $rline"
    puts $pipe $rline
    flush $pipe
    after 200 [list append rwait ""]
    vwait rwait
}
close $pipe
puts "Finished Octave pipe!"
exit 0
======

And here the output (not perfect, but a starter):

======
> x=1
> disp(x)
> x = 1
> 1
> y=x+1
> y = 2
> disp(y)
> 2
> # error ?
> disp(z)
> error: 'z' undefined near line 1, column 1
> x=x+2
> x = 3
> disp(x)
> 3
====== 

** Example: Julia pipe **

[DDG] - 2023-05-24: Here as well an pipe example for Julia-[Octave] (there are problems with buffering, Julia is not fast echoing to the terminal and even flushing every line does not help, plotting is very slow, so we give one second for every line, so that Julia has time to evaluate the code:

======
set code "x=1
println(x)
y=x+1
println(y)
println(z)
x=x+2
println(x)
using PyPlot ;# fast than using Plots, but still slow!
ioff();
x = range(0, 1, length=20)
y = x .^ 2
plot(x, y) ;# figure plotting is sooo slow!!
savefig(\"jpipe.png\")
println(\"Done!\")        
"

proc piperead {pipe args} {
    if {![eof $pipe]} {
        #puts "read $pipe : $args" 
        set got [gets $pipe]
        if {$got ne ""} {
            puts "$got"
        }
    } elseif {[fblocked $pipe]} {
        # No output
        break
    } else {
        close $pipe
    }
}

set pipe [open "|julia -i --threads 1 2>@1" r+]
fconfigure $pipe -buffering line -blocking true
fileevent $pipe readable [list piperead $pipe]
set rwait [list]
foreach rline [split $code \n] {
    puts $pipe $rline
    puts $pipe "flush(stdout)"
    flush $pipe
    if {[regexp {^[^s]} $rline]} {
        puts "julia> $rline"
    }
    # need some delay for the pipe
    after 1000 [list append rwait ""] ; # wait for Julia to evaluate this line of code
    vwait rwait
}
======

And here the output:

======
[bariuke]$ tclsh julia-pipe.tcl
julia> x=1
1
julia> println(x)
1
julia> y=x+1
2
julia> println(y)
2
julia> println(z)
ERROR: UndefVarError: z not defined
Stacktrace:
 [1] top-level scope
   @ none:1
julia> x=x+2
3
julia> println(x)
3
julia> using PyPlot
julia> ioff();
julia> x = range(0, 1, length=20)
julia> y = x .^ 2
PyObject <matplotlib.pyplot._IoffContext object at 0x7fb7cf0351b0>
0.0:0.05263157894736842:1.0
julia> plot(x, y)
20-element Vector{Float64}:
 0.0
 0.0027700831024930744
 0.011080332409972297
 0.02493074792243767
 0.04432132963988919
 0.06925207756232686
 0.09972299168975068
 0.13573407202216065
 0.17728531855955676
 0.22437673130193903
 0.27700831024930744
 0.33518005540166207
 0.39889196675900274
 0.46814404432132967
 0.5429362880886426
 0.6232686980609419
 0.709141274238227
 0.8005540166204986
 0.8975069252077561
 1.0
julia> println("Done!")        
1-element Vector{PyCall.PyObject}:
 PyObject <matplotlib.lines.Line2D object at 0x7fb7cc92c1f0>
Done!
======

DDG: 2023-05-24 - Anyone an idea to speedup things in Julia?

<<categories>> Example | Channel | Interprocess Communication