samoc: How to do binary-safe "exec":

======
proc bexec {command input} {
    # Execute shell "command", send "input" to stdin, return stdout.
    # Ignores stderr (but "2>@1" can be part of "command").
    # Supports binary input and output. e.g.:
    #     set flac_data [bexec {flac -} $wav_data]

    # Run "command" in background...
    set f [open |$command {RDWR BINARY}]
    fconfigure $f -blocking 0

    # Connect read function to collect "command" output...
    set ::bexec_done.$f 0
    set ::bexec_output.$f {}
    fileevent $f readable [list bexec_read $f]

    # Send "input" to command...
    puts -nonewline $f $input
    close $f write

    # Wait for read function to signal "done"...
    vwait ::bexec_done.$f

    # Retrieve output...
    set result [set ::bexec_output.$f]
    unset ::bexec_output.$f
    unset ::bexec_done.$f

    fconfigure $f -blocking 1
    try {
        close $f
    } trap {CHILDSTATUS} {options info} {
        dict set info -errorinfo $result
        return -options $info $result
    }

    return $result
}


proc bexec_read {f} {
    # Accumulate output in ::bexec_output.$f.

    append ::bexec_output.$f [read $f]
    if {[eof $f]} {
        fileevent $f readable {}
        set ::bexec_done.$f 1
    }
}
======


[PYK] 2014-05-11:  The code above does not need to go to the effort of setting
the channel to non-blocking, as Tcl will manage the buffer behind the scenes.
Forget about setting the channel to non-blocking, dispense with the
`[vwait]`, send the desired data into the channel, close the `write` side of
the channel, and then read the output. For code that does need to use
non-blocking channels, it's generally best not to interfere with the event
loop by using `[vwait]` as the code above does. Instead, consider structuring
the code such that it works in an [event-oriented programming%|%event-oriented]
manner. See also [example of reading and writing to a piped command], which
provides a template for conducting an interactive conversation with another
process.

[samoc] 2014-05-16: It would be great to have a simpler way to execute a command in the background.
However, I can't see how to make it work for large binary data without `fconfigure $f -blocking 0`.
Example (platform::identify > macosx10.9-x86_64):

======
set wav [read [open ~/stuff/foo.wav {RDONLY BINARY}]]
set flac [bexec {flac - --totally-silent} $wav]
======

In this case `fconfigure $f -blocking 0` is required.
Without non-blocking mode, "puts -nonewline $f $input" never returns.
My assumption is that flac reads some (but not all) of the input from its stdin side of the pipe, then tries to write some output to its stdout side of the pipe.
There is no-one reading flac's output pipe yet, so it blocks on output.
We're blocked on sending input to flac, it is blocked on sending output back to us.
Deadlock.

I expect that this behaviour is common for filter type programs that deal with large amounts of data. It is impractical for these programs to buffer the entire input in RAM, so they process a chunk at a time and send the output to stdout as they go.

WRT being "event-oriented". The aim here is to expose a simple, blocking, non-event oriented interface for executing an external command. The whole point is to hide the messy event processing detail.

It would be nice if `exec flac - << $wav` worked. But exec is broken for binary input/output.

[PYK] 2014-05-16: `exec flac - <@$wavchan` works fine for me.  Do you have any
evidence to support the claim that ''"exec is broken for binary input/output"''
?  It is best to [Update considered harmful%|%avoid vwait].  Below are three
different methods for piping binary data to [https://xiph.org/flac/%|%flac].
The third method polls and uses `[after]`, but even that is preferable to
`[vwait]` if the script author doesn't want to design the code around `[chan
event%|%file events]`.

======
#! /bin/env tclsh

#method 1:  redirection of stdin and stdout
set chan [open ztest1.wav {RDONLY BINARY}]
exec flac - <@$chan 2>@stderr >ztestout1.flac

#method 2:  redirect data into flac, read flac output into Tcl,  then write to channel 
seek $chan 0
set chan2 [open |[list flac - <@$chan] {RDONLY BINARY}]
set chanout [open ztestout2.flac {BINARY CREAT WRONLY}]
while {![eof $chan2]} {
    puts -nonewline $chanout [read $chan2 32768]
}
close $chan2
close $chanout

#method 3: read output, write to flac, read flac output, write to channel
seek $chan 0
set chan2 [open |[list flac -] r+]
chan configure $chan2 -translation binary -blocking 0
set chan3 [open ztestout3.flac {BINARY CREAT WRONLY}]
set done 0
while 1 {
    if {!$done} {
        if {[eof $chan]} {
            close $chan2 write
            set done 1
        } else {
            set data [read $chan 32768]
            puts -nonewline $chan2 $data 
        }
    }
    set data [read $chan2 32768]
    if {$data eq {}} {
        if {[eof $chan2]} {
            break
        }
    } else {
        puts -nonewline $chan3 $data 
    }
    after 1
}

close $chan
close $chan3
======


<<categories>> Binary Data