Version 4 of bexec

Updated 2014-05-16 10:35:28 by RLE

samoc: How to do binary-safe "exec":

proc bexec {command input} {
    # Execute shell "command", send "input" to stdin, return stdout.
    # Ignores stderr (but "2>@1" can be part of "command").
    # Supports binary input and output. e.g.:
    #     set flac_data [bexec {flac -} $wav_data]

    # Run "command" in background...
    set f [open |$command {RDWR BINARY}]
    fconfigure $f -blocking 0

    # Connect read function to collect "command" output...
    set ::bexec_done.$f 0
    set ::bexec_output.$f {}
    fileevent $f readable [list bexec_read $f]

    # Send "input" to command...
    puts -nonewline $f $input
    close $f write

    # Wait for read function to signal "done"...
    vwait ::bexec_done.$f

    # Retrieve output...
    set result [set ::bexec_output.$f]
    unset ::bexec_output.$f
    unset ::bexec_done.$f

    fconfigure $f -blocking 1
    try {
        close $f
    } trap {CHILDSTATUS} {options info} {
        dict set info -errorinfo $result
        return -options $info $result
    }

    return $result
}


proc bexec_read {f} {
    # Accumulate output in ::bexec_output.$f.

    append ::bexec_output.$f [read $f]
    if {[eof $f]} {
        fileevent $f readable {}
        set ::bexec_done.$f 1
    }
}

PYK 2014-05-11: The code above does not need to go to the effort of setting the channel to non-blocking, as Tcl will manage the buffer behind the scenes. Forget about setting the channel to non-blocking, dispense with the vwait, send the desired data into the channel, close the write side of the channel, and then read the output. For code that does need to use non-blocking channels, it's generally best not to interfere with the event loop by using vwait as the code above does. Instead, consider structuring the code such that it works in an event-oriented manner. See also Example of reading and writing to a piped command, which provides a template for conducting an interactive conversation with another process.

samoc 2014-05-16: It would be great to have a simpler way to execute a command in the background. However, I can't see how to make it work for large binary data without "fconfigure $f -blocking 0". Example (platform::identify > macosx10.9-x86_64):

set wav [read [open ~/stuff/foo.wav {RDONLY BINARY}]]
set flac [bexec {flac - --totally-silent} $wav]

In this case "fconfigure $f -blocking 0" is required. Without non-blocking mode, "puts -nonewline $f $input" never returns. My assumption is that flac reads some (but not all) of the input from its stdin side of the pipe, then tries to write some output to its stdout side of the pipe. There is no-one reading flac's output pipe yet, so it blocks on output. We're blocked on sending input to flac, it is blocked on sending output back to us. Deadlock.

I expect that this behaviour is common for filter type programs that deal with large amounts of data. It is impractical for these programs to buffer the entire input in RAM, so they process a chunk at a time and send the output to stdout as they go.

WRT being "event-oriented". The aim here is to expose a simple, blocking, non-event oriented interface for executing an external command. The whole point is to hide the messy event processing detail.

It would be nice if "exec flac - << $wav" worked. But exec is broken for binary input/output.