Version 22 of The filter idiom

Updated 2010-10-12 21:16:24 by Rainald

Richard Suchenwirth 2003-07-09 - After having written the following code repeatedly, I just declare it to be an idiom. The filter function is frequently seen on Unix (and even DOS) tools, and I think was published in the Software Tools book(s):

  • read from stdin (or files specified on command line)
  • write to stdout

Such tools (grep, sed, awk, sort, ...) can easily be combined ("glued") in pipes to do powerful things. A little framework for Tcl scripts with the filter function:

 set about "usage: myFilter ?file...?

 Does something meaningful with the specified files, or stdin.
 "

 #-- This handler contains the real functionality for one stream (or file)
 proc myHandler channel {
   ...
 }
 #-- This prevents lengthy errors if a filter|more is terminated with 'q'
 proc puts! string {catch {puts stdout $string}}

 if {[lsearch $argv "--help"]>=0} {puts $about; exit}

 if {[llength $argv]==0} {
    puts! [myHandler stdin]
 } else {
        foreach file $argv {
            set fp [open $file]
            puts! [myHandler $fp]
            close $fp
        }
 }

Mike Tuxford is a little confused here, although that in itself is not unusual. It appears to me that you provide a method of repeating a single set of functions upon a multiple set of files, whereas unix pipes provide multiple functions upon the returned data passed between the functions. That is, the 1st command passes its stdout to the 2nd command as its stdin, and so on... Perhaps an example of usage might clarify things for me. - RS: Well, the above is the framework for one filter, which you can put into a pipe, but also can draw input from files specified on command line, like this (and similar to e.g. cat or more):

 echo Data | myFilter | more
 cat data.file | myFilter | more
 myFilter *.file | more
 more data.File

You're right that real filters read from stdin only, but many have the added convenience of filenames on command line - that's what I meant, and implemented.

Mike Tuxford says: OK, I see, and that's still a useful thing. Not to mention that you got me thinking and learning. Thanks for the quick response.


Rainald - 2010-10-12 05:34:55

What is wrong with this command line (Windows with installed tcl)?

 blockwise_average.tcl 8 <huge.csv >large.csv

The behaviour: A gray window pops up titled "blockwise_average" and a 0-byte file large.csv is created.

The content of blockwise_average.tcl, probably not related to my problem:

 # STDIN-to-STDOUT filter
 # Single command-line parameter is the number of lines consumed per line of output, default = 100
 # Input is comma-separated, timestamps in ms, followed by 3 integer numbers
 # Output too, timestamps in s, followed by 3 floats

 set nAv [expr {$argc >= 1 ? [lindex $argv 0] : 100}]
 set iAv $nAv while {[gets stdin line] > 0} {
        if {$iAv == $nAv} {set iAv 0; set st 0.; set sx 0.; set sy 0.; set sz 0.}
        set fields [split $line ","]
        lassign $fields t x y z
        set st [expr {$st + $t}]
        set sx [expr {$sx + $x}]
        set sy [expr {$sy + $y}]
        set sz [expr {$sz + $z}]
        incr iAv
        if {$iAv == $nAv} {
                set st [expr {$st / $iAv / 1000.}]
                set sx [expr {$sx / $iAv}]
                set sy [expr {$sy / $iAv}]
                set sz [expr {$sz / $iAv}]
                puts [format "%.1f,%.2f,%.2f,%.2f" $st $sx $sy $sz] }}

AM Nothing is wrong :) What you do not realise is that not tclsh was started but wish, the graphical interface that comes with Tcl/Tk. wish presents an empty window and a console on Windows. You can type in your commands in the console.


Rainald - 2010-10-12 09:38:49

Thank you. I may want to apply a chain of filters (as mentioned in this thread) to lots of input files. Using tclsh instead of wish maybe more appropriate(?)

Therefore, I changed the file association of *.tcl from wish to tclsh and started over again. The result:

 channel "stdin" wasn't opened for reading

As I'm pretty sure that the code has worked before, I'm confused.

P.S.: I backed out to solid ground, implemented the filter in C, but am still interested in a hint to a solution in tcl.


Arts and crafts of Tcl-Tk programming