named pipe

Description

Named pipes in *nix

(This section is adapted from the Linux man page pipe (7) .)

Pipes and FIFOs (also known as named pipes) provide a unidirectional interprocess communication channel. A pipe has a read end and a write end. Data written to the write end of a pipe can be read from the read end. A FIFO (short for "First In, First Out") has a name within the file system, and is opened as a regular file. Any process may open a FIFO, assuming the file permissions allow it, and may then read or write to it as appropriate.

I/O on Pipes and FIFOs

Pipes and FIFOs have exactly the same semantics once open. They provide a byte stream communication channel with no concept of message boundaries.

Pipe Capacity

A pipe has a limited capacity. If the pipe is full, then a write will block or fail, depending on whether the pipe was opened in non-blocking mode. In Linux the pipe capacity is 65536 bytes, but applications should not rely on a particular capacity.

Portability notes

According to POSIX, pipes only need to be unidirectional. Linux implements unidirectional pipes. Portable applications should avoid reliance on bidirectional pipe semantics.

Named pipes in Windows

Windows also supports named pipes, albeit with different semantics. TWAPI provides commands for communicating over named pipes from Tcl.

Event-driven I/O on named pipes

In an interesting c.l.t posting, Colin Macleod describes what it takes to make a Unix FIFO (first in, first out) or named pipe, event-based:

I happen to be working on a project just now where I want to have an event-driven Tcl process read from a fifo. (This is on Solaris 2.6, in case it makes a difference.) My first attempts suggested that this could only be done by polling, but after digging around more I found that I could get fileevent to work as follows:

proc readpipe pipe {
    set data [read $pipe]
    ... process $data ...
}

### Open the fifo, set up event-driven input from it.
proc open_pipe pipename {
    exec /bin/sh -c "sleep 10 > $pipename" &
    set pipe [open $pipename r]
    fconfigure $pipe -blocking 0
    fileevent $pipe readable [list readpipe $pipe]
    set dummy [open $pipename w]
    after 20000 {exec /bin/true}; # to get zombie sleep reaped
}

The tricky bits are:

The process opening the fifo to read will block until another process opens it to write. I get past this by exec-ing a shell which opens the pipe, writes nothing, then exits after 10 seconds. Note - it is critical that the "> $pipename" redirection is done by the shell (after it is forked as a new process) and not by Tcl.
If the fifo is closed by the writing process, fileevent will fire continuously because it sees end-of-file on the fifo. We can avoid this by making sure that at least one process always has the fifo open for writing, so I also open the fifo for writing from the same Tcl process and never write to it or close it. The 10-second delay in step 1 is to give this time to happen.
Finally, I noticed that the exec-ed shell would hang around as a zombie after the sleep finished. The delayed and apparently pointless exec at the end is because running exec again has the side-effect of wait-ing for terminated background processes - documented under Tcl_DetachPids(3).

Lars H, 2008-08-10: My experience on the opening problem is slightly different:

Opening a pipe for reading is no problem, provided that you use the {RDONLY NONBLOCK} access mode; open returns immediately, and you can configure the channel as you wish.
Opening a pipe for writing is hazardous, because the open will block until the pipe has also been opened for reading; the access mode {WRONLY NONBLOCK} is rejected by the OS (throws error). The work-around is to first open the pipe for reading as above, second open it for writing (no NONBLOCK), and third close the read channel to the pipe.

DKF, 2011-09-30: The WRONLY NONBLOCK combo is correct, but you have to deal with a POSIX ENXIO error by rescheduling the attempt to open the pipe a bit later. I've seen reports that without NONBLOCK you can end up having problems in all threads in a process, but I don't know how much credence to give to those reports; after all, it might be a different bug...

Without suggesting that it is 100% bulletproof, this works fairly well:

## ********************************************************
##
## Name: fifo
##
## Description:
## Provide an anonymous fifo using cat.
##
## Parameters:
##
## Usage:
## trivial handler example:
##
## proc handle { fifo } {
##      puts -nonewline [ gets $fifo ]
## }
##
## set fifo [ fifo handle ]
## puts $fifo peep!
##
## Comments:
## The input side of the fifo should never be flushed!
## The fifo can be closed like an ordinary file.

proc fifo { { handler "" } } {
    if { [ catch {
        set fifo [ open |cat a+ ]
        fconfigure $fifo -blocking off
        fconfigure $fifo -buffering none
        if { [ string length $handler ] } {
            fileevent $fifo readable "$handler $fifo"
        }
    } err ] } {
        return -code error "[ myName ]: $err"
    }
    return $fifo
}
## ********************************************************

(Explain how general file append operations can be detected as events only through polling.)

An example can be seen in tailf.

fifo module

dbohdan 2019-04-13: The following is a Tcl 8.6 module for working with named pipes. It creates named pipes and disposes of them automatically using the with pattern.

Use example

% fifo::with-fifos -template /tmp/pipe.io pipe1 pipe2 {
    puts [list fifos: $pipe1 $pipe2]
    exec tac $pipe1 &
    exec wc $pipe2 &
    exec echo hello\nworld | tee $pipe1 $pipe2 > /dev/null
}
fifos: /tmp/pipe_gwrbgf.io /tmp/pipe_7kG3E9.io
      2       2      12 /tmp/pipe_7kG3E9.io
world
hello

Source code

# Copyright (c) 2019, 2021, 2024 D. Bohdan
# License: MIT

package require Tcl 8.6 9

namespace eval fifo {
    variable version 0.1.0
}


proc fifo::mkfifo {{template {}}} {
    close [file tempfile path $template]
    file delete $path

    exec mkfifo $path

    return $path
}


proc fifo::with-fifos args {
    set template {}
    if {{-template} eq [lindex $args 0]} {
        set args [lassign $args _ template]
    }

    if {[llength $args] == 0} {
        error "wrong # args: should be\
               \"with-fifos ?-template template? ?varName ...? script\""
    }

    set varNames [lrange $args 0 end-1]
    set script [lindex $args end]

    try {
        foreach varName $varNames {
            upvar 1 $varName v
            set v [mkfifo $template]
        }

        uplevel 1 $script
    } finally {
        foreach varName $varNames {
            upvar 1 $varName v
            file delete $v
        }
    }
}