Version 57 of coroutine

Updated 2009-11-14 12:05:48 by jima

Consequence of NRE.

It is an implementation of asymmetric stackful coroutines (see Lua's team survey (pdf!) [L1 ] and tutorial [L2 ], and/or the Wikipedia article [L3 ]).

(DKF: Seems to be actually a "generator" according to the definitions referred to on Wikipedia.) (NEM: The definition on wikipedia seems wrong to me. The difference between a generator and a coroutine is that the former can only produce values, whereas the latter can be both a producer and a consumer. In practical terms, this means that yield can both generate a value and receive a value to return: set y [yield $x]. C.f. Python which extended its generators to full coroutines in version 2.5 by adding exactly this capability.)

Three new commands are provided:

For a period these commands resided in tcl::unsupported, but they have been promoted to full Tcl commands.

For the time being MS is maintaining docs at [L4 ].


See also TIP#328. http://tip.tcl.tk/328


CGM has a rough demo of coroutine-enabled event handling.

NEM and Coroutines for event-based programming.

JBR and Coroutines for cooperative multitasking -- Coroutines for pre-emptive multitasking

NEM - Going Loopy with Coroutines.

jima thinks Arjen Markus's page Discrete event modelling with coroutines should be also mentioned here.


jima 2009-11-11 notices goroutines in [L5 ] (and, incidentally, the use of UTF-8 in go language).

  • DKF: Their documentation is not quite as clear as it should be (notably on whether there's shared mutable state available or if not, just how things are protected) but it looks like what they're really doing is lightweight threads, not coroutines. We might want to suggest “borrowing” their channel abstraction though; it's a model that works really well.
  • jima 2009-11-14 ... what about this quick and (very) dirty channel example --> playing channels with coroutines?

From c.l.t.[L6 ]

Just stating what I understand and where I stand.

- A - Experimental coroutines are in HEAD and 8.6a2

     * they do allow new and simpler ways to express things
     * what's in there is experimental infrastructure
     * please play with code and concept, help us make it better

- B - Some design decisions I consider necessary, where the alternatives I saw
fall in one or more of the following categories:

     (a) semantics not clear (to me and/or generally)
     (b) no hope in hell to have it in 8.6 (compat)
     (c) no hope in hell to have it in 8.6 (time)

    Among these:
     * the coroutine bodies run in [uplevel #0]
     * coroutines are not serializable
     * coroutines are like one-shot continuations, not restartable

- C - Some design decisions where made with possibly insufficient info:

    (0a) [coroutine] creates a command and not a special object with subcommands
         Justification:
           (a) provides all that is needed
           (b) does not require new (sub)commands to resume, delete, ...
           (c) leverages Tcl's command management code (for instance, automatic cleanup
               on namespace deletion)
           (d) implementation simplicity
    (0b) [coroutine] runs the body up to the first [yield] instead of just creating the
         command
         Justification:
           (a) this is not really restrictive
           (b) it does not create commands that take different arguments at first and
               successive calls
           (c) implementation simplicity
    (1)  coroutines are self-cleaning
    (2)  [coroutine] requires a command name ([cmd] in the rest of this post]
    (3)  [cmd] takes a single argument: [yield]'s inside return value
    (4)  [yield] takes a single argument: the outside return value
    (5)  [yield] always return an "ok" code, both inside and outside

    Of these, only (5) restricts functionality; it was taken because *I* can
not see how to make the alternative (a) useful, (b) semantically clear, (c) play
nicely with (1). I may be wrong, eager to learn.

     (1) I think is valuable: when the coroutine is exhausted there's nothing
else that can be done with it, so let the command disappear instead of burdening
the caller to clean up.

     The other three are ... irrelevant at this stage! Simple script wrappers
can provide alternative syntax at minimal perf cost. They do not constrain
experimentation in any way. Should we want more than one available, standard
wrappers could be in tcllib or TIPped and "builtin". Should the perf penalty be
too large, they can at any time be coded in C (or even bytecompiled!) as solving
perf bugs.

- D - I am eager to get help making this better

     (a) Looking mostly for "mental bugs", things that are stopping me from
writing a TIP right away:
       * does it make sense? are there inconsistencies and semantic snafus?
       * are there B-type restrictions that could be lifted in time for 8.6?
       * is there more C-type stuff that I am missing, and that is losing
generality?

     (b) Looking for arguments that would tip C-decisions one way or the other;
this includes primarily real use cases. Mainly interested in (1) and (5), and
possibly other decisions that I may have made unconsciously (help me find them!).

AMG: According to MS's documentation, "a yield is only allowed if the C stack is at the same level as it was when the coroutine is invoked." Does this mean that yield cannot be used as the SQLite progress callback?

DKF: In theory, it could. In practice, not with the current code because it recursively calls the interpreter rather than using the NRE evaluation mechanism. The tdbc version might work though (it's founded on NRE-enabled stuff).


AMG: [yield] don't work across multiple interps.

% coroutine foo [interp create] eval yield
yield can only be called in a coroutine
% coroutine foo info coroutine
::foo
% coroutine foo [interp create] eval info coroutine
% # returns empty string

I was considering a design where a script run in a safe interp repeatedly yields to the invoking interp, but I guess I can't do this.

MS Your example is designed to illustrate that you cannot directly yield across interps, which is true. Trying to guess the real aim of the exercise, I suggest something like

set i [interp create]
$i eval coroutine foo mycmd ...
interp alias {} foo $i foo

You can now call foo from the main interp and have the coroutine named foo run in the slave; when the coroutine yields, you are back in the main interp. Is this somehow related to what you really wanted to do?

AMG: Very interesting! I'm not entirely sure I can use this on my current project (I was just casting about for ideas), but I'm sure this will come in handy at some point. Thank you for explaining how to use coroutines across interpreters and for writing NRE/coroutine in the first place.

AMG, much later: Here's more code that mixes coroutines and interps. Unfortunately, it doesn't work, at least not in Tcl 8.6b1. TclExecuteByteCode() fails with "C stack busy".

proc splitscript {input} {
    set slave [interp create -safe]
    $slave eval {unset {*}[info vars]}
    foreach command [$slave eval {info commands}] {
        $slave hide $command
    }
    $slave alias unknown apply {{args} {yield $args}}
    coroutine nextline apply {{slave input} {
        yield
        $slave eval $input
    }} $slave $input
    set result {}
    while {1} {
        set line [nextline]
        if {$line ne ""} {
            lappend result $line
        } else {
            interp delete $slave
            return $result
        }
    }
}

Is this by design, am I up against an unimplemented feature, is this a Tcl bug, or is there a mistake in my code or my understanding?

MS same as above: you cannot yield across interp boundaries; more concretely, cross-interp aliases are not nre-enabled, they consume stack ("C stack busy") and block yield. I know I thought of some security risk in allowing this, but that is not the only reason: enabling this would have meant bigger changes in the core which I thought were unwarranted for a first version.

AMG: This is not quite the same as before, as indicated by the different error message. The yield happens in the same interpreter as the coroutine, which is the requirement not met by the code I posted long ago. An [info coroutine] inside [unknown] returns nextline. But between the invocation of the coroutine and the yield, there are two interpreter switches: the coroutine calls into the slave interpreter, which then calls back into the master, which yields. And as you say, cross-interp aliases are not NRE-enabled. I can't think of any security problems with this, since all the coroutine stuff is done inside a single interpreter. However, I certainly do understand your second concern. You can't NRE-enable the world, not all at once anyway. :^)

AMG: By the way, here's a non-coroutine version of the above:

proc splitscript {input} {
    set slave [interp create -safe]
    $slave eval {unset {*}[info vars]}
    foreach command [$slave eval {info commands}] {
        $slave hide $command
    }
    $slave expose lappend lappend*
    $slave invokehidden proc unknown* {args} {
        lappend* ::script* $args
    }
    $slave invokehidden namespace unknown unknown*
    $slave invokehidden set script* {}
    $slave eval $input
    set script [$slave invokehidden set script*]
    interp delete $slave
    return $script
}

DKF: You can do a "global coroutine" like this:

coroutine foo eval {yield "in the coro 1"; yield "in the coro 2"; return "finishing"}

But that doesn't mean that I think it is a good idea. For example, you don't get nice variable handling this way.

NEM Or, using a lambda:

coroutine foo apply {{} { yield "in the coro 1"; ... }}

DKF: Yes, but with lambdas you get that nice variable handling. That's because it comes from the local variable context that gets pushed as part of the apply (or the procedure if you're using a "traditional" coroutine).


Unconditionally terminate the currently running coroutine:

    rename [info coroutine] ""; yield

Terminating the currently running coroutine, catchable within the coroutine:

    return -level [info level]

LV In an environment which supports threads and safe interps, what questions should a new developer ask when determining the appropriate technology? And are there situations where one might want/need to combine these in ways?


DKF: This example (from the Tcl docs) shows how to generate primes with the sieve of eratosthenes in Tcl using coroutines...

proc filterByFactor {source n} {
    yield [info coroutine]
    while 1 {
        set x [$source]
        if {$x % $n} {
            yield $x
        }
    }
}
coroutine allNumbers apply {{} {while 1 {yield [incr x]}}}
coroutine eratosthenes apply {c {
    yield
    while 1 {
        set n [$c]
        yield $n
        set c [coroutine prime$n filterByFactor $c $n]
    }
}} allNumbers
for {set i 1} {$i <= 20} {incr i} {
    puts "prime#$i = [eratosthenes]"
}

See also:

It might be interesting if someone (TM) attempted a translation of A Curious Course on Coroutines and Concurrency .