jima 2009-11-11 notices goroutines in [L5 ] (and, incidentally, the use of UTF-8 in go language).

• DKF: Their documentation is not quite as clear as it should be (notably on whether there's shared mutable state available or if not, just how things are protected) but it looks like what they're really doing is lightweight threads, not coroutines. We might want to suggest “borrowing” their channel abstraction though; it's a model that works really well.
• jima 2009-11-14 ... what about this quick and (very) dirty channel example --> playing channels with coroutines?

From c.l.t.[L6 ]

Just stating what I understand and where I stand.

- A - Experimental coroutines are in HEAD and 8.6a2

     * they do allow new and simpler ways to express things
     * what's in there is experimental infrastructure
     * please play with code and concept, help us make it better

- B - Some design decisions I consider necessary, where the alternatives I saw
fall in one or more of the following categories:

     (a) semantics not clear (to me and/or generally)
     (b) no hope in hell to have it in 8.6 (compat)
     (c) no hope in hell to have it in 8.6 (time)

    Among these:
     * the coroutine bodies run in [uplevel #0]
     * coroutines are not serializable
     * coroutines are like one-shot continuations, not restartable

- C - Some design decisions where made with possibly insufficient info:

    (0a) [coroutine] creates a command and not a special object with subcommands
         Justification:
           (a) provides all that is needed
           (b) does not require new (sub)commands to resume, delete, ...
           (c) leverages Tcl's command management code (for instance, automatic cleanup
               on namespace deletion)
           (d) implementation simplicity
    (0b) [coroutine] runs the body up to the first [yield] instead of just creating the
         command
         Justification:
           (a) this is not really restrictive
           (b) it does not create commands that take different arguments at first and
               successive calls
           (c) implementation simplicity
    (1)  coroutines are self-cleaning
    (2)  [coroutine] requires a command name ([cmd] in the rest of this post]
    (3)  [cmd] takes a single argument: [yield]'s inside return value
    (4)  [yield] takes a single argument: the outside return value
    (5)  [yield] always return an "ok" code, both inside and outside

    Of these, only (5) restricts functionality; it was taken because *I* can
not see how to make the alternative (a) useful, (b) semantically clear, (c) play
nicely with (1). I may be wrong, eager to learn.

     (1) I think is valuable: when the coroutine is exhausted there's nothing
else that can be done with it, so let the command disappear instead of burdening
the caller to clean up.

     The other three are ... irrelevant at this stage! Simple script wrappers
can provide alternative syntax at minimal perf cost. They do not constrain
experimentation in any way. Should we want more than one available, standard
wrappers could be in tcllib or TIPped and "builtin". Should the perf penalty be
too large, they can at any time be coded in C (or even bytecompiled!) as solving
perf bugs.

- D - I am eager to get help making this better

     (a) Looking mostly for "mental bugs", things that are stopping me from
writing a TIP right away:
       * does it make sense? are there inconsistencies and semantic snafus?
       * are there B-type restrictions that could be lifted in time for 8.6?
       * is there more C-type stuff that I am missing, and that is losing
generality?

     (b) Looking for arguments that would tip C-decisions one way or the other;
this includes primarily real use cases. Mainly interested in (1) and (5), and
possibly other decisions that I may have made unconsciously (help me find them!).

AMG: According to MS's documentation, "a yield is only allowed if the C stack is at the same level as it was when the coroutine is invoked." Does this mean that yield cannot be used as the SQLite progress callback?

DKF: In theory, it could. In practice, not with the current code because it recursively calls the interpreter rather than using the NRE evaluation mechanism. The tdbc version might work though (it's founded on NRE-enabled stuff).

AMG: yield don't work across multiple interps.

% coroutine foo [interp create] eval yield
yield can only be called in a coroutine
% coroutine foo info coroutine
::foo
% coroutine foo [interp create] eval info coroutine
% # returns empty string

I was considering a design where a script run in a safe interp repeatedly yields to the invoking interp, but I guess I can't do this.

MS Your example is designed to illustrate that you cannot directly yield across interps, which is true. Trying to guess the real aim of the exercise, I suggest something like

set i [interp create]
$i eval coroutine foo mycmd ...
interp alias {} foo $i foo

You can now call foo from the main interp and have the coroutine named foo run in the slave; when the coroutine yields, you are back in the main interp. Is this somehow related to what you really wanted to do?

AMG: Very interesting! I'm not entirely sure I can use this on my current project (I was just casting about for ideas), but I'm sure this will come in handy at some point. Thank you for explaining how to use coroutines across interpreters and for writing NRE/coroutine in the first place.

AMG, much later: Here's more code that mixes coroutines and interps. Unfortunately, it doesn't work, at least not in Tcl 8.6b1. TclExecuteByteCode() fails with "C stack busy".

proc splitscript {input} {
    set slave [interp create -safe]
    $slave eval {unset {*}[info vars]}
    foreach command [$slave eval {info commands}] {
        $slave hide $command
    }
    $slave alias unknown apply {{args} {yield $args}}
    coroutine nextline apply {{slave input} {
        yield
        $slave eval $input
    }} $slave $input
    set result {}
    while {1} {
        set line [nextline]
        if {$line ne ""} {
            lappend result $line
        } else {
            interp delete $slave
            return $result
        }
    }
}

Is this by design, am I up against an unimplemented feature, is this a Tcl bug, or is there a mistake in my code or my understanding?

MS same as above: you cannot yield across interp boundaries; more concretely, cross-interp aliases are not nre-enabled, they consume stack ("C stack busy") and block yield. I know I thought of some security risk in allowing this, but that is not the only reason: enabling this would have meant bigger changes in the core which I thought were unwarranted for a first version.

AMG: This is not quite the same as before, as indicated by the different error message. The yield happens in the same interpreter as the coroutine, which is the requirement not met by the code I posted long ago. An [info coroutine] inside [unknown] returns nextline. But between the invocation of the coroutine and the yield, there are two interpreter switches: the coroutine calls into the slave interpreter, which then calls back into the master, which yields. And as you say, cross-interp aliases are not NRE-enabled. I can't think of any security problems with this, since all the coroutine stuff is done inside a single interpreter. However, I certainly do understand your second concern. You can't NRE-enable the world, not all at once anyway. :^)

AMG: By the way, here's a non-coroutine version of the above:

proc splitscript {input} {
    set slave [interp create -safe]
    $slave eval {unset {*}[info vars]}
    foreach command [$slave eval {info commands}] {
        $slave hide $command
    }
    $slave expose lappend lappend*
    $slave invokehidden proc unknown* {args} {
        lappend* ::script* $args
    }
    $slave invokehidden namespace unknown unknown*
    $slave invokehidden set script* {}
    $slave eval $input
    set script [$slave invokehidden set script*]
    interp delete $slave
    return $script
}

AMG: See Config file using slave interp for another variation on the above script, plus discussion on its use.

DKF: You can do a "global coroutine" like this:

coroutine foo eval {yield "in the coro 1"; yield "in the coro 2"; return "finishing"}

But that doesn't mean that I think it is a good idea. For example, you don't get nice variable handling this way.

NEM Or, using a lambda:

coroutine foo apply {{} { yield "in the coro 1"; ... }}

DKF: Yes, but with lambdas you get that nice variable handling. That's because it comes from the local variable context that gets pushed as part of the apply (or the procedure if you're using a "traditional" coroutine).

Unconditionally terminate the currently running coroutine:

    rename [info coroutine] ""; yield

Terminating the currently running coroutine, catchable within the coroutine:

    return -level [info level]

LV In an environment which supports threads and safe interps, what questions should a new developer ask when determining the appropriate technology? And are there situations where one might want/need to combine these in ways?

DKF: This example (from the Tcl docs) shows how to generate primes with the sieve of eratosthenes in Tcl using coroutines...

proc filterByFactor {source n} {
    yield [info coroutine]
    while 1 {
        set x [$source]
        if {$x % $n} {
            yield $x
        }
    }
}
coroutine allNumbers apply {{} {while 1 {yield [incr x]}}}
coroutine eratosthenes apply {c {
    yield
    while 1 {
        set n [$c]
        yield $n
        set c [coroutine prime$n filterByFactor $c $n]
    }
}} allNumbers
for {set i 1} {$i <= 20} {incr i} {
    puts "prime#$i = [eratosthenes]"
}

It might be interesting if someone (TM) attempted a translation of A Curious Course on Coroutines and Concurrency .

CMcC - 2010-04-14 23:41:52

The case for multi-arg coroutines

jbr - May 5 2010

Coroutine names don't appear in the info procs list? Is this by design? Below I can call "xxx" but I cannot see if its currently defined by asking info.

 proc yield-once {} { yield; puts Called  }
 coroutine xxx yield-once

 puts [info procs *]

 xxx

 exit

Output does not contain "xxx":

 auto_load_index unknown auto_import throw auto_execok clock auto_qualify auto_load yield-once try tclLog

AMG: Coroutines aren't procs, they're commands.

proc yield-once {} {yield; puts Called}
coroutine xxx yield-once
info commands x* ;# returns "xxx"

yield-once is a proc, but yield-once is not a coroutine. The coroutine command is created by [yield], not [coroutine]. In this case [yield] calls the coroutine "xxx", since that's the name provided to it by [coroutine].

jbr: Thanks - the documentation is correct and makes sense now that you point is out, coroutine creates a command just like the man page says.

AMG: Is there a way to clone a coroutine? I'm not talking about making a new coroutine that has the same command and arguments as an existing coroutine. Cloning a coroutine would produce a new command whose only difference from the original coroutine command is return value of [info coroutine]. The two coroutine commands would not be linked, they'd just have the same state at the moment of cloning.

% coroutine keaton apply {{} {
      yield
      foreach i {0 1 2 3} {
          puts "$i [info coroutine]"
          if {$i == 1} {
              yield
          }
      }
  }}
% keaton
0 keaton
1 keaton
% CloneCoroutine -old keaton -new kid_a
% kid_a
2 kid_a
3 kid_a
% keaton
2 keaton
3 keaton

DKF 01-Feb-2012: As things stand, no.

AMG: Basically this would establish a means to checkpoint a coroutine and resume execution at any old saved checkpoint, independent of what that coroutine has done (or what has been done to that coroutine) since the time the checkpoint was created.

I wonder what would come of coroutines cloning themselves. I'm afraid that with that kind of power they might try to take over the world. ;^) The [CloneCoroutine] command would return empty string when it is (initially) called, but when the newly created coroutine commands are called with an argument, they would see [CloneCoroutine] return that argument, similar to [yield]. That way, they could tell themselves apart without having to rely on [info coroutine], plus they could be passed arbitrary data. Why would it be advantageous to not depend on [info coroutine]? A coroutine could be cloned, then later that clone could be renamed on top of the original. In this situation, [info coroutine] would not be sufficient to distinguish between the original and the cloned invocations, though a global variable could also be used for this purpose. I don't know if this would be useful, I'm just playing with the idea.

AMG: Are there fundamental technical restrictions preventing the implementation of coroutine cloning? I can understand there being a lack of interest, since the subject hasn't come up before. But is that the only real obstacle?

MS The real obstacle is that each coroutine has its own tcl stack, which stores (among other things) pointers to itself. Making a copy of that stack implies fixing those autorefs, which in turn implies storing the nature of each entry in the stack to know if it has to be fixed or not (and how), which in turn implies a redesign of the tcl stack.