Version 34 of multi-arg coroutines

I wish to consider amending "TIP 328 Coroutines" [L1 ], along these lines:

Invocation of a coroutine should accept multiple arguments, those arguments should be returned to the coroutine's yield as a list of actual parameters.

The current implementation forbids invoking a coroutine with more than one argument. I have found, in practice, that many of my coroutine invocations are naturally like other command invocations, and take multiple arguments.

The case for providing multi-arg'd coroutines is:

1. coroutines should be able to emulate any command, not just any single-arg'd command [generality]
2. to implement single-arg's coroutines in multi-arg'd coroutines is trivial - nothing needs to be done. The converse (implementing multi-arg'd coroutines under coroutine is inefficient and difficult. [increased expressive power]
3. there is no sound reason that the invocation of a coroutine should not resemble that of any other command [principle of minimal surprise]

For these reasons, the core should be modified to accept multiple actual parameters to a coroutine invocation, ::yield should be modified to resemble ::yield2 below, and a new interface ::yieldm should be created to return the actual parameter list as it's passed.

Arguments in favour of single-arg'd coroutine:

1. extra cost in packing (at invocation) and unpacking (in yield) the actual parameters. Total cost is building a 1-element list and 1 lindex, both in C. I consider this to be insignificant compared to the cost of passing more than one argument, which is the more usual case in my experience.
2. the potential that coroutine invocation might require options to control the invocation in ways simple command call doesn't. Examples, IIRC, were [$coro -terminate] which would be interpreted as an unconditional termination. I find this hypothetical need unconvincing, as it would be as easily served by other kinds of interface to the coroutine (much as info has been pressed into service to provide information about a coroutine, some other hypothetical command like [corocontrol $coro] could be used to control the coro.
3. (a) [$coro] and [yield] are properly considered as symmetrical operations, (b) [$coro] v. [yield] is an attractive/consonant way to represent their symmetry, (c) having multi-arg'd [$coro] would violate or misrepresent that symmetry.
4. Are there any other objections to multi-arg coros?

Demonstration of point 2 [increased expressive power]

CASE 1: (counterfactual)

Assume a [Coroutine] which generates a command taking multiple args, to implement coroutine as we have it implemented:

[Coroutine] would require no wrapping or changes to function as coroutine does now. Only yield would have to change.

To provide precisely the same functionality as yield currently does it is necessary to strip off a single layer of list:

proc ::yield2 {value} {
    return [lindex [::yield $value] 0]
}

No other changes are necessary. More likely, one would define ::yield like that, and create a new ::yield-variant which returned the whole invocation arg list.

CASE 2: [Coroutine] in [coroutine] - implementing multi-arg'd coroutines over singe-arg'd coroutine

proc Coroutine {name command args} {
    set ns [namespace qualifiers $name]
    if {$ns eq ""} {
        set ns [uplevel 1 {namespace current}]
    }
    set name [namespace tail $name]

    set coco [::coroutine ${ns}::_C$name $command {*}args]
    trace add command ${ns}::_C$name delete "rename ${ns}::name {}"
    proc ${ns}::$name {args} {
        set name [lindex [info level 0] 0]
        set ns [namespace qualifiers $name]
        if {$ns eq ""} {
            set ns [uplevel 1 {namespace current}]
        }
        set name [namespace tail $name]
        
        tailcall ${ns}::_C$name $args
    }
}

The predominant cost in this is that of tracing intermediate commands to avoid leakage. Even if this were not the case, the cost of calling a proc to wrap the extra args is considerable. The only alternative is to wrap the args on *each* invocation.

One can provide variable assignment by signature (or Occam-like protocol):

proc entrypoint {value args} {
    uplevel 1 lassign [::yield $value] {*}$args
}

This is possible in current coroutine the same way, but requires the caller to form args into lists on each invocation.

MS Notes that this would require that the invocation's arguments be a list of {name,value} pairs. This breaks the wanted analogy to proc, where the assignment to variables is positional and not by name. It would be possible to mimic proc perfectly, but in that case scripting the current coroutine functionality becomes cumbersome.

CMcC was thinking of it as a wrapper to yield with the effect of lassigning the actual parameters to caller-local variables. The coro invocation would only provide the values, the caller of entrypoint would provide the formal parameters (that's the analogy.)

jmn 2010-04-15 I totally agree. I was thoroughly dismayed by the single arg coroutine thing. It just seems to go against the grain of the “Tcl way” - for no real advantage. If it was multi-arg'd it would present an interesting way to build some command alternatives along the lines of existing mechanisms such as 'interp alias'. Having to wrap it to achieve this is ugly enough to discourage this sort of innovation especially if the whole point of the innovation was to do so in a situation where dispatch performance matters.

nem notes that performance will be dominated by the cost of the coroutine context switch (quite high currently). Also, wrapping a coro to accept multiple args is trivial:

proc apply-list {cmd args} { $cmd $args }
do stuff [list apply-list $coro]

On the other hand, forcing all coros be multi-arg means that yield then must return a list, and the extremely common single arg case then needs to remember to use [lindex [yield] 0] everywhere.

MS An alternative is to provide a new command (coroutine2?) which creates multi-arg commands. In that case, yield can be modified in C to do the right thing depending on the nature of the enclosing coroutine. For some reason which I'm not clear about (paternity?), this would be my current preference.

A second command would also make it easy to allow the actual arguments to be passed positionally, as in proc. Some syntax would be needed to allow the invocation to also pass the internal result from yield, if one is wanted. Maybe something like

  coroutine coro2Cmd ::apply {{x1 x2} {...}} 42 42
  coro2Cmd -yieldResult foo 11 22

that would cause the coro to be created with x1=x2=42. When it is later resumed: x1 is set to 11, x2 is set to 22, and yield returns foo. If no option -yieldResult is specified yield returns {}.

CMcC this really doesn't make coro invocation look like normal command invocation, though. My preference is for something where a caller doesn't need to know that what it's calling is a coro. The reason I prefer this is that I can't see why a caller should need to know, or should have to consider anything unusual when calling a command which might happen to be implemented as a coro. The specific implementation you sketch, with special options to be interpreted, doesn't provide for justification (1) above. Additionally, it's not possible to pass -yieldResult in if you wanted to.

MS At least one of us misunderstands the other :P In my proposal, if the caller doesn't know this is a coro, he uses it as a normal command

   coro2Cmd 11 12

OTOH, if he does know it is a coro he has the option of also sending in a result for yield. Note that I am not really proposing this as a solution, just an option that really mimics normal command invocation - with positional semantics for the arguments.

CMcC I did misunderstand. You were suggesting that the coroutine command somehow ascertain by inspection the formal parameter set of the its second arg, and use that as a kind of occam protocol for later interaction. I don't think that can work, can it? You have no way to inspect, for example, a C-coded command to discover its formal parameter signature, nor (really) to be certain that a given command was created by a given implementation of a given other command. I think what you seem to be suggesting is impossible, unless I still misunderstand it.

I am attracted to the idea that a coro is just like any other command in terms of its invocation (because it enables a coro to simulate any other command.)

I dislike the -yieldResult idea because I can think of no arguments one can pass to a command which are interpreted by the invocation mechanism itself prior to the command gaining control. What you propose with your example -yieldResult is to make command invocation get involved, at a C level, unable to be intercepted by Tcl, to interpret (as modifiers of the invocation) things which would normally be considered arguments to the command, and this only for coros. To me, that seems like a special case with wide implications (you can't pass “-yieldResult” as the first argument to any coro) and no evident benefit (at least, I don't understand what it buys you that can't be achieved differently.)

I understand there's also an argument that there may be other things one would like to do to a coroutine than invoke it in the usual manner one invokes commands. For things which operate *on* a coro rather than through it (say, for example, injecting an error into it) I would suggest a completely different command, say [coro_op kill $coro] or indeed [coro_op error $coro $eo] to cause yield to complete with error dict specified by $eo.

CMcC I wouldn't mind if there were two ways to create coroutines, with a new form creating a multi-arg coro, but I don't think that solution is necessary, or as neat as providing a second yield and making multi-arg the standard behaviour of coroutine.

What's your objection to this change, Miguel? By the time the coro is invoked, one has already parsed the arguments t o its invocation in order (currently) to complain that they don't form a singleton set. I presume that by that stage they exist in a list form anyway, and that coro invocation needs to pick out the first argument to return as yield's result anyway, surely the performance cost of duplicating the [lrange $invocation 1 end] is not significantly greater than that of [lindex $invocation 1], and in any case is exactly what a proc command invocation would have to do (without the necessity of then assigning them to their corresponding formal parameters.)

Lars H: How about letting yield accept a second argument that (like the args argument of proc) specifies the arguments the coroutine will take the next time it is called? This would mean that after

  yield $value {foo bar {baz "apa"} args}

the variables foo, bar, baz, and args in the local context would all have been assigned according to the values supplied in the coro call (which would throw an error unless at least two arguments are supplied). This does open up for coroutine commands having wildly varying syntaxes, but on the other hand it is already a consequence of the fact that you can yield just about anywhere that coroutine commands can have semantics that vary wildly from call to call.

CMcC - 2010-04-20 03:18:35

I want to explore the taxonomy of commands which create commands here: Creating Commands