Version 20 of promise

2016-01-26: promise 1.0a1 released

Promises are concurrency primitives that let you write asynchronous code in a sequential style. The promise package is a Tcl based implementation of promises modeled for the most part on the Javascript/ECMAScript 6 standard.

Project page and downloads are at http://sourceforge.net/projects/tcl-promise/ .

Reference documentation is at http://tcl-promise.sourceforge.net but it's probably best to start with the posts at http://www.magicsplat.com/blog/tags/promises/ for an introduction with examples.

APN In response to a question on the chat about running multiple du programs in parallel, here is a promise based solution. Besides printing the output of each du invocation, it also prints the total disk usage for the given paths once all invocations exit.

First define the procedures to handle successful and failed (for example, non-existent path) completions.

proc on_success {du_output} {
    puts $du_output
    # Assumes a very specific format for du output and return the used space.
    return [scan $du_output %d]
}

proc on_failure {message error_dictionary} {
    puts $message
    # On errors, return 0 as used space after printing the error
    return 0
}

Assume paths contains the list of paths of interest.

set paths [list c:/Tcl c:/Temp c:/nosuchpathexists]

Create a promise for each invocation of du, passing it the commands to call on successful and unsuccessful completions.

set promises [lmap path $paths {
    set promise [promise::pexec du -sk $path]
    $promise then on_success on_failure
}]

Finally, combine all the promises into one which will calculate the total once all promises are fulfilled.

set totaller [promise::all $promises]
$totaller done [promise::lambda {outputs} {
    puts "Total: [tcl::mathop::+ {*}$outputs]"
}]

Note that as always, promises require the Tcl event loop to be running.

The following output is produced:

/usr/bin/du: cannot access `c:/nosuchpathexists': No such file or directory
180149        c:/Temp

229933        c:/Tcl

Total: 410082

Discussion

PYK 2015-04-02: Promises are primarily useful as a stop-gap until a language grows real coroutines. Fortunately, Tcl already has coroutines. For comparison here's a coroutine implementation of the example:

#! /bin/env tclsh

proc du_multi {varname args} {
    upvar 1 $varname vname
    set chans {}
    foreach arg $args {
        set chan [open |[list du -sk $arg]]
        chan configure $chan -blocking 0
        chan event $chan readable [list [info coroutine] [list $arg $chan]]
        dict set chans $chan {}
    }
    while {[llength [dict keys $chans]]} {
        lassign [yield] dirname chan
        if {[eof $chan]} {
            set dirsize [scan [dict get $data $chan] %d]
            puts [list $dirname $dirsize]
            incr total $dirsize
            close $chan
            dict unset chans $chan
        } else {
            dict append data $chan [read $chan]
        }
    }
    set vname $total
}


proc main {argv0 argv} {
    variable total
    coroutine dm du_multi total {*}$argv
    vwait total
    puts [string repeat _ [string length "total $total"]]
    puts [list total $total]
}

main $argv0 $argv

APN Humbly begs to differ. I see coroutines as one way that promises could be implemented in a language. Async callbacks via the event loop is another. What I found most useful about promises is the "contracts" they define that allows them to be combined in various ways (error handling also being a big part of that). Could you do something similar with coroutines? Of course you could but you would have simply reimplemented a promise library on top of coroutines.

PYK 2016-04-03: I'm just really curious about all the noise around promises and futures. I keep reading about them, but each example I've seen looks to me like it would have been better-written as a coroutine. Maybe eventually this page will sport some code that shows something that looks better with promises than as coroutines. I'll also hazard a prediction that once coroutines really get rolling in Javascript, promises will fade away.

APN 2016-04-04: FWIW, I first looked at promises about a year and a half ago and could not really understand the fuss and gave up on a Tcl version. It was only when I revisited them much later that things sunk in. Let me try to summarize the core points keeping in mind promises are a useful tool for specific use cases and not a panacea.

Computation involves production and consumption of values.
The production of values *may* happen asynchronously but not necessarily so.
Consumption may involve multiple values produced by independent producers.
Consumers do not know how values are produced (not even whether the value is produced asynchronously or synchronously)
Producers do not know how the produced values will be used (essentially producers and consumers are decoupled).
The completion of one or more computations *may* lead to one or more computations. Computations thus may be a chain (directed graph actually) in which each step is dependent on the results of one or more prior steps.
Any computation may produce errors or exceptions.

In the simple example above, each invocation of du is a computation that produces a value, the disk usage, or an error. The totaling of the resulting value is also a computation, one that is chained to the multiple du invocations. The du computation does not know how the value it produces will be used. The totaller does not know how its input values were produced.

Now suppose I would like to change the application to also print out the max usage. All I need to do is to add the following fragment.

set maxer [promise::all $promises]
$maxer done [promise::lambda {outputs} {
    puts "Max: [tcl::mathfunc::max {*}$outputs]"
}]

Of course I could have added this to the totaller fragment as well but I want to assume that the consumers are independent.

Furthermore, once both results are done, I would like to email them so I add this.

set emailer [promise::all [list $totaller $maxer]]
$emailer done {promise::lambda {total_and_max} {
    email [email protected] "Total/max were [join $total_and_max /]"
}

Another enhancement would be to add a timeout to the whole computation to abort if it does not complete (see the blog for an example).

Now, with your coro-based du, what would it take to implement similar enhancements that too in a generalized manner, not just for the specific example? Could it be done? Of course. But I think that you will find that by the time you refactor as needed, you would have implemented some form of the promise abstraction. If you are arguing that promises could be better implemented using coroutines instead of the event loop, that is a completely different argument or discussion but I don't think that was your point.

As an aside, I don't think your coro has the same behaviour as the promise example in terms of error handling. Consider errors from (a) a dir not existing, (b) dir existing but no access, (c) the du program not being present on the system. Part of promises has to do with the relative ease with which errors can be handled in a common fragment similar to try blocks in synchronous code.

One final comment - regarding your comment regarding promises fading, it's possible but you might want to see the Scala documentation as an example of languages supporting both facilities and how they are presented. You may also find this post from one of the ES6 promise architects illuminating.

PYK 2016-04-06: That's the most cogent and accessible description of promises I've read yet, and it's refreshing to read an explanation that isn't steeped in Javascript. I left out error handling in my example because I would just write any desired error handling into the body of the coroutine, as usual. It would be trivial to modify the ecoroutine version of du_multi to produce values that could then be fed to a maxer or emailer command. The money quote from You're missing the point... is

: More importantly, if at any point that process fails, one function in the composition chain can throw an exception, which then bypasses all further compositional layers until it comes into the hands of someone who can handle it with a catch.

Then there's an example of an asynchronous promise chain and its synchronous version. The synchronous version flows better, but it's got that pesky problem of being synchronous. In Tcl, it seems like a couple of helper procs and a coroutine take care of that:

# Cooperating producers call this to deliver results
proc deliver {cmd to} {
    catch $cmd result options
    after 0 [list after idle [list $to $result $options]
}

proc order args {
    upvar result myresult
    {*}$args [info coroutine]
    lassign [yieldto return -level 0] myresult options
    return -options $options $myresult
}

coroutine [info cmdcount] {{} {
    ...
    try {
      set tweets [order [getTweetsFor] domenic]
      set shortUrls [parseTweetsForUrls tweets]
      set mostRecentShortUrl [lindex shortUrls 0]
      set responseBody [order httpGet [expandUrlUsingTwitterApi[mostRecentShortUrl]]]
      console log [list {Most recent link text} responseBody]
    } catch {
      console log [list {Error with the twitterverse} error
    }
}}

I don't see this as a promise implementation because it doesn't conform to the promise API, but it certainly shares traits of promises such as passing error information along with results. Instead of being locked down to success/failure responders, scripts retain access to the standard Tcl error handling facilities. It's something along these lines that I suspect will supersede promises.

APN One of us is missing something. I freely claim that's you :-) To use an (imperfect) analogy, it feels to me like I am promoting the benefits of a mail protocol like SMTP and you come back stating your belief that SMTP will be supplanted by TCP/IP. And provide examples of how a SMTP-like mail protocol can be implemented on raw TCP/IP as proof of this! It might help a continuing discussion if you implemented the maxer and email enhancements, along with the error handling, to my example on top of your du_multi code.

Category Concurrency

coroutine