2016-01-26: promise 1.0a1 released

''Promises'' are concurrency primitives that let you write
asynchronous code in a sequential style. The `promise` package
is a Tcl based implementation of promises
modeled for the most part on the Javascript/ECMAScript 6 standard.

Project page and downloads are at http://sourceforge.net/projects/tcl-promise/.

Reference documentation is at http://tcl-promise.sourceforge.net but
it's probably best to start with the posts at http://www.magicsplat.com/blog/tags/promises/
for an introduction with examples.

[APN] In response to a question on the chat about running multiple du programs
'''in parallel''', here is a promise based solution. Besides printing the output
of each du invocation, it also prints the total disk usage for the given
paths once all invocations exit.

First define the procedures to handle successful and failed
(for example, non-existent path) completions.
                                                             
======
proc on_success {du_output} {
    puts $du_output
    # Assumes a very specific format for du output and return the used space.
    return [scan $du_output %d]
}

proc on_failure {message error_dictionary} {
    puts $message
    # On errors, return 0 as used space after printing the error
    return 0
}
======

Assume paths contains the list of paths of interest.

======
set paths [list c:/Tcl c:/Temp c:/nosuchpathexists]
======

Create a promise for each invocation of du, passing it the commands
to call on successful and unsuccessful completions.

======
set promises [lmap path $paths {
    set promise [promise::pexec du -sk $path]
    $promise then on_success on_failure
}]
======

Finally, combine all the promises into one which will calculate the
total once all promises are fulfilled.

======
set totaller [promise::all $promises]
$totaller done [promise::lambda {outputs} {
    puts "Total: [tcl::mathop::+ {*}$outputs]"
}]
======

Note that as always, promises require the Tcl event loop to be running.

The following output is produced:

======
/usr/bin/du: cannot access `c:/nosuchpathexists': No such file or directory
180149        c:/Temp

229933        c:/Tcl

Total: 410082
======

** Discussion **

[PYK] 2015-04-02: Promises are primarily useful as a stop-gap until a language
grows real [coroutine%|%coroutines].  Fortunately, Tcl already has coroutines.
For comparison here's a [coroutine] implementation of the example:

======
#! /bin/env tclsh

proc du_multi {varname args} {
    upvar 1 $varname vname
    set chans {}
    foreach arg $args {
        set chan [open |[list du -sk $arg]]
        chan configure $chan -blocking 0
        chan event $chan readable [list [info coroutine] [list $arg $chan]]
        dict set chans $chan {}
    }
    while {[llength [dict keys $chans]]} {
        lassign [yield] dirname chan
        if {[eof $chan]} {
            set dirsize [scan [dict get $data $chan] %d]
            puts [list $dirname $dirsize]
            incr total $dirsize
            close $chan
            dict unset chans $chan
        } else {
            dict append data $chan [read $chan]
        }
    }
    set vname $total
}


proc main {argv0 argv} {
    variable total
    coroutine dm du_multi total {*}$argv
    vwait total
    puts [string repeat _ [string length "total $total"]]
    puts [list total $total]
}

main $argv0 $argv
======

----

[APN] Humbly begs to differ. I see coroutines as one way that promises could be implemented in a language. Async callbacks via the event loop is another. What I found most useful about promises is the "contracts" they define that allows them to be combined in various ways (error handling also being a big part of that). Could you do something similar with coroutines? Of course you could but you would have simply reimplemented a promise library on top of coroutines.

[PYK] 2016-04-03:  I'm just really curious about all the noise around promises
and futures.  I keep reading about them, but each example I've seen looks to me
like it would have been better-written as a coroutine.  Maybe eventually this
page will sport some code that shows something that looks better with promises
than as coroutines.  I'll also hazard a prediction that once coroutines really
get rolling in [Javascript], promises will fade away.

[APN] 2016-04-04: FWIW, I first looked at promises about a year and a half ago
and could not really understand the fuss and gave up on a Tcl version. It was 
only when I revisited them much later that things sunk in. Let me try to summarize
the core points keeping in mind promises are a useful tool for specific use cases
and not a panacea.

   * Computation involves production and consumption of values.
   * The production of values *may* happen asynchronously but not necessarily so.
   * Consumption may involve multiple values produced by independent producers.
   * Consumers do not know how values are produced (not even whether the value is produced asynchronously or synchronously)
   * Producers do not know how the produced values will be used (essentially producers and consumers are decoupled).
   * The completion of one or more computations *may* lead to one or more computations. Computations thus may be a chain (directed graph actually) in which each step is dependent on the results of one or more prior steps. 
   * Any computation may produce errors or exceptions.

In the simple example above, each invocation of du is a computation that produces a
value, the disk usage, or an error. The totaling of the resulting value is also
a computation, one that is chained to the multiple du invocations. The du computation
does not know how the value it produces will be used. The totaller does not
know how its input values were produced.

Now suppose I would like to change the application to also print out the max usage.
All I need to do is to add the following fragment.

======
set maxer [promise::all $promises]
$maxer done [promise::lambda {outputs} {
    puts "Max: [tcl::mathfunc::max {*}$outputs]"
}]
======

Of course I could have added this to the totaller fragment as well but ''I want to
assume that the consumers are independent''.

Furthermore, once both results are done, I would like to email them so I add
this.

======
set emailer [promise::all [list $totaller $maxer]]
$emailer done {promise::lambda {total_and_max} {
    email someone@somewhere.com "Total/max were [join $total_and_max /]"
}
======

Another enhancement would be to add a timeout to the whole computation
to abort if it does not complete (see the blog for an example).

Now, with your coro-based du, what would it take to implement similar enhancements
that too in a generalized manner, not just for the specific example?
Could it be done? Of course. But I think that you will find
that by the time you refactor as needed, you would have implemented some
form of the promise abstraction. If you are arguing that promises could be better implemented using
coroutines instead of the event loop, that is a completely different
argument or discussion but I don't think that was your point.

As an aside, I don't think your coro has the same behaviour
as the promise example in terms of
error handling. Consider errors from (a) a dir not existing, (b) dir
existing but no access, (c) the du program not being present on the system.
Part of promises has to do with the relative ease with which errors
can be handled in a common fragment similar to try blocks in synchronous
code.

One final comment - regarding your comment regarding promises fading,
it's possible but you might want to see the Scala documentation
as an example of languages supporting both facilities and how they
are presented. You may also find https://blog.domenic.me/youre-missing-the-point-of-promises/%|%this post%|%
from one of the ES6 promise architects illuminating.

[PYK] 2016-04-06:  That's the most cogent and accessible description of
promises I've read yet, and it's refreshing to read an explanation that isn't
steeped in Javascript.  I left out error handling in my example because I would
just write any desired error handling into the body of the coroutine, as usual.
It would be trivial to modify the ecoroutine version of `du_multi` to produce
values that could then be fed to a `maxer` or `emailer` command.  The money
quote from
[https://blog.domenic.me/youre-missing-the-point-of-promises/%|%You're missing
the point...] is

    :   ''More importantly, if at any point that process fails, one function in the composition chain can throw an exception, which then bypasses all further compositional layers until it comes into the hands of someone who can handle it with a catch.''

Then there's an example of an asynchronous promise chain and its synchronous
version.  The synchronous version flows better, but it's got that pesky problem
of being synchronous.  In Tcl, it seems like a couple of helper procs and a
coroutine take care of that:

======
# Cooperating producers call this to deliver results
proc deliver {cmd to} {
    catch $cmd result options
    after 0 [list after idle [list $to $result $options]
}

proc order args {
    upvar result myresult
    {*}$args [info coroutine]
    lassign [yieldto return -level 0] myresult options
    return -options $options $myresult
}

coroutine [info cmdcount] {{} {
    ...
    try {
      set tweets [order [getTweetsFor] domenic]
      set shortUrls [parseTweetsForUrls tweets]
      set mostRecentShortUrl [lindex shortUrls 0]
      set responseBody [order httpGet [expandUrlUsingTwitterApi[mostRecentShortUrl]]]
      console log [list {Most recent link text} responseBody]
    } catch {
      console log [list {Error with the twitterverse} error
    }
}}
======

I don't see this as a promise implementation because it doesn't conform to the
promise API, but it certainly shares traits of promises such as passing error
information along with results.  Instead of being locked down to
success/failure responders, scripts retain access to the standard Tcl error
handling facilities.  It's something along these lines that I suspect will
supersede promises. 


<<categories>> Concurrency | coroutine