2017-11-06: promise 1.0.3 released

''Promises'' are concurrency primitives that let you write
asynchronous code in a sequential style. The `promise` package
is a Tcl based implementation of promises
modeled for the most part on the Javascript/ECMAScript 6 standard.

Project page and downloads are at http://sourceforge.net/projects/tcl-promise/.

Reference documentation is at http://tcl-promise.magicsplat.com but
it's probably best to start with the posts at http://www.magicsplat.com/blog/tags/promises/
for an introduction with examples.

[APN] In response to a question on the chat about running multiple du programs
'''in parallel''', here is a promise based solution. Besides printing the output
of each du invocation, it also prints the total disk usage for the given
paths once all invocations exit.

First define the procedures to handle successful and failed
(for example, non-existent path) completions.
                                                             
======
proc on_success {du_output} {
    puts $du_output
    # Assumes a very specific format for du output and return the used space.
    return [scan $du_output %d]
}

proc on_failure {message error_dictionary} {
    puts $message
    # On errors, return 0 as used space after printing the error
    return 0
}
======

Assume paths contains the list of paths of interest.

======
set paths [list c:/Tcl c:/Temp c:/nosuchpathexists]
======

Create a promise for each invocation of du, passing it the commands
to call on successful and unsuccessful completions.

======
set promises [lmap path $paths {
    set promise [promise::pexec du -sk $path]
    $promise then on_success on_failure
}]
======

Finally, combine all the promises into one which will calculate the
total once all promises are fulfilled.

======
set totaller [promise::all $promises]
$totaller done [promise::lambda {outputs} {
    puts "Total: [tcl::mathop::+ {*}$outputs]"
}]
======

Note that as always, promises require the Tcl event loop to be running.

The following output is produced:

======
/usr/bin/du: cannot access `c:/nosuchpathexists': No such file or directory
180149        c:/Temp

229933        c:/Tcl

Total: 410082
======

** Discussion **

[PYK] 2015-04-02: Promises are primarily useful as a stop-gap until a language
grows real [coroutine%|%coroutines].  Fortunately, Tcl already has coroutines.
For comparison here's a [coroutine] implementation of the example:

======
#! /bin/env tclsh

proc du_multi {varname args} {
    upvar 1 $varname vname
    set chans {}
    foreach arg $args {
        set chan [open |[list du -sk $arg]]
        chan configure $chan -blocking 0
        chan event $chan readable [list [info coroutine] [list $arg $chan]]
        dict set chans $chan {}
    }
    while {[llength [dict keys $chans]]} {
        lassign [yield] dirname chan
        if {[eof $chan]} {
            set dirsize [scan [dict get $data $chan] %d]
            puts [list $dirname $dirsize]
            incr total $dirsize
            close $chan
            dict unset chans $chan
        } else {
            dict append data $chan [read $chan]
        }
    }
    set vname $total
}


proc main {argv0 argv} {
    variable total
    coroutine dm du_multi total {*}$argv
    vwait total
    puts [string repeat _ [string length "total $total"]]
    puts [list total $total]
}

main $argv0 $argv
======

----

[APN] Humbly begs to differ. I see coroutines as one way that promises could be implemented in a language. Async callbacks via the event loop is another. What I found most useful about promises is the "contracts" they define that allows them to be combined in various ways (error handling also being a big part of that). Could you do something similar with coroutines? Of course you could but you would have simply reimplemented a promise library on top of coroutines.

[PYK] 2016-04-03:  I'm just really curious about all the noise around promises
and futures.  I keep reading about them, but each example I've seen looks to me
like it would have been better-written as a coroutine.  Maybe eventually this
page will sport some code that shows something that looks better with promises
than as coroutines.  I'll also hazard a prediction that once coroutines really
get rolling in [Javascript], promises will fade away.

[APN] 2016-04-04: FWIW, I first looked at promises about a year and a half ago
and could not really understand the fuss and gave up on a Tcl version. It was 
only when I revisited them much later that things sunk in. Let me try to summarize
the core points keeping in mind promises are a useful tool for specific use cases
and not a panacea.

   * Computation involves production and consumption of values.
   * The production of values *may* happen asynchronously but not necessarily so.
   * Consumption may involve multiple values produced by independent producers.
   * Consumers do not know how values are produced (not even whether the value is produced asynchronously or synchronously)
   * Producers do not know how the produced values will be used (essentially producers and consumers are decoupled).
   * The completion of one or more computations *may* lead to one or more computations. Computations thus may be a chain (directed graph actually) in which each step is dependent on the results of one or more prior steps. 
   * Any computation may produce errors or exceptions.

In the simple example above, each invocation of du is a computation that produces a
value, the disk usage, or an error. The totaling of the resulting value is also
a computation, one that is chained to the multiple du invocations. The du computation
does not know how the value it produces will be used. The totaller does not
know how its input values were produced.

Now suppose I would like to change the application to also print out the max usage.
All I need to do is to add the following fragment.

======
set maxer [promise::all $promises]
$maxer done [promise::lambda {outputs} {
    puts "Max: [tcl::mathfunc::max {*}$outputs]"
}]
======

Of course I could have added this to the totaller fragment as well but ''I want to
assume that the consumers are independent''.

Furthermore, once both results are done, I would like to email them so I add
this.

======
set emailer [promise::all [list $totaller $maxer]]
$emailer done {promise::lambda {total_and_max} {
    email someone@somewhere.com "Total/max were [join $total_and_max /]"
}
======

Another enhancement would be to add a timeout to the whole computation
to abort if it does not complete (see the blog for an example).

Now, with your coro-based du, what would it take to implement similar enhancements
that too in a generalized manner, not just for the specific example?
Could it be done? Of course. But I think that you will find
that by the time you refactor as needed, you would have implemented some
form of the promise abstraction. If you are arguing that promises could be better implemented using
coroutines instead of the event loop, that is a completely different
argument or discussion but I don't think that was your point.

As an aside, I don't think your coro has the same behaviour
as the promise example in terms of
error handling. Consider errors from (a) a dir not existing, (b) dir
existing but no access, (c) the du program not being present on the system.
Part of promises has to do with the relative ease with which errors
can be handled in a common fragment similar to try blocks in synchronous
code.

One final comment - regarding your comment regarding promises fading,
it's possible but you might want to see the Scala documentation
as an example of languages supporting both facilities and how they
are presented. You may also find https://blog.domenic.me/youre-missing-the-point-of-promises/%|%this post%|%
from one of the ES6 promise architects illuminating.

[PYK] 2016-04-06:  That's the most cogent and accessible description of
promises I've read yet, and it's refreshing to read an explanation that isn't
steeped in Javascript.  I left out error handling in my example because I would
just write any desired error handling into the body of the coroutine, as usual.
It would be trivial to modify the ecoroutine version of `du_multi` to produce
values that could then be fed to a `maxer` or `emailer` command.  The money
quote from
[https://blog.domenic.me/youre-missing-the-point-of-promises/%|%You're missing
the point...] is

    :   ''More importantly, if at any point that process fails, one function in the composition chain can throw an exception, which then bypasses all further compositional layers until it comes into the hands of someone who can handle it with a catch.''

Then there's an example of an asynchronous promise chain and its synchronous
version.  The synchronous version flows better, but it's got that pesky problem
of being synchronous.  In Tcl, it seems like a couple of helper procs and a
coroutine take care of that:

======
# Cooperating producers call this to deliver results
proc deliver {cmd to} {
    catch $cmd result options
    after 0 [list after idle [list $to $result $options]
}

proc order args {
    upvar result myresult
    {*}$args [info coroutine]
    lassign [yieldto return -level 0] myresult options
    return -options $options $myresult
}

coroutine [info cmdcount] {{} {
    ...
    try {
      set tweets [order [getTweetsFor] domenic]
      set shortUrls [parseTweetsForUrls tweets]
      set mostRecentShortUrl [lindex shortUrls 0]
      set responseBody [order httpGet [expandUrlUsingTwitterApi[mostRecentShortUrl]]]
      console log [list {Most recent link text} responseBody]
    } catch {
      console log [list {Error with the twitterverse} error
    }
}}
======

I don't see this as a promise implementation because it doesn't conform to the
promise API, but it certainly shares traits of promises such as passing error
information along with results.  Instead of being locked down to
success/failure responders, scripts retain access to the standard Tcl error
handling facilities.  It's something along these lines that I suspect will
supersede promises. 

[APN] One of us is missing something. I freely claim that's you :-)
To use an (imperfect) analogy, it feels to me
like I am promoting the benefits of a mail protocol like SMTP and you
come back stating your belief that SMTP will be supplanted by TCP/IP. And
provide examples of how a SMTP-like mail protocol can be implemented
on raw TCP/IP as proof of this! It might help a continuing discussion if you
implemented the maxer and email enhancements, along with the error handling,
to my example on top of your du_multi code.

[PYK] 2016-04-13:  So true -- I'm quite often missing something!  I happen to
be in the middle of wrangling coroutines at the moment, and will be publishing
some utilties that have come of the experience.  I'll also work through the
exercies you suggested.

----

[JHJL] 2018-01-16 Thank you for providing this package, [APN], it is certainly getting my brain cells busy trying to work out how to best use promises; I am learning many Tcl tips by reading the source code.

Currently,  I am struggling to work out a pattern for accumulating a set of record field values from a REST API, which returns N records at a time, using pgeturl. I can repeatedly fetch each batch of records OK, but my pgeturl wrapper blocks. which defeats the object of using promises!  Any guidance would be welcome.

As an aside, I had a situation this morning where the target server was down causing a socket error. I am pondering whether or not this error should be trapped by pgeturl (and presumably other socket based calls) and returned through the  REJECTED mechanism.

[APN] Can you show the code? Difficult to make a suggestion as it is not clear exactly what you are doing. Do you have to wait for a previous request to complete before sending the next one? If so, the async/await command pair will likely help. Regarding the socket error, how does the error manifest itself? Again without seeing code, it is hard to make any suggestions.

[JHJL] Abridged lowlights ;) Basically, I want to call pickfields products {id sku name} {limit 100} and get notified when a list of  all  {id sku name} records (possibly 1000s) are available

======
async pickfields {path fields {params {}}} {

  set result [list]
  set defaults [dict create limit 20 page 0]
  set params [dict merge $defaults $params]

  set got [set limit [dict get $params limit]]

  while {$got == $limit }  {

    dict incr params page

    set state [await [pg $path $params]]

    set jsn [dict get $state body]

    if {$fields != {}} {
      lappend result [pickkeyed $jsn {*}$fields]
    } else {
      lappend result [json get $jsn]
    }
    set got [jsoncount $jsn] 

  }
  return $result
}

# A little helper, returns a promise
proc pg {path params} {
  # get API URL with parameters e.g. given products {limit 100} ->  https://host.com/path?limit=100
  set url     [_mkurl    $path $params]
  # OAUTH signature
  set oauth   [_getoauth GET $path $params]

  pgeturl \
    $url  \
    -blocksize [expr {2 ** 20}] \
    -headers [list \
        Authorization $oauth \
        Content-type= "application/json" \
    ]
}

# Get named fields from either a JSON object or a JSON array
proc pickkeyed {json args} {
    
    if {[llength $args] == 0}  {
        # No field list - return everything
        return [::rl_json::json get $json]
    }

    set type [rl_json::json type $json]

    if { $type eq "object"} {
        return [rl_json::json lmap {key val} $json {pick $val {*}$args}]
    } else {
        return [rl_json::json lmap {val} $json {pick $val {*}$args}]
    }
}

# Number of records in a JSON object/array
proc jsoncount {jsn} {
  expr { [rl_json::json type $jsn] eq "object" ? [rl_json::json get $jsn ?size] : [rl_json::json get $jsn ?length]}
}

# get named fields from JSON object
proc pick {jsn args} {
  foreach a $args {
    if {[rl_json::json exists $jsn $a]} {
      lappend result $a [rl_json::json get $jsn $a]
    } else {
      return -code error "unknown field '$a'"
    }
  }
  return $result
}

======

[APN] At first glance, that code looks reasonable. Obviously cannot run it to 
try it out and see the failure since it needs the authentication bits and url.
I'll try to modify it to an example that works similarly and see if I can
troubleshoot it. I presume you are doing something like

======

set prom [pickfields http://www.example.com {fieldA fieldB}]
$prom done [lambda args {puts "Success: $args"}] [lambda args {puts "Error: $args"}]

======

and that by "blocks" you mean neither of the actions passed to the `done` method fires.
Am I right?

Regarding the socket error, do you have a stack trace? All errors within `pgeturl` 
should be converted to REJECT actions. I would consider it a bug (hopefully fixable)
if that is not the case. A stack trace (errorInfo) would be useful to see if that
is the case.


<<categories>> Concurrency | coroutine