2024-10-04: promise 1.2.0 released
Promises are concurrency primitives that let you write asynchronous code in a sequential style. The promise package is a Tcl based implementation of promises modeled for the most part on the Javascript/ECMAScript 6 standard.
Project page and downloads are at http://sourceforge.net/projects/tcl-promise/ .
Reference documentation is at http://tcl-promise.magicsplat.com but it's probably best to start with the posts at http://www.magicsplat.com/blog/tags/promises/ for an introduction with examples.
APN In response to a question on the chat about running multiple du programs in parallel, here is a promise based solution. Besides printing the output of each du invocation, it also prints the total disk usage for the given paths once all invocations exit.
First define the procedures to handle successful and failed (for example, non-existent path) completions.
proc on_success {du_output} {
puts $du_output
# Assumes a very specific format for du output and return the used space.
return [scan $du_output %d]
}
proc on_failure {message error_dictionary} {
puts $message
# On errors, return 0 as used space after printing the error
return 0
}Assume paths contains the list of paths of interest.
set paths [list c:/Tcl c:/Temp c:/nosuchpathexists]
Create a promise for each invocation of du, passing it the commands to call on successful and unsuccessful completions.
set promises [lmap path $paths {
set promise [promise::pexec du -sk $path]
$promise then on_success on_failure
}]Finally, combine all the promises into one which will calculate the total once all promises are fulfilled.
set totaller [promise::all $promises]
$totaller done [promise::lambda {outputs} {
puts "Total: [tcl::mathop::+ {*}$outputs]"
}]Note that as always, promises require the Tcl event loop to be running.
The following output is produced:
/usr/bin/du: cannot access `c:/nosuchpathexists': No such file or directory 180149 c:/Temp 229933 c:/Tcl Total: 410082
PYK 2015-04-02: Promises are primarily useful as a stop-gap until a language grows real coroutines. Fortunately, Tcl already has coroutines. For comparison here's a coroutine implementation of the example:
#! /bin/env tclsh
proc du_multi {varname args} {
upvar 1 $varname vname
set chans {}
foreach arg $args {
set chan [open |[list du -sk $arg]]
chan configure $chan -blocking 0
chan event $chan readable [list [info coroutine] [list $arg $chan]]
dict set chans $chan {}
}
while {[llength [dict keys $chans]]} {
lassign [yield] dirname chan
if {[eof $chan]} {
set dirsize [scan [dict get $data $chan] %d]
puts [list $dirname $dirsize]
incr total $dirsize
close $chan
dict unset chans $chan
} else {
dict append data $chan [read $chan]
}
}
set vname $total
}
proc main {argv0 argv} {
variable total
coroutine dm du_multi total {*}$argv
vwait total
puts [string repeat _ [string length "total $total"]]
puts [list total $total]
}
main $argv0 $argvAPN Humbly begs to differ. I see coroutines as one way that promises could be implemented in a language. Async callbacks via the event loop is another. What I found most useful about promises is the "contracts" they define that allows them to be combined in various ways (error handling also being a big part of that). Could you do something similar with coroutines? Of course you could but you would have simply reimplemented a promise library on top of coroutines.
PYK 2016-04-03: I'm just really curious about all the noise around promises and futures. I keep reading about them, but each example I've seen looks to me like it would have been better-written as a coroutine. Maybe eventually this page will sport some code that shows something that looks better with promises than as coroutines. I'll also hazard a prediction that once coroutines really get rolling in Javascript, promises will fade away.
APN 2016-04-04: FWIW, I first looked at promises about a year and a half ago and could not really understand the fuss and gave up on a Tcl version. It was only when I revisited them much later that things sunk in. Let me try to summarize the core points keeping in mind promises are a useful tool for specific use cases and not a panacea.
In the simple example above, each invocation of du is a computation that produces a value, the disk usage, or an error. The totaling of the resulting value is also a computation, one that is chained to the multiple du invocations. The du computation does not know how the value it produces will be used. The totaller does not know how its input values were produced.
Now suppose I would like to change the application to also print out the max usage. All I need to do is to add the following fragment.
set maxer [promise::all $promises]
$maxer done [promise::lambda {outputs} {
puts "Max: [tcl::mathfunc::max {*}$outputs]"
}]Of course I could have added this to the totaller fragment as well but I want to assume that the consumers are independent.
Furthermore, once both results are done, I would like to email them so I add this.
set emailer [promise::all [list $totaller $maxer]]
$emailer done {promise::lambda {total_and_max} {
email [email protected] "Total/max were [join $total_and_max /]"
}Another enhancement would be to add a timeout to the whole computation to abort if it does not complete (see the blog for an example).
Now, with your coro-based du, what would it take to implement similar enhancements that too in a generalized manner, not just for the specific example? Could it be done? Of course. But I think that you will find that by the time you refactor as needed, you would have implemented some form of the promise abstraction. If you are arguing that promises could be better implemented using coroutines instead of the event loop, that is a completely different argument or discussion but I don't think that was your point.
As an aside, I don't think your coro has the same behaviour as the promise example in terms of error handling. Consider errors from (a) a dir not existing, (b) dir existing but no access, (c) the du program not being present on the system. Part of promises has to do with the relative ease with which errors can be handled in a common fragment similar to try blocks in synchronous code.
One final comment - regarding your comment regarding promises fading, it's possible but you might want to see the Scala documentation as an example of languages supporting both facilities and how they are presented. You may also find this post from one of the ES6 promise architects illuminating.
PYK 2016-04-06: That's the most cogent and accessible description of promises I've read yet, and it's refreshing to read an explanation that isn't steeped in Javascript. I left out error handling in my example because I would just write any desired error handling into the body of the coroutine, as usual. It would be trivial to modify the ecoroutine version of du_multi to produce values that could then be fed to a maxer or emailer command. The money quote from You're missing the point... is
Then there's an example of an asynchronous promise chain and its synchronous version. The synchronous version flows better, but it's got that pesky problem of being synchronous. In Tcl, it seems like a couple of helper procs and a coroutine take care of that:
# Cooperating producers call this to deliver results
proc deliver {cmd to} {
catch $cmd result options
after 0 [list after idle [list $to $result $options]
}
proc order args {
upvar result myresult
{*}$args [info coroutine]
lassign [yieldto return -level 0] myresult options
return -options $options $myresult
}
coroutine [info cmdcount] {{} {
...
try {
set tweets [order [getTweetsFor] domenic]
set shortUrls [parseTweetsForUrls tweets]
set mostRecentShortUrl [lindex shortUrls 0]
set responseBody [order httpGet [expandUrlUsingTwitterApi[mostRecentShortUrl]]]
console log [list {Most recent link text} responseBody]
} catch {
console log [list {Error with the twitterverse} error
}
}}I don't see this as a promise implementation because it doesn't conform to the promise API, but it certainly shares traits of promises such as passing error information along with results. Instead of being locked down to success/failure responders, scripts retain access to the standard Tcl error handling facilities. It's something along these lines that I suspect will supersede promises.
APN One of us is missing something. I freely claim that's you :-) To use an (imperfect) analogy, it feels to me like I am promoting the benefits of a mail protocol like SMTP and you come back stating your belief that SMTP will be supplanted by TCP/IP. And provide examples of how a SMTP-like mail protocol can be implemented on raw TCP/IP as proof of this! It might help a continuing discussion if you implemented the maxer and email enhancements, along with the error handling, to my example on top of your du_multi code.
PYK 2016-04-13: So true -- I'm quite often missing something! I happen to be in the middle of wrangling coroutines at the moment, and will be publishing some utilties that have come of the experience. I'll also work through the exercies you suggested.
JHJL 2018-01-16 Thank you for providing this package, APN, it is certainly getting my brain cells busy trying to work out how to best use promises; I am learning many Tcl tips by reading the source code.
Currently, I am struggling to work out a pattern for accumulating a set of record field values from a REST API, which returns N records at a time, using pgeturl. I can repeatedly fetch each batch of records OK, but my pgeturl wrapper blocks. which defeats the object of using promises! Any guidance would be welcome.
As an aside, I had a situation this morning where the target server was down causing a socket error. I am pondering whether or not this error should be trapped by pgeturl (and presumably other socket based calls) and returned through the REJECTED mechanism.
APN Can you show the code? Difficult to make a suggestion as it is not clear exactly what you are doing. Do you have to wait for a previous request to complete before sending the next one? If so, the async/await command pair will likely help. Regarding the socket error, how does the error manifest itself? Again without seeing code, it is hard to make any suggestions.
JHJL Abridged lowlights ;) Basically, I want to call pickfields products {id sku name} {limit 100} and get notified when a list of all {id sku name} records (possibly 1000s) are available
async pickfields {path fields {params {}}} {
set result [list]
set defaults [dict create limit 20 page 0]
set params [dict merge $defaults $params]
set got [set limit [dict get $params limit]]
while {$got == $limit } {
dict incr params page
set state [await [pg $path $params]]
set jsn [dict get $state body]
if {$fields != {}} {
lappend result [pickkeyed $jsn {*}$fields]
} else {
lappend result [json get $jsn]
}
set got [jsoncount $jsn]
}
return $result
}
# A little helper, returns a promise
proc pg {path params} {
# get API URL with parameters e.g. given products {limit 100} -> https://host.com/path?limit=100
set url [_mkurl $path $params]
# OAUTH signature
set oauth [_getoauth GET $path $params]
pgeturl \
$url \
-blocksize [expr {2 ** 20}] \
-headers [list \
Authorization $oauth \
Content-type= "application/json" \
]
}
# Get named fields from either a JSON object or a JSON array
proc pickkeyed {json args} {
if {[llength $args] == 0} {
# No field list - return everything
return [::rl_json::json get $json]
}
set type [rl_json::json type $json]
if { $type eq "object"} {
return [rl_json::json lmap {key val} $json {pick $val {*}$args}]
} else {
return [rl_json::json lmap {val} $json {pick $val {*}$args}]
}
}
# Number of records in a JSON object/array
proc jsoncount {jsn} {
expr { [rl_json::json type $jsn] eq "object" ? [rl_json::json get $jsn ?size] : [rl_json::json get $jsn ?length]}
}
# get named fields from JSON object
proc pick {jsn args} {
foreach a $args {
if {[rl_json::json exists $jsn $a]} {
lappend result $a [rl_json::json get $jsn $a]
} else {
return -code error "unknown field '$a'"
}
}
return $result
}
APN At first glance, that code looks reasonable. Obviously cannot run it to try it out and see the failure since it needs the authentication bits and url. I'll try to modify it to an example that works similarly and see if I can troubleshoot it. I presume you are doing something like
set prom [pickfields http://www.example.com {fieldA fieldB}]
$prom done [lambda args {puts "Success: $args"}] [lambda args {puts "Error: $args"}]
and that by "blocks" you mean neither of the actions passed to the done method fires. Am I right?
Regarding the socket error, do you have a stack trace? All errors within pgeturl should be converted to REJECT actions. I would consider it a bug (hopefully fixable) if that is not the case. A stack trace (errorInfo) would be useful to see if that is the case.
JHJL Thanks for your reply APN, I must apologise for using some loose terminology, muddled thinking and trying new tricks whilst coming down with the flu. My use of "block" was completely wrong - I was using $prom done puts which dumped a huge list into TkCon and the latter struggled to cope. My code seems to be working better than I first thought, initial testing seemed to stall or only produce the first batch of items.
The socket error manifested because the server was temporarily suspended by the hosting company. The URL/OAuth I used was valid but a socket error was being received. My confusion was that TkCon popped up an error dialog for this failure so my on_reject handler wasn't called. Needless to say, the server is back now and I can't reproduce the error.
APN Ok, so I presume now things are working as you expect. As an aside, the tkcon (or maybe it is inherent in the wish console) struggles with long lines has bitten me more than once thinking my code was stuck in an infinite loop somewhere.