PURPOSE: Examine options for keeping a GUI up to date while doing something that is expected to take a long time. (also see [Countdown program]) ---- [MS] TODO: add an example using 8.6's [coroutine] [KBK] - 19 April 2001 The single thread model that the [Tcl event loop] uses makes for simpler programming in most cases, because the Tcl programmer doesn't generally need to worry about thread safety. Most programmers with experience in dealing with threaded applications are happy to avoid the insidious bugs that result when the complex and subtle rules for threads are violated. Unfortunately, one place where threads really shine is where two components of an application compete for resources. Perhaps the commonest example that new Tcl programmers call to mind arises in dealing with long-running calculations. If the GUI is to be kept alive, then it must get some CPU resources, even if the long-running calculation doesn't relinquish them willingly. The easiest way to structure this sort of thing is to use another process, which you launch with the [open] command. You use a [fileevent] on the resulting pipe to monitor the process's output and do whatever you need to do in the GUI, and your user interface stays alive. ---- [WGRM] - 26 August 2004 That might be a problem when you want the exit code. However you can get around that on Linux using a compound command that echoes the exit code. Another possibility, surely, is to use send -async? [AF] You can get the exit status of a pipe from the global variable errorCode. '''CHILDSTATUS pid code''' - This format is used when a child process has exited with a non-zero exit status. [AMG], 2 November 2005: I describe the child process method on [Threads vs. events]. ---- Sometimes, though, using another process isn't feasible (you may, for instance, need to be portable to Macintosh) or you have other reasons for wanting to keep your calculations in the same address space as your GUI. In this case, your best bet is to structure your code so that it does a little bit of work at a time. Let's work through some examples. The first example plots trajectories of a cannonball at various elevation angles of the cannon. Full code for the example is shown at the bottom of this page. A second, much simpler example, follows. Starting with the cannonball example, let's just note that the calculation begins by calling the [[sim::init]] procedure and that it does a small amount of work by calling [[sim::one_step]]. The [[sim::one_step]] procedure returns 1 if there is more work to be done and 0 if it's finished. We'll be plotting the results on a canvas, and we want to keep a set of cross-hairs on the canvas up to date at all times: proc update_crosshairs { w x y } { $w coords xhair $x 0 $x [winfo height $w] $w coords yhair 0 $y [winfo width $w] $y $w raise xhair $w raise yhair } grid [canvas .c -width 640 -height 480] .c create line 0 0 0 480 -tags xhair -fill white .c create line 0 0 640 0 -tags yhair -fill white bind .c {update_crosshairs %W %x %y} A naive Tcl programmer would then want to implement the long-running calculation using [[update]]: sim::init .c 640 480 while { [::sim::one_step] } { update } This approach has a number of pitfalls, which are discussed in [Update considered harmful] and [Tcl event loop]. The primary problem is unexpected recursion: the [[update]] command enters the event loop recursively, and it in turn may enter event handlers recursively. If the event handlers aren't expecting to be re-entrant, chaos will result. ---- What could go wrong in this specific example? -davidw ---- A far better practice is to return to the event loop between steps, using [[after idle]] to schedule the next calculation once the events settle down. Alas, simply doing this isn't safe! An 'after idle' that reschedules itself causes trouble, as the manual warns: At present it is not safe for an idle callback to reschedule itself continuously. This will interact badly with certain features of Tk that attempt to wait for all idle callbacks to complete. If you would like for an idle callback to reschedule itself continuously, it is better to use a timer handler with a zero timeout period. Even this warning is oversimplified. Simply scheduling a timer handler with a zero timeout period can mean that the event loop will never be idle, keeping other idle callbacks from firing. The truly safe approach combines both: proc doOneStep {} { if { [::sim::one_step] } { after idle [list after 0 doOneStep] } return } sim::init .c 640 480 doOneStep This skeleton should be considered the basic framework for performing long running calculations within a single Tcl interpreter. It seems to me that this operation should be explicitly supported by the event loop. This is precisely the kind of obscure yak shaving that makes Tcl hard to get up to speed on. -- [Peter da Silva] ---- For those who want to try this out, I include the complete program below. # We'll create a namespace that holds the state of the calculation at a given time: namespace eval sim { variable dt 0.2; # Integration step size variable v 98; # Muzzle velocity (m/s) variable g 9.8; # Acceleration of gravity variable stepNum; # Number of steps performed so far variable maxSteps; # Maximum number of steps needed variable canvas; # Path name of the canvas that # displays results variable xorig; # X coordinate of the cannon on canvas variable yorig; # Y coordinate of the cannon on canvas variable scale; # Scale factor (pixels per meter) } # The plotting can be initialized with a call to the 'sim::init' procedure: proc sim::init { w width height } { variable stepNum variable maxSteps variable dt variable v variable g variable canvas variable xorig variable yorig variable scale set stepNum 0 set maxSteps 101 set vy [expr { sqrt( 0.5 ) * $v }] set t [expr { $vy / $g }] set x [expr { 2 * $vy * $t }] set canvas $w set xorig [expr { $width / 2 }] set yorig [expr { $height * 0.95 }] set scale [expr { 0.45 * $width / $x }] return } # A single trajectory will be plotted each time the 'sim::one_step' procedure is called. This procedure will return 1 if there is still work to be done, or 0 if the last trajectory has been calculated. proc sim::one_step {} { variable stepNum variable maxSteps variable dt variable v variable g variable canvas variable xorig variable yorig variable scale if { $stepNum >= $maxSteps } { return 0 } # Run one simulation run set angle [expr { 3.14159 * $stepNum / ( $maxSteps - 1 ) }] set vx [expr { $v * cos($angle) }] set vy [expr { $v * sin($angle) }] set x 0.0 set y 0.0 set coords {} while { $y >= 0.0 } { set cx [expr { $scale * $x + $xorig }] set cy [expr { -$scale * $y + $yorig }] lappend coords $cx $cy set x [expr { $x + $vx * $dt }] set y [expr { $y + $vy * $dt }] set vy [expr { $vy - $g * $dt }] } eval [list $canvas create line] $coords incr stepNum return 1 } # The 'update_crosshairs' procedure keeps the cross-hairs on the canvas aligned with the mouse pointer. proc update_crosshairs { w x y } { $w coords xhair $x 0 $x [winfo height $w] $w coords yhair 0 $y [winfo width $w] $y $w raise xhair $w raise yhair } # The 'doOneStep' procedure is discussed above. proc doOneStep {} { if { [::sim::one_step] } { after idle [list after 0 doOneStep] } return } # The main program creates the canvas and the cross-hairs, and establishes bindings to keep the cross-hairs up to date. grid [canvas .c -width 640 -height 480] .c create line 0 0 0 480 -tags xhair -fill white .c create line 0 0 640 0 -tags yhair -fill white bind .c {update_crosshairs %W %x %y} # There are two alternative ways to manage switching between handling GUI events and doing a piece of calculation. The second one is preferred. # set naive 1; # to show the behavior of the naive implementation if { [info exists naive] } { sim::init .c 640 480 while { [::sim::one_step] } { update } } else { sim::init .c 640 480 doOneStep } ---- '''Simplified Example: Stopping a Long Calculation''' This very simple example illustrates the same principle as the cannonball example, but with much less code. The GUI is simply a label and start and stop buttons. A global variable stores the value displayed by the label. The "long running" calculation just increments the variable. # very simple GUI for long running calculation grid [label .l -textvariable count] grid [button .b1 -text start -command start] grid [button .b2 -text stop -command stop] set count 0 The calculation is performed one step at a time, as recommended in the earlier example. One approach uses a global variable as a flag which is checked as the calculation progresses. proc start { } { set ::running 1 doOneStep } proc stop {} { set ::running 0 } proc doOneStep {} { if { $::running } { incr ::count after idle [list after 0 doOneStep] } return } Another approach would be to stop the calculation by intervening directly into the event queue. At the time the stop routine is called, one of the two scripts "doOneStep" or "after 0 doOneStep" is in the queue. Cancelling the event stops the calculation. proc start { } { doOneStep } proc stop {} { foreach e [after info] { set script [lindex [after info $e] 0] if { $script == "doOneStep" || \ $script == "after 0 doOneStep"} { after cancel $e } } } proc doOneStep {} { incr ::count after idle "after 0 doOneStep" } ---- A different treatment of some of the same material appears in "Scripted wrappers for legacy applications, Part 2" [http://www.itworld.com/AppDev/4061/swol-0105-regex/]. ---- [[We should write up a [thread]ed example, at least for completeness. Threading *does* remain relevant for lots of database work, for instance.]] [TV] The idea of the title of this page is of course important: often, when the computation model resembles [Model / View / Controller], the maker of a computer program with [GUI] expects the computer it runs on to have sufficient horsepower to complete a requested operation in interaction time, and that other programs aren't in the way to prevent this. For computation tasks / computer power combinations where waiting is to be expected, probably indeed one wants the UI to allow interactions in the meanwhile, but this would probably also require another UI/application interaction pattern definition. Assuming that a database is a seperate process, [IPC] seems reasonable enough to deal with that particular case, and then read in return data in the normal event loop, which is safer and usually more programming-decent then threading, though milage and opinions may vary. ----- 2005, Jan 27 [Eric Amundsen] - cross posting from news:comp.lang.tcl (c.l.t) Anyone got a pure tcl firework display? I'm putting together a little program to drill my kids on their math facts (add/sub/mult/div), and I want a "reward" for getting x in a row right kind of thing. My wife is insisting on a firework display as one of the rewards. However, my life is insisting that I continue to do all those other things like work, shovel snow, blah blah blah. Consequently I don't have the time to climb the animation learning curve. I've looked at the particles thing on the wiki http://wiki.tcl.tk/3252 but I don't see a "quick" way to adjust it to a firework burst and decay. If I get a good solution I'll post it on the wiki. Thanks, Eric Amundsen Rochester MN [RA2] Move South, Eric! This way you won't have to shovel snow and you'll have all the time in the world to study TCL animation. :-) Seriously, [ULIS] knows quite a lot on the subject (TCL animation; not snow removal! :-) Please post this question on his home page and I'm sure he'll be glad to help... I hope I did not hurt your feelings by suggesting to move South since you seem to have a famous explorer of the Northern boundary in your ancestry. Any relationship with Amundsen, the Arctic explorer? [Eric Amundsen] - I did solve this myself then with [Firework Display]. Should have updated this page. Animation was actually very easy, and I've essentially been doing it along with this type of thing - [Keep a GUI alive during a long calculation]. No hurt feelings. While the lineage hasn't ever been determined with certainty, there does seem to be some affinity to cold weather in my blood (or at least my heritage) [http://en.wikipedia.org/wiki/Roald_Amundsen] ---- [AMG], 2005-Mar-18: First let's see if I understand this. If not, correct me and disregard everything else I say. :^) It's not safe to recursively [[[update]]], but if you desire the same effect you must schedule your [continuation] to be run after all pending idle tasks and timed tasks have completed. This is very, very much like yielding to the OS and letting it reschedule your task. (I sorely wish we had [[[yield]]] instead of [[[update]]] to schedule the current [continuation], jump to the [event loop], then jump back when it is fair to do so.) I'm looking at the documentation for [sqlite]'s eval and progress (sub)commands [http://www.sqlite.org./tclsqlite.html]. [[$db eval]] allows you to specify code to run for every row returned. [[$db progress]] schedules a callback to be executed after doing however-many sqlite virtual machine opcodes. How do I use these facilities to keep my GUI alive? Reading between the lines, I sense that I am expected to supply [[[update]]] as the callback paramter to [[$db progress]]. In other news, I'm pondering the new API for my [itunesdb], which is still slow (takes more than one second) for large music collections. I'd like for it to avoid blocking the GUI for more than a heartbeat, especially since the display system expects the application to redraw itself on expose events, and "painting" looks very ugly. So how do I accomplish this? When reading playlists, a single record can be very large (especially the master playlist, which contains all songs), therefore returning to the event loop in between each read isn't good enough. [Peter Newman] 19 March 2005: I don't understand why reading a record from a file should be slow - but assuming it is:- * Perhaps you could write a Tcl program to translate the ''itunesDB'' file to one from which individual records can be retrieved quickly, or; * Run the GUI, and the ''itunesDB'' file parser, as two separate processes. You want the file parser to load the records into memory, for example. So that the GUI can quickly retrieve records, using some form of inter-process communication. * If keeping the GUI alive is the only issue (and not the speed with which it's drawn), the simplest and safest way is - assuming the GUI comprises whatever it comprises - and a listbox or table that you want to fill with data from the ''itunesDB'' file:- proc myFantasticItunesApp { } { ...draw the GUI - with the listbox(s)/table EMPTY... after idle AddOneRecordToTheListbox/Table --OR-- after 10-or-some-other-small-number-of-milliseconds AddOneRecordToTheListbox/Table } with:- proc AddOneRecordToTheListbox/Table { } { ...add one or some other small number of records to the listbox - however many you can get without blocking for too long - say half a second max... after idle AddOneRecordToTheListbox/Table --OR-- after 10-or-some-other-small-number-of-milliseconds AddOneRecordToTheListbox/Table } Your GUI should be as responsive as possible - even if reading an individual record is so slow that they just dribble into the listbox. Also this method is basically the same as using ''update'' - but, unlike ''update'', it's perfectly safe. P.S. NEVER, EVER use ''update'' (unless someone is quite literally holding a gun to your head). [AMG], 2005-Mar-18 (I suspect we may be in different timezones!): Sorry, I based the slow-record-reading comment on the performance of [itunesdb] 0.01, which was quite atrocious. Back then, reading the master playlist (with 4137 tracks) of my sample iTunesDB file took 8.5 seconds (400 MHz machine). Now, with 0.03 on the same machine, it's down to 1.8 seconds, or 0.45 seconds on a 1500 MHz. That's not so bad after all. Plus I have further speed optimizations in mind. Yum. Here's a proc that does hoever many units of work it can in some period (say, 100 ms), then updates a status widget, reschedules itself with [[[after]]], and returns to the event loop. It works by measuring the time to do the first unit of work then divides that from the update period to get the number of work units it can do per period. It also tracks the actual time spent doing that many work units, and if it's off by more than 50% (or some amount) it remembers to recalibrate the next time it runs. package require Tk set work(total) 10000 ;# Total number of times to call do_work. set work(done) 0 ;# Number of times do_work has been called. set work(period) 100 ;# Time in milliseconds between display updates. set work(steps) 0 ;# Number of times to call do_work between updates. set work(recalib) 1 ;# If true, recalibrate before next update. set work(start) 0 ;# Start time of calculation. proc do_work {} { for {set foo 0} {$foo < 1000} {incr foo} {} } proc next {} { global work if {$work(recalib)} { set start [clock microseconds] do_work; incr work(done) set dur [expr {[clock microseconds] - $start}] set work(steps) [expr {int(1e3 * $work(period) / $dur)}] if {$work(steps) == 0} { set work(steps) 1 } .steps-v configure -text [format %d $work(steps)] } if {$work(total) - $work(done) < $work(steps)} { set work(steps) [expr {$work(total) - $work(done)}] } set start [clock milliseconds] for {set step 0} {$step < $work(steps)} {incr step} { do_work; incr work(done) } set dur [expr {[clock milliseconds] - $start}] .percent-v configure -text [format %.2f [expr { 100.0 * $work(done) / $work(total)}]] .actual-v configure -text [format "%d ms" $dur] .elapsed-v configure -text [format "%d ms" [expr { $dur + $start - $work(start)}]] if {abs($dur - $work(period)) > $work(period) / 2} { set work(recalib) 1 } if {$work(done) < $work(total)} { after idle [list after 0 next] } } proc main {} { foreach {name text value} [list \ percent "Percent complete:" 0.00 steps "Steps per period:" ? \ target "Target period:" ? actual "Actual period:" ? \ elapsed "Time elapsed:" "0 ms" ] { label .$name-l -text $text label .$name-v -text $value grid .$name-l .$name-v grid configure .$name-l -sticky w grid configure .$name-v -sticky e } .target-v configure -text [format "%d ms" $::work(period)] set ::work(start) [clock milliseconds] next } main I wrote a Window Maker dockapp (in Tcl/Tk!!) that lets me switch between 400 MHz and 1500 MHz (terrific for benchmarking). Anyway, if I click it while running the above code, the "steps per period" value automatically changes to keep the period more-or-less constant. [Peter Newman] 19 March 2005: I'd say that's about the best you can do using standard Tcl (and without going to more exotic solutions like ''threads'' or ''separate processes'' - and without speeding up the read routines somehow). Is the GUI now responsive enough (it should be - once you've tuned it right) - both at 400 MHz and 1500 MHz? [AMG]: Thanks, it's very nice. There's not much concern for responsiveness, since the only thing worth clicking is Cancel. ---- [Category GUI]