Version 7 of Distributed computation

Updated 2006-10-13 22:10:54

* agents


AM (10 october 2006) I have been sketching a package that will do what I want with distributed computation:

  • make setting up such a computation as simple as possible from the client side.
  • make the server side as transparant as possible
  • first: whether threads are used or separate processes to gain concurrency is up to the server and what resources it has available)
  • second: whether there is a fixed pool of worker interpreters or not, depends on the kind tasks the client asks for (independent tasks versus tasks that require synchronisation)
  • third: if synchronisation is required, then all tasks to be synchronised will be waited upon

Zarutian 13. oktober 22:09 2006: on last point: will the server handle that?

This is just meant to preserve my early-morning thoughts on the matter - scribbles on the back of a print-out may get lost.

AM (12 october 2006) Some further remarks: the packages Thread, comm and tie will be very helpful in the actual implementation.

Here is a sketch of one possible application - searching the web for interesting pages:

   set server [connectServer "servername" -poolsize 10]
   set keywords [list ...]
   set interesting_urls {}
   set urls [list ...] ;# Initial list

   exportProcs $server [list getWebPage checkWebPage] ;# The procedures that do the actual work
   registerHandler $server storeUrls ;# procedure that gathers the results in a convenient form

   set count 0
   while { 1 } {
       # Set the workers to work
       foreach u $urls {
           independentTask $server task$count [list \
               checkWebPage $u
           }
           incr count 
       }

       set tasks [waitForTasks $server] ;# Get a list of tasks that have finished, might be empty 

       foreach task $tasks {
           lappend urls $result($task) 
       }

       #
       # Collect the new URLs that were found 
       #
       ...

       #
       # To be added: administration and suitable stopping criteria
   }

Well, this is just a sketch - most important aspects:

  • Scheduling tasks
  • Waiting for results
  • All details are taken care of

Category Interprocess Communication