[MJ] - YATS (Yet Another [Tuplespace]) that supports:
   * duplicate tuples
   * persistence with sqlite (create in memory db if persistence is not needed)
   * asynchronous and synchronous destructive and non destructive reads
   * running as a server
   * Unicode and binary tuples (thanks to sqlite3)
'''Missing functionality'''

   * Selecting tuples with matches on anything but the tuplename (fixed in version below) tuples can now be matched on any part. String matching is used .e.g: {* test? 4}
   * Error checking on sqlite operations (error checking is now added).
   * protection from SQL injection attacks e.g. (fixed in version below) only variable bindings are being used
   * The Gelernter Linda paper also allows underspecified tuples to be written in the tuplespace, not only read from. For instance one should be able to output {test * 2} which could be read by a client requesting the tuple {test 1 2} as well as one requesting {test 2 2} (added by letting ''tuple_match'' do a two way match)

'''Possible improvements'''

   * Currently the tuples are stored in one field in the database, this will probably make searching for tuples slow if the tuplespace contains a lot of tuples. Possible improvement would be to create a seperate table with the tuples. e.g.
 tuplespace contains tuid

extra table contains:
 tuid field value
 id   0  <number of elements N>
 id   1  <value of 1st elelemt>
 .
 .
 id   N  <value of Nth element>

[MJ] - This is not really useful because selecting matching tuples is non-trivial in a DB design like this.


'''Tupleserver commands'''

The following commands are supported by the server:

   * ''out tuplespace tuple'': write ''tuple'' to ''tuplespace''.
   * ''dir tuplespacepattern ?pattern?'': list the tuples matching ''pattern'' in tuplespaces matching ''tuplespacepattern'' if the pattern is not provided list all tuples.
   * ''dir'': list all tuplespaces.
   * ''purge tuplespace'': remove all tuples from ''tuplespace''.
   * ''exit'': disconnect session.
   * ''read space pattern'': read a tuple matching ''pattern'' from tuplespace ''space''. The tuple will remain in the space.
   * ''in space pattern'': read and remove a tuple matching ''pattern'' from tuplespace ''space''.
   * ''ain'', ''aread'': non-blocking versions of ''read'' and ''in''. Will return an empty result if no matching tuple is available.
   * ''ping'': check connection with server.
   * ''help'': returns all available server commands

The server will respond with a 2 element list. The first element will contain the status of the command (ok or error). The second element will be the actual result.

'''Code for the Tupleserver'''

 # tupleserver.tcl -
 #
 # Copyright (c) 2008 Mark Janssen
 #
 # A Tcl implementation of a Linda Tuplespace

 package require Tk
 catch {
     package require Iocpsock
     rename socket socketo
     rename socket2 socket
 }

 proc main {} {
     ts::init
     set server_port 4040
     while { [catch {socket -server handle_socket $server_port}] } {after 1000}
     label .l -text "Server listening on port $server_port, F5 for console , F6 for restart"
     grid .l -sticky ew
     grid columnconfigure . 0 -weight 1
     bind . <F5> {console show}
     bind . <F6> {exec wish $argv0 &; exit}

     # aliases that can be used from console
     interp alias {} ts_out {} ::ts::out stdout
     interp alias {} ts_ain {} ::ts::ain stdout
     interp alias {} ts_in {} ::ts::in stdout
     interp alias {} ts_aread {} ::ts::aread stdout
     interp alias {} ts_read {} ::ts::read stdout
     interp alias {} ts_dir {} ::ts::dir stdout
     interp alias {} ts_purge {} ::ts::purge stdout
     interp alias {} ts_cb {} ::ts::cb stdout

     wm protocol . WM_DELETE_WINDOW {exit}
 }

 proc handle_socket {socket host port} {
     fconfigure $socket -buffering line -blocking no -encoding utf-8
     fileevent $socket readable [list ::ts::handle_request $socket {}]
 }

 # namespace for server commands
 namespace eval ts {

 # commands supported by the tupleserver
 set commands {purge cb dir ain in aread read out exit ping help}

     proc init {} {
     # open persistent database
         package require sqlite3 3.3

         sqlite3 db ~/ts.sqlite
         # sqlite3 db {}
         # db eval {PRAGMA synchronous=OFF;}

         # create schema
         db eval {CREATE TABLE IF NOT EXISTS tuples(id INTEGER PRIMARY KEY,space,value)}
         db eval {DROP TABLE IF EXISTS callbacks}
         db eval {CREATE TABLE callbacks(id INTEGER PRIMARY KEY,type,space,pattern,socket)}

         # register the tuple_match function
         db function tuple_match [namespace code tuple_match]
     }

     # check if argument is a list. TODO replace with [string is list] in 8.5 beta
     proc islist {string} {
     # check if a string is a valid list before doing list ops
         if {[catch {lindex $string 0}]} {
             return false
         } else {
             return true
         }
     }

     # match a tuple with a pattern
     proc tuple_match {tuple pattern} {
         if {[llength $tuple] ne [llength $pattern]} {
             return 0
         }
         foreach tup $tuple pat $pattern {
             if {![string match $pat $tup] && ![string match $tup $pat]} {
                 return 0
             }
         }
         return 1
     }

     proc add_callback {socket type tuplespace pattern} {
             # puts "$socket - $type callback registered"
             db eval {INSERT INTO callbacks VALUES(NULL,$type,$tuplespace,$pattern,$socket)}
     }

     proc remove_callback {socket} {
         db eval {delete from callbacks where socket=$socket}
     }

     # handle incoming request on the tupleserver
     proc handle_request {socket data} {
         variable commands
         if {[catch {set rc [gets $socket line]}] } return
         if {[chan eof $socket]} {
             puts "connection with $socket closed"
             chan close $socket
             # remove pending callbacks for this socket
             remove_callback $socket
             return
         }
         if {$rc == -1} {
            # not a full line yet
            return
         }
         lappend data $line

         if {![info complete [join $data \n]]} {
            fileevent $socket readable [list ::ts::handle_request $socket $data]
            return
         } else {
             fileevent $socket readable [list ::ts::handle_request $socket {}]
         }

         set data [join $data \n]
         if {![islist $data]} {
             catch {puts $socket [list error "request should be a valid list"]}
             return
         }

         set cmd [lindex $data 0]
         set params [lrange $data 1 end]
         # puts "$socket -> $data"
         if {[lsearch -all $commands $cmd] eq {}} {
             catch {puts $socket [list error "unknown server command $cmd"]}
             return
         }
         set status [catch {$cmd $socket {*}$params} result]
         switch $status {
             5 {
                set result [list nomatch $result]
             }
             2 {
                 # command resulted in a registered callback
                 return
             }
             1 {
                 set result [list error $result]
             }
             0 {
                 set result [list ok $result]
             }
         }
         # puts "$socket <- $result"
         catch {puts $socket $result}
     }

     proc purge {_ args } {
         if {[llength $args] != 1} {
             error "wrong # of args, should be \"purge tuplespace\""
         }
         set tuplespace [lindex $args 0]
         db eval {DELETE FROM tuples where space=$tuplespace}
         return
     }

    proc cb {_ args} {
        set callbacks {}
        db eval {select * from callbacks} {
            lappend callbacks [list $type $space $pattern]
        }
        return $callbacks
    }

     proc ping {_ args } {
         return
     }
     proc help {_ args} {
         variable commands
         return $commands
     }

     proc exit {socket} {
         chan close $socket
         # remove pending callbacks for this socket
         remove_callback $socket
         return -code return
     }

     proc dir {_ args } {
         set tuplespace [lindex $args 0]
         set pattern [lindex $args 1]
         switch -- [llength $args] {
             default {
                 error "wrong # of args, should be \"dir ?tuplespace? ?pattern?\""
             }
             0   {
                 return [db eval {SELECT DISTINCT space FROM tuples}]
             }
             1   {
                 set tuples [db eval {SELECT value FROM tuples WHERE space GLOB $tuplespace}]
                 return $tuples
             }

             2   {
                 if {![islist $pattern]} {
                     error "pattern should be a valid list"
                 }
                 return [db eval {SELECT value FROM tuples WHERE space GLOB $tuplespace AND tuple_match(value,$pattern)}]
             }
         }
     }

     proc out {_ args} {
         if {[llength $args] != 2} {
             error "wrong # of args, should be \"out tuplespace tuple\""
         }
         set space [lindex $args 0]
         set tuple [lindex $args 1]
         if {![islist $tuple]} {
             error "tuple should be a valid list"
         }
         # check for pending callbacks
         db eval {select id,type,socket from callbacks where space=$space and tuple_match($tuple,pattern)} {
             # puts "$socket (callback) <- ok {$tuple}"
             set err [catch {puts $socket [list ok $tuple]}]
             db eval {delete from callbacks where id=$id}
             if {$type eq "in" && !$err} {
                 # do not add the tuple when a in callback is registered and the tuple was succefully delivered
                 return
             }
         }
         return [db eval {INSERT INTO tuples VALUES(NULL,$space,$tuple)}]
     }

     proc aread {socket args} {
         areq read $socket {*}$args
     }

     proc ain {socket args} {
         areq in $socket {*}$args
     }

     proc read {socket args} {
         req read $socket {*}$args
     }

     proc in {socket args} {
         req in $socket {*}$args
     }

     proc areq {type socket args} {
         if {[llength $args] != 2} {
             error "wrong # of args, should be \"a$type tuplespace pattern\""
         }
         set tuplespace [lindex $args 0]
         set pattern [lindex $args 1]
         set id {}
         set value {}
         db eval {SELECT id,value FROM tuples where space=$tuplespace AND tuple_match(value,$pattern) LIMIT 1} {
             if {$type eq "in"} {
                 db eval {DELETE FROM tuples WHERE id=$id}
             }
         }
         if {$id eq {}} {
            return -code 5 {}
         } else {
             return $value
         }
     }

     proc req {type socket args} {
         if {[llength $args] != 2} {
             error "wrong # of args, should be \"$type tuplespace pattern\""
         }
         set tuplespace [lindex $args 0]
         set pattern [lindex $args 1]
         set resp [a$type $socket $tuplespace $pattern]
         if {$resp eq {}} {
             add_callback $socket $type $tuplespace $pattern
             return -level 2 -code return
         } {
             return $resp
         }
     }
 }

 main
 vwait forever


And a client using the Tupleserver:

 proc connect_tuplespace {name namespace} {
  set s [socket localhost 4040]
  fconfigure $s -buffering line
  fileevent $s readable [list handle_command $s $name $namespace]
  puts $s "in $name {* * *}"
 }

 # connect to server
 connect_tuplespace LOGsrv logsrv
 puts "Client connected to server"

 # define commands understood by logsrv
 namespace eval logsrv {
  proc log {severity description} {
    puts "logging $severity: $description"
  }
 }

 proc handle_command {s ts ns} {
  gets $s resp
  # get next lines if response has embedded new lines
  while {![info complete $resp]} {
     append resp \n[gets $s]
  } 
  set status [lindex $resp 0]
  set resp [lindex $resp end]
  if {$status eq "error"} {
    puts stderr $resp
    return 
  }
  set cmd [lindex $resp 0]
  set args [lrange $resp 1 end]
  if {[catch {::${ns}::$cmd {*}$args} error]} {
    puts stderr $error
  }
  puts $s "in $ts {* * *}"
 }

 vwait forever

If you now connect to the server running the tupleserver on port 4040 the following command:

 out LOGsrv {log INFO {testing info message}}

will cause one (of possibly many clients listening to the LOGsrv tuplespace) to pick up the message and display the log line.

[MJ] - Below an initial version (using Snit) to simplify writing of TS clients.

 package require snit
 snit::type client {
     variable connection {}
     variable active_tuplespace {}
 
     method connect {server port} {
         if {$connection ne {}} {
             return -code error "already connected close current connection first"
         }
         set connection [socket $server $port]
         fconfigure $connection -buffering line -encoding utf-8
         return     
     }
 
     method close {} {
         if {$connection eq {}} {
             return -code error "not connected"
         }
         close $connection
         set connection {}
     }
     method send {cmd args} {
         puts $connection [list $cmd {*}$args]
         return [$self response] 
     }
 
     method response {} {
         gets $connection resp
         while {![info complete $resp]} {
             append resp \n[gets $connection]
         }
         return $resp
     }
 
     method open {tuplespace} {
         set active_tuplespace $tuplespace
     }
 
     method out {args} {
         if {[llength $args]==2} {
             $self send out {*}$args
         } elseif {[llength $args]==1} {
             $self send out $active_tuplespace {*}$args
         } else {
             return -code error "wrong # of args: should be \"$self out ?tuplespace? tuple\""
         }
     }
 
     method in {args} {
         if {[llength $args]==2} {
             $self send in {*}$args
         } elseif {[llength $args]==1} {
             $self send in $active_tuplespace {*}$args
         } else {
             return -code error "wrong # of args: should be \"$self in ?tuplespace? tuple\""
         }
     }
 }

 # testing
 set c [client ts%AUTO%]
 $c connect localhost 4040
 $c open Test
 $c out {1 2 3 4} ;        # takes the open TS
 $c out Test {4 5 6 7} ;   # explicitly specify it
 puts [$c in {* * * *}]
 puts [$c in {* * * *}]
 
 console show

----
[LV] 2007 Jan 19 - Across this wiki, various styles of managing the page
are practiced. In some cases, one ''authority'' code is edited and improved.
In other cases, people add corrections and improvements as follow on 
comments. There are likely other methods in use as well.

I just wonder if, for this page for instance, the original author is
monitoring the page and updating some authority copy of the code, so that
someone could come along and not have to make an attempt to ''merge''
changes and proc replacements into a copy of the code and then try to
get it to work....

''[escargo] 19 Jan 2007'' - This is also an issue for programs such as [wish-reaper] that extract code
from the pages. Where there are multiple versions, or even code intended for separate files on the
same page, grabbing all the code in one (somewhat undifferentiated) block doesn't work very well.
It's a problem without a consensus solution.

[LV] Good point - if I had remembered wish-reaper, I would have mentioned that problem. Thanks!

[MJ] 05 Feb 2008: Good point, which is why I moved this implementation to a page of its own. 

----
[AM] I am interested in distributed computing - that is: let different (instances of) programs (or threads) 
cooperate with each other to get the job done. I do not see, however, from the examples how you could 
create such a system using tuplespaces (more the practical implementation, I guess, than the principle).
Anything that is easier than mutexes and their fiendish friends is welcome :).
----
[MJ] - Without knowing the details of your exact problem, it is difficult to be precise. Maybe a small example will help. Consider a simulation you want to run with a large number of parameters a possible use of tuplespaces could be:

   * Create 3 types of programs:
   1. A program generating the simulation parameters, paramgen
   1. A program performing the simulation, simu
   1. A program analyzing the results, ana
   * Create a tuplespace.
   * Let the paramgen program post parameters in the tuplesspace for instance in the form {simulate parameters}
   * The (possibly multiple) simu instances running will check for tuples matching {simulate ....} take them from the tuplespace and start simulation
   * When the simulation is done, the simu program will put a tuple {result results} in the tuplespace.
   * The ana instance running will check for tuples matching {result ....} take them from the tuplespace and analyze the result.

This will allow you to start additional simu instances on different nodes without any changes to the code (the Tuplespace will handle the coordination). If analyzing takes a long time, multiple instances could be started as well. To easily monitor progress one could image a Log process (as in the example above) that checks for tuples in the form {log src severity msg} and writes them to a central log file. All other processes can then do logging using the tuplespace. For instance the simu processes could post for instance {log simu info {simulation finished}}.

Does this help? 

[AM] I think it does. I wll try and see how this fits into the sort of systems I am thinking about (domain decomposition
for the solution of PDEs for instance where values on the boundaries of the various domains are exchanged and the
processes need to wait for the results of their neighbours to become available), but this may be a small modification
of the above.

[AM] (Mostly as a reminder for myself) Alternative applications of tuplespaces:

   * Online visualisation
   * Computational steering
   * Cooperating agents

(Well, these are examples I have been thinking about from time to time, nothing very original, but
they could be well implemented using tuplespaces instead of via the "observer" pattern)

Oh, I did find a nice tutorial on the subject, with a rather lame elaborated example (lame, because the
technique used for solving the number crunching is awkward to say the least, but perhaps shows
the principle better): http://www2.java.no/web/files/moter/tuplespace.pdf

[MJ] - The original paper "Generative communication in Linda" by David Gelernter contains some more examples and provides some higher abstractions(like P2P communication) based on the basic primitives. Unfortunately I cannot find a free download, but the article is available from the ACM http://portal.acm.org/citation.cfm?id=2433.

[MJ] - 2007-11-16 Added updated version that:
   * uses the ''db function'' functionality to add ''tuple_match'' as an SQLite function. This makes selecting tuples much easier (and probably faster)
   * changed callbacks to be stored in a table
   * removed open and close. Just ''out'' a tuple to a tuplespace and it will be created.

[MJ] - Feb 2008 Added abillity to use multiline tuples (as long as it is a valid Tcl list). Added a return code for ain and aread if there was no match.

[AM] (5 february 2008) I have started to develop a practical application of tuplespaces. The idea is that 
you can use the tuple server to manage data (both input and output) so that you can easily do computations on 
various machines. 

More specifically:

   * The tuple server runs on machine 1
   * A "client" which has the task of preparing the input for the computations and sending them off to the tuple server runs on machine 2. For this task it can use a Tcl procedure or an external program.
   * Running on machine 3 (or 4 or 5 or ...) a "computational client" just connects to the tuple server, waiting for suitable tasks. These consist of a bunch of input files, which it receives and stores in a convenient place. Once all the data are there, the external program is started. 
   * The results from that program are sent back to the tuple server.
   * The client waits for the message that a task has been completed. If it receives such a message, it picks up the results from the tuple server and stores them locally.

Right now it is getting in the stage where it actually works (I had to cope with a few oddities in my code that led to the 
loss of files, but solving them was actually easy). More to come - like the actual source code, but that will be on a separate 
page with a better description.

----
%|[Category Concept] | [Category interprocess communication]|%