WebSockets are a nice alternative to XMLHTTPRequest for bi-directional communication between a web browser and a server application. They need a browser, e.g. Chrome, that supports the WebSocket API and a server that supports the WebSocket protocol.
AMG: I moved the older discussion to the bottom of the page [L1 ] so the current code can be right at the top.
AMG: Here's the code. It requires the latest Wibble [L2 ].
EF: Note that a complete client library, which code inherits from the implementation below, is available in WebSocket Client Library. The library is also able to provide protocol framing support within a Web server, if necessary.
dzach: 2013-05-03 Any reason why the query key is not included in the request dict?
AMG: I don't see why it wouldn't be. query and rawquery are set very near the top of [getrequest]. dzach You are right. Strangely enough, my working copy of wibble, which I thought was "clean", doesn't, but a fresh one I just ran does include the query in the request dict.
package require sha1
# Define the ::wibble::ws namespace.
namespace eval ::wibble::ws {
namespace path ::wibble
}
# Send a WebSocket frame.
proc ::wibble::ws::send {type {msg ""} {final 1}} {
log "[info coroutine] Tx $type $msg"
# Compute the opcode. The opcode is zero for continuation frames.
upvar #1 fragment fragment
if {[info exists fragment]} {
set opcode 0
} else {
set opcode [dict get {text 1 binary 2 ping 9} $type]
}
if {!$final} {
set fragment ""
} else {
unset -nocomplain fragment
}
# Encode text.
if {$type eq "text"} {
set msg [encoding convertto utf-8 $msg]
}
# Assemble the header.
set header [binary format c [expr {!!$final << 7 | $opcode}]]
if {[string length $msg] < 126} {
append header [binary format c [string length $msg]]
} elseif {[string length $msg] < 65536} {
append header \x7e[binary format Su [string length $msg]]
} else {
append header \x7f[binary format Wu [string length $msg]]
}
# Send the frame.
set socket [namespace tail [info coroutine]]
chan puts -nonewline $socket $header$msg
chan flush $socket
}
# Close the current WebSocket connection.
proc ::wibble::ws::close {{reason ""} {description ""}} {
icc put ::wibble::[namespace tail [info coroutine]] exception close\
$reason $description
}
# WebSocket analogue of [::wibble::process]
proc ::wibble::ws::process {state socket request response} {
# Get configuration options.
if {![dict exists $state options maxlength]} {
set maxlength 16777216
} elseif {[dict get $state options maxlength] eq ""} {
set maxlength 18446744073709551615
} else {
set maxlength [dict get $state options maxlength]
}
# Create WebSocket event handler wrapper coroutine.
set fid [namespace current]::$socket
cleanup ws_unset_feed [list icc destroy $fid]
icc configure $fid accept connect disconnect text binary close
coroutine $socket apply {{handler state} {
try {
while {1} {
foreach event [icc get [info coroutine] *] {
{*}$handler $state {*}$event
if {[lindex $event 0] eq "disconnect"} {
return
}
}
}
} on error {"" options} {
panic $options "" "" "" "" [dict get $state request]\
[dict get $state response]
}
} ::wibble::zone} [dict get $state options handler] $state
# Respond to WebSocket handshake.
chan puts $socket "HTTP/1.1 101 WebSocket Protocol Handshake"
chan puts $socket "upgrade: websocket"
chan puts $socket "connection: upgrade"
chan puts $socket "sec-websocket-accept: [binary encode base64 [sha1::sha1 -bin\
[dict get $request header sec-websocket-key\
]258EAFA5-E914-47DA-95CA-C5AB0DC85B11]]"
chan puts $socket ""
chan flush $socket
chan configure $socket -translation binary
# It's necessary to bypass [icc put] in this one case, because it defers
# event delivery when called from a coroutine. Consequentially, before it
# would attempt to send the event, the feed will be destroyed.
cleanup ws_disconnect [list $fid disconnect]
# Invoke connect handler.
icc put $fid connect
set reason 1000
try {
foreach event [icc catch {
# Main loop.
set msg ""
set mode ""
while {1} {
# Get basic header. Abort if reserved bits are set, mask bit
# isn't set, unexpected continuation frame, fragmented or
# oversized control frame, or the opcode is unrecognized.
binary scan [getblock 2] Su header
set opcode [expr {$header >> 8 & 0xf}]
set len [expr {$header & 0x7f}]
if {($header & 0x7080 ^ 0x80) || ($opcode == 0 && $mode eq "")
|| ($opcode > 7 && (!($header & 0x8000) || $len > 125))
|| $opcode ni {0 1 2 8 9 10}} {
# Send close frame, reason 1002: protocol error.
set reason 1002
break
}
# Determine the effective opcode for this frame.
if {$mode eq ""} {
set mode $opcode
} elseif {$opcode == 0} {
set opcode $mode
}
# Get the extended length, if present.
if {$len == 126} {
binary scan [getblock 2] Su len
} elseif {$len == 127} {
binary scan [getblock 8] Wu len
}
# Limit the maximum message length.
if {[string length $msg] + $len > $maxlength} {
# Send close frame, reason 1009: frame too big.
set reason [list 1009 "limit $maxlength bytes"]
break
}
# Use an alternate message buffer for control frames.
if {$opcode > 7} {
set oldmsg $msg
set msg ""
}
# Get mask and data. Format data as a list of 32-bit integer
# words and list of 8-bit integer byte leftovers. Then unmask
# data, recombine the words and bytes, and append to the buffer.
binary scan [getblock [expr {4 + $len}]] II*c* mask words bytes
for {set i 0} {$i < [llength $words]} {incr i} {
lset words $i [expr {[lindex $words $i] ^ $mask}]
}
for {set i 0} {$i < [llength $bytes]} {incr i} {
lset bytes $i [expr {[lindex $bytes $i] ^
($mask >> (24 - 8 * $i))}]
}
append msg [binary format I*c* $words $bytes]
# If FIN bit is set, process the frame.
if {$header & 0x8000} {
switch $opcode {
1 {
# Text: decode and notify handler.
icc put $fid text [encoding convertfrom utf-8 $msg]
} 2 {
# Binary: notify handler without decoding.
icc put $fid binary $msg
} 8 {
# Close: decode, handle, send close frame, terminate.
if {[string length $msg] >= 2} {
binary scan [string range $msg 0 1] Su reason
icc put $fid close $reason [encoding convertfrom\
utf-8 [string range $msg 2 end]]
} else {
icc put $fid close
}
set reason ""
break
} 9 {
# Ping: send pong to client, don't notify handler.
chan puts -nonewline $socket \x8a[binary format c\
[string length $msg]]$msg
chan flush $socket
}}
# Prepare for the next frame.
if {$opcode < 8} {
# Data frame: reinitialize parser.
set msg ""
set mode ""
} else {
# Control frame: restore previous message buffer.
set msg $oldmsg
}
}
}
}] {
# Catch exception events and translate them into close reason codes.
if {[lrange $event 0 1] eq {exception close}} {
set reason [lrange $event 2 3]
break
}
}
} on error {result options} {
# Default error close reason.
set reason {1001 "internal server error"}
return -options $options $result
} finally {
# Send close frame with reason, if one was given.
catch {
if {[llength $reason]} {
set msg [string range [binary format Su [lindex $reason 0]\
][encoding convertto utf-8 [lindex $reason 1]] 0 124]
chan puts -nonewline $socket \x88[binary format c\
[string length $msg]]$msg
} else {
chan puts -nonewline $socket \x88\x00
}
}
}
# Ask Wibble to close the connection.
return 0
}
# WebSocket upgrade zone handler.
proc ::wibble::zone::websocket {state} {
set header [dict get $state request header]
if {[dict exists $header sec-websocket-key]
&& ([dict exists $header sec-websocket-origin] || [dict exists $header origin])
&& [dict exists $header sec-websocket-version]
&& [dict exists $header connection]
&& [dict exists $header upgrade]
&& [lsearch -nocase [dict get $header connection] upgrade] != -1
&& [string equal -nocase [dict get $header upgrade] websocket]} {
sendresponse [list nonhttp 1 sendcommand [list ws::process $state]]
}
}sbron 2013-10-27: I have removed the dependency on the base64 package. Wibble requires Tcl 8.6 to run and 8.6 has base64 encoding/decoding built-in. So it seemed silly to use an external package for that.
AMG: Agreed, thank you.
dzach 26-12-2012: It looks like Firefox does not send the "sec-websocket-origin" header. I have not dug into the details but it seems to me that the "origin" header should serve the same purpose, so, for the above WebSocket zone handler to work with Firefox (and Opera), a check for the "origin" header can be added:
proc ::wibble::zone::websocket state {
set header [dict get $state request header]
if {[dict exists $header sec-websocket-key]
&& ([dict exists $header sec-websocket-origin] || [dict exists $header origin])
&& [dict exists $header sec-websocket-version]
&& [dict exists $header connection]
&& [dict exists $header upgrade]
&& [lsearch -nocase [dict get $header connection] upgrade] != -1
&& [string equal -nocase [dict get $header upgrade] websocket]} {
sendresponse [list nonhttp 1 sendcommand [list ws::process $state]]
}
}UKo 2015-11-22: sec-websocket-origin is no longer part of the WebSocket standard (now part of the living html standard [L3 ]). The last version it was mentioned is 10 [L4 ] and it was removed in 11 [L5 ].
sbron 2017-09-23: RFC6455 specifies that the origin header is optional. It should therefore probably not be used as a check if a request is for a websocket. If desired, the header may be used later to protect against malicious connections. But then the contents of the header must be checked, not just its presence.
Here's an example WebSocket handler:
proc ::wibble::zone::ws-demo {state event args} {
upvar #1 cancel cancel
log "[info coroutine] Rx $event $args"
# Perform initialization and cleanup.
if {$event eq "connect"} {
icc configure [info coroutine] accept tick
} elseif {$event eq "disconnect"} {
after cancel $cancel
}
# Process timer and text events.
if {$event in {connect tick}} {
set cancel [after 1000 [list ::wibble::icc put [info coroutine] tick]]
ws::send text "time [clock format [clock seconds]]"
} elseif {$event eq "text"} {
if {[lindex $args 0] eq "close"} {
ws::close
} else {
ws::send text "expr [expr [lindex $args 0]]"
}
}
}Also, put "::wibble::handle /ws-demo websocket handler ws-demo" at the top of your zone handler list. And here's a demo HTML5 file, usable in Chrome:
UKo 2015-11-23: For me this demo works in all tested modern browsers:
<!DOCTYPE html>
<html lang="en">
<body>
<table>
<tr><th>State</th><td id="state">initializing</td></tr>
<tr><th>Time</th><td id="time">???</td></tr>
<tr><th>Result</th><td id="result">???</td></tr>
<tr><th>Expression</th><td><form>
<input type="text" id="expression" />
</form></td></tr>
<script>
var ws = new WebSocket("ws://"+window.location.host+"/ws-demo");
var state = document.getElementById("state");
var time = document.getElementById("time");
var result = document.getElementById("result");
var expression = document.getElementById("expression");
var sendcount = 0;
var receivecount = 0;
ws.onopen = function () {
state.innerHTML = "connected";
};
ws.onmessage = function (evt) {
receivecount += 1;
state.innerHTML = "sent " + sendcount + ", received " + receivecount;
if (evt.data.substr(0, 4) == "time") {
time.innerHTML = evt.data.substr(5);
} else if (evt.data.substr(0, 4) == "expr") {
result.innerHTML = evt.data.substr(5);
}
};
ws.onclose = function (evt) {
state.innerHTML = "disconnected";
};
expression.form.addEventListener("submit", function (evt) {
evt.preventDefault();
if (ws.readyState === 1) {
ws.send(expression.value);
expression.value = "";
sendcount += 1;
state.innerHTML = "sent " + sendcount + ", received " + receivecount;
}
return false;
});
</script>
</body>
</html>As promised, this demo incorporates both a clock and a calculator. I did this to better exercise the asynchronous nature of WebSocket. Events come both from the client and from the timer.
Now that I know the [tailcall] trick [L6 ], I'm considering writing an [icc] utility proc or two for the purpose of easing event handler dispatch. It/they would wrap around this idiom, which I've had to write more than once in my own code:
foreach event [icc get $feed {*}$filters] {
switch -glob [lindex $event 0] {
filter/*/whatever {puts 1}
filter2/whatever/* {puts 2}
etc {puts 3}
}
}Or in the case of the above WebSocket code, replace [icc get $feed {*}$filters] with [icc catch $script].
icc get2 $feed $filters\
filter/*/whatever event {puts [lindex $event 1]}\
filter2/whatever/* "" {puts 2}\
etc "" {puts 3}
icc catch2 $script\
filter/*/whatever event {puts [lindex $event 1]}\
filter2/whatever/* "" {puts 2}\
etc "" {puts 3}\
exception "" {puts 4}The name(s) and syntax are definitely subject to change. Suggestions? Here are some ideas:
Pulling all this together might yield:
icc switch -feeds $feed -filters $filters -eventvar event\
-identvar ident -identsplit / {
filter/*/whatever {puts [lindex $event 1]}
filter2/whatever/* {puts [lindex $ident 2]}
etc {puts 3}
}
icc switch -script $script -eventvar event\
-identvar ident -identregexp {(.*)/(.*)/(.*)} {
filter/*/whatever {puts [lindex $event 1]}
filter2/whatever/* {puts [lindex $ident 2]}
etc {puts 3}
exception {puts 4}
}But now I think that's too many pairs of options which must go together, and that complicates parsing. Maybe replace "-feeds $feed -filters $filters" with "-get [list $feed $filters]". Hmm, that [list] in there makes it seem even less clean, though. I dunno.
agb Dec. 2010. Chrome now supports an updated version of the websocket protocol so the wibble example previously here no longer works (to see it have a look in this page's history) . The changes to get the current version working are non-trivial.
jbr 2010-12-20 - Here is code that will allow wibble to handshake with the new spec. The version of Chrome that I have (8.0.552.231) has the new handshake but sends the old data framing. I can send data from the client, but, I haven't gotten it to accept data messages from the server. Wibble.tcl needs to be patched to add a way for it to release the socket that will be the websocket channel without responding and closing it:
agb 2010-12-21 - I made a small change to ::wibble::ws-handle to check that chan read actually reads a byte. With this change I have successful, bi-directional messages over the web socket with chrome 8.0.552.224. Thanks for updating wibble.
# Abort processing on this client.
proc wibble::abortclient {} {
return -code 7
}AMG: I take it that this command is to be called by a zone handler in order to get Wibble to terminate the coroutine without closing the socket. Correct?
Also, see my comments on Wibble wish list ([L7 ]) for an alternative, less invasive approach.
jbr: Set keepalive 0 at the top of ::wibble::process and then change the exceptional return handling like this:
} on 7 outcome {
set keepalive 1
} finally {
if { !$keepalive } {
catch {chan close $socket}
}
}AMG: Some time after you wrote the above, I have changed Wibble to have customizable cleanup handlers. With the latest version of Wibble, instead of modifying the finally block, change the initialization of the cleanup list (top of [process]) to the following:
set cleanup {
{chan close $file}
{if {!$keepalive} {chan close $socket}}
{dict unset ::wibble::icc::feeds $coro}
}jbr: Add this to the zone handlers:
wibble::handle /ws websocket handler ws-demo
This is your server side callback:
proc ::ws-demo { event sock { data {} } } {
switch $event {
connect {}
message {
puts "WS-Demo: $event $sock $data"
}
}
::wibble::ws-send $sock "Hello"
}AMG: Are connect and message the only two events that can happen?
jbr: Connect and message are the only two events. WebSockets is a very low level thing (data packets) with the application specific messaging completely undefined.
jbr: Utility to help the server send data frames, doesn't work yet!!
proc ::wibble::ws-send { sock message } {
# New data framing?
#puts -nonewline $sock [binary format cc 4 [string length $message]]$message
# Old data framing?
puts -nonewline $sock "\x00"
puts -nonewline $sock $message
puts -nonewline $sock "\xFF"
flush $sock
}Handler to accept data from browser. Uses old data framing.
proc ::wibble::ws-handle { handler sock } {
if { [chan eof $sock] } {
puts "Closed $sock"
close $sock
} else {
set code [read $sock 1]
if {[binary scan $code c code]} { ; # Do I need this? I think so.
switch $code {
0 {
set message {}
while { [set c [read $sock 1]] != "\xFF" } {
append message $c
}
$handler message $sock $message
}
default {
puts "Bad Blocking: $c"
}
}
}
}
}The Zone Handler
package require md5
proc ::wibble::websocket { state } {
set upgrade {}
set connection {}
dict with state request header {}
if { $connection ne "Upgrade" || $upgrade ne "WebSocket" } {
return
}
set sock [dict get $state request socket]
puts "WebSocket Connect: $sock"
set key1 [regsub -all {[^0-9]} ${sec-websocket-key1} {}]
set spc1 [string length [regsub -all {[^ ]} ${sec-websocket-key1} {}]]
set key2 [regsub -all {[^0-9]} ${sec-websocket-key2} {}]
set spc2 [string length [regsub -all {[^ ]} ${sec-websocket-key2} {}]]
set key3 [read $sock 8]
set handler [dict get $state options handler]
chan event $sock readable [list ::wibble::ws-handle $handler $sock]
set key1 [expr $key1/$spc1]
set key2 [expr $key2/$spc2]
set challenge [binary format II $key1 $key2]$key3
set response [md5 $challenge]
puts $sock "HTTP/1.1 101 WebSocket Protocol Handshake"
puts $sock "Connection: Upgrade"
puts $sock "Upgrade: WebSocket"
puts $sock "Sec-WebSocket-Origin: http://localhost:8080" ; # This shouldn't be hard coded!!
puts $sock "Sec-WebSocket-Location: ws://localhost:8080/ws/demo"
puts $sock ""
chan configure $sock -translation binary
puts $sock $response
chan flush $sock
$handler connect $sock ; # There should be an option to pass a session Id here.
abortclient
}AMG: Thanks for the code, guys. I will need to ponder some more before integrating this into Wibble, but I do think I want this feature. However, I think it would benefit from tighter integration. As far as I can tell, it leverages Wibble for establishing the connection but then takes over all I/O. This concept is quite similar to something JCW shared with me the other day, namely an implementation of Server-Sent Events [L8 ] [L9 ]. Whatever I do, I would like it to support both protocols, or at least their common requirements.
If you're wondering why I haven't integrated all this sooner, it's because AJAX was my priority. It may be terribly clumsy compared to WebSockets and Server-Sent Events, but it also has the most browser support.
jcw Neat... jbr's return 7 and keepalive idea look like a very useful tweak:
jbr 2011-05-01 Andy has offered a better way to handle this by removing the socket from the coroutines list and returning an uncaught error. No need to hack Wibble's main zone handler body.
} on 7 outcome {
set keepalive 1
} finally {
if { !$keepalive } {
catch {chan close $socket}
}
}Better than what I'm doing right now, which is to do an "icc get" to grab control over the socket by suspending the co-routine indefinitely. The problem with that is that I always get an error on socket close, as wibble tries to resume and send a response to the (now closed) socket. What's not clear to me is whether the "return 7" also causes the request's co-routine to be cleaned up right away (seems like a good idea).
AMG: The coroutine will always be cleaned up, thanks to the "finally" clause inside [process]. The only way to avoid the "finally" clause is to delete the current coroutine command (rename [info coroutine] "") then yield.
A few days ago I came up with another approach that I prefer to any presented on this page or the Wibble wish list: define a new key in the response dict that defines a custom I/O handler that [process] will execute instead of doing its normal post-[getresponse] activities. This way, more of the Wibble infrastructure is available to the custom code: error handling, automatic cleanup, and the ability to loop again and get another HTTP request from the same socket.
2011.1013 jbr Here we are almost a year later with an update.
WebSocket has again moved to a new handshake & framing. Here is a zone handler for Andy's newest wibble and chrome 14 websockets.
package require sha1
package require base64
# Utility proc to frame and send short strings up to 126 chars
#
proc ::wibble::ws-send { sock message } {
puts -nonewline $sock [binary format cc 0x81 [string length $message]]$message
flush $sock
}
# WebSocket handler proc to receive short (up to 126 chars) text format frames
#
proc ::wibble::ws-handle { handler sock } {
if { [chan eof $sock] } {
close $sock
} else {
binary scan [read $sock 1] c opcode
binary scan [read $sock 1] c length
set opcode [expr $opcode & 0x0F]
set length [expr $length & 0x7F]
binary scan [read $sock 4] c* mask
binary scan [read $sock $length] c* data
set msg {}
set i 0
foreach char $data {
append msg [binary format c [expr { $char^[lindex $mask [expr { $i%4 }]] }]]
incr i
}
$handler message $sock $msg
}
}
# Zone handler
#
proc ::wibble::websocket { state } {
set upgrade {}
set connection {}
dict with state request header {}
if { $connection ne "Upgrade" || $upgrade ne "websocket" } {
return
}
set sock [dict get $state request socket]
puts "WebSocket Connect: $sock"
set response [base64::encode [sha1::sha1 -bin ${sec-websocket-key}258EAFA5-E914-47DA-95CA-C5AB0DC85B11]]
set handler [dict get $state options handler]
puts $sock "HTTP/1.1 101 WebSocket Protocol Handshake"
puts $sock "Upgrade: websocket"
puts $sock "Connection: Upgrade"
puts $sock "Sec-WebSocket-Accept: $response"
puts $sock ""
chan configure $sock -translation binary
chan event $sock readable [list ::wibble::ws-handle $handler $sock]
$handler connect $sock
return -code 7
}At the top of the ::wibble::process proc I initialize the keepsock variable and modified the cleanup procs like this:
set keepsock 0
set cleanup {
{chan close $file}
{ if { !$keepsock } { chan close $socket } }
{dict unset ::wibble::icc::feeds $coro}
}Then I added a clause in the try structure to catch the return -code 7 from the zone handler:
} on 7 outcome {
set keepsock 1
} finally {Works for me. Thanks Andy.
AMG: I made an update of the above for version 2011-11-24 [L10 ]. Thanks to some spiffy new features in Wibble, it doesn't require any core modifications. However, I'm not quite ready to post it here, since I need to test it. I grabbed a copy of Chrome, but I don't know how to get it to talk to WebSocket anything. If you could please post an example HTML/JavaScript file that makes use of the above, I'll debug what I've got and post the combined demo here.
A more flexible alternative to the WebSocket handler proc receiving the event (connect or message) as its first argument is to support separate connect and message handler command prefixes in the options dict. For compatibility with existing handlers, the connecthandler and messagehandler could name the same proc, but with an extra connect or message argument. Does this sound like a worthwhile change? Also, how about a disconnect handler? Would that ever have any value? The ICC feed mechanism already has a lapse system which is designed to detect application-level user disconnects, as opposed to protocol-level TCP socket disconnects.
In my development code, I got rid of the socket arguments since the socket name is always [namespace tail [info coroutine]].
In addition to the max-126-character string protocol, is there a way to send and receive binary data? Should we support it?
AMG: Disregard the above, it's overcome by events. I redid my update; it's totally new code now. It still doesn't require any core modifications, which is good, and I'm still not ready to post it here, which is bad. However, I did test it, and it works so far. I ran into some fundamental design issues, which I'll discuss below. I decided against separate connecthandler and messagehandler; a single handler works well enough, and it can dispatch to separate procs if needed. I kept the disconnect event, on the theory that a handler might want an opportunity to clean up after itself. The socket arguments are still gone. I added support for binary data, frames longer than 126 bytes, ping/pong, close frames, fragmentation and continuation frames, basically everything I could find in the WebSocket protocol draft document [L11 ].
The big design problem derives from the fact that WebSocket doesn't require the client and server to take turns the way HTTP does. Wibble successfully models HTTP with this main loop, running inside a coroutine:
while {1} {
set request [getrequest $port $socket $peerhost $peerport]
set response [getresponse $request]
if {![{*}$sendcommand $socket $request $response]} {
chan close $socket
break
}
}With WebSocket, both sides can talk simultaneously; yet this is done with only a single TCP connection. This does not map well to Wibble's one-coroutine-per-socket model, shown above. The Wibble coroutine cycles between three states: get the client data, generate the response, send the response, then repeat or terminate.
Let's take a simple example: a calculator with a clock. Implement these two features as separate connections, and there's little challenge. But multiplex them together, and you've got problems. The client should be able to send a new math expression at any time, to which the server should rapidly respond with the result. Every second, the server should also send the new time, without the client asking.
My first inclination is to use [icc] to wait for either readability or timeout. ([icc] can also be used to wait for events coming from other coroutines, but that's not important here.) That would work, if not for the problem of incomplete reads. Reading the WebSocket data is nontrivial. You have to read the first two bytes for the basic header, then read zero, two, or eight more for the extended length (depending on the length given in the basic header), plus four more for the mask, and then you have to read the actual data. For extra joy, consider that a message can be fragmented across multiple frames, and that control frames can be interleaved with the fragmented data frames. Also consider that each frame can be up to 14+2**64 bytes in size. Socket readability certainly does not guarantee that the entire frame is immediately available for reading. This isn't UDP. ;^)
Since reading the data isn't atomic, it's a stateful process, with a potential wait state everywhere a read happens. At every wait state the system needs to return to the event loop so that a different socket can be serviced. Sounds like a job for coroutines. ;^) But... what if the other operation waiting to be processed exists in the same coroutine as the read? That doesn't work.
Here's what I'm considering. Create a second coroutine for each WebSocket connection which doesn't directly fit the HTTP turn-taking model. Name the coroutine the same as the socket, but put it in the ::wibble::ws namespace, such that it's separate from the one in the ::wibble namespace yet [namespace tail [info coroutine]]] is still the socket name. The original Wibble coroutine is in charge of reading the client data, and whenever a message is received, it sends it to the new coroutine via [icc]. The new coroutine also roughly follows the HTTP model shown above, but with [getrequest] replaced with [icc get]. This way it aggregates messages from multiple sources and gets them atomically. When it wants to send, it writes to the socket. Meanwhile, the first coroutine only writes to the socket when initiating and tearing down the connection.
To make this efficient, I would have to modify [icc] to not concatenate the event identification and data together into a single string. Instead, the payload would need to be a separate Tcl_Obj. (AMG, update: [icc] already supports arbitrary payloads in addition to the event identifier! No change needed.)
Now I'm gonna digress from WebSocket in the interest of searching for a unified architecture, since I'm always looking for ideas. :^)
This two-coroutine model isn't necessary for any of the pure-HTTP concepts, since the communication direction strictly alternates. Imagine if I went ahead and made two coroutines anyway. They'd always be taking turns. Each one would do its thing, notify the other that it's done, then wait for the other to finish. Therefore they might as well just be a single coroutine, which is exactly what I have.
Actually, there is one major exception, one time when the HTTP client can do something that's of interest to the server even though it's the server's turn to talk. That one thing is: disconnect. In an AJAX long polling situation, the server can hang for minutes at a time waiting for there to be something worthwhile to report. During that time, the client can disconnect, but the server wouldn't be able to tell until it tries to read from the client. Of course, the server can't try to read until after it sends something. My current solution is to have the server periodically send a no-op, just as an excuse to check if the client is still alive. If I had a two-coroutine model for AJAX long polling, the client read coroutine could always be on the lookout for client disconnect; when it happens, it can use [icc] to let the other coroutine know it's quitting time. (More likely, it'll use [icc] to let all the other coroutines know that the one client died, and it'll just directly terminate the coroutines associated with the socket.)
However, there's still a problem: unclean disconnects. If the client makes a request, it's now the server's turn to respond. While the server is hanging out waiting for something worthwhile to happen, the client sends some more data. The only way the read coroutine can tell the difference between the client sending data and the client disconnecting is to try to read from the client, so that's what it does. It can't do this without bound, so eventually it'll have to stop reading. At this point, the read coroutine again loses its ability to know when the client disconnects. One possible solution is to read without bound, but that opens up a DoS attack. Another possible solution is to forcibly disconnect the client if it sends too much data out-of-turn, but this interferes with HTTP pipelining. Everything comes down to the fact that the portable C I/O API doesn't have an out-of-band notification of client disconnects. So, I can't think a solution solid enough to justify the complexity of adding another coroutine, and I strongly lean against adopting a two-coroutine model for the Wibble core. But I'm definitely open to suggestions!
jbr 2011-11-26 Andy, my thinking here is that once you've gotten an accept via the websocket zone handler then you just have a socket. You can use icc if you like or you can just register a fileevent. There is no reason to tie it to any other wibble infrastructure or HTTP concepts, its just a socket connected to some javascript in a web browser. The wibble co-routine should just return without closing the socket.
P.S. - Thanks for coding up the rest of the message encoding. I've just been passing simple events to my server from user interactions with forms, but completeness is a good thing.
AMG: But be careful, the WebSocket code is running inside the same thread of execution as the rest of the web server. The code you posted above is extremely easy to DoS. All you have to do is perform a valid WebSocket handshake, then send one byte and one byte only. The code will block indefinitely waiting for the rest of the WebSocket header. During that time, the rest of the server is blocked too. Using [getblock] instead of [read] solves the problem, since it yields rather than block the whole process. So hey, there's a case where the Wibble infrastructure is useful inside WebSocket code.
jbr - I can't argue with that using [getblock] (or something like it) isn't a good idea. I'm just suggesting that the web socket zone handler allow the application specific code do its own thing an not impose a framework that might not fit so well. What if I want to hand the socket to a 3rd party library that already handles a web socket connection in its own code?
AMG: Tonight I hope to post the new code I've got so you can better see what I have in mind. I don't plan for WebSocket to be integral to Wibble, especially if that means its design can constrain the application. Rather, WebSocket should be an add-on product, mostly as a demonstration of Wibble's flexibility. If implementing WebSocket cleanly requires a change to Wibble, I'll make that change, but I'll do so in a generic enough fashion that it'll also work for Server-Sent Events or other connection upgrades. If an interesting design idea crops up in the course of implementing WebSocket or any other zone handler or application, I'll consider adopting it (or its cousin) into Wibble proper. But I'm not going to change Wibble's scope in the process. [::wibble::process] shall forever deal only in HTTP, and if an application wants to switch from HTTP to something else, it should do so using the sendcommand response dict entry, to which [process] will hand over control.
Here's the relationship between Wibble and WebSocket. This is the same relationship as between Wibble and other zone handlers, except (1) Wibble doesn't attempt to send anything to the client, and (2) the WebSocket code does all reads after the initial HTTP-like handshake.
I do have a framework in mind for WebSocket. It does resemble Wibble in some ways, but only because I think it's a sensible approach. :^) The fact is that it also resembles the code you started with; I just beefed it up for completeness and to avoid the DoS problem mentioned above.