**Wibble web server**
[AMG]: Wibble is a small, [pure-Tcl] web server inspired by [Wub], [DustMote], [Coronet], and [Templates and subst]. One fine day I wanted to put together a site using Wub, but I needed help and couldn't find [CMcC] or [JDC] in the [Tcl chatroom]. The need to hack would not be denied! So I wrote this.
This code is intended to be customized for your application. Start by changing the zone handlers and root directory. Feel free to create your own zone handlers; it's very easy to do so. Just make another [proc].
***Name***
Regarding the name: "Wibble" is similar in sound to "Wub", and http://en.wikipedia.org/wiki/Wibble%|%according to Wikipedia%|%, it is one possible pronunciation of "[www]".
***Zone handlers***
Zones are analogous to domains in Wub and [TclHttpd]. Zones are matched against the [URI] by prefix, with an important exception for directory names. I use the name "zone" instead of "domain" because I don't want any confusion with [DNS] domain names. Someday I may add support for virtual hosts via the Host: header, which means that DNS domain/host names will be included in zone names.
Handlers can be stacked, so many zones are defined by a combination of handlers which are executed in sequence. Each handler receives a request [dict]ionary as its last argument, augmented with configuration options and a few extra parameters. The request dictionary is derived from the HTTP request. The configuration options are defined in the '''$::zones''' variable. The extra parameters indicate the match prefix and suffix, plus the (possible) filesystem path of the requested object. The handler then uses the [[operation]] command to return an opcode and operand. The opcode specifies which operation should be performed to continue or finish processing.
Zones also stack. For example, if no handlers for zone /foo return a response, then the handlers for / are tried. Just as the handlers within a zone must be specified in the order they are to be executed, the zones themselves must be specified in order of decreasing specificity. To inhibit this stacking behavior, be sure that a default handler is defined for the zone, e.g. '''notfound'''.
There is a list of requests, initialized to contain only the request received from the client. Requests can be added, modified, or deleted by zone handlers. Each request is tried in turn against each handler in the zone.
***Request dictionary***
* '''socket''': The name of the Tcl [channel] that is connected to the client.
* '''peerhost''': Network address of the client.
* '''peerport''': TCP port number of the client.
* '''method''': HTTP method (GET, PUT, POST, HEAD, etc.).
* '''uri''': HTTP URI, including query string.
* '''path''': Path of client-requested object, excluding query string.
* '''protocol''': HTTP/1.0 or HTTP/1.1, whatever the client sent.
* '''header''': Dictionary of HTTP header fields from the client.
* '''query''': Dictionary of query string elements.
* '''querytext''': Query string in text form.
***Extra parameters***
* '''prefix''': Zone prefix name.
* '''suffix''': Client-requested object path, sans prefix and query string.
* '''fspath''': Object path with '''root''' option prepended. Only defined if '''root''' option is defined.
***Configuration options***
* '''root''': Filesystem directory corresponding to zone root directory.
* '''indexfile''': Name of "index.html" file to append to directory requests.
Support for configuration options varies between zone handlers.
***Supported zone handler return opcodes***
* '''prependrequest''': Merge operand with request dictionary and insert into list of requests with high priority.
* '''replacerequest''': Merge operand with request dictionary, replacing current request.
* '''deleterequest''': Remove current request from request list.
* '''sendresponse''': Send operand to client as response dictionary.
* '''pass''': Fall through to next request or next handler.
***Predefined zone handlers***
* '''vars''': Echo request dictionary plus any extra or optional arguments.
* '''dirslash''': Redirect directory requests lacking a trailing slash.
* '''indexfile''': Add '''indexfile''' to directory requests.
* '''static''': Serve static files (not directories).
* '''template''': Serve data generated from .tmpl files.
* '''dirlist''': Serve directory listings.
* '''notfound''': Send 404.
----
**Sample index.html.tmpl**
======
% dict set response header Content-Type "text/html; charset=utf-8"
$uri
% set rand [expr {rand()}]
% if {$rand > 0.5} {
random number [format %.3f $rand] > 0.5
% } else {
random number [format %.3f $rand] <= 0.5
% }
[clock format [clock seconds]]
======
----
**Wibble source code**
======
#!/bin/sh
# The next line restarts with tclsh.\
exec tclsh "$0" ${1+"$@"}
package require Tcl 8.6
package provide wibble 0.1
# Define the wibble namespace.
namespace eval wibble {
namespace export filejoin unhex operation handle listen
variable zones {}
}
# Echo request dictionary.
proc wibble::vars {request} {
dict set response status 200
dict set response header content-type "text/html; charset=utf-8"
dict set response content {}
dict for {key val} $request {
if {$key in {header query}} {
set newval ""
dict for {subkey subval} $val {
append newval "[list $subkey] [list $subval] "
}
set val $newval
}
dict append response content | $key | $val |
}
dict append response content
\n
operation sendresponse $response
}
# Redirect when a directory is requested without a trailing slash.
proc wibble::dirslash {request} {
dict with request {
if {[file isdirectory $fspath]
&& [string index $suffix end] ni {/ ""}} {
dict set response status 301
dict set response header location $path/$querytext
operation sendresponse $response
} else {
operation pass
}
}
}
# Rewrite directory requests to search for an indexfile.
proc wibble::indexfile {request} {
dict with request {
if {[file isdirectory $fspath]} {
if {[string index $path end] ne "/"} {
append path /
}
dict set request path $path$indexfile
operation prependrequest $request
} else {
operation pass
}
}
}
# Generate directory listings.
proc wibble::dirlist {request} {
dict with request {
if {![file isdirectory $fspath]} {
# Pass if the requested object is not a directory or doesn't exist.
operation pass
} elseif {[file readable $fspath]} {
# If the directory is readable, generate a listing.
dict set response status 200
dict set response header content-type "text/html; charset=utf-8"
dict set response content
foreach elem [concat [list ..]\
[lsort [glob -nocomplain -tails -directory $fspath *]]] {
dict append response content\
"$elem
"
}
dict append response content \n
operation sendresponse $response
} else {
# But if it isn't readable, generate a 403.
dict set response status 403
dict set response header content-type "text/plain; charset=utf-8"
dict set response content Forbidden\n
operation sendresponse $response
}
}
}
# Process templates.
proc wibble::template {request} {
dict with request {
if {[file readable $fspath.tmpl]} {
dict set response status 200
dict set response header content-type "text/plain; charset=utf-8"
dict set response content ""
set chan [open $fspath.tmpl]
applytemplate "dict append response content" [read $chan]
chan close $chan
operation sendresponse $response
} else {
operation pass
}
}
}
# Send static files.
proc wibble::static {request} {
dict with request {
if {![file isdirectory $fspath] && [file exists $fspath]} {
dict set response status 200
dict set response contentfile $fspath
operation sendresponse $response
} else {
operation pass
}
}
}
# Send a 404.
proc wibble::notfound {request} {
operation sendresponse [dict create status 404\
content "can't find [dict get $request uri]"\
header [dict create content-type "text/plain; charset=utf-8"]]
}
# Apply a template.
proc wibble::applytemplate {command template} {
set script ""
set pos 0
foreach pair [regexp -line -all -inline -indices {^%.*$} $template] {
lassign $pair from to
set str [string range $template $pos [expr {$from - 2}]]
if {$str ne ""} {
append script "$command \[" [list subst $str\n] \]\n
}
append script [string range $template [expr {$from + 1}] $to]\n
set pos [expr {$to + 2}]
}
set str [string range $template $pos end]
if {$str ne ""} {
append script "$command \[" [list subst $str] \]
}
uplevel 1 $script
}
# Get a line or a block of data from a channel.
proc wibble::get {chan {size line}} {
if {$size eq "line"} {
# Receive a line of text.
while {1} {
if {[chan pending input $chan] > 4096} {
chan puts stderr "line length greater than 4096"
chan close $chan
return -level [info level]
} elseif {[chan gets $chan line] >= 0} {
return $line
} elseif {[chan eof $chan]} {
chan close $chan
return -level [info level]
} else {
yield
}
}
} else {
# Receive a block of data.
while {1} {
set chunklet [chan read $chan $size]
set size [expr {$size - [string length $chunklet]}]
append chunk $chunklet
if {$size == 0} {
return $chunk
} elseif {[chan eof $chan]} {
chan close $chan
return -level [info level]
} else {
yield
}
}
}
}
# Version of [file join] that doesn't do ~user substitution and ignores leading
# slashes, for all elements except the first.
proc wibble::filejoin {args} {
for {set i 1} {$i < [llength $args]} {incr i} {
lset args $i ./[lindex $args $i]
}
string map {./ ""} [file join {*}$args]
}
# Decode hexadecimal URL encoding.
proc wibble::unhex {str} {
set pos 0
while {[regexp -indices -start $pos {%([[:xdigit:]]{2})} $str range code]} {
set char [binary format H2 [string range $str {*}$code]]
set str [string replace $str {*}$range $char]
set pos [expr {[lindex $range 0] + 1}]
}
return $str
}
# Zone handler return operation.
proc wibble::operation {opcode {operand ""}} {
return -level 2 -opcode $opcode $operand
}
# Register a zone handler.
proc wibble::handle {zone command args} {
dict lappend wibble::zones $zone [list $command $args]
}
# Get an HTTP request from a client.
proc wibble::getrequest {chan peerhost peerport} {
# The HTTP header uses CR/LF line breaks.
chan configure $chan -translation crlf
# Parse the first line.
regexp {^\s*(\S*)\s+(\S*)\s+(.*?)\s*$} [get $chan] _ method uri protocol
regexp {^([^?]*)(\?.*)?$} $uri _ path query
set path [unhex $path]
set path [regsub -all {(?:/|^)\.(?=/|$)} $path /]
while {[regexp -indices {(?:/[^/]*/+|^[^/]*/+|^)\.\.(?=/|$)}\
$path range]} {
set path [string replace $path {*}$rng ""]
}
set path [regsub -all {//+} /$path /]
# Start building the request structure.
set request [dict create socket $chan peerhost $peerhost peerport\
$peerport method $method uri $uri path $path protocol $protocol\
header {} query {} querytext $query]
# Parse the headers.
set headers {}
while {1} {
set header [get $chan]
if {$header eq ""} {
break
}
if {[regexp {^\s*([^:]*)\s*:\s*(.*)\s*$} $header _ key val]} {
dict set request header [string tolower $key] $val
}
}
# Parse the query string.
foreach elem [split [string range $query 1 end] &] {
regexp {^([^=]*)(?:=(.*))?$} $elem _ key val
dict set request query [unhex [string map {+ " "} $key]]\
[unhex [string map {+ " "} $val]]
}
# Get the request body, if there is one.
if {$method in {POST PUT}} {
if {[dict exists $request header transfer-encoding]
&& [dict get $request header transfer-encoding] eq "chunked"} {
# Receive chunked request body.
set data ""
while {1} {
set length [get $chan]
if {$length == 0} {
break
}
chan configure $chan -translation binary
append data [get $chan $length]
chan configure $chan -translation crlf
}
} else {
# Receive non-chunked request body.
chan configure $chan -translation binary
set length [dict get $request header content-length]
set data [get $chan $length]
chan configure $chan -translation crlf
}
dict set request content $data
}
return $request
}
# Get a response from the zone handlers.
proc wibble::getresponse {request} {
variable zones
set requests [list $request]
# Process all zones.
dict for {prefix handlers} $zones {
# Process all handlers in this zone.
foreach handler $handlers {
lassign $handler command options
# Try all requests against this handler.
for {set i 0} {$i < [llength $requests]} {incr i} {
set request [lindex $requests $i]
# Skip this request if it's not for the current zone.
set path [dict get $request path]
set length [string length $prefix]
if {[string index $prefix end] eq "/"} {
set matchprefix $prefix
set matchlength $length
} else {
set matchprefix $prefix/
set matchlength [expr {$length + 1}]
}
if {$path ne $prefix
&& ![string equal -length $matchlength $matchprefix $path]} {
continue
}
# Compile a few extra arguments to pass to the handler.
set extras [dict create\
prefix $prefix suffix [string range $path $length end]]
if {[dict exists $options root]} {
dict set extras fspath [filejoin\
[dict get $options root]\
[dict get $extras suffix]]
}
# Invoke the handler.
set arguments [dict merge $request $options $extras]
if {[catch {{*}$command $arguments} operand opcode]} {
return -options $opcode $operand
}
# Process the handler's result operation.
switch -- [dict get $opcode -opcode] {
prependrequest {
# Put a new higher-priority request in the list.
set operand [dict merge $request $operand]
set requests [linsert $requests $i $operand]
incr i
break
} replacerequest {
# Replace the request.
set operand [dict merge $request $operand]
set requests [lreplace $requests $i $i $operand]
break
} deleterequest {
# Delete the request.
set requests [lreplace $requests $i $i]
break
} sendresponse {
# A response has been obtained. Return it.
return $operand
} pass {
# Fall through to try next request.
} default {
error "invalid opcode \"[dict get $opcode -opcode]\""
}}
}
}
}
# Return 501 as default response.
return [dict create status 501\
content "not implemented: [dict get $request uri]"\
header [dict create content-type "text/plain; charset=utf-8"]]
}
# Main connection processing loop.
proc wibble::process {socket peerhost peerport} {
try {
chan configure $socket -blocking 0
while {1} {
# Get request from client, then formulate a response to the reqeust.
set request [getrequest $socket $peerhost $peerport]
set response [getresponse $request]
# Get the content size.
if {[dict exists $response contentfile]} {
set size [file size [dict get $response contentfile]]
if {[dict get $request method] ne "HEAD"} {
# Open the channel now, to catch errors early.
set file [open [dict get $response contentfile]]
chan configure $file -translation binary
}
} elseif {[dict exists $response content]} {
dict set response content [encoding convertto iso8859-1\
[dict get $response content]]
set size [string length [dict get $response content]]
} else {
set size 0
}
# Parse the Range request header if present and valid.
set begin 0
set end [expr {$size - 1}]
if {[dict exists $request header range]
&& [regexp {^bytes=(\d*)-(\d*)$} [dict get $request header range]\
_ begin end]
&& [dict get $response status] == 200} {
dict set response status 206
if {$begin eq "" || $begin >= $size} {
set begin 0
}
if {$end eq "" || $end >= $size || $end < $begin} {
set end [expr {$size - 1}]
}
}
# Add content-length and content-range response headers.
set length [expr {$end - $begin + 1}]
dict set response header content-length $length
if {[dict get $response status] == 206} {
dict set response header content-range "bytes $begin-$end/$size"
}
# Send the response header to the client.
chan puts $socket "HTTP/1.1 [dict get $response status]"
dict for {key val} [dict get $response header] {
set normalizedkey [lsearch -exact -sorted -inline -nocase {
Accept-Ranges Age Allow Cache-Control Connection
Content-Disposition Content-Encoding Content-Language
Content-Length Content-Location Content-MD5 Content-Range
Content-Type Date ETag Expires Last-Modified Location Pragma
Proxy-Authenticate Retry-After Server Set-Cookie Trailer
Transfer-Encoding Upgrade Vary Via Warning WWW-Authenticate
} $key]
if {$normalizedkey ne ""} {
chan puts $socket "$normalizedkey: $val"
} else {
chan puts $socket "$key: $val"
}
}
chan puts $socket ""
# If requested, send the response content to the client.
if {[dict get $request method] ne "HEAD"} {
chan configure $socket -translation binary
if {[dict exists $response contentfile]} {
# Send response content from a file.
chan seek $file $begin
chan copy $file $socket -size $length
chan close $file
} elseif {[dict exists $response content]} {
# Send buffered response content.
chan puts -nonewline $socket [string range\
[dict get $response content] $begin $end]
}
}
# Flush the outgoing buffer.
chan flush $socket
}
} on error {"" options} {
# Log errors.
set message "[clock format [clock seconds]]: INTERNAL SERVER ERROR\n"
dict for {key val} $options {
append message "$key = $val\n"
}
chan puts -nonewline stderr $message
try {
set message [encoding convertto iso8859-1 $message]
chan configure $socket -translation crlf
chan puts $socket "HTTP/1.1 500 Internal Server Error"
chan puts $socket "Content-Type: text/plain; charset=utf-8"
chan puts $socket "Content-Length: [string length $message]"
chan puts $socket "Connection: close"
chan puts $socket ""
chan configure $socket -translation binary
chan puts -nonewline $socket $message
} finally {
chan close $socket
}
}
}
# Accept an incoming connection.
proc wibble::accept {socket peerhost peerport} {
chan event $socket readable [namespace code $socket]
coroutine $socket process $socket $peerhost $peerport
}
# Listen for incoming connections.
proc wibble::listen {port} {
socket -server [namespace code accept] $port
}
# Demonstrate Wibble if being run directly.
if {$argv0 eq [info script]} {
# Guess the root directory.
set root [file normalize [file dirname [info script]]]
# Define zone handlers.
wibble::handle /vars vars
wibble::handle / dirslash root $root
wibble::handle / indexfile root $root indexfile index.html
wibble::handle / static root $root
wibble::handle / template root $root
wibble::handle / dirlist root $root
wibble::handle / notfound
# Start a server.
wibble::listen 8080
# Enter the event loop.
vwait forever
}
# vim: set sts=4 sw=4 tw=80 et ft=tcl:
======
----
**Discussion**
[CMcC] likes this, it's neat and minimal, but flexible and fully functional. A couple of observations as they arise: all the header keys have to be set to a standard case after you parse them out of the request stream, as the spec doesn't require a client to use a standard case.
[AMG]: HTTP allows case insensitivity? Damn. Case insensitivity will be the death of us all! HTTP specifications (or at least clients) probably require a specific case for the server, which unfortunately is neither all-lowercase nor all-uppercase. What a pain!
[CMcC] AFAIK you can return whatever you like (case-wise) from the server, so no ... no requirement. It's all case-insensitive for the field names.
[AMG]: Still, I must be able to deal with clients which assume HTTP is case sensitive and that it requires the case style shown in the examples. Most folks only read the examples, so they draw this conclusion: [http://cr.yp.to/proto/design.html]. Just look at Wibble itself! It fails when the client uses unexpected letter case in the request headers! I didn't spot where the specification allowed case independence, and none of the examples suggested this to me.
[AMG]: Update: I now force all request headers to lowercase and normalize all known response headers to the "standard" case observed at [http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html]. Unrecognized response headers are left untouched.
[CMcC]: Also, I'm not sure what you do with multiple query args which have the same name, you have to be prepared for that, and do you handle query args without value? Unsure.
[AMG]: An earlier revision supported multiple query arguments with the same name, plus query arguments without values. Then I decided those two features weren't really important to me, that it was simpler to just require that sites made using Wibble wouldn't depend on those features. But if you can give me a compelling application for multiple like-named and argument-less queries, I'll re-add support. For now, later query arguments replace like-named earlier query arguments, and query arguments without value are taken as having empty string as the value. My earlier solution was for queries with arguments to be in a key-value list, then for all query argument names (even those without values) to be in a second list, sorted by the order in which they appeared in the URI.
[CMcC] yeah, it's much easier if you can ignore that requirement. Wibble's free to do that, Wub's not (sadly.)
[AMG]: How does Wub make use of argument-less query arguments and repeated query argument names?
[CMcC]: Adding virtual host support is trivial as you've noted, you just need to combine the host with your zone regexp.
I note you don't fall-back to HTTP/1.0 (not really necessary, I guess,)
[AMG]: I have the beginnings of support for falling back to HTTP/1.0, in that I remember the protocol advertised by the client. In the future I can use that information.
[CMcC] I really wouldn't bother - there's no real need to support HTTP/1.0 IMHO - the only existing client still using it is the Tcl client (and that should be fixed soon.) Again, Wub doesn't have the option of taking the sensible path.
[AMG]: I'll have to check a couple streaming music players to see if they all grok HTTP/1.1. They would have to if they support seeking.
[CMcC]: nor do you stop processing input on POST/PUT as the spec requires (you ought to make sure this is done, as some things require it.) Your pipeline processing requires run-to-completion of each request processing, I think, but there are definitely cases where you would not want this (they're not common, but when you get such a requirement there's no way around it, I think) so that's a limitation, although not a show-stopper.
[AMG]: I don't have any experience with POST/PUT. I just put in the few rudiments I could figure out from the HTTP specification. I'll have to read up on POST/PUT in more detail.
[CMcC] the spec says you can't process any requests (and the client oughtn't to send any requests) on a pipeline until the POST/PUT is handled completely. It's subtle, but it's (just) conceivable that something could be screwed up by it. Since your approach is to do things which make sense for most apps, you could probably get away with it by just documenting the behaviour.
[AMG]: Wibble processes each request immediately after reading it. Its main loop is: get request, compute response, send response, repeat. Computing a response for a POST/PUT necessarily involves committing its side effects to whatever database is used. So subsequent responses remain unread, waiting in the receive buffer, until Wibble is completely finished with the POST/PUT, or any other type of request.
[CMcC]: I like the way zone handlers stack, but the necessity of returning a list is less good, IMHO - I prefer the default case to be easy. I'd consider using a [[return -code]] to modify the return behaviour, or perhaps using a pseudo response key element to specify further processing.
[AMG]: I haven't done much with [[return -code]], so it hadn't occurred to me. That's an interesting idea, thanks. I think I'll change it to return the operation as the return code and the operand as the return value.
[CMcC] yah, you might want to map the normal error codes from Tcl (including Tcl_OK) to reasonable values (e.g. Tcl_OK=>200, Tcl_Error=>500)
[AMG]: I wound up using [[return -opcode]] (wrapped by the [[operation]] sugar proc), which puts a custom "-opcode" key in the -options dict, then I receive this opcode using [catch]. The purpose of [[return -code]] is already defined, and it requires integers or a predefined enumeration, so I decided not to mess with it. Also the purpose of this general mechanism is not to report errors or status, but rather to tell the response generator what operation to take next: modify request, send response, etc. I do map error to HTTP 500 using '''[try] {...} on error {...}''', then I print the error options dictionary to both stderr and (if possible) to the client socket. On error, I always close the client socket, forcing tenacious clients to reconnect, which is almost like rebooting a computer following a crash.
[CMcC]: I think the idea of starting a new dictionary for response is pretty good (as it means you don't have to filter out the old stuff,) but I'm not sure that doing so retains enough data for fully processing the original request. Do you pass the dict to the zone handlers converted to a list? That's not so good, as it causes the dict to shimmer.
[AMG]: Both the request and response dictionaries are available everywhere in the code. They're just in separate variables. Yeah, I convert the dict to a list by means of [{*}] into [args]. If that causes shimmering, I'll change the zone handlers to accept a single normal argument. By the way, extra non-dict arguments can be passed to the zone handler by making the command name a list. This makes it possible to use [namespace ensemble] commands, etc. as zone handlers.
[AMG]: Update: I have made this change. The shimmering is eliminated.
[CMcC]: I'm not sure that the idea of jamming new requests into the pipeline is a good one.
[AMG]: It was the best solution I could think of for handling index.html in the face of the template generation. With the example setup, if there is a file called index.html in the directory being requested, '''static''' will serve it straight from disk. If not, '''template''' will try to make one from index.html.tmpl. And--- very important!--- if that doesn't work, '''dirlist''' will generate a listing. If '''indexfile''' simply replaced requests for a directory with requests for index.html, '''dirlist''' could never trigger. And if '''indexfile''' only did this replacement if index.html actually existed on disk, '''template''' would not be used to generate the index.html. I couldn't think of any other way to get all these different handlers to work together.
[CMcC] this is one of the subtle and intriguing differences between Wub and Wibble architectures - firstly you don't transform the request dict, you create a new one, and as a consequence you have to keep the original request around, and as a consequence of that you have to be able to rewrite the current request (if I understand it correctly.) Those are all (slightly) negative consequences of that architectural decision. The upside is that you don't have to keep track of protocol and meta-protocol elements of the response dict as tightly as Wub does - Wub has to cleanse the fields which make no sense in response, and that's time-consuming and unsightly - Wibble doesn't, and can also easily treat those elements using [[[dict] with]] which is a positive consequence of the decision.
[AMG]: Keeping the original request is easy and natural for me; all I had to do was use two variables: `set response [[getresponse $request]]`. To be honest, I didn't notice that Wub operated by transmuting the request dictionary into the response dictionary, so I didn't try to emulate that specific behavior. Instead it made sense to generate a new response from the request: from the perspective of a packet sniffer, that is what all web servers do. Also I enjoy the ability to not only rewrite requests, but also to create and delete alternative requests which are processed in a specific priority order. Branch both ways! The goal is to get a response, and handlers are able to suggest new requests which might succeed in eliciting a response. Or maybe they won't, but the original request will. Rewriters can leap without looking: they don't have to predict if the rewritten request will succeed. And in the '''indexfile'''/'''template'''/'''static'''/'''dirlist''' case, '''indexfile''' doesn't have the power to make this prediction.
[CMcC]: You should probably add gzip as an output mode, if requested by the client, as it speeds things up.
[AMG]: I figured [gzip] can wait until later. It's more important for me to bang out a few sites using this. Also I need to look into multithreading so that clients don't have to wait on each other.
[CMcC] [gzip]'s easy to add, and well worth adding. I should probably get around to adding Range to Wub, too.
[AMG]: Okay, I'll look into [gzip] and [zlib deflate]. Wibble never sends chunked data, so it should be as easy as you say. I'll just look at the response headers to see if I need to compress before sending.
Wibble doesn't support multipart ranges. I doubt any web servers do; it's complicated and it's worthless. Clients are better off making multiple pipelined individual range requests.
[AMG]: Update: I'm not sure how encodings and ranges are supposed to interact. Content-Length gives the number of bytes being transmitted; that much is clear. What about Content-Range? Do its byte counts reflect the encoded or unencoded data? And the request Range--- surely its byte counts are for the unencoded data.
I think I'll ignore the qvalues; they're tough to parse and kind of dumb. Why would a client advertise that it accepts gzip but prefers uncompressed? Or why would it give something a qvalue of 0 rather than simply not listing it?
[CMcC]: All in all, good fun. I still wish you'd applied this to Wub, and improved it, rather than forking it, but oh well. I wish you'd been around when I was developing Wub, as some of your ideas would have (and could still) contribute to Wub's improvement. I definitely like the simplicity of your processing loop, although I think that Wub Nub's method of generating a switch is faster and probably better (having said that, it's harder to get it to handle stacking.)
[AMG]: Yeah, I wish the same. I really haven't spent much time on Wibble. I wrote it in one afternoon, then a week later, spent a couple hours tidying it up for publication. I don't have a huge amount of time to spend on this sort of thing, so when I hack, I hack ''furiously!'' And I really couldn't wait for you and [JDC] to answer my [Wub] questions. Sorry! :^) I invite you to absorb as much as you like into Wub. If you give me direction and I have time, I'll gladly help.
Now that this code is written, I think it should stay up as a separate project. I think it fills a slightly different niche than Wub. Both are web servers, of course. But Wub is a large and complete server made up of many files, whereas Wibble is a smallish server hosted entirely on the Wiki. That makes it a lot more accessible and useful for educational and inspirational purposes, kind of like [DustMote]. Maybe think of it as Wub-lite, a gateway or gentle introduction to some of the concepts that undergird Wub.
Thank you for your comments.
[CMcC] You're welcome - as I say, this is interesting. It's interesting to see where you've taken the request- response- as dict paradigm, it's also interesting to see how you've used [coroutine]s - very clean indeed. Wub has two coros per open connection, and a host of problems with keeping them synchronised. The idea was to keep protocol-syntax and semantics distinct, and therefore to make the server more responsive. I'm scratching my head, wondering whether to move to single-coro per pipeline, as Wibble does, but have to think through the implications.
It's good to see Wibble, because you started with [dict] and [coroutine] as given, and evolved it with them in mind, where Wub evolved in mid-stream to take advantage of them, Wibble seems to make better considered use of the facilities as a consequence.
I would definitely advise keeping Wibble as a distinct project - it addresses the problem of a minimal server (like Dustmote et al,) but still tries to provide useful functionality (unlike Dustmote et al.)
I'd be interested to see what you make of Wub/TclHttpd's Direct domain functionality.
[AMG]: I started with [Coronet], which you and [MS] wrote. I dropped everything I didn't need, merged [[get]] and [[read]], absorbed [[terminate]] and $maxline into [[get]], eliminated the initial [yield]s in [[get]], renamed [[do]] to [[process]] and [[init]] to [[accept]], changed [[accept]] to use the [socket] name as the [coroutine] name, and defined the readability handler before creating the coroutine. I did that last step because the coroutine might [close] the socket and [return] without yielding. Yeah, for an instant I have the readability handler set to a command that doesn't yet exist (or, in case of return without yield, will never exist), but this is safe since I don't call [update].
I will try to look into Direct later today. I see Wub has directns and directoo; I am guessing that these map URIs to command invocations, possibly in the manner of [Zope].
Hey, for a fun read, check out [http://www.and.org/texts/server-http]. Does Wub support individual headers split across multiple lines?
----
!!!!!!
%| [Category Webserver] |%
!!!!!!