AMG: This page is for Wibble news and announcements. Discussion is welcome, but if it gets lengthy it should be moved to the Wibble discussion page. Announcements prior to the 2010-11-06 are embedded in Wibble discussion and Wibble discussion archive. Detailed change reports are at Wibble change log.
Change log: [L1 ]
AMG: I fixed two problems which I never really noticed before. One, ' (apostrophe) needs to be quoted when used inside HTML attributes, since the attribute itself might be quoted with apostrophes, a.k.a. single quotes. I hadn't noticed, since I always use double quotes. Some other folks prefer apostrophes, so it's good to support both. Two, the Tk non-console command entry widget should be automatically focused. Again I hadn't noticed, since the way to get this interface is to run Wibble on Unix using wish rather than the default tclsh.
So, what do I still need to work on? Here's a brief summary of the Wibble wish list and Wibble bugs page.
As mentioned on some of the linked pages, I'm considering modifying the zone handler concept to support character encoding, virtual hosts, and gzipped payloads. This may break compatibility with existing Wibble installations, but since I haven't made a 1.0 release yet, I don't feel a strong obligation to support a particular API. If this is a problem for you, let me know and I'll see if I can accommodate.
Speaking of 1.0, once I make this release I won't be able to break incompatibility ever again, unless I move to a revision control system that supports branch development. Sure, I could do that by having separate Wibble implementation pages for each branch, but that would be a major pain, even more so than the current system.
I don't know if the "coroutine already running" problem still exists, so the first order of business is to try to reproduce it. Same goes for the Apache Benchmark problem.
I need to contribute a fix to SQLite since the [$db incrblob] feature would be very useful for Wibble but currently doesn't work in the asynchronous configuration required by Wibble. AK, DRH, and I discussed this issue at the Eighteenth Annual Tcl/Tk Conference (2011), but so far no action has been taken.
Change log: [L22 ]
AMG: This release has some performance improvements suggested by SEH [L23 ], plus it has a few new features thanks to MaxJarek. The new features are support for custom [socket] commands, used for SSL and possibly others [L24 ], and support for custom POST content-types, such as application/json-rpc [L25 ]. There's also a new [dict getnull] command which returns empty string instead of error when asked to retrieve a key that doesn't exist.
Change log: [L26 ]
AMG: Thanks to steveb and DKF, I was able to fix [icc catch]! See [L27 ] for the discussion, including a contrived example of [icc catch] and some other [icc] commands.
Change log: [L28 ]
AMG: While working on WebSocket, I identified a deficiency in [icc]: there's no way to interrupt an [icc get] other than sending one of the events expected by that [icc get] invocation. I tried a couple design approaches and finally settled on defining a new event type called "exception", which [icc get] handles specially. Instead of directly returning this event, [icc get] will return it with an abnormal code (#7) which is caught elsewhere by [icc catch]. The idea is that you put [icc catch] somewhere near the top of your execution stack, and it'll be there to collect exceptional events, presumably for the purpose of cleanly bringing down the coroutine.
Unfortunately, [icc catch] doesn't work. I left the code for it in Wibble anyway, since the workaround is to copy it and incorporate it into your code. See [L29 ] for more information. (If that link doesn't work for you, see also [L30 ].)
I also added the [icc destroy] procedure to cleanly destroy an ICC feed. One interesting thing it has to do is check if there are any coroutines that would sleep forever as a result of the feed going away. If it finds any such coroutines, it wakes them up immediately. Their [icc get] invocations will return an empty list, signifying that nothing happened, not even a timeout.
I modified the demo code to use the docroot directory if it exists. I did this to make the demo more directly useful for simple websites, such that you can just put your stuff in docroot and run the stock wibble.tcl, and the job's done.
Change log: [L31 ]
AMG: I really shouldn't have held on to this release for as long as I did. I kept it bottled up for several reasons:
Well, it's Thanksgiving, I have a few days off work, this new code's been begging for a release, you all deserve it, I received massive encouragement at the Eighteenth Annual Tcl/Tk Conference (2011), and so I took the plunge. I hope this code is useful to everyone.
First, I must apologize to jnc: now you will have to merge this new code with your git repository [L32 ]. Had I released more frequently, you'd have an easier time. I see you and dzach went a different way than I did on a few design points. I'll list 'em out here as a way to show what's new. Please post discussion on the Wibble discussion page.
Hmm, I guess I should directly discuss what's changed! Go see the change log [L33 ] for a dry list, or read on for exposition. Since there's so much to cover, and since it's been such a long time since the last update, this announcement will go into much greater detail than the previous ones.
This new version is definitely backward incompatible, but in ways that I think make sense. Wibble's still a growing boy, it's gonna outgrow its clothes a few more times before maturity. Maybe I'm just flattering myself, but it seems to me that the ways in which you'll have to update your code are ways in which you wish you could have written your code in the first place. For example, the zone handlers are better organized now; they're grouped into a dedicated zone namespace. If that's a problem, you can just explicitly give the namespace name when you call [handle]. Also, some things are given better names. Useless commands have been removed, remaining commands have been made more flexible, new commands have been added. The [template] command is made available for direct use in zone handlers, including script files. [cleanup] is now practical. But the really, really big change is in the way response headers are formatted. Now their structure mirrors that of the request handlers. The most obvious benefit of this change is in setting cookies. Here's an example cookie.script file you can put in your docroot:
dict set state response header set-cookie cookiename "" "cookie value"
dict set state response header set-cookie cookiename domain .whatever
dict set state response header set-cookie cookiename path /whatever
dict set state response header set-cookie cookiename port {80 8080}
dict set state response header set-cookie cookiename discard ""
dict set state response header set-cookie cookiename httponly ""
dict set state response header set-cookie cookiename secure ""
dict set state response header set-cookie cookiename expires {reltime 60}
dict set state response header set-cookie cookie2name "" "cookie2 value"
dict set state response header set-cookie cookie3name "" "cookie3 value"
vars $stateThe first thing you'll notice is that multiple cookies are being set. Also, not only are the cookies being set, their extra attributes are being set too. See, the expiration time here is specified as relative, but there's also an abstime option; see [entime] for details on how it works. Also, you're sure to notice the length of these commands. If that's a problem, you can build up the dictionaries by parts; for instance, make a set-cookie dictionary, then put it in the state/response/header dictionary.
Now, this next part doesn't have anything to do with cookies, but the last line of the above script is a call to [vars]. This is an example of one zone handler directly chaining to another, bypassing the normal zone handler search mechanism. I must emphasize that this is a perfectly valid thing to do. I touched on this concept in the previous release announcement, but it bears repeating. Here, the [vars] zone handler command will pretty-print the request dictionary, but it'll also pass the cookies along so they're set in the browser. Fine point: [scriptfile] never gets a chance to call [sendresponse], since [vars] does it first, which immediately terminates the response generation process. Another thing. Since the response dictionary is now formatted the same as the request dictionary, [vars] now does a much better job of displaying it. Or if there's an error, the new [panic] procedure will do basically the same thing as [vars], albeit in plain text.
Another big, new feature is the custom sendcommand. Most of the time, [defaultsend] is used to send the response to the client, then return to the processing loop. With HTTP upgrades (WebSocket, Server-Sent Events), returning to normal HTTP processing is undesirable. Instead, the upgrade zone handler supplies a custom sendcommand in the response dictionary. The custom sendcommand can take over all remaining processing for the socket, speaking whatever protocol it desires. Despite the name, the sendcommand is free to read from the socket, implement its own zone handler-like mechanism, whatever it wants. When it's done, it probably ought to return false so that [process] closes the socket. By the way, the sendcommand doesn't have to be a distinct proc; it can be an apply lambda. By the way #2, [defaultsend] returning false is how I chose to implement Connection: Close.
There's a new zone handler to imbue the response headers with a content-type, and the example code at the end of wibble.tcl gives a sample configuration for many common file formats. To make this work right, I had to fix a longstanding bug where the zone handlers wouldn't inherit any headers set by preceding zone handlers. This new [contenttype] zone handler is free to make its best guess about the type, based on the filename extension present in the URI, but it doesn't send a response, only put its guess in the state dictionary inherited by later zone handlers. Follow-up handlers get this header for free, though they're also able to override it. For instance, if the client requests "404.jpeg", [contenttype] adds "content-type: image/jpeg" to the response headers, but [notfound] later overrides that header with "content-type: text/plain". This is a prime example of the stacking zone handler concept.
I want to mention something else about inheriting the state dictionary. Your zone handlers can put whatever they want in that state dictionary to be received by later zone handlers. The state dictionary starts off with options, request, and response sub-dictionaries. Not only are zone handlers free to read and update these dictionaries; they can also create new application-specific dictionaries.
The example code is much enhanced to now include a Tk console. Run wibble.tcl, it'll pop up a window, you edit the code, hit Ctrl-R, and it'll reload in the middle of execution. Inspect whatever you want inside the console. Well, on second thought, you might have trouble looking at the stack variables of suspended coroutines...
A major weakness remaining in Wibble is its charset handling, or lack thereof. See Wibble discussion for more information, and post your suggestions there. I know this makes Wibble unusable for some of you guys, but you can work around it by patching your local copy. As for myself, I'm very hesitant to put in a change which I consider to be a "band-aid"; I'm looking for a proper solution, the way I did with header encoding and sendcommand and my other hopefully brilliant ideas.
Change log: [L34 ]
AMG: Lots of new features!
The biggest change is that I replaced the [suspend] and [resume] commands with a much more sophisticated inter-coroutine communication system, accessed using [icc get], [icc put], and [icc configure]. (I developed the [icc] commands for a chat application I hope to publish soon.) Here's the basic outline of ICC:
Coroutines send events to other coroutines via feeds. [icc put] sends events, [icc get] receives events, and [icc configure] configures a feed. Feeds filter which events they will accept, and [icc get] can apply additional filters. Filtering is done by applying [string match] to event names. Events can carry arbitrary data in addition to their names. [icc get] can monitor multiple feeds simultaneously, and [icc put] can send to multiple feeds. A feed can be configured to "lapse" if no [icc configure] or [icc get] is done on it within a timeout; upon lapse, the feed is destroyed and a custom script is executed, e.g. to announce that the feed has lapsed. A feed is automatically created for each client connection coroutine, and additional feeds can be created using [icc configure]. [icc get] accepts an optional timeout argument, and it can also monitor the current coroutine's input socket for readability.
Another important change is the addition of text/xml POSTs, which are needed by WebServices. Whenever data is POSTed using the text/xml enctype, it decodes the full POST into a single "xml" element of the post dictionary. (I'm still waiting for word on whether I should use [deurl] or [dehex] to do the decoding. The difference is that [deurl] treats + as space.)
Also there's now a way to register cleanup scripts that will be run whenever the current connection coroutine terminates, even if it's terminating due to an error.
Zone handlers can now send the client data from an open channel, in addition to on-disk files or in-memory strings. One day this may be useful for sending from an SQLite [incrblob]; however, this functionality currently doesn't work due to what I believe is an oversight in the implementation of [incrblob].
By request, I changed "wibble" to "::wibble".
This new version contains an example of one zone handler chaining to another. I think it was JBR who was asking about this. The [dirslash] zone handler chains to the new [redirect] zone handler. Because of the way I implemented [sendresponse] (also [nexthandler]), [redirect] does not return to [dirslash]. Ditto [dirlist] and [forbidden].
Change log: [L35 ]
AMG: I removed the query and post dicts when no query or post was made. Also, I fixed a bug that only strikes when the zone handlers fail to produce a response.
Change log: [L36 ]
AMG: JBR pointed out that most uses of [nexthandler] are really no-ops and can be removed. I have now removed them from Wibble and the examples on this wiki. When a zone handler returns, that's the same as if it did [nexthandler $state] without modifying $state. I updated Wibble detailed description to say this.
Change log: [L37 ]
AMG: Along with several bug fixes, I tightened up the way zone handlers work. Now they take a single, unified state dict instead of two request and response dicts. This is cool for (at least) three reasons: One, it rarely made sense to receive a response dict, so why require it? Two, this way custom options don't have to be inserted into and removed from the request dict. Three, zone handlers can create custom dicts inside the argument(s) to [nexthandler], which will be picked up by later handlers.
Of course, custom zone handlers need to be updated in order to work with the new version of Wibble. I've updated all the examples on this wiki.
I also changed the way coroutines are suspended and resumed. Now there's a [suspend] command that takes arguments naming the events that it should wait for. The coroutine will stay sleeping until one of those events happen. The original version of Wibble didn't need this, since there was only one thing that could wake up the coroutine (chan readable). Later I added asynchronous [chan copy], which constituted another event. In discussions with JCW, I found that AJAX can have other events as well. To support AJAX, I unified the interface to suspending and resuming. The [resume] command constructs a command prefix that will resume the coroutine.
I put up a basic AJAX example here: [L38 ].
I updated Wibble detailed description somewhat, incorporating some stuff from Wibble discussion. Also I moved the older stuff from Wibble discussion to a new page, Wibble discussion archive.
Change log: [L39 ]
AMG: I can't believe I didn't notice this problem before. The [chan copy] invocation blocks until it's complete. I changed it to use the -command option, instructing it to resume the connection's coroutine when it's done. Then I [yield]. The [llength] of [yield]'s return value shows why the coroutine was resumed:
Also I put in workarounds for the two bugs currently posted on the Wibble bugs page. I'll back these out again when they're no longer needed.
Change log: [L40 ]
AMG: I guess I'm making up for lost time; it's been well over a year since my last big update. This update isn't nearly as big as yesterday's, but it's still progress.
I added another zone handler called script that's meant as a companion to template. Previously they were combined into a single template zone handler.
I made the vars zone handler much nicer, and error reporting is nicer too. They now share a common library routine to format the request data.
I added a few more keys to the request dictionary (time, rawtime, port).
PUT is gone.
Change log: [L41 ]
AMG: Tired of having one giant page for everything Wibble, I reorganized into several pages.
This level of content shuffling would make RA2 happy, but I figured I'd avoid one of his major mistakes and let the Wiki do the indexing work for me. All Wibble-related pages link to Category Wibble, whose backrefs are conveniently repeated on the main Wibble page. But I can't have everything. Page history suffers, since it obviously doesn't carry over to the new pages. For history prior to today, see [L42 ].
I took this opportunity to publish the latest version of Wibble, found at Wibble implementation. You may notice that I turned off syntax highlighting for that page. Trust me, the syntax highlighting made the code far less readable (scroll through [L43 ] to see what I mean). We need to fix that. :^(
The new version digs deeper into the headers, breaking them into lists and dicts. Among other things, this made POST file uploads possible. That's another new feature I've added. I don't know if I'll ever support PUT; does anyone even use PUT? If no one speaks up, I'll drop it entirely.
I think I have more writing to do: updated documentation, new examples, etc.