Version 123 of Wub

Updated 2009-04-13 14:29:45 by CMcC

Introduction

Wub is a web-server written in pure-Tcl. It requires Tcl 8.6. It should help in creating highly-dynamic (and portable) web applications. It is the successor of Tclhttpd, which is (2008-07-05) not maintained anymore.

Wub's documentation is here [L1 ]. The author CMcC is currently in a documenting mood, and would appreciate feedback on where the documentation lacks clarity, information, specificity, accuracy, etc etc ... all those banes of documenters.

Wub is essentially an interpreter for HTTP 1.1 requests, and so it's very appropriate to use Tcl for that kind of application.

Here's Wub's Read-Eval-Print loop ... couldn't be simpler.

http://wiki.tcl.tk/_repo/images/arch.png

Semantics for the interpretation are provided by Domains, which are (beginning to be) documented here [L2 ]

Digression

RLH Why does wub require 8.6 when it isn't even out yet?
CMcC because it depends on facilities which are only in 8.6, of course.
RLH That is obvious. However, I would think it would be a very few that are going to run 8.6 at this point and so...can't use Wub. Of course, it is yours to do with what you will I just think that a shame.
CMcC well, it's a tradeoff I guess. I can develop faster and better with 8.6 facilities (coros made a big difference in performance and code maintainability, TclOO made code structure a bit better,) and so I'm gambling that by the time I've run out of things to do, people will be using 8.6.
JE Also worth noting: by using new features like dictionaries, OO, and coroutines before they're even out of alpha, Wub has helped to make Tcl better. Feedback from early adopters on the bleeding edge is very valuable to the folks trying to invent the bleeding edge.

News

13Jan09:

Version 4.0 of the Httpd server engine released. Now it's purely and only a coroutine-driven engine.

Application Changes: Responder::post and Responder::do become ::Httpd::post and ::Httpd::do.

Quick Start 

   svn checkout http://wub.googlecode.com/svn/trunk/ wub-read-only
   cd wub/Wub/
   tclsh Application.tcl

(there may, of course, be transient svn server outages.)

See http://code.google.com/p/wub/source/checkout for details.

Default comes up at port 8080. There's a control port at 8082 - tcl commands into running app from localhost only.

External Dependencies

Optional: tls for SSL

Configuration

Wub/Application.tcl is a demo application showing all available Wub Domains. It may be configured by reference to Site.tcl in the same directory. Wub/Minimal.tcl is an extremely minimal demo application which provides only a file system based domain.

Wub Architecture

Wub is a framework for writing web servers.

It provides a collection of utilities enabling a programmer to assemble a range of useful dedicated web servers. While the programmer is free to assemble the components in any way they wish, some approaches are better supported than others, and this could be called Wub's architecture.

Wub's Httpd module converts HTTP protocol interactions into tcl dicts containing all the request's protocol fields. These 'request dicts' are transformed into 'response dicts' by the user-supplied application using Wub Utilities and Wub Domains (Utilities/ and Domain/ respectively) and are returned via HTTP to the client.

A request/response dict resembles a whiteboard upon which Domains and Utilities operate, reading fields left by earlier processing, creating and modifying fields as a result of their own operation. Most modules act as functions, taking a request dict argument and returning a response dict result.

One convention which is rigorously enforced is that non-protocol fields in request/response dicts have names of the form '-*', distinguishing them from HTTP fields which will eventually be sent to the client.

Dataflow

Httpd package receives protocol transactions and generates a request dict. After performing some analytic transformations, such as Url and Query processing, User Agent processing and malicious client Blocking, Httpd (optionally) consults one of two Cache packages. If the Cache can satisfy the request, it does so. Otherwise, a configurable Incoming proc is called with the request. A sample Incoming is provided in the supplied example Application.tcl, which acts through the Response module to dispatch incoming requests to an appropriate Domain by means of a switch over some fields in the request dict - specifically and primarily the -url field which contains the incoming request URL. After Domain package handling, the returned response dict is queued up to be returned to the client via Httpd.

In addition to the Cache module, there are two other filtering modules: Convert (which handles mime type content negotiation and conversion) and Session (which manages sessions) which can apply to the output from any other Domain.

Domain packages provide application-level semantics to a web server. A range of example domains are provided.

Wub documentation is proceeding slowly. http:/_wub/ contains what will eventually be a discursive description with examples. http:/_doc/ contains a class/namespace level live comment-scrape from the source actually running what you're reading now.

Related

Wub Suspend - how to suspend under Wub Wub Consumer - how to interact with HTTP clients via Wub - the consumer API

Archeology

Wub's complete rewrite is complete, and is running this wikit. (25Apr07)

Wub's available here: http://code.google.com/p/wub/ under subversion control.

Try grabbing the wikit code from http://code.google.com/p/wikitcl/ into a sibling directory, and running tclsh Site.tcl from that (wikit) directory.

More later.

CMcC

Wub was underperforming relative to Apache et al, so I've rewritten it all to be multithreaded and lighter-weight.

The pursuit of a Lego-like plugging little language has been abandoned, under the following reasoning: (a) the audience for lego-plugability is probably not an audience that should be building websites anyway, (b) building nice things out of Lego is hard, (c) the overhead of maintaining a plug-together API was considerable although I did learn a lot about the new return functionality, (d) tcl is amply competent at parsing and dispatching URLs - which is all the plugin/domain model really had to offer, (e) multithreading made site writing easier (giving a thread-level execution environment), (f) multithreading made the simple domain plugging model non-viable (caching, for example, has to be in the top level dispatcher thread.)

Preliminary benchmarks suggest a rough parity with Apache out-of-the-box.

New caching architecture will make it easier to interact with a decent caching reverse-proxy if one's ever written.

If you consider tclhttpd, take a look at Wub too. Wub will probably see more active development than tclhttpd in the future (as the wub author is/was one of the main contributors to the tclhttpd code...).

Wub is a complete redesign of a http (nearly-1.1) webserver using dicts for state instead of all the global vars used in tclhttpd. Wub owes a kind of filial respect to tclhttp, as a few core techniques are derived from it, but I hope Wub will be more convivial for developers.

The Wub architecture is designed to be highly modular. All protocol operations occur in two modules Listener and Httpd. Once an http request has been parsed to a dictionary (and after possible further processing by Cookie, Entity and Query modules) the request dict is transformed by a series of Domains.

I would like Wub to resemble a lego set of modules out of which one can assemble a pachinko or a pinball machine for handling http requests (the requests and responses are like the balls ... that's the metaphor).

The following Domains (and more) are available:

Session
handles cookie-based sessions
Cache
provides caching of content
File
reflects a filesystem to the internet
Convert
converts content to an arbitrary mime-type using a graph of individual conversions, so xyz->html->pdf type conversions are possible.
Mason
emulates a quite good PERL CMS-like system called Mason (http://www.masonhq.com/ ).

Db and Sql: reflect a metakit and sqlite database (respectively) to the internet. Http query language is used to construct SQL-ish queries, and the results returned. Coupling this with the Convert module, one can generate CSV or HTML or Sylk or whatever form seems most convenient. (Not part of the kit distro - I'm still working on authentication of access - ask CMcC if you're interested.)

RLH 2006-09-07: Is the Mason portion done? I would have thought that Template-Toolkit for Tcl would have been a better fit, but you probably have you reasons for the Mason version as well. Where did Wub come from? And lastly...if you need help give a shout out (docs, testing as I am on 3 different operating systems, etc.).

CMcC RLH, Mason's done, yeah. What is Template-Toolkit for Tcl? Templating in tcl's pretty easy really, subst does the job very nicely. Wub the name? I just thought it was a lame-sounding variant on Web ... and there's the PK Dick story 'beyond lies the Wub,' which sounds good.

RLH Is there any documentation on the Mason part? I am very interested in seeing the Tcl version (as there is a Python version as well).

CMcC The parts of Mason I thought worth lifting, and acknowledging as lifted by the use of the name, are the wrapper and the notfound scripts, and the idea of searching up the file system hierarchy for candidates. I didn't see anything else worth emulating, but if there's something worthwhile, I'm happy to consider them.

In answer to the question about documentation: what docco there is (and there is some) is to be found within the Wub distribution - start it running and look in http://localhost:8080/docs/ . The source itself it (I hope) reasonably well internally narrated.

RLH Just a note that 8.5 is required to run it. At least that is the error message I got when running it under 8.4.

GP-L I downloaded wub.kit but the source of the pages is served. It seems that no content-type is returned... Lynx -head displays:

 HTTP/1.0 200 OK                                                                                  
 Date: Tue, 06 Feb 2007 22:16:58 GMT                                                              
 Server: Wub 1.0                                                                                  
 connection: close                                                                                
 set-cookie2: session="4 C6A76862EBC01B6542D3332C7A8B09AB 0";version=1                            
 etag: 6D084DA7CBBF3F51674F9CC444FC7BC0                                                           
 last-modified: 1170800126                                                                        
 vary: cookies                                                                                    
 set-cookie: session="4 C6A76862EBC01B6542D3332C7A8B09AB 0";version=1

and the log in the terminal says:

 [Tue Feb 06 23:17:53 CET 2007] [8080] [info] 'Send code: 200 reply: -code 200 accept {text/html, text/plain, text/sgml, */*;q=0.01} etag 6D084DA7CBBF3F51674F9CC444FC7BC0
 -session 5 accept-language fr -ipaddr 127.0.0.1 -dynamic 0 -port 8080 -scheme http -Query {} -host xxxxx.xxxxxx -version 1.0
 set-cookie {session="5 291F3BE767501A5C875EF7BCE19FFA58 0";version=1} accept-encoding {gzip, compress} -uri http://xxxxx.xxxxxx:8080/
 -shandler ::session host 127.0.0.1:8080 -fd file14 -entry http://xxxxx.xxxxxx:8080/ -times {reply 158737238453437 read 158737233680374
 dispatch 158737233919356 start 158737232717051 seconds 1170800273} -depends {/private/tmp/docroot /private/tmp/docroot/index.tml}
 -url http://xxxxx.xxxxxx:8080/ -rtype CachedContent -generation 4 content-type text/html content-length 3535 -path / -cookies
 {session {-expires {Mon, 06 Aug 2007 21:17:53 GMT} -value {5 291F3BE767501A5C875EF7BCE19FFA58 0} -path / -changed 1}} last-modified 1170800126
 -continuations {} user-agent {Lynx/2.8.5rel.2 libwww-FM/2.14} -transaction 1 -listener ::Listener1 -rport 61612 -http ::Pool::Httpd1 -sock sock12
 -dispatch_count 0 set-cookie2 {session="5 291F3BE767501A5C875EF7BCE19FFA58 0";version=1} -modified 1170800126 -method HEAD -vary {cookies 1}'

RLH 2007-02-19: Can you publish a tar.gz or zip with the source along with the kit?

The source is already contained in the kit-file, simply UNWRAP the kit-file using http://www.equi4.com/pub/sk/sdx.kit , see sdx for further doco.

RLH Yes, I know but I really don't like messing with kit files.

CMcC Yes. Now it's under subversion (as detailed above)

JAH - 2007-02-20: Downloaded wub.kit and am running on OS X 10.4 with swisskit, which has the tcl8.5 engine. wub.kit grashes on startup with:

Well, lots of error messages ... Basically swisskit has the 8.5 version of tcl, it does not have the mk4tcl database which wub needs. So, don't use swisskit to run wub. I finally found a compatible tclkit at http://www.equi4.com/pub/tk/downloads.html which does have the required version and mk4tcl database. End of round one.

JAH - 2007-02-20: Round 2:

Problems running 'out of the box' with wub. It does serve pages, but does not convert the content-type text/x-html-fragment into text/html as it passes through the mason and expansion steps. Neither Safari, firefox, nor Opera likes x-html-fragment. Is there a 'logical' place that fragments turn into full fledged html? Is that in mason or somewhere else?

CMcC 25Apr07

Found and solved that problem. Turns out that some browsers send the header 'Accept: */*' which means 'the user will accept any type at all', but of course that's not true. Now, Wub assumes '*/*' means text/html.

There have been a lot of changes to get wikit up under Wub. I'll have to go back and fix the default Site.tcl file to reflect those changes.

tedickey - 2007-05-01

It delivers text/html, but lynx and w3m are not seeing the connection close in a timely fashion.

RLH 2007-05-02: I do not see anything on the Google site as far as "source" goes. LV As was mentioned about, you download the starkit version of wub, and use sdx to unwrap it. The source will be in the extracted directory. MHo Can't locate wub.kit, too...

CMcC There is a 'source' link on the google site given. The kit version is totally out of date.

LES 2007-05-31: How do you use it? I downloaded the source and there is no documentation, not even about how to run it. It looks like I am supposed to run ../wub/Wub/wub-daemon, but nothing happens when I run it.

CMcC Yeah, rushed release - sorry. Try grabbing the wikit code from http://code.google.com/p/wikitcl/ into a sibling directory, and running tclsh Site.tcl from that (wikit) directory.

MHo: Hmmm... I tried tclkit site.tcl. That results in can't find package snit.....

Vtk: Several suggestions: 1. Documentation -- the major reason why projects die or never take off is the lack of documentation. I see you are using Snit in your code so you are familiar with the way Snit documantation is done. It is the best way to do it -- by giving working examples of each command/option. DO NOT do a man page type documentation. It is a waste of time and is NOT helpful. DO NOT tell people to read the source code as a way of figuring out how to use it. Some experienced people can do it but still it takes a lot of time and most users are not willing to do that. So, they leave in search of a better documented project (let's take PHP, for example. It is not the best programming language but the creator(s) had lots of documentation from the start and lots of working examples. That's the main reason (IMHO) PHP has become popular). 2. Lego type plugins/modules -- consider using XOTcl instead of Snit. It is more friendly for creating modular design (look at filters/mixins, for example) and is more advanced. Once you know Snit, XOTcl is easy to learn and converting your code from Snit to XOTcl is not difficult. I have myself invested a lot of time into Snit, to find out later that it was easier and shorter to write my code in XOTcl. Snit is a great OO language to start with (especially for learning OO way of programming), but XOTcl is a step up from there. Good luck and I hope your project doesn't stagnate like many other promissing projects out there.

CMcC Thanks, Vtk, that's all good advice - particularly the idea of using examples. In the interim I've set up http://wiki.tcl.tk/_doc/ which rips namespace, class and proc/method level docs straight out of the code and formats it. That covers the man-style documentation with minimal effort. I'll think about examples in-line as well.

apw 2007-12-21 - These are the steps I used to make a local wiki running:

Vtk 2008-03-30 - Re: the steps to take to make it run by apw. The installation steps above are exactly what drives people away. If you have to go through all kinds of dancing to make it work, who knows what else is the problem. From my many experiences with other code, I don't feel too excited to install it. Getting old and tired of spending hours to (may be) make it work. And that "may be it will work" stops me right there. I would prefer the three steps: 1.download 2. unpack 3. run ;) ;) - I bet everyone would like that - that's why I like tclhttpd: download, gunzip, "tclsh httpd.tcl -debug 1" - is all I need to do.

Sarnold 2008-07-05 - At a first look, Wub runs on Microsoft Windows XP with the constraint that the path of the Tcl files does not contain spaces. In short, don't install Wub to C:\Program Files\ or C:\Documents and Settings\. The demo application seems to work, at least it is serving files. Larry Smith Or <shudder> refer to it as Progra~1

LV Perhaps there are coding changes in Wub that would enable it to be free of this restriction?

fhou 2008-07-20 - The QuickStart seems to work fine for me. The mailing list [email protected] ("the best source of current information about Wub") does not seem to exist anymore. Is there another one, or is this the place?

boune 2009-01-09 - Why not other wiki system? See http://www.wikimatrix.org/ , there are dozens of good wiki systems.

CMcC 2009-01-09 - Wub's the web server, not the wiki. Why another wiki system, something wrong with wikit? If so, what? Finally, there's some virtue to it all being written in tcl.

EMJ 2009-02-17 I'm writing an application that has tclhttpd incorporated, so I should look at Wub, yes? So...

  • Set up a side install of Tcl 8.5 (too much stuff on the machine to retest when I move off 8.4 - one of these days...)
  • Get and build Subversion - if I can't build it myself I don't use it on this server, period. (There are exceptions, but this is not enough to justify one.) Luckily, I don't have to actually install it to use it to ...
  • Get Wub into a suitable directory.
  • Try to run Application.tcl, find I need TclOO.
  • Get and install TclOO.
  • Now I have invalid command name "coroutine", does this mean I need 8.6 - can't do that, the application I'm writing is for someone else and I can't use beta releases!
  • Oh, well, I didn't want that evening anyway (I had a break to do some ironing - wow!).

CMcC Yes, Wub uses coroutines extensively now in Httpd.tcl (the protocol package.) I'm sorry that bit you, but if you saw how much simpler the code is, you'd understand.

MHo:

  • Is there a chance to turn the server into a windows service with the help of winserv and to wrap it as a single executable starpack? In theory, it should not be more complex as with tclhttpd, I guess...
  • Just tried to start wub with the tclkitsh.exe v8.6 from http://www.patthoyts.tk/tclkit.html - and get a can't find package fileutil error... is there a predefined place where I can copy the routines into or should I hack elsewhere...?
Category Internet Category Webserver Category Application