Version 191 of Wub

Updated 2010-10-01 20:17:08 by jnc

Where is Wub?

  1. Wub is hosted here http://code.google.com/p/wub/
  2. Wub has a discussion group here http://groups.google.com/group/wub-discussion
  3. Wub code can be fetched from svn with svn checkout http://wub.googlecode.com/svn/trunk/ wub if you are running the bleeding edge Tcl8.6 from HEAD, or for current stable branch: svn checkout http://wub.googlecode.com/svn/branches/RB-2.0.0 wub if you have garden-variety Tcl8.6
  4. Occasional releases can be found here: [L1 ]
  5. Documentation can be browsed here: [L2 ]
  6. Isssssssssues with Wub can be reported here: [L3 ]
  7. Wub tutorial presented at 8th European Tcl/Tk Users Meeting, available from http://code.google.com/p/wubwikit/downloads/list

What is Wub: Introduction

Wub is a web-server written in pure-Tcl. It runs the Wiki you're now reading. It absolutely requires Tcl 8.6 and tracks the HEAD closely. It should help in creating highly-dynamic (and portable) web applications. It is the successor in spirit of Tclhttpd, aiming to preserve the best of it while using the bleeding-edge latest Tcl facilities to simplify and extend.

Wub's documentation is here [L4 ]. The author CMcC is currently in a documenting mood, and would appreciate feedback on where the documentation lacks clarity, information, specificity, accuracy, etc etc ... all those banes of documenters.

Wub is essentially an interpreter for HTTP 1.1 requests, and it's very appropriate to use Tcl for that kind of thing. Here's Wub's Read-Eval-Print loop ... couldn't be simpler.

http://wiki.tcl.tk/_repo/images/arch.png

Semantics for the interpretation are provided by Domains [L5 ]. Domains are Tcl modules which transform requests into responses, possibly producing side-effects (such as e.g. DB updates.) This simple transformational semantic model underpins all Wub processing - all Wub's components are functions which take a request dict and return a response dict.

The Httpd module [L6 ] operates and binds a pair of coroutines per connection. They are called reader and consumer, and here's what they do:

HTTP requests and responses are nothing more than a collection of fields and an optional entity, and so representing them as Tcl dicts is a natural and obvious choice. A reader coroutine takes requests from a connection and converts them to Tcl dicts which are queued for a consumer coroutine to process.

The consumer coroutine pre-processes the request, decodes the URL and dispatches to the appropriate Domain handler. The return result of the Domain handler invocation is a response dict which consumer post-processes, then queues up for reader which converts it to HTTP and sends it down the wire to the connected client.

Wub is extensible, and so the Httpd module which supervises all of this can be modified to customize pre- and post- processing and also to customize the URL decode and dispatch operation.

While it's conceptually simple to write dispatch code (which lives, incidentally, in ::Httpd::do) the implementation can get complex and messy as it deals with details of decoding, switching and interfacing. Most web sites and applications merely decode URLs into components, then dispatch on the URL path component. So, to make it easy to customize Wub's dispatch behavior, the Nub module [L7 ] compiles an abstract web-site description language (which is just Tcl, suitably constrained) to generate the dispatcher. The Nub generator supports URL rewriting, client redirection and blocking. You can also write Nubs which generate literal text [L8 ] and Nubs containing scripts which generate output directly (who needs CGI when you have a Tcl interpreter always present?)

Contrast the Nub approach with that of Apache: Nubs are simple declarative expressions in Tcl which are compiled into pure Tcl to decode and dipatch. Apache's equivalent is the .htaccess file, which is interpreted by a hand-written C interpreter.

That's really all there is. Domains plug into this architecture, and may be as simple or complex as needed. A collection of useful Domains come built-into Wub [L9 ] and more are being added all the time.

Working Example of Wub: This Wiki

This Wiki runs under Wub. It's derived from a wiki system called wikit written by jcw.

The Wiki's pages are served from a custom-built Domain which interfaces to the original wikit code (which is almost completely unmodified) and provides additional functionality (edit history, diffs, etc.)

CSS files, javascript, images, and other similarly static content are served via the File Domain [L10 ] which simply delivers files from a file system hierarchy.

The detailed documentation being referenced throughout this exposition is served via an augmented File-like Domain called Mason [L11 ]

The layout of the site, the mapping of URLs to Domains is handled by Nub, which can be seen in operation here [L12 ] ... if differently configured, Nub actually allows you to dynamically reconfigure the server dispatch mechanism on-the-fly.

The Nub Domain interface is implemented using the jQ Module [L13 ] to deliver jQuery javascript plugins and scripting to the client. So that's Web 2.0 via Wub 5.0. Woo Woo.

Quick Start

Dependencies:

Must have: tcllib Optionally: tls for SSL

  1. Download the code: svn checkout http://wub.googlecode.com/svn/trunk/ wub
  2. Change into the Wub directory: cd wub/Wub/
  3. Configure the site.ini: cp example.ini site.ini
  4. Edit site.ini to your taste
  5. Run the demo application: tclsh8.6 Application.tcl

Default comes up at port 8080. There's a control port at 8082 - tcl commands into running app from localhost only.

Configuration

Wub/Application.tcl is a demo application showing all available Wub Domains [L14 ]. It may be configured by a file called site.ini in the same directory (an example.ini in that directory may be used as a template.) Default nubs [L15 ] are found in Domains/nubs/ and may be edited in a site nub file of your choosing. Nubs may also be edited and saved online using the nub domain, an example of which can be seen here [L16 ] (those nubs run this wiki.)

Wub Architecture

Wub is a framework for writing web servers.

It provides a collection of utilities enabling a programmer to assemble a range of useful dedicated web servers. While the programmer is free to assemble the components in any way they wish, some approaches are better supported than others, and this could be called Wub's architecture.

Wub's Httpd module converts HTTP protocol interactions into tcl dicts containing all the request's protocol fields. These 'request dicts' are transformed into 'response dicts' by the user-supplied application using Wub Utilities and Wub Domains (Utilities/ and Domain/ respectively) and are returned via HTTP to the client.

A request/response dict resembles a whiteboard upon which Domains and Utilities operate, reading fields left by earlier processing, creating and modifying fields as a result of their own operation. Most modules act as functions, taking a request dict argument and returning a response dict result.

One convention which is rigorously enforced is that non-protocol fields in request/response dicts have names of the form '-*', distinguishing them from HTTP fields which will eventually be sent to the client.

Dataflow

Httpd package receives protocol transactions and generates a request dict. After performing some analytic transformations, such as Url and Query processing, User Agent processing and malicious client Blocking, Httpd (optionally) consults one of two Cache packages. If the Cache can satisfy the request, it does so. Otherwise, a configurable ::Httpd::do proc is called with the request, usually to dispatch incoming request dicts to an appropriate Domain by means of a switch over some fields in the request dict - specifically and primarily the -url field which contains the incoming request URL. After Domain package handling, the returned response dict is queued up to be returned to the client via Httpd.

In addition to the Cache module, there are other filtering modules: Convert (which handles mime type content negotiation and conversion).

Domain packages provide application-level semantics to a web server. A range of example domains are provided.

Digression

RLH Why does wub require 8.6 when it isn't even out yet?
CMcC because it depends on facilities which are only in 8.6, of course.
RLH That is obvious. However, I would think it would be a very few that are going to run 8.6 at this point and so...can't use Wub. Of course, it is yours to do with what you will I just think that a shame.
CMcC well, it's a tradeoff I guess. I can develop faster and better with 8.6 facilities (coros made a big difference in performance and code maintainability, TclOO made code structure a bit better,) and so I'm gambling that by the time I've run out of things to do, people will be using 8.6.
JE Also worth noting: by using new features like dictionaries, OO, and coroutines before they're even out of alpha, Wub has helped to make Tcl better. Feedback from early adopters on the bleeding edge is very valuable to the folks trying to invent the bleeding edge.

Related

Wub Suspend - how to suspend under Wub Wub Consumer - how to interact with HTTP clients via Wub - the consumer API

Archeology

Wub's complete rewrite is complete, and is running this wikit. (25Apr07)

Wub's available here: http://code.google.com/p/wub/ under subversion control.

Try grabbing the wikit code from http://code.google.com/p/wikitcl/ into a sibling directory, and running tclsh Site.tcl from that (wikit) directory.

More later.

CMcC


Wub was underperforming relative to Apache et al, so I've rewritten it all to be multithreaded and lighter-weight.

The pursuit of a Lego-like plugging little language has been abandoned, under the following reasoning:

  1. the audience for lego-plugability is probably not an audience that should be building websites anyway,
  2. building nice things out of Lego is hard,
  3. the overhead of maintaining a plug-together API was considerable although I did learn a lot about the new return functionality,
  4. tcl is amply competent at parsing and dispatching URLs — which is all the plugin/domain model really had to offer,
  5. multithreading made site writing easier (giving a thread-level execution environment),
  6. multithreading made the simple domain plugging model non-viable (caching, for example, has to be in the top level dispatcher thread).

Preliminary benchmarks suggest a rough parity with Apache out-of-the-box.

New caching architecture will make it easier to interact with a decent caching reverse-proxy if one's ever written.


If you consider tclhttpd, take a look at Wub too. Wub will probably see more active development than tclhttpd in the future (as the wub author is/was one of the main contributors to the tclhttpd code...).

Wub is a complete redesign of a http (nearly-1.1) webserver using dicts for state instead of all the global vars used in tclhttpd. Wub owes a kind of filial respect to tclhttp, as a few core techniques are derived from it, but I hope Wub will be more convivial for developers.

The Wub architecture is designed to be highly modular. All protocol operations occur in two modules Listener and Httpd. Once an http request has been parsed to a dictionary (and after possible further processing by Cookie, Entity and Query modules) the request dict is transformed by a series of Domains.

I would like Wub to resemble a set of modules out of which one can assemble a pachinko or a pinball machine for handling http requests (the requests and responses are like the balls ... that's the metaphor).

The following Domains (and more) are available:

Cache
provides caching of content
File
reflects a filesystem to the internet
Convert
converts content to an arbitrary mime-type using a graph of individual conversions, so xyz->html->pdf type conversions are possible.
Mason
emulates a quite good PERL CMS-like system called Mason (http://www.masonhq.com/ ).
Db and Sql
reflect a metakit and sqlite database (respectively) to the internet. Http query language is used to construct SQL-ish queries, and the results returned. Coupling this with the Convert module, one can generate CSV or HTML or Sylk or whatever form seems most convenient. (Not part of the kit distro - I'm still working on authentication of access - ask CMcC if you're interested.)

RLH 2006-09-07: Is the Mason portion done? I would have thought that Template-Toolkit for Tcl would have been a better fit, but you probably have you reasons for the Mason version as well. Where did Wub come from? And lastly...if you need help give a shout out (docs, testing as I am on 3 different operating systems, etc.).

CMcC RLH, Mason's done, yeah. What is Template-Toolkit for Tcl? Templating in tcl's pretty easy really, subst does the job very nicely. Wub the name? I just thought it was a lame-sounding variant on Web ... and there's the PK Dick story 'beyond lies the Wub,' which sounds good.

tom.rmadilo CMcC, Template Toolkit for Tcl is my templating toolkit. Templating in Tcl is not easy, subst is very dangerous and can't be fixed using options -nobackslashes and/or -nocommands. For instance: $var([rm -rf /]) will remove everything on your hard drive. Very nice! My template language is safe and expands the template text into a Tcl script which avoids dangerous code. The basic safety mechanism is a non-recursive expansion of the template into Tcl code. A nice side-effect of compilation of the template is validation of the template language. The compiler is written in flex/bison and an invalid template leaves a partially compiled remnant which helps identify the location of the syntax error. If you have ever tried to track down template coding errors, you will appreciate this feature.

RLH Is there any documentation on the Mason part? I am very interested in seeing the Tcl version (as there is a Python version as well).

CMcC The parts of Mason I thought worth lifting, and acknowledging as lifted by the use of the name, are the wrapper and the notfound scripts, and the idea of searching up the file system hierarchy for candidates. I didn't see anything else worth emulating, but if there's something worthwhile, I'm happy to consider them.

In answer to the question about documentation: what docco there is (and there is some) is to be found within the Wub distribution - start it running and look in http://localhost:8080/docs/ . The source itself it (I hope) reasonably well internally narrated.

RLH Just a note that 8.5 is required to run it. At least that is the error message I got when running it under 8.4.

GP-L I downloaded wub.kit but the source of the pages is served. It seems that no content-type is returned... Lynx -head displays:

 HTTP/1.0 200 OK                                                                                  
 Date: Tue, 06 Feb 2007 22:16:58 GMT                                                              
 Server: Wub 1.0                                                                                  
 connection: close                                                                                
 set-cookie2: session="4 C6A76862EBC01B6542D3332C7A8B09AB 0";version=1                            
 etag: 6D084DA7CBBF3F51674F9CC444FC7BC0                                                           
 last-modified: 1170800126                                                                        
 vary: cookies                                                                                    
 set-cookie: session="4 C6A76862EBC01B6542D3332C7A8B09AB 0";version=1                             

and the log in the terminal says:

 [Tue Feb 06 23:17:53 CET 2007] [8080] [info] 'Send code: 200 reply: -code 200 accept {text/html, text/plain, text/sgml, */*;q=0.01} etag 6D084DA7CBBF3F51674F9CC444FC7BC0
 -session 5 accept-language fr -ipaddr 127.0.0.1 -dynamic 0 -port 8080 -scheme http -Query {} -host xxxxx.xxxxxx -version 1.0
 set-cookie {session="5 291F3BE767501A5C875EF7BCE19FFA58 0";version=1} accept-encoding {gzip, compress} -uri http://xxxxx.xxxxxx:8080/
 -shandler ::session host 127.0.0.1:8080 -fd file14 -entry http://xxxxx.xxxxxx:8080/ -times {reply 158737238453437 read 158737233680374
 dispatch 158737233919356 start 158737232717051 seconds 1170800273} -depends {/private/tmp/docroot /private/tmp/docroot/index.tml}
 -url http://xxxxx.xxxxxx:8080/ -rtype CachedContent -generation 4 content-type text/html content-length 3535 -path / -cookies
 {session {-expires {Mon, 06 Aug 2007 21:17:53 GMT} -value {5 291F3BE767501A5C875EF7BCE19FFA58 0} -path / -changed 1}} last-modified 1170800126
 -continuations {} user-agent {Lynx/2.8.5rel.2 libwww-FM/2.14} -transaction 1 -listener ::Listener1 -rport 61612 -http ::Pool::Httpd1 -sock sock12
 -dispatch_count 0 set-cookie2 {session="5 291F3BE767501A5C875EF7BCE19FFA58 0";version=1} -modified 1170800126 -method HEAD -vary {cookies 1}'

RLH 2007-02-19: Can you publish a tar.gz or zip with the source along with the kit?

The source is already contained in the kit-file, simply UNWRAP the kit-file using http://www.equi4.com/pub/sk/sdx.kit , see sdx for further doco.

RLH Yes, I know but I really don't like messing with kit files.

CMcC Yes. Now it's under subversion (as detailed above)


JAH - 2007-02-20: Downloaded wub.kit and am running on OS X 10.4 with swisskit, which has the tcl8.5 engine. wub.kit grashes on startup with:

Well, lots of error messages ... Basically swisskit has the 8.5 version of tcl, it does not have the mk4tcl database which wub needs. So, don't use swisskit to run wub. I finally found a compatible tclkit at http://www.equi4.com/pub/tk/downloads.html which does have the required version and mk4tcl database. End of round one.


JAH - 2007-02-20: Round 2:

Problems running 'out of the box' with wub. It does serve pages, but does not convert the content-type text/x-html-fragment into text/html as it passes through the mason and expansion steps. Neither Safari, firefox, nor Opera likes x-html-fragment. Is there a 'logical' place that fragments turn into full fledged html? Is that in mason or somewhere else?

CMcC 25Apr07

Found and solved that problem. Turns out that some browsers send the header 'Accept: */*' which means 'the user will accept any type at all', but of course that's not true. Now, Wub assumes '*/*' means text/html.

There have been a lot of changes to get wikit up under Wub. I'll have to go back and fix the default Site.tcl file to reflect those changes.


tedickey - 2007-05-01

It delivers text/html, but lynx and w3m are not seeing the connection close in a timely fashion.


RLH 2007-05-02: I do not see anything on the Google site as far as "source" goes. LV As was mentioned about, you download the starkit version of wub, and use sdx to unwrap it. The source will be in the extracted directory. MHo Can't locate wub.kit, too...

CMcC There is a 'source' link on the google site given. The kit version is totally out of date.


LES 2007-05-31: How do you use it? I downloaded the source and there is no documentation, not even about how to run it. It looks like I am supposed to run ../wub/Wub/wub-daemon, but nothing happens when I run it.

CMcC Yeah, rushed release - sorry. Try grabbing the wikit code from http://code.google.com/p/wikitcl/ into a sibling directory, and running tclsh Site.tcl from that (wikit) directory.

MHo: Hmmm... I tried tclkit site.tcl. That results in can't find package snit.....

Vtk: Several suggestions:

  1. Documentation — the major reason why projects die or never take off is the lack of documentation. I see you are using Snit in your code so you are familiar with the way Snit documantation is done. It is the best way to do it — by giving working examples of each command/option. DO NOT do a man page type documentation. It is a waste of time and is not helpful. Do not tell people to read the source code as a way of figuring out how to use it. Some experienced people can do it but still it takes a lot of time and most users are not willing to do that. So, they leave in search of a better documented project (let's take PHP, for example. It is not the best programming language but the creator(s) had lots of documentation from the start and lots of working examples. That's the main reason (IMHO) PHP has become popular).
  2. Lego type plugins/modules — consider using XOTcl instead of Snit. It is more friendly for creating modular design (look at filters/mixins, for example) and is more advanced. Once you know Snit, XOTcl is easy to learn and converting your code from Snit to XOTcl is not difficult. I have myself invested a lot of time into Snit, to find out later that it was easier and shorter to write my code in XOTcl. Snit is a great OO language to start with (especially for learning OO way of programming), but XOTcl is a step up from there.

Good luck and I hope your project doesn't stagnate like many other promissing projects out there.

CMcC Thanks, Vtk, that's all good advice - particularly the idea of using examples. In the interim I've set up http://wiki.tcl.tk/_doc/ which rips namespace, class and proc/method level docs straight out of the code and formats it. That covers the man-style documentation with minimal effort. I'll think about examples in-line as well.

apw 2007-12-21 — These are the steps I used to make a local wiki running:

Vtk 2008-03-30 — Re: the steps to take to make it run by apw. The installation steps above are exactly what drives people away. If you have to go through all kinds of dancing to make it work, who knows what else is the problem. From my many experiences with other code, I don't feel too excited to install it. Getting old and tired of spending hours to (may be) make it work. And that "may be it will work" stops me right there. I would prefer the three steps: 1.download 2. unpack 3. run ;) ;) - I bet everyone would like that - that's why I like tclhttpd: download, gunzip, "tclsh httpd.tcl -debug 1" - is all I need to do.


Sarnold 2008-07-05 - At a first look, Wub runs on Microsoft Windows XP with the constraint that the path of the Tcl files does not contain spaces. In short, don't install Wub to C:\Program Files\ or C:\Documents and Settings\. The demo application seems to work, at least it is serving files. Larry Smith Or <shudder> refer to it as Progra~1

LV Perhaps there are coding changes in Wub that would enable it to be free of this restriction?


fhou 2008-07-20 — The QuickStart seems to work fine for me. The mailing list [email protected] ("the best source of current information about Wub") does not seem to exist anymore. Is there another one, or is this the place?

boune 2009-01-09 — Why not other wiki system? See http://www.wikimatrix.org/ , there are dozens of good wiki systems.

CMcC 2009-01-09 — Wub's the web server, not the wiki. Why another wiki system, something wrong with wikit? If so, what? Finally, there's some virtue to it all being written in tcl.


EMJ 2009-02-17: I'm writing an application that has tclhttpd incorporated, so I should look at Wub, yes? So...

  • Set up a side install of Tcl 8.5 (too much stuff on the machine to retest when I move off 8.4 - one of these days...)
  • Get and build Subversion - if I can't build it myself I don't use it on this server, period. (There are exceptions, but this is not enough to justify one.) Luckily, I don't have to actually install it to use it to ...
  • Get Wub into a suitable directory.
  • Try to run Application.tcl, find I need TclOO.
  • Get and install TclOO.
  • Now I have invalid command name "coroutine", does this mean I need 8.6 - can't do that, the application I'm writing is for someone else and I can't use beta releases!
  • Oh, well, I didn't want that evening anyway (I had a break to do some ironing - wow!).

CMcC Yes, Wub uses coroutines extensively now in Httpd.tcl (the protocol package.) I'm sorry that bit you, but if you saw how much simpler the code is, you'd understand.

MHo:

  • Is there a chance to turn the server into a windows service with the help of winserv and to wrap it as a single executable starpack? In theory, it should not be more complex as with tclhttpd, I guess...
  • Just tried to start wub with the tclkitsh.exe v8.6 from http://www.patthoyts.tk/tclkit.html - and get a can't find package fileutil error... is there a predefined place where I can copy the routines into or should I hack elsewhere...?

See also:


jima (2009-04-18)

I have downloaded Wub and started it successfully following its Quick Start note.

When I point my browser to local port 8080 I am informed that mk4tcl package is missing.

I thought the only explicit dependency was tcllib...

Is this mk4tcl message due to the fact I am starting with "Application.tcl" and this needs mk4tcl?

Can I have a site without this package ?

jdc Try to remove the session line from Domains/nubs/nub.nub

CMcC Thanks for this, both the query and the fix. I'm removing Session from nub, (a) because it doesn't work very well, (b) because it introduces this spurious dependency, (c) because there's a better way to do it.


aa (2009-05-04)
I'm trying to get Wub up and running on a Windows XP desktop. I downloaded wub-1.0.tar.gz and extracted it into the C:/wub directory. I downloaded ActiveTcl8.6.0.0b1.289036-win32-ix86-threaded.exe and installed it. Running tclsh Application.tcl in the C:/wub/Wub/ directory gives a bunch of startup messages, all of which look reasonable except for this:

 TLS:-host dl889kh91 -port 8081 -httpd Httpd -id 2 -tls {}
 error @@{constructor err: -code 1 -level 0 -errorcode NONE -errorinfo {dl889kh91:8081 couldn't open socket: address already in se\ncmd=tls::socket -server {::Listener::accept {-host dl889kh91 -port 8081 -httpd Httpd -id 2 -tls {\n    -ssl3 1 -ssl2 0 -tls1 1\n           } -certfile server-public.pem -keyfile server-private.pem}} -command ::Listener::progress -ssl3 1 -ssl2 0 -tls1 1 8081\n    while executing\n"error "[dict get $args -host]:[dict get $args -port] $listen\ncmd=$cmd""} -errorline 33}

Then when I launch IE and attempt to view http://localhost:8080 , it redirects to http://localhost:8080/wub/ and gives an error page: Server Error: can't read "defs(wub)": no such element in array.

The full errorinfo is this:

 can't read "defs(wub)": no such element in array while executing "$defs(wub) do $r" ("*,/wub/*" arm line 3) invoked from within "switch -glob -- [dict get $r -host],[dict get $r -path] { "*,/wub/docs/*" { Debug.nub {Dispatch [dict get $r -url] via *,/wub/docs/*} ..." (procedure "do" line 65) invoked from within "do REQUEST [pre $r]"

and the command window where the server is running shows

 log @@{Listening on https://dl889kh91:8081/ using docroot C:/wub/Wub/docroot}
 site @@{NUB:nubs {nub.nub bogus.nub}}
 error @@{Nub Definition Error: 'could not read "C:/wub/Wub/docroot": no such file or directory' in running "Mason create wub auth .before wrapper .after root C:/wub/Wub/docroot mount /wub/".  (-code 1 -level 0 -errorcode {POSIX ENOENT {no such file or directory}} -errorinfo {could not read "C:/wub/Wub/docroot": no such file or directory\n    while executing\n"file lstat $root attr"\n    (class "::Mason" constructor line 48)\n    invoked from within\n"Mason create wub auth .before wrapper .after root C:/wub/Wub/docroot mount /wub/"} -errorline 61)}
 error @@{Server Error: 'can't read "defs(wub)": no such element in array' (-code 1 -level 0 -errorcode {TCL READ VARNAME} -errorinfo can't\ read\ \"defs(wub)\":\ no\ such\ element\ in\ array\n\ \ \ \ while\ executing\n\"\$defs(wub)\ do\ \$r\"\n\ \ \ \ (\"*,/wub/*\"\ arm\ line\ 3)\n\ \ \ \ invoked\ from\ within\n\"switch\ -glob\ --\ \[dict\ get\ \$r\ -host\],\[dict\ get\ \$r\ -path\]\ \{\n\t\t\t\n\t\t\t\ \ \ \ \"*,/wub/docs/*\"\ \{\n\t\t\t\tDebug.nub\ \{Dispatch\ \[dict\ get\\$r\ -url\]\ via\ *,/wub/docs/*\}\n\t...\"\n\ \ \ \ (procedure\ \"do\"\ line\ 65)\n\ \ \ \ invoked\ from\ within\n\"do\ REQUEST\ \[pre\ \$r\]\" -errorline 35)}

What am I missing?

CMcC is investigating. What I have now is: The problem arises because Wub can't find the default document root. In a customised installation, site.ini would define docroot (there's an example.ini which can be customised to suit.) The server is intended to fail safe, and use sensible defaults, but obviously that's not happening in your case. My best guess is that ActiveTcl defines ::starkit::toplevel, which the server misinterprets as meaning that it's running within a starkit, and so it's picking the wrong default toplevel directory. I'm investigating this possibility. In the meantime, suggest you copy example.ini to site.ini, which should pick up the correct defaults. Thanks for spotting this problem.

aa: That's more like it! Now it looks the way I expect, and I can start trying to use it instead of trying to troubleshoot it. :)


AMG: I am toying with the idea of putting together a site for serving a particular Web comic [L17 ]. Necessary features:

  • HTTP 1.0 (I doubt any 1.1 features are needed)
  • Serving static files from the filesystem
  • CGI-like functionality from templates in the "Templates and subst" format
  • Ability to access SQLite databases
  • User accounts authenticated with HTML forms and cookies, not HTTP authentication
  • Uploading of "avatar" images for user-posted comments

While it would be nice to support administration through the Web (e.g. using a browser to upload new pages or delete unwanted comments), I think it would also be acceptable to have a collection of scripts usable from a ssh login shell. (The comic artist may disagree; I haven't consulted with her yet.) For this reason I don't think SSL is necessary. Of course, features can be added in time.

Is Wub a good fit for this task, or is it overkill? Should I be looking at something simpler? Right now I'm simply hacking on DustMote.

The reason I'm contemplating this project is that the artist has wanted for years to set up ComicPress [L18 ] but cannot master MySQL's administrative complexity. I think she would be much better off with SQLite.

CMcC As to requirements:

  • HTTP 1.1 - one of the features of 1.1 is pipelining which means significantly faster loading (because the client doesn't need to re-open a connection for each entity they want to fetch.) There's really no excuse for serving HTTP/1.0 anymore.
  • Wub's File and Mason domains handles static file serving
  • Wub's Mason allows files to be interpreted directly within tcl, no need for CGI (which is slow, gross and grossly slow.)
  • Anything running under tcl can access SQLite.
  • Wub's Login domain does this. ATM it uses Metakit, I'll coerce it to use SQLite if you like. It does Login/Auth using cookies (or HTTP auth) and also gives you per-user rows for storing stuff.
  • Uploading, yeah, easy enough. Wub's Form utility also enables you to do nice form handling in conjunction with its Direct domains. Wub's jQ domain lets you use jQuery for making it nice (multiple file uploads, etc.)

Additionally, Wub has support for site-wide page customisation.

As to 'overkill': serving pages is part of it. Serving pages efficiently is more of it (Wub does server side caching.) Serving pages in the presence of malicious bots and spiders is a lot more of it. While you can certainly use DustMote for the first part (serving pages) you're left with a lot more to do to achieve much of the rest of it. Wub's not particularly minimal, but I think it's about as heavyweight as the full task requires. If you do choose to use DustMote, or write your own, I'd encourage you to look at Wub's Utilities/ directory as it has a lot of functionality you're going to need anyway.

I'm more than happy to work with people who want to try this stuff out.

AMG: Okay, I have the Wub SVN installed, along with its dependencies. (I even have mk4tcl, which was a pain because the latest release doesn't compile with Tcl 8.6; the latest SVN works, with a lot of warnings and one test failure.) It runs. But I don't have any idea how to set it up. I don't know where to begin customizing it, configuring it, or even learning it. My impression is that it's bigger and more complicated than is warranted for my needs. But I could be wrong--- maybe my needs are more complex than I realize, and/or maybe Wub is simpler than I think it is.

So I will explain where I am at with DustMote. I added template [subst]'ification of *.tmpl files, added directory redirection (301's /dirname to /dirname/), and changed index.html defaulting to search for index.tmpl or index.html. I will need to bring in the utility procedures I previously wrote for my own Web site [L19 ] to handle quoting, etc., and I'll also bring in handling of HTTP headers and URL query parameters. But still, this would leave me without pipelining, uploads, cookies, sessions, or authentication. I'm sure these can be added. Server-side caching can be accomplished by remembering the [subst] scripts generated from the *.tmpl files, but I'd have to profile first to see if it's worth it. (Hooray for [time]!) I don't know how to deal with malicious bots or spiders.

jdc Have a look at [L20 ]. I'm trying to put a collection of examples there as part of a tutorial on Wub. Each example introduces a Wub feature. This is still work-in-progress.

CMcC let's do some problem analysis:

  1. receiving, decoding, dispatching-upon and replying-to HTTP protocol streams is one part of the task of serving a web page. A solution to that can be found in [L21 ]. It's a pretty good solution, it is not as tiny as Dustmote because it does a lot more, only some of which is able to be removed.
  2. interpreting received HTTP request URLs semantically and transforming them into responses is covered by all the stuff in [L22 ] where you will find several modules of code which perform generally useful transformations. You can choose to use whichever of them suits your needs, or none of them, or modify them as you wish.
  3. interpreting various fields in the HTTP request and giving them semantics like 'cookie', 'query', 'user agent', 'url' etc, generating forms, html, report-form tables, etc are covered in [L23 ]

Is it possible to take Dustmote (which does a severely restricted subset of 1. above,) add 2. and 3. avove to it, and get a solution? Sure. Is it possible to write brand new code to do 1., 2. and 3. above? Sure.

The only reasons I can imagine for doing that (rewriting 1. 2. and 3.) would be:

  1. the provisions above are insufficient in some regard.
  2. one just wants to learn what goes into doing it.
  3. it's too hard to learn how to use the existing code

If The provisions are insufficient, I would like to know why, so I can improve them. If you just want to write it from scratch, that's fine too ... I will learn from the sub-problems you have had to solve and incorporate the solutions you arrive at (if they're better than mine.) If you just find it hard to learn to use existing code, part of that is my fault - there needs to be more and better documentation ('specially for the Utilities,) but I think that part of it is an underestimation of the relative difficulty of finding and solving the problems you will confront while trying to provide the facilities.

You should understand, I think, that I didn't set out to write a replacement for tclhttpd (and scads of tcllib stuff like ncgi) merely because I thought it'd be cool. I did it because I wanted to do stuff which neither tclhttpd or things like ncgi did properly. I did it after I tried to work with tclhttpd authors to make it work better. I did it as a last resort, in effect.

It's been an interesting journey. I think you should at least consider the use of a map and compass before undertaking it.

AMG: For this project, I wrote Wibble. Perhaps some ideas from Wibble can be incorporated into Wub. Have a look.

CMcC: I've taken a radical simplification from Wibble, and reduced the number of coroutines per connection down to 1. It's still experimental right now, but as it's currently (3Jun09) running this wiki, it seems to be working fine.


aa (2009-06-06)

Trying to walk before running, I set up a File domain to serve up a folder tree full of static pages. Some of the html files have spaces in the path, and those files yield a "File filename doesn't exist" error. The spaces in the URL get escaped into %20's, but Wub apparently doesn't unescape them before trying to open the file. I'm losing my way trying to trace through the program flow, and I figure the author would know what procedure to focus on in order to fix it. I think I successfully reported the problem at [L24 ], but in case I bungled it, I'm mentioning it here as well.

CMcC Fixed in svn HEAD, thanks.


SAF - 2009-09-20 10:38:05

We all know about the limitations of Rails (what a paradigm: "You don't have a scalability problem until you have one."). Unfortunately, in the real world, being scalable takes careful planning and not just throwing more cpus and band-aids after the fact.

With that, I'm curious if any thought has been given to clustering or scalability in general. I would very much like to see TCL as a viable option to backend something the size of Twitter or Facebook.


apw - 2009-11-08 Congratulations CMcC and jdc to the current version of Wub. I had always problems in the past, when trying to run Wub "out of the box". My last try was a few weeks ago and for the first time it did run "out of the box".

Today I have tried to add new Domains and within a day I was able to mostly understand the needed internals (only a small part of Wub) and - with your help on the chat - I could build a scgi client (the scgi server is already built in and running) and I could create a Domain and run my own code (a mini web frame work). Now I will continue to work with Wub, as it is really very flexible and has a lot of interfaces to make extensions easy. I especially like the use of dicts on all places.

mz - 2010-01-28 We want to build a web client of oure application how is build in tcl/tk. A part of the application we have build with wub, but with the login/logout, the session handling and the ssl encyption we have stil the problem to implement. Have you some documentation or examples about login/logout, session handling and ssl implementation?


gasty - 2010-03-31 15:34:39

Hi. I just tried the latest SVN version of Wub and doesn't work. Wub run ok, but the pages are never displayed in the browser (either firefox or explorer).

I think the problem is a configuration option missing in the default Application.tcl, since at the end of the console shows this:

579-1mS site @@{No application specified}
579-0mS site @@{entered event loop}

The 'wub-1.0.tar.gz' version in the Downloads tab works just fine (after removing '.svn' directory in the 'images' directory).

Related to this, the latest version of wubwikit neither works for me. Maybe I need a newest version of tcl8.6? I'm using tclkit8.6b1 for windows from http://www.patthoyts.tk/tclkit/


jima (2010-04-19)

Websocket plans?

I see that there are incipient server implementations in python to support websockets. More info in [L25 ] and in [L26 ]

Are there any plans to support these in wub?

Perhaps the python code can serve as basis for a tcl implementation...

CMcC 2010Jul31. We don't need no steeenkin' python reference.

Ok, I've modified the server to handle WebSockets, with what I consider to be a standard-compliant implementation (pick your standard ... the latest one I could find.) I also added an attempt at a domain for handling just websockets (although any domain could be fitted to handle them) and ... I can't get it to work. My FF doesn't grok them, my chrome says it does but just sits there and blinks at me ... so completion of this will have to await better browser adoption/standardisation.


jnc 2010-10-01: Step 3 of the quick start states: "Configure the site.ini: cp example.ini site.ini" ... Should that be 'cp example.conf site.ini' ? In the SVN I just checked out I do not see an example.ini file. Assuming 'cp example.conf site.ini' is correct, I get the following error:

 can't read "docroot": no such variable
    while executing
 "Block new logdir $docroot [config section Block]"
    (procedure "modules" line 23)
    invoked from within
 "modules                "
    (procedure "::Site::start" line 5)
    invoked from within
  "Site start "
    (file "Application.tcl" line 16)

I can comment that line out in Site.tcl but that just leads to another error when it tries to process a request. Unsure if they are related. Any thoughts?