AMG: This page is for general discussion on the subject of the Wibble web server. For discussion of bugs, see Wibble bugs. Older discussion can be found in the Wibble discussion archive.
2011 11 25 Andy - Thanks for the new wibble code dump. I agree that we need to move to a real source control system. Please just make a choice and go there. I thought that I mailed you feedback but sorry if I didn't. I posted new working WebSocket code to the wiki. I still have several zone handlers that I haven't shared yet though.
The chunked encoding exists so that the server can start sending content without knowing the content size.
AMG: Thanks for the report. See [L1 ] for my thoughts on chunked encoding from server to client. At present, Wibble doesn't try to send chunked, only receive. A custom sendcommand can use chunked if it likes, but I really ought to put chunked into [defaultsend] for the case of contentchan with no contentsize.
As for the source control system, I don't want to officially make the move until I find a way to drag along all the documentation, including page revision history.
jblz: UPDATE 1-1-11 (HAPPY NEW YEAR!)
I may move this to its own page if i end up using htmlgen in a serious way, but for the meantime, this is how to use htmlgen with wibble:
# this is a reimplementation of the code in the /vars zonehandler
# adapted to using htmlgen
package require htmlgen
namespace import ::htmlgen::*
dict set response header content-type text/html
::htmlgen::buffer ::DOC {
html ! {
head ! {
title - Playing Vars
style type=text/css - {
body {font-family: monospace}
table {border-collapse: collapse; outline: 1px solid #000}
th {white-space: nowrap; text-align: left}
th, td {border: 1px solid #727772}
tr:nth-child(odd) {background-color: #ded}
tr:nth-child(even) {background-color: #eee}
th.title {background-color: #8d958d; text-align: center}
}
}
body ! {
div id=wrapper ! {
table ! {
foreach {dictname dictvalue} $state {
if {$dictname in {request response}} {
set dictvalue [dumpstate $dictvalue]
}
tr ! {
th class=title colspan=2 + {
[enhtml $dictname]
}
}
foreach {key value} $dictvalue {
tr ! {
th + {[enhtml $key]}
td + {[enhtml $value]}
}
}
}
}
}
}
}
}
dict append response content $::DOCA few points worth mentioning
Thanks to kbk in tcler's chat for helping me with wrangling htmlgen.
AMG: I updated the above code for the change from [dumprequest] to [dumpstate]. I don't have htmlgen, so I'd appreciate it if you could test for me.
jblz: These procs are taken from the ngci package, and modified for use with wibble.
proc ::wibble::cookie {cookie} {
upvar header cheader
if {[dict exists $cheader cookie $cookie ""]} {
return [dict get $cheader cookie $cookie ""]
}
}
proc ::wibble::setcookie {args} {
upvar response cresponse
array set opt $args
dict set cresponse header set-cookie $opt(-name) "" $opt(-value)
foreach extra {path domain} {
if {[info exists opt(-$extra)]} {
dict set cresponse header set-cookie $extra $opt(-$extra)
}
}
if {[info exists opt(-expires)]} {
dict set cresponse header set-cookie expires [list abstime $opt(-expires)]
}
if {[info exists opt(-secure)]} {
dict set cresponse header set-cookie secure ""
}
}AMG: I updated this code for the new header formatting style, though I admit I didn't test. This should fix the problem you were having with not being able to set more than one cookie per request. Personally, I wouldn't use these wrapper procs; I'd just directly read and manipulate the request and response dicts; it's easy now. ;^)
dzach: What is the purpose of this line in ::wibble::process?
} elseif {[dict exists $response content]} {
dict set response content [encoding convertto iso8859-1 [dict get $response content]]Content that is already in iso8859-1 is not converted. Content that has been already converted to utf-8 by a user defined zone handler cannot be converted back to iso8859-1 with this code.
AMG: The problem is that Wibble doesn't officially support anything other than ISO8859-1, since that is HTTP's assumed character set when nothing else is explicitly specified. It's quite possible to add that feature, but I just haven't done so. Actually, it's now more possible than ever due to the vastly improved header parsing code.
AMG: I put a little bit of thought into encodings. It seems to me that it's a mistake to have the zone handler advertise the encoding for outgoing data, since at that point it's just a Tcl string. Instead, the send procedure should just convert to the client's favorite encoding, disregarding those not supported by Tcl, then add the charset to the content-type. I'll have to make a table mapping from HTTP encoding names to Tcl encoding names. Has someone already made this table?
I'm not entirely sure what to do with data streamed from disk or other channel. Unlike the string case, I don't positively know the source encoding. Neither can I convert it to the client's favorite encoding, nor can I correctly advertise the existing charset. For the moment I stream it unmolested and hope the browser guesses right. I guess a zone handler could be written to inject a charset according to user-supplied configuration options, then the send code can convert if the charset isn't among those accepted by the client.
Another issue: What do I do about characters not in the target charset? Better question: how do I know which characters aren't in the target charset? They get translated to "?", but so do real "?"'s, and thanks to variable-width encodings (e.g. utf-8), I wouldn't be able to reliably map from the encoded string back to the source, short of encoding one character at a time. Also, is "?" present in every charset? For HTML and maybe XML, I should be able to encode the foreign characters as HTML entities, if only I knew which ones needed to be encoded. It looks like I'm at the mercy of the [encoding] command here. Dear everybody, Please use Unicode! Love, Andy.
AMG: Several folks have suggested that Wibble has outgrown its current ad-hoc revision control and issue management system, which would be this wiki. I mostly agree. The current system requires a lot of manual labor, and I don't often have the time. Also I would like to expand the discussion, examples, documentation, etc. without feeling guilty that I'm hijacking somebody else's wiki. But I'm conflicted; there's a lot that I like about hosting Wibble on this wiki. In particular, it's very open; anyone can make changes and comments without getting an account or using a special client or anything like that. It's popular, it's maintained, it looks nice, it has a couple spiffy features, there are many great local pages to link to, other pages link back to Wibble, you can search page contents as well as titles, etc.
Fossil comes closest to the current system, and I am seriously considering it. In fact, I've already made a couple abortive attempts to import Wibble into a Fossil repository. The main thing that's stopping me is Fossil's markup interpretation for commit and ticket messages. If I could get a decoration mode instead of a markup mode, I'd be happy, but the current system requires a lot of work in order to get correct formatting (e.g. not losing square brackets, which all by itself is a deal-breaker). Also those workarounds break the command-line interface in order to get the common web interface pages right, and I will have to undo them again later when I finally get the decoration mode I need. (Tcl/Tk's own Fossil import has the same issues that Wibble does.)
Fossil has a built-in wiki which is nice for hosting discussion. I just played with the Fossil wiki a bit. It works, but it's as minimal as can be. However, it does support most HTML, so tables and such are possible. All in all, I'd prefer a few extra features, mostly to ease importing the pages and history from this wiki into Fossil. For instance, italic text and bold text are bracketed with <i>...</i> and <b>...</b> instead of double and triple apostrophes, respectively. Likely I will have to write a script to convert the markup. Also there's no double-bracket rule, so inserting brackets verbatim (which will be a very common task) requires that the brackets be spelled "&\#91;" and "&\#93;" (without the \'s; I had to insert those to dodge a bug in our wiki [L2 ]), or "<nowiki>[</nowiki>" and "<nowiki>]</nowiki>", which is even uglier. In fact, I think the lack of a double-bracket rule is also a deal-breaker, the same as the message markup problem described above.
Fossil also has a versioned documentation system, which makes it easy to get the documentation that matches any particular version of the code. That's a definite plus over the current ad-hoc system Wibble currently has, where you have to do some sleuthing to find the right version of the documentation pages. Of course, I haven't been good about updating the documentation anyway. ;^) But if I could version the documentation alongside the code, I would definitely make the effort to keep them in sync.
The Fossil wiki versioning is separate from the code/documentation versioning. They're on the same timeline, which makes correlation possible but not automatic. But they're still essentially separate. This creates a problem. I'd like for the documentation, examples, etc. pages to be in the wiki, open to editing and discussion. But for me to version them with the code, they have to be files. The only thing I can figure is to have manually synchronized parallel copies, both as files and as wiki pages. The wiki pages will only be regarded as documenting the latest version of Wibble, whereas the files are authoritative for whichever version of Wibble they came with. This seems like I'm trading one problem for another. Right now I have to tie all the pages together manually, but that's easier than manually copying everything between the wiki and files. I don't have to change the formatting of the files, but I might still want to, since if they're wiki-formatted then the user will need Fossil to properly read them.
I think what I need is a tool to extract a snapshot of the wiki, which I can commit as files. Maybe this tool can also prerender everything into HTML. If I am to open up the official documentation pages for editing and commenting, there will be discussion interspersed with normative verbiage. Most of the time when reading documentation you will want to skip the discussion, but it can be valuable on occasion. So it makes sense to be able to specially bracket the discussion and show/hide it using JavaScript/CSS trickery or simply by exporting two copies of each file, the official version and the annotated version.
Suggestions? Comments?
escargo 2011-03-19 - OK. These comments are from somebody sitting on the sidelines, but maybe they will be helpful, or at least thought-provoking.
Some of the problems you are having sound like features missing from Fossil. Perhaps they can be added. Where you say, "I will have to write a script...," that sounds like where a feature needs to be added to Fossil. Alternatively, maybe there needs to be a feature added to this wiki that makes exporting easier for other programs to import, an exchange format almost.
Where you mention, "manually synchronized parallel copies," that sounds like a need for a function that can link file and wiki page contents. (If I recall correctly, Fossil is really storing things in a data base, and probably exporting them or serving them as files or pages respectively as required. That would mean making Fossil understand some embedded query string that tells it to provide some tagged version of data as an exported file or current wiki page, if I understand correctly.) Perhaps the way to proceed is to enhance the underlying software so that moving to Fossil is less painful and that Fossil would better support how you need it to work. Your requirements might not be unique.
jnc: I don't know how you feel about Git, but I imported Wibble into a git repo: http://github.com/jcowgar/wibble . When working with GitHub, you have a decent bug tracker, a wiki and a **very** easy means to have user contribute changes/bug fixes.
dzach: Good to see that jnc. I added some code that I had hanging around for some time to the GitHub repo: gzip support, Cross-Origin Resource Sharing HTTP response headers [L3 ] , and a few more things. Hopefully this will be the new home for wibble's code from now on. AMG ?
AMG: Thanks for the show of support, but I still don't feel positively about officially hosting Wibble in git. Today I took another stab at getting Wibble into Fossil. I seem to have overcome the wikification problem for ticket titles and comments, but I still have to take a look at check-in comments. Also I need to find a way to semi-automate importing this wiki, or portions thereof, into the Fossil database. This is because this wiki serves as Wibble's documentation, and the documentation should be versioned and delivered alongside the source. Opening the versioned wiki (or a branch thereof) to editing via the web interface also remains an open issue.
jnc (Nov 25 2011): Just happy to see Wibble progressing and moving into any SCM system. Fossil is a good choice. About documentation, I am not sure what you mean. You wish to maintain documentation here on wiki.tcl.tk? Why not just make a doc/ directory in fossil and put the contents of the wiki pages into the doc/ directory? Update on the wiki, then when making a release, just wget/curl the wiki source pages you wish to include in the fossil repo?
AMG: I consider all this discussion to be part of the documentation. Yes, I plan to make a directory like you describe, but this wiki has a different format than that of Fossil. I need a conversion script, but none exists at the moment. This script needs to do more than simply convert formats; it must also be able to differentiate between internal links (pointers to other pages included in the Wibble archive) and external links (pointers to pages here in the Tcl Wiki). But yes, once I have that script, I plan to do just as you say.
jnc: I'd see maintaining documentation in two different formats and a conversion script as problematic. The fossil wiki format is very limited, making this conversion more difficult. Then each time either format changed or allowed new syntax, the script would be broken. I would choose just one format and stick with that. With fossil you do not have to use the wiki. You can disable access to the whole thing, to everyone. Or you can create a wiki page that simply points to the wiki here. It'd be a shame to have wibble progress waiting on a conversion script between incompatible wiki formats.