[AMG]: This page documents bugs in the [Wibble] web server, as well as bugs in [Tcl] that affect Wibble. There may be some overlap with the [Wibble discussion] page. ---- **[[getline]] line limit check** [APN] Does the getline proc have a bug in its line limit check? It appears it will raise an error if the ''total'' buffered input exceeds maxline, as opposed to individual lines exceeding that limit. For example, if it receives 50 lines of 100 chars each in a single network chunk, it will raise an error when it should not. [AMG]: I think you're right, but at the moment I don't have the means to test. If a client sends a bunch of short lines while the server is taking a nap, when the server wakes, it will act as if the client had sent one long line. For some reason I had thought [[[chan pending]]] would tell me how many characters would be in the next line returned by [[[chan gets]]], but here this is only the case when there's at most one line per packet. By "packet" I mean the batch of input characters which triggered the readable event. What's the right way to handle this? When I detect that the next [[gets]] ''may'' exceed $maxline, I could instead do a [[[chan read] [[expr {$maxline + 1}]]]] and check for newline. If there's a newline, everything up to the (first) newline is the next line to process, and the newline itself is discarded. But what do I do with the rest of the input? What has been read, cannot be unread. Basically I would have to move the input buffering facility from the Tcl I/O subsystem into my own script. Things get really nasty when the read contains both HTTP header and body text, because the header text needs CR/LF translation and the body text does not. I can't put CR's back into the body text, because I don't know if they were there originally. So I would not be able to use Tcl's CR/LF translation for the header; instead I'd need to strip CR's in the script. I think the goal of [[chan pending]] was to avoid having to do this, because the approach I have described is the same as what is required in the absence of [[chan pending]]. [APN] How about doing the chan gets before the chan pending? Something like - ====== proc wibble::getline {chan} { while {1} { if {[chan gets $chan line] >= 0} { # Got a whole line, may be more than 4096, but that's ok return $line } elseif {[chan blocked $chan]} { # Partial line, see if max length exceeded if {[chan pending input $chan] > 4096} { # Be anal. Do a gets again since data might have come in # between the chan gets and the chan pending calls. # For example, there are 4000 bytes without crlf when # the chan gets was called. Then another 1000 bytes # arrive with a crlf at the front before the chan pending # call. The line length limit is not exceeded in that # case even through chan pending returns > 4096. if {[chan gets $chan line] >= 0} { return $line } else { error "line length greater than 4096" } } else { # Incomplete line, but not exceeding max length. Wait for more. yield } } else { # EOF chan close $chan return -level [info level] } } } ====== ---- **Invalid command name process, C stack busy** [APN]: When run against Tcl built against CVS head as of 12/18/2009, wibble fails with the error "invalid command name process". The fix is to change the coroutine call in wibble::accept to pass in [[[namespace current]]]::process instead of process. The question is whether this change in coroutine command in how it resolves names was intentional or not. [AMG]: That's odd. [[namespace current]]::process works fine, but [[[namespace code] process]] does not. When I try the latter, I get "cannot yield: C stack busy". [MS] (after today's fix, but it shouldn't matter) % proc a {} {yield; return hello} % coroutine foo {*}[namespace code a] % foo hello [APN] The current coroutine docs state '''At the point when command is called, the current namespace will be the global namespace and there will be no stack frames above it (in the sense of upvar and uplevel). However, which command to call will be determined in the namespace that the coroutine command was called from.''' Based on this I would presume the original code should have worked as well (without qualifiers) since the coroutine command should have resolved '''process''' to be '''wibble::process''' based on the namespace it was called from. This behaviour seems to have changed very recently, perhaps the documentation has not been updated? [MS] 2009-12-19 looks like [[bug #2917627]] [https://sourceforge.net/tracker/?func=detail&atid=110894&aid=2917627&group_id=10894], fixed in head. [APN] Confirmed. [AMG]: MS, thanks for your fix. I reverted my workaround, since it's no longer needed. New topic: Why does [[namespace code]] add a frame to the C stack? ---- **Wibble hangs on IE8 POST** [dzach] 2010-05-12: I've encountered an odd problem with proc ''getblock'', which hangs wibble when sending a POST request from an IE8 in Windows XP: the ''append chunk $chunklet'' line seems to be failing to initialize variable ''chunk''. IE8, in my configuration, sends the POST content in two packets, the first of which is empty (size is 0). When this happens wibble hangs for ever. Firefox sends the POST content in one packet and the problem does not appear. Initializing ''chunk'' outside the ''while'' loop with: ''set chunk ""'', solves the problem. In the test setup, Wibble runs on Linux with Tcl8.6b1.2. [AMG]: That is very odd. I should get IE8 for myself to test, but I don't have it right this minute. Two-argument [[[append]]] always creates the variable if it doesn't exist, regardless of its second argument. In tclExecute.c, doStoreStk is part of the implementation of compiled append, and it calls TclObjLookupVarEx() with the "create" flags set.h. For the sake of argument, let's say that [[append]] is buggy and fails to create chunk when chunklet is empty. $size should be nonzero in this case, and [[chan eof $chan]] is false because IE8 hasn't closed the channel yet, so [[[yield]]] is executed. Nowhere in this code path does chunk's existence matter, and there's no reason why pre-creating chunk would have an effect. The little information available suggests there's a bug in Tcl, but I can't say for sure. Please put in some debug prints to stderr. In particular, I want prints before and after the yield, to see which branch of the [[[if]]] is being taken and whether the coroutine is ever resumed. Do this both without and with your ''set chunk ""'' workaround. Oh, another question: When Wibble hangs, does it take up 0% or 100% CPU time? [dzach]: The tests run with utf-8 encoding. I noticed that when I insert the ''puts stderr $chunklet'' line, the problem does not appear. If I take out that line, I reproduce the problem. CPU load is 100%. Here come the tests: # test proc proc ::wibble::getblock {chan size} { puts stderr size=$size while {1} { set chunklet [chan read $chan $size] puts stderr "{$size - [string length $chunklet]}" set size [expr {$size - [string length $chunklet]}] append chunk $chunklet if {$size == 0} { puts stderr "return size==0" return $chunk } elseif {[chan eof $chan]} { chan close $chan puts stderr "return eof" return -level [info level] } else { puts stderr "before yield" yield puts stderr "after yield" } } } # output size=56 {56 - 0} before yield after yield {56 - 56} # test proc (work-around) proc ::wibble::getblock {chan size} { puts stderr size=$size set chunk "" while {1} { set chunklet [chan read $chan $size] puts stderr "{$size - [string length $chunklet]}" puts stderr chunklet=$chunklet set size [expr {$size - [string length $chunklet]}] append chunk $chunklet if {$size == 0} { puts stderr "return size==0" return $chunk } elseif {[chan eof $chan]} { chan close $chan puts stderr "return eof" return -level [info level] } else { puts stderr "before yield" yield puts stderr "after yield" } } } # output (work-around) size=56 {56 - 0} chunklet= before yield after yield {56 - 56} chunklet=name=&geometry=38.079226,23.930992&description=&dtstart= chunk=name=&geometry=38.079226,23.930992&description=&dtstart= return size==0 [AMG]: I created a simple method="post" form, installed IE8, and submitted data several times. IE8 always sent the full form data as a single packet, with no incomplete or zero-sized reads. Is there anything special about your form that's making IE8 do what it does? I'm running the Tcl CVS HEAD compiled earlier today (12 May 2010, ChangeLog revision 1.5108), on [Slackware] 13.0. How about you? Also, please explain why you stated that you used UTF-8 encoding. At present, Wibble is only designed to work with ISO8859-1, because it doesn't have the smarts to figure out what encodings are acceptable to the client, and ISO8859-1 is the legal fallback in that case. (I have a project to fix this, but it's stalled due to lack of time.) How is it that you're using UTF-8? If simply [puts]'ing the value of $chunklet makes the problem go away, most likely there is a Tcl bug. Please explore this a little more. What happens if you take the value of $chunklet and ignore it? For example, "list $chuklet". If adding that makes the problem go away, something is definitely going wacky under the hood, and you should file a bug report. Or maybe it's the puts that's having an effect, and it doesn't matter what text is being printed. Again, that's a Tcl problem, since the output channel is unrelated to network communication with the client. 100% CPU load suggests that [[[chan read]]] is returning empty string, yet the channel is still readable and [[[chan event]]] keeps calling into the [coroutine]. That doesn't make sense to me. I see one more oddity I can't explain. Your debug print shows geometry to be "38.079226,23.930992", yet both Firefox 3.6.3 and IE8 encode this as "38.079226%2C23.930992". What's going on here? Why isn't your comma also encoded? By the way, this page is getting too long for my liking. I wish some benevolent [wikignomes] would clean it up for me! :^) [dzach]: Wibble, in my tests, substitutes a hand crafted -for speed- web server, and runs on a Kubuntu (2.6.32-22 kernel) with tcl from CVS HEAD (changelog 1.5098/Sat May 1 11:20:12 2010). IE8 is on the same machine running windows XP in a Sun's VirtualBox. The POST is sent using a typical javascript AJAX XMLHttpRequest: `req.setRequestHeader("Content-type", "application/x-www-form-urlencoded; charset=utf-8")`. In my narrow scope application, Wibble serves UTF encoded (mostly dynamic) content (in Greek) and user POSTed submissions get decoded inside a modified `wibble::process` using `encoding convertto UTF-8 [[dict get $response content]]` (could as well do it outside Wibble, since all this is done after `getblock` finishes reading the channel). I suspect UTF-8 might have something to do with the problem. The POST content send by the browser using AJAX is encoded with javascript's ''encodeURI()'' which leaves some characters (, / ? : @ & = + $ #) unencoded. I'll check if something is missing or if changing this makes a difference. I'll also try to run IE8 on another machine using a native windows XP installation. (A little later): IE8 hangs the server from a native XP installation too. Changing the ''encodeURI'' to ''encodeURIComponent'' (which encodes comma too) has no effect to the result. Away from the UTF issue, it seems that the problem is cured as soon as ''chunklet'' gets "stringified". I guess it would be worth while to test if ''chan read'' returns a string type when it reads nothing. Having said that, initializing ''chunklet'' instead of ''chunk'' doesn't help, so my guessing might be wrong. [AMG]: dzach, thanks for your testing. Please keep us posted. If anyone has a clue what might be going on here, please jump in and give me a hint, 'cuz I'm mystified. :^) [dzach]: I like Wibble's Zen, so testing it is fun, given the existance of a work-around for this problem. If I get the time, I'll try to write a simple AJAX test so that others can reproduce it. [dzach] 2010-5-17: The problem appears with an unmodified Wibble (ISO8859-1 encoding), ruling out a UTF-8 involvement. To reproduce the error, use the code in http://paste.tclers.tk/2087 and Internet Explorer 8. [AMG]: Works fine for me. Am I doing something wrong? ;^) I get the same behavior for both Firefox and IE8: ====== size=22 {22 - 22} return size==0 ====== Tonight I plan to cobble together a HTTP client that will split its POSTs over two packets, just to see if this really is the cause of the problem. [AMG]: Looks like I haven't found time to do this, yet. [AMG], update: This appears to be a Tcl bug, which I have now reported on Sourceforge [http://sourceforge.net/tracker/?func=detail&aid=3091701&group_id=10894&atid=110894]. Thanks [dzach] for bringing it to my attention! It sure took me a long time to reproduce it. [dzach] I'm glad to see the bug nailed. [AMG]: The bug is fixed in the Tcl CVS HEAD, as of 2010-09-15. See [http://sourceforge.net/tracker/?func=detail&aid=3067036&group_id=10894&atid=110894]. Thanks [DKF]! <> Wibble