Version 0 of How To Read Large Strings without Newlines

Updated 2011-06-01 21:58:50 by XAN

I've run into a kind of surprising limitation(?) and it has me stumped. Maybe someone else here has done something similiar?

The Problem

I am attempting to use TclCurl to have a "conversation" with an ASP driven webpage. A little background... these are corporate in-house webpages over a VPN and there is little to no hope of getting them to change anything. And there is a major design flaw in these webpages - the designer has decided to use the __VIEWSTATE field to save state for everything... including the kitchen sink! The result...

The stumbling block is a __VIEWSTATE field that is approximately 500K large! To communicate with the server, I have to capture this awful chunk of data and send it back in the POST request. I have code that does this, and I know that it works because it is heavily tested on pages with a sane-sized __VIEWSTATE. Trying to work with these pages, however, just causes TCL to hang.

Possible Cause

I have no problem reading a file megabytes large into a variable, so somothing else is going on here. I looked for "offensive" characters that may be choking the interpreter but none seem to exit. The strking detail seems to to me to be the absense of newlines. The ~500K is all on one line... could it be that TCL cannot handle this scenario, or handle it well?

Things I've Tried and The Outcome

  1. I've taken a chunk of about 10K and tried building a 500K string with no newlines using append. The interpreter does get slower and slower and eventually grinds to a halt. When doing this with newlines, no slowdown occurs.
  2. I've tried every possible method of reading this data into a variable...
  • different encodings and translations on fconfigure -> read into variable
  • set variable directly using copy/paste into console
  • breaking up into chunks and rebuilding these chunks into one variable
  1. I've looked for a way to have TclCurl pull the postfields from a file. No such method seems to exist - it must come from a variable.
  2. I've tried using external options like CMD.exe type filename. Same problem. Interestingly, CMD.exe has no problem typing this file out in a CMD.exe instance.

*Questions*

  1. Why can't TCL handle this data?
  2. What can I do? Anyone have any suggestions or similiar experience?