2008-02-24
1. I keep a large collection of text data as a list in memory (lappend x {text...} etc)
2. I want to search this data.
Here's what happens.
set match [lsearch -regexp $x {needle}]-> memory usage of the tcl process more than doubles (before: 80MB, after: 200MB)
foreach k $x { if {[regexp -nocase {needle} $k]} {puts "match"} - >ditto
foreach k $x { if {[regexp -nocase {needle} [list $k]]} {puts "match"} -> Heureka! total memory usage stays at 80MB.
I'm still not quite sure what's going on, it's about keeping lists 'pure' I guess. I'm now consulting these pages:
6am EDIT: I think I almost get it now. regexp treats $x as string and forces every element into a string representation AS WELL as a list representation. sigh. lsearch is forcing a string representation of all elements. seems unavoidable. I could also lappend items as strings not lists, but I'm saving my data as TCL source for various reasons, and of course { } is much cleaner in that case.
-hans
I see a common misconception about Tcl here. Please notice using curly braces to quote a command argument *does not* turn the contents of the {...} into a list. IOW, your lappend x {text...} etc should result in exactly the same value as lappend x "text..." etc. As to the original issue of memory growth, I would be curious to read what the core maintainers have to say. Also interesting would be a -glob search. - RT 24Feb08
Lars H: Are you taking about a the object getting a string representation, or getting a String internal representation? (They're not the same.) For
foreach k $x { if {[regexp -nocase {needle} [list $k]]} {puts "match"} to make a difference, it seems it'd have to be the latter (in order to get a string rep for [list $k], one first needs the string rep of $k, but since [list $k] is not the same Tcl_Obj as $k, an intrep imposed upon [list $k] by regexp will not be retained in the elements of $x); that String intreps use 2 bytes for every character is consistent with a jump from 80MB to 200MB, if you have about 60M ASCII characters without intrep in your big list.
I once had a similar difficulty, but the other way round, with a program that dumped a list of lists of integers to a text file. In order to generate a stringrep for a list of integers, Tcl first had to generate stringreps for all the integers, and that similarly doubled the memory usage during the final dump. In that case I solved it by feeding each integer through format before I did anything stringy to it, since that gave a Tcl_Obj that wasn't shared with the big list of lists of integers...
See also: