Version 1 of Compact data storage

Updated 2002-11-11 22:19:08

Purpose: To collect and discuss programming techniques for avoiding unnecessary allocation of memory. Example: Decide whether a value is a string or a list, and treat it accordingly. If you treat it as both, it will become both a string and a list, and that means your data is stored twice.


Some hard results on how many bytes Tcl uses for various kinds of things can be found on Memory costs with Tcl.

Some scripts that can be used for measuring how much memory is allocated by a sequence of commands can be found on Measuring memory usage. For example the growth and Delta procedures used below can be found on that page.


100000 random digits

is the name of a classical table of random numbers -- depending on one's point of view, it is either one of the most boring books ever published, or one of the least boring books ever published (since you never can tell what is going to happen next) -- but it also makes a nice example. Suppose you want to make a list of 100000 (pseudo-)random digits. The obvious way of doing this is

 set L [list]
 for {set n 0} {$n<100000} {incr n} {
    lappend L [expr {int(floor(rand()*10))}]
 }

and the test

 Delta [growth {
    set L [list]
    for {set n 0} {$n<100000} {incr n} {
       lappend L [expr {int(floor(rand()*10))}]
    }
 }]

will tell you that this causes Tcl to allocate 3207168 (or thereabout) bytes. However, since equal values will appear over and over again in this list, it is more efficient to make sure that they really are the same (as objects) values. One way of doing this is

 set L [list]
 set digits [list]
 for {set n 0} {$n<10} {incr n} {lappend digits $n}
 for {set n 0} {$n<100000} {incr n} {
    lappend L [lindex $digits [expr {int(floor(rand()*10))}]]
 }

The test

 Delta [growth {
    set L [list]
    set digits [list]
    for {set n 0} {$n<10} {incr n} {lappend digits $n}
    for {set n 0} {$n<100000} {incr n} {
       lappend L [lindex $digits [expr {int(floor(rand()*10))}]]
    }
 }]

returns (for me) a mere 798720 bytes. The number of correct digits in these measurements is probably rather low, but the difference in magnitude is quite significant. -- Lars Hellstr�m