Richard Suchenwirth - Robust software should withstand all kinds of mistreatment (calling with improper arguments) without ending up in core dump, segmentation or "general protection" faults. One way to test this property is made easy by Tcl's info commands that lists everything you could call - except for procedures from not yet sourced files, but these are a subset of [array names ::auto_index].
The following proc executes a stress test by calling (almost) all commands with varying nonsense arguments. Errors are expected (and caught), but execution of this proc should not result in a crash. If such dangerous commands are found, edit them into the taboo list, which initially contains only exit and stresstest itself.
proc stresstest {} {
rename exit tcl::exit
proc exit args {return -code 111} ;# prevent early death
set taboo {exit stresstest vwait}
#-- add known troublemakers here until the script terminates
lappend taboo
#-- make sure all indexed commands are in view
foreach i [array names ::auto_index] {
puts "*** $i"
catch {$i} res
puts $res
}
foreach i [lsort [info commands]] {
if {[lsearch $taboo $i]>=0} continue
# Richard, are you tempted to define a little pattern
# language, so the following inline code would collapse
# to
# foreach pattern {{} foo {foo bar} ...} {
# eval catch [list $i $pattern] res
# puts "*** $i $pattern"
# }
# ? The presumed attraction is that it'd encourage
# composition of new patterns.
puts "*** $i"
catch {$i} res
puts $res
puts "*** $i foo"
catch {$i foo} res
puts $res
puts "*** $i foo bar"
catch {$i foo bar} res
puts $res
}
puts "successfully passed stress test"
} A standard tclsh passes this test. It gets interesting if you have your own extensions, which may not be as robust as Tcl itself is...
This is an analogue to fuzz ([L1 ], Debian package [L2 ], citations [L3 ]). -- RS grins at how easy it is in Tcl to reinvent wheels (which still fit between thumb and index finger ;-)
Endless loops (well, potentially..) can also be found in stress testing. The innocent command
for {set i 0} {$i<$max} {incr i} {...}will never terminate (by itself) if max is a not-numeric string whose first character sorts behind "9", because the end test is done as string comparison if not both arguments are numeric. This may happen when max is a parameter passed in, and not tested to be numeric. For integers (using doubles in for loops could cause a real problem), this test is easily done with
incr max 0
before the loop - does nothing if max was an integer, and raises an error otherwise.
The stress test cannot detect such subtle bugs by itself, so best don't redirect its stdout to a file and walk away for the night...
Another idea I had last night: for stress testing a proc, insert deliberate errors (e.g. error "stress 1001") in all branches of if,for,foreach,while.. of that proc; run test suite, catch errors, remove them from instrumented proc, until no more errors are caught. If the proc still has such fabricated errors, the test suite isn't complete.