testing

Tcl has long traditions in testing [put in some general-purpose testing reference here--Beizer, ...]. Especially notable are Expect, Oracle, Sybase, NASA , and other projects.

See Also

Generating random tests
Another good page on testing.
opensourcetesting.org
MR A good directory of systems... which unfortunately only lists 3 tools for doing tcl unit testing...
IEEE 829 Standard for Software and System Test Documentation
Stress testing
Techniques for 'driving' Windows applications
Top Five (Wrong) Reasons You Don't Have Testers ,Joel Spolsky ,2000-04-30

Quotes

“Program testing can be used to show the presence of bugs, but never to show their absence!”
Edsger Dijkstra

Tools

AutoIt
A Windows-specific DLL that can simulate key presses, mouse movements, and so on. MPJ: I added the Ffidl wrapper code to access all the AutoIt functions.
Eggplant, by Redstone Software
Addresses the same need in the Mac OS X world.
Faking It in Tcl
FIT
Jacks
A Java compiler testing framework that was developed for open-source Java compiler projects because the official Sun development kits have not been released. Jacks rivals the Sun kits in capability, and may soon even have the same number of compatibility tests.
cwind
FDR2
According to Stephan Kuhagen (of FDR on comp.lang.tcl) this "makes heavy use of Tcl/Tix".
HATE
Harness for Algorithm Testing and Evaluation. Can do both regression tests and statistical analysis of different algorithms applied to accomplish a single task.
tcltest
Distributed with Tcl, Used for testing Tcl itself.
TesTcl
A Tcl library for unit testing iRules which are used when configuring F5 BigIP devices. The goal of this library is to make it easy to unit test iRules used when load balancing HTTP traffic.
TETware
Test Execution Management System.
tclwebtest
For testing web applications.
Pipeline programming
SimpleTest
A tcltest work-alike SimpleTest is a tcltest work-alike for projects using The Simple Library.
STAF (Software Testing Automation Framework]
Includes a Tcl API.
tclunit
A little GUI for driving tcltest suites.
TkXext
WinTclSend
zzuf
A transparent application input fuzzer. changes "random bits in" the file and socket accesses it detects on its own. There's no particular connection to Tcl, I believe, except that combines with it nicely.

Case Studies

The GNU C Compiler
tomk: uses Tcl scripts and Expect to do regression testing.
COMPANY: Deutsche Börse Systems
OpenCascade
The Open CASCADE 3D modeling application development platform relies on Tcl for its testing framework [L1 ].
QuaSR
testing system for Sybase. One of the largest Tcl code bases.

Description

JCG writes: "All of tcltest, simpletest and DejaGNU have a similar goal, IMHO: that of providing a /framework/ for testing, although both tcltest and simpletest are conceived for Tcl testing while DejaGNU seems more suited towards C/C++. None of them provide a full fledged environment useful for non-programmers, though."

And don't forget about the tcltest package, which is a test harness for writing and running tests in the Tcl test suite or for creating a customized test harness for an extension. It includes both documentation and a huge set of examples (the Tcl and Tk test suites).

RS: A basic testing skeleton (further developed in e.g.) might look like this:

proc test {command expected} {
    catch {uplevel 1 $command} result ;# so we can use variables in caller's scope
    if {$result ne $expected} {error "$command->$result, expected $expected"}
}

Instead of error, you might use puts, so the test suite doesn't stop at the first surprise. Example for testing the test:

test {expr {2+3}} 5

The Suitability of Tcl for Testing

In response to a question about testing C and Java from Tcl, Cameron Laird responded eloquently:

"Tcl is the single most successful language for test automation of the kind you're describing. That's so undeniable that I'd forgotten it deserves to be documented for new generations of developers. DejaGnu is the testing framework that's made gcc possible. Companies like Oracle and Sybase have MILLIONS of lines of Tcl testing code on which they rely to assure the operation of their 'flagship' products.

Given enough motivation, I could do a whole book on use of Tcl in testing. Tcl tests automobile engines, emergency telephone circuits, chemical sensors, microprocessors, rocket components, industrial ovens, and much, much more. In the absence of any other knowledge, ANY software project should think of Tcl as its first choice for testing."

What more do you need?

RLH How about the book? : )

Dart

The original Dart framework was written in Tcl. Recently, it was rewritten in Java.

The Dart framework produces the pretty test status dashboards that one sees for Vtk. More information about Dart is at http://public.kitware.com/Dart/HTML/Index.shtml . Some examples of Dart's output are at [L2 ] and [L3 ]. Dart, like Vtk, is open-source.

The CVS repository for Vtk is

cvs -d :pserver:[email protected]:/cvsroot/VTK checkout VTK

Password is "vtk".

The repository for Dart is

cvs :pserver:[email protected]:/cvsroot/Dart checkout Dart

Password is "dart"

Techniques

Test in a simulated environment that makes all external effects deterministic. See discrete event simulation
Execute the program in a single-threaded mode for testing.
Produce multiple implementations of a function, and test that the produce substantially similar results.
Introduce errors and challenge the testing suite to find them.
Create a test case generator that produces tests a fuzz tester can use for reference.
Fuzz test with as much tracing as possible enabled in order to quickly see where things are going wrong.
test a variety of possible inputs, particularly the edge cases. See mutation testing .

Articles

Articles:

  • CL's article about Mo DeJong, detailing the benefits of a strong regression test policy. [L4 ]
  • Michael J. Norton's "Network Test Automation with Mac OS X and Tcl" [L5 ]
  • Ahmet C. Keskin, Till I. Patzchke, and Ernst vonVoight. "TclTk: A Strong Basis for Complex Load Testing Systems." Proc. 7th Intl. Tcl/Tk Conf. Austin, Texas: USENIX, 14--18 February 2000, pp. 53--60. [L6 ]
  • Carsten H. Lawrenz and Rajkumar Madhuram. "Using Tcl to Build a Buzzword-Compliant Environment That Glues Together Legacy Analysis Programs" Proc. 7th Intl. Tcl/Tk Conf. Austin, Texas: USENIX, 14--18 February 2000, pp. 61--70. [L7 ]
  • Paul Amaranth. "A Tcl-based Multithreaded Test Harness." Proc. 6th Intl. Tcl/Tk Conf. San Diego, California: USENIX, 14--18 September 1998, pp. 69--78. [L8 ]
  • C. Allen Flick and S. Dixson. "Using TCL/TK for an Automatic Test Engine." Proc. 6th Intl. Tcl/Tk Conf. San Diego, California: USENIX, 14--18 September 1998, pp. 79--88. [L9 ]
  • David E. Smyth. "Tcl and concurrent object-oriented flight software: Tcl on Mars." Proc 2nd Tcl/Tk Workshop. New Orleans: CPU, 23--25 June, 1994, pp. 3--10. [L10 ]
  • Arjen Markus' paper "Generating test programs with TestMake" [L11 ] Dead link US
  • ...
  • [ Android, ...]
  • ...
  • http://www.commsdesign.com/design_corner/OEG20020123S0058

'Nother testing project: describe a Tcl perspective on

  • unit
  • component
  • system and
  • functional

testing, then do the same for

  • standard conformance
  • dependency analysis
  • coverage
  • complexity

then integrate stress testing and other performance metrics.