Version 0 of New Test Package

Updated 2004-08-19 15:21:46 by RHS

RHS Tcltest is an amazing little package for writing simple unit type tests. However, there are a lot of things that tcltest makes it very hard to do. There are other things that aren't hard to do, but make the tests that do them hard to read. I'd like to put together a list of desired functionality for a new test package, in hopes of eventually writing such a beast (or, someone else coming along and writing it). The things I'm looking for are:

• Simple to write simple tests, like tcltest. If I just want to call a proc and test the return value, it should be a very small bit of code to do so
• Things that aren't part of the actual testing framework should be in a seperate package. For example, things like makeFile and makeDirectory should be in a seperate package that comes with the framework.
• Testing the errorCode should be as easy as testing the return code
• Complex test cases should still be easy to read and understand.
• Assertions should be available, and the test cases should know that the test failed if any assertions failed
• It should be easy to say If this assertion fails, the test is finished, don't bother with the rest

 test myComplexTest-1.1 {
     A sample of a complex test, with comments
 } -setup {
     set data {a 1 b 2 c 3}
     set filename [extrapackage::makeFile $data myComplexTestFile-1.1]
     catch {unset myArray}
     catch {unset expectArray} ; array set expectArray $data
 } -body {
     set code [readFileToArray $filename myArray]

     assertEquals -nofail 1 $code "The read failed, don't bother with other assertions"
     assertEquals -nofail 1 [info exists myArray] "The array did not get created"
     assertArrayEquals expectArray myArray "The array results were incorrect"
 }

• Tests w/o a -result flag are assumed to return with a code of 0/2, and their actual result doesn't matter. The success/failure of such a test case (if it returns with code 0 or 2) depends on the assertions in the test
• It should be possible to create "Test Suites" that group a bunch of tests/test files together
• It should be possible to programatically run a test, a whole file of tests, a test suite, a while directory of tests
• It should be possible to retrieve the results of programatically run tests

I'm sure I can come up with other requirements. More importantly, though, is... what would other folks want in a testing package?