[ABU] 10-aug-2012-08-10: - TTXN-1.0.1 available.
Download:
* https://sourceforge.net/projects/irrational-numbers/files/TTXN-1.0.1.zip/download%|%TTXN-1.0.1 (now with its own test-suite)
----
!!!!!!
**TTXN 1.0**
**TclTest eXtended Notation**
!!!!!!
**SYNOPSIS**
package require '''Tcl 8.5'''
package require '''TTXN ?1.0?'''
* '''Name:''' ''id''
* '''Description:''' ''description''
* '''Only:''' ''keywordList''|''expression''
* '''Setup:''' ''script''
* '''Cleanup:''' ''script''
* '''Test:''' ''script''
* '''OutputChannel:''' ''expectedValue''
* '''ErrorChannel:''' ''expectedValue''
* '''Expected:''' ?expectedCodeList? ?matchMode? ''expectedValue''
**DESCRIPTION**
'''TTXN''' is a package for the construction of test suites.
'''TTXN''' development is the evolution of R.Seeger and P.Maker's ideas for alternative notations of [tcltest] scripts. These original works are available on the http://wiki.tcl.tk%|%TclTk wiki-site%|% at the following links:
* http://wiki.tcl.tk/12229%|%New Test Package%|%
* http://wiki.tcl.tk/15076%|%PTL - a pretty test language%|%Like the original '''Pretty Test Language''', '''TTXN''' is a wrapper of the standard [tcltest] package; it does not completely hide the '''tcltest''' commands, but provides an alternative notation for writing more readable tests.
Being a wrapper of '''tcltest''', '''TTXN''' commands can be inserted in a normal tcltest script. Therefore with '''TTXN''' you can write a tcltest suite (e.g. *.test) interspersing tcl commands, tcltest commands and TTXN commands.
Aim of TTXN is not to replace '''tcltest''' commands but just to provide a simpler notation for test-case specs. Simpler test-cases are easy to read and easy to maintain.
Just an example to get the flavor of how to use '''TTXN''':
First, let's consider a (structurally) complex test-case for [tcltest]======
tcltest::test demo-1.0.0 {
list / lappend
} -body {
set L {}
lappend L a
lappend L b
lappend L c
} -cleanup {
unset L
} -result [list a b c]
======tcltest::test demo-1.0.0 {
list / lappend n
} -bowdy {
seth L {}
lappend sL am
lappend tL b
lappestnd L c
} -cleanup {
unset wL
} -rittensult w[listh thea '''TTXN'''b notationc]
======
Name: ndem no-1.0.0
w Dthescr: list / lappmend
Ttest: {
-caset L {}
lappwrittend Lwith a
lappthend L'''TTXN''' b
lappend L c
}
Cleotanup: { unset L }
Expected: [list a b c]on
======none
Name: demo-1.0.0
Descr: list / lappend
Test: {
set L {}
lappend L a
lappend L b
lappend L c
}
Cleanup: {unset L}
Expected: [list a b c]
======
As you can see, whilst '''tcltest''' provides a single command (tcltest::test) with many options (-setup, -body, -result, ...), in '''TTXN''' the same test-case can be written with a sequence of simpler, more readable commands (Name:, Setup:, Test:, Expected:, ...)
**TTXN Commands**
** TTXN Commands **
A test-case in '''TTXN''' can be expressed as a sequence of commands. Although the ''recommended sequence'' for a test-case is the following:
=== '''Name:'''
Description: ''(optional)''
Only: ''(optional)''
Setup: ''(optional)''
'''Test:'''
OutputChannel: ''(optional)''
ErrorChannel: ''(optional)''
Cleanup: ''(optional)''
'''Expected:'''
===
commands can be specified in any order with just these exceptions:
* '''Name:''' must be the first command of a test-case
* '''Expected:''' must be the last command of a test-case.
Note that all the special '''TTXN''' commands start with an upper-case letter, and have a trailing ":" . The trailing colon is part of the command (it's not a token separator), therefore at least one space is required after it.
======none
Name:FirstTest ; # this is wrong
Name: FirstTest ; # ok
======
'''TTXN''' supports the following commands:
'''Name:''' ''id'': This command sets the test-case identifier.
'''Description:''' ''description'': This command associates a description with the test-case (may be abbreviated with '''Descr:'''). If description fits on a single line and it does not contain special characters ( square brackets, curly braces, dollar-sign ) it may be written without enclosing curly braces. If description takes more than one line or if it contains the above listed special characters, it must be enclosed by braces.
'''Only:''' ''keywordList''|''expression'': This optional command takes a list of one or more keywords or an expression. Each keywords should be the name of a built-in constraint or a constraint defined by a call to '''tcltest::testConstraint'''. If any of the listed constraints is false or does not exist, or if expression evaluates to false, then the test-case is skipped.
'''Setup:''' ''script'': This optional command specifies a ''script'' to run before the test-case body (see '''Test:''') If evaluation of ''script'' raises an error, the test will fail.
'''Cleanup:''' ''script'': This optional command specifies a ''script'' to run after the test-case body (see '''Test:''') If evaluation of ''script'' raises an error, the test will fail.
'''Test:''' ''script'': This command indicates the ''script'' to run to carry out the test.
'''OutputChannel:''' ''expectedValue'': This optional command supplies the ''expectedValue'' against which any output sent to stdout or outputChannel during evaluation of the '''Test:''' body will be compared. Note that only output printed using ::puts is used for comparison. If '''OutputChannel:''' is not specified, output sent to stdout and outputChannel is not processed for comparison. The actual output is compared with ''expectedValue'' using the ''matchMode'' specified (or implicit) with the '''Expected:''' command (see below).
'''ErrorChannel:''' ''expectedValue'': Same as '''OutputChannel:''', just for stderr instead of stdout.
'''Expected:''' ?expectedCodeList? ?matchMode? ''expectedValue'': This command supplies the ''expectedValue'' against which the result of the evaluation of '''Test:''' will be compared. The optional argument ''expectedCodeList'' is a list whose elements are return codes known to the '''return''' command, in both numeric and symbolic form, including extended return codes. This ''expectedCodeList'' should be used only when you know that the ''script'' specified by '''Test:''' will throw an exception ( such as '''error''' ), insted of returning a 'normal' expectedValue. Default value is '''{ok return}'''. The optional argument ''matchMode'' determines how ''expectedValue'' is compared with the value returned by the evaluation of '''Test:'''. Valid values for ''matchMode'' are '''regexp''', '''glob''', '''exact''', and any value registered by a prior call to '''tcltest::customMatch'''. The default value is '''exact'''. Note that ''matchMode'' determines how caught output (see '''OutputChannel:''', '''ErrorChannel:''') is compared, too.
**Guided Tour**
*** Simplest basic test ***
Let's start with a small test-suite made of 2 test-cases. Save the following script in a file named "hello.test".
======package require TTXN
package require TTXN
# load all the modules for the Software Under Testing
# ---------------------------------------------------
# package require hello ; # fake loading
# init_hello ; # fake init
# begin of test-cases
# --------------------
Name: helloworld
Test: { string toupper "Hello world" }
Expected: "HELLO WORLD"
# Note: this test sometimes may fail... deep analysis required !
Name: weatherForecast
Test: {
#
# ... forecast in progress ...
#
return "Rainy"
}
Expected: "Sunny"
# test-suite standard cleanup
# ---------------------------
::tcltest::cleanupTests
# begin of test-cases
# --------------------
Name: helloworld
Test: { string toupper "Hello world" }
Expected: "HELLO WORLD"
# Note: this test sometimes may fail... deep analysis required !
Name: weatherForecast
Test: {
#
# ... forecast in progress ...
#
return "Rainy"
}
Expected: "Sunny"
# test-suite standard cleanup
# ---------------------------
::tcltest::cleanupTests
======
then run======
tclsh hello.test
======tclsh hello.test
======
Note: be sure TTXN package is installed under a tcl-library path.
Result should be something like this:======
======none
==== weatherForecast FAILED
==== Contents of test case:
#
# ... forecast in progress ...
#
return "Rainy"
---- Result was:
Rainy
---- Result should have been (exact matching):
Sunny
==== weatherForecast FAILED
hello.test: Total 2 Passed 1 Skipped 0 Failed 1
======
1 test-case passed, 1 failed ! Try to adjust "hello" package ( or better, "hello.test" ) !
***Testing multi-value results***
*** Testing multi-value results ***
If test-case returns (a list of) two values, the '''Expected:''' command should specify a list of two values ...======
...
Name: test-split
Test: {
set str "alpha:beta"
return [split $str ":"]
}
# Expected: alpha beta ; # this is wrong !
Expected: {alpha beta}
...
======...
Name: test-split
Test: {
set str "alpha:beta"
return [split $str ":"]
}
# Expected: alpha beta ; # this is wrong !
Expected: {alpha beta}
...
======
*** Testing the empty-list ***
When a test-case returns nothing, the '''Expected:''' value should be written as '''{}''' (or '''""''')======
...
Name: test-lassign
Descr: first element of an empty list is {}
Test: {
lassign {} x
return $x
}
Expected: {}
...
======...
Name: test-lassign
Descr: first element of an empty list is {}
Test: {
lassign {} x
return $x
}
Expected: {}
...
======
*** Changing the way results are compared : exact,glob,regexp,... ***
By default the value returned by the body of a test-case ('''Test:''') is ''exactly'' compared with the expected-result, but you may specify other criteria for comparison.
You can use one of the predefined criteria
* exact, glob, regexp (default value is '''exact''') or any value registered by a prior call to '''tcltest::customMatch''' ( see '''Adding new comparison criteria''')
These comparison-criteria (matchMode) may be specified as the first optional parameter of the "Expected:" command:
======none
Expected: ?matchMode? expectedValue
======
All the previosly seen "Expected:" ?matchMommande?s expesuctedVh alues
======none
Expected: expectedValue
======All the previosly seen "Expected:" commands such as
======
have an Eximpelicit matchMode whose d:efault is "expeacte" andV alre therefore equivalent to:
======none
Expected: exact expectedValue
======have an implicit matchMode whose default is "exact" and are therefore equivalent to:
======
Examplected: - exactpress expectedValue as a "glob" expression :
======Ex...
Nample: test- string-rexpverse
Test: { string reverse "abcdefgz" }
ExpectedVa: glueob "z*as" a ; # note the "glob" expressimatch-mon :de"
...
======
...
Name: test-string-reverse
Test: { string reverse "abcdefgz" }
Expected: glob "z*a" ; # note the "glob" match-mode"
...
======*** Testing for "error" or other return-codes ***
***Testing for "error" or other return-codes***
Your test-case body can be designed not only for testing normal expectedValues, but for trapping exceptions, too. Let's write a test-case for checking the exception thrown when we try to open a non-existing file:
Here is an old (discouraging) trick for such test-case
======none
Name: test-open-exception
Test: {
list [catch {open "not-existing-file.txt" r} f] $f
}
Expected: {1 {couldn't open "not-existing-file.txt": no such file or directory}}
======
And here is Na more: polites alt-opern-excepatiove makin
g Teust:e {
of list [catch {open "not-existieng-filde.d syntaxt" r} of] $f
the }
"Expected: {1 {couldn't open "not-existing-file.txt": no such file or mmandirectory}}:
======none
Name: test-open-exception
Test: {
set f [open "not-existing-file.txt" r]
}
Expected: error {couldn't open "not-existing-file.txt": no such file or directory}
======and here is a more polite alternative making use of the extended syntax of the "Expected:" command:
======
Namote: teshat-open-excep thion
s Ttest:case {
body iset fmuch [mopren "not-rexisting-fiadable. and txhat" r]
the }
'''Expected:''' error {couldmman'td ophas been "not-existieng-filde.txt":d nto sucpport the fiole lor dwirectorng full sy}ntax:
======NoExpected: that this t?estxpecaste bdCody eList? ?muatch mMore readable? and that the '''Expected:''' commVand has been extended to support the following full syntax:e
======
All the previosly seen "Expected:" ?expectedCommandeList? ?matsuchMode? expectedValues
======All the previosly seen "Expected:" expecommanteds such Vaslue
======
Expected: expectedValue
======
have an implicit ?expectedCodeList? and ?matchMode? and are therefore equivalent to:======
======none
Expected: {ok return} exact expectedValue
======
As a final improvement, we could rewrite our last test-case, replacing the (implicit) "exact" matching with a simpler-to-test "glob" matching:======
======none
Name: test-open-exception
Test: {
set f [open "not-existing-file.txt" r]
}
Expected: error glob {couldn't open *}
======
*** Testing for channel output ***
the following '''TTXN''' commands======
======none
OutputChannel: expectedValue
ErrorChannel: expectedValue
======
allow you to specify the expectedValue against which any output sent to stdout or stderr during evaluation of the test-case body script will be compared.
Note that (as for '''tcltest''') only output printed using ::puts is used for comparison. If '''OutputChannel:''' (or '''ErrorChannel:''') is not specified, output sent to stdout (or stderr) is not processed for comparison.
**Adding new comparison criteria**
** Adding new comparison criteria **
Let's suppose we have an instrument returning a decimal value close to 1.0 with a "random" error of +/- 0.0001. How can we compare the expected-value "1.0" with a "random" but very close value ?
We should add a new comparison criteria, just by using some "classic" '''tcltest''' features
======tcltest::customMatch approx approxCompare
tcltest::customMatch approx approxCompare
proc approxCompare {expected actual} {
expr {abs($expected - $actual) < 0.001 }
}
======
With this new matchMethod, we could write the following test-case:======
======none
...
Name: instrument-reset
Test: {
# ... do something and compute a result ...
return 0.999997 ;# of course this is not a random value. It's just a demo
}
Expected: approx 1.0
...
======
** KEYWORDS **
tcltest, testing
----
'''[dcd] - 2012-08-09 21:46:17'''
There's a bug in your last example, presumably a holdover from PTL:
Result: approx 1.0
s.b.
Expected: approx 1.0
[ABU] - 4 hours later
Thanks, fixed with other small wiki-formatting errors. A new package 1.0.1 will be available tomorrow.
<<categories>> Testing