Agent SMITH: Evolution of a Test Tool in Tcl/Tk

Agent SMITH: Evolution of a Test Tool in Tcl/Tk ("A happy success story") a Tcl2008 talk given by John Seal (JJS).

Story Maker not available during development. Extremely complicated accept/reject logic (messages contain many interrelated fields. Teller performs validity and consistency checks. Literally hundreds of test cases. Certification required to play with those other systems.) Several communities with different needs.

Story Teller project engineer sketched out an idea (drew a picture of a possible GUI) and asked "how hard would it be..." for a tool that would let you:

• Build a test message
• Describe the required initial conditions and expected outputs
• Send it to Teller
• See the response
• See the response
• See any error messages
• Indicate ...

Tcl/Tk makes networking easy and rapid prototyping. Already used in Story series and related systems. Development proceeded in four stages:

1. Prototype proposed solution for proof-of-concept demo
2. Automate the tool with playback of test cases
3. Hook into Teller to monitor internal and external data
4. Parse the data into human-readable format

Prototype proposed solution

• Add detailed expected response fields
• Eliminate confusing Expected Output fields
• There are really two kinds of response (the actual response message back from teller; actions taken by teller and the other system as a result)
• Can automatically check expected vs actual response
• Still need a way to describe external reactions, but how?
• Add generic Event Log pane (scrolling text widget log: combines test case, initial conditions, result and error text)
• Add destination host:port fields for message

Demonstrated proof-of-concept. Add menus to aid in data entry. First tried to use tk_optionMenu widgets; not good for labels. They show their current selection. Made the window really big and cluttered. Switched to menubuttons with down-arrow character in name. Should we have used combo boxes? (Maybe still cluttered?) Also reduced the size of the main window. Made the event log a separate resizable window. Move the exit button into a File menu. Move the destination specification into a Server menu. Added an entry to File menu to open a test case file (which was sourced by Tcl using a simple domain specific language (DSL)). It's easy to create Domain Specific Languages with Tcl.

(... showed sample test case file ...)

Automated test case: added transport controls for playback (back, stop, step, play, delay)

Testers don't like to use a text editor to write test cases (they aren't programmers). They saw the blindingly obvious solution to add a button to "record" new test cases. Lesson: always listen to feedback from the user community.

Integration with Teller: Teller uses a peculiar homegrown IPC scheme internally. Source code was available, but: functions are compiled into static libraries; options:

• CriTcl (haven't used it before, feared it, though after Steve Landers tutorial at Tcl2008 he doesn't fear it anymore)
• Ffidl (ditto)
• Custom C extension in shared library (experienced in doing it)

Built shared library with wrappers for Teller IPC functions. Use it to register for Teller internal messages. Open socked to receive other messages intended for Maker. Lesson learned: It's easy to extend Tcl with other languages.

Extended the test case DSL to specify outgoing messages expected to be generated and incoming messages expected to get back. Also added a log menu (controls what to log, how to log it). Hex dump log code contributed to the wiki Dump a file in hex and ASCII.

Messages going out and coming back in are in a format called "Generic Tactical Information Message Format" (GTIMF). Teller already had a C tool to dump TIMF messages. But it parses fields and displays them numerically--doesn't interpret values, so not truly human-readable. Hard to correlate with Agent SMITH event log. Decided to write GTIMF parser in Tcl. Message structure from ICD encoded in simple Tcl data structures. Helper procedures parse message hierarchically. Could easily be extended to other protocols. Less than a week to design, code, test and document! Lesson learned: Once again, Tcl excels at network programming.

Tc/Tk helped ensure success of initial effort

• Simple enough for rapid prototyping
• Flexible enough for changing requirements
• Powerful enough for delivered applications

Agent SMITH has been well received by multiple user communities and used in some unexpected ways:

• System Engineers use it to explore "What If?" scenarios
• Software Engineers use it to debug other Teller changes
• Test Engineers use it to provide repeatable stimulus

Lesson: Tcl/Tk is ideal for developing tools and test harnesses.