Version 23 of fickle

Updated 2004-09-30 15:07:39 by jt

fickle -- the fast-ish lexical analyzer generator

fickle is a Tcl utility that generates Tcl code, much like flex does for C code. From the README:

 This is a scanner generator program much like flex(1) is to C.  If you
 have no desire to author Tcl programs, particularly those that
 manipulate text, fickle is not for you.  A passing knowledge of flex
 or some other lex-like program would be useful as that fickle uses
 nearly identical syntax and commands as flex.  Two good references are
 the flex(1) man page and the O'Reilly book 'lex & yacc' by Levine,
 Mason, and Brown.

fickle differs from other Tcl scanner generators. Whereas yeti dynamically generates the scanner fickle is designed to be used at compile time (and thus in theory faster). tcLex[L1 ] is a C binary in contrast to fickle which is written in pure Tcl (tested under both Tcl 8.3 and 8.4). Like lex is to yacc, fickle is designed to co-exist with my LALR(1) parser generator taccle.

The fickle package comes with several examples derived from lex & yacc[L2 ]. One example I wrote is tsa, a Tcl source code analyzer. It tries to calculate source lines of code (sloc).

fickle version 2.02 has the following:

  • corrects error with %buffersize option
  • reformatted code and added some more comments

fickle version 2.01 had the following:

  • interactive mode (with -I flag or %option interactive)
  • corrects errors in yy_scan_string and definitions containing backslashes
  • generated comments are now TclDoc friendly

fickle version 2.00 was a near complete rewrite of the previous one. Its features are:

  • much faster than previous versions
  • start states, both inclusive and exclusive (with %option stack)
  • case sensitive and insensitive matching (-i flag, %option caseful, and %option caseless)
  • yy_flex_debug to aid debugging (with -d flag or %option debug)
  • yylineno to keep track of input line numbers (with %option yylineno)
  • yywrap, yyrestart, yy_scan_string to manipulate input buffers
  • input, unput, yyless, and YY_INPUT to modify how fickle reads from yyin
  • change default yy prefix via -P flag

A full list of features is described in the README[L3 ].

Download fickle from below:

fickle is protected by the GNU General Public License. Of course, any code derived from using fickle is protected by whatever copyright you impose.

Invoke fickle before running the program much akin to using flex:

 > tclsh fickle.tcl foo.fcl
 > tclsh foo.tcl

whereas with flex it would be:

 > flex foo.f
 > gcc -o foo lex.yy.c -lfl
 > ./foo

For comparison purposes here is a fickle specification file to match some English verbs:

 # Recognizes various English verbs in sentences.
 # This is based upon example 'ch1-02.l' from "lex & yacc" by John
 # R. Levine, Tony Mason, and Doug Brown (by O'Reilly & Associates, ISBN
 # 1-56592-000-7).  For more information on using lex and yacc, see
 # http://www.oreilly.com/catalog/lex/.
 %{
 #!/usr/bin/tclsh
 %}

 %%
 [\t ]+        # ignore whitespace
 is |
 am |
 are |
 were |
 was |
 will          puts "$yytext: is a verb"
 [a-zA-Z]+     puts "$yytext: is not a verb"
 .|\n          ECHO  ;# normal default anyway
 %%
 yylex

And the equivalent flex version:

 %{
 /*
  * this sample demonstrates (very) simple recognition:
  * a verb/not a verb.  this example derived from 'ch1-02.l'.
  */
 %}

 %%
 [\t ]+        /* ignore whitespace */
 is |
 am |
 are |
 were |
 was |
 will          { printf ("%s: is a verb\n", yytext); }
 [a-zA-Z]+     { printf ("%s: is not a verb\n", yytext); }
 .|\n          { ECHO;  /* normal default anyway */ }
 %%
 main()
 {
     yylex();
 }

27sep04 jcw - There's a small typo, I've changed a line containing:

    "%buffersize " {

to:

    "%buffersize" {

This showed up when running "make" in the examples/ directory.

jt Thanks for spotting this. Version 2.02 includes this correction.

Return to Jason Tang

Category Application | Category Dev. Tools