Version 15 of fickle

Updated 2004-07-23 19:55:43

fickle -- the fast-ish lexical analyzer generator

fickle is a Tcl utility that generates Tcl code, much like flex does for C code. From the README:

 This is a scanner generator program much like flex(1) is to C.  If you
 have no desire to author Tcl programs, particularly those that
 manipulate text, fickle is not for you.  A passing knowledge of flex
 or some other lex-like program would be useful as that fickle uses
 nearly identical syntax and commands as flex.  Two good references are
 the flex(1) man page and the O'Reilly book 'lex & yacc' by Levine,
 Mason, and Brown.

fickle differs from other Tcl scanner generators. Whereas yeti dynamically generates the scanner fickle is designed to be used at compile time (and thus in theory faster). tcLex[L1 ] is a C binary in contrast to fickle which is written in pure Tcl (tested under both Tcl 8.3 and 8.4).

The fickle package comes with a bunch of example files derived from lex & yacc[L2 ]. One example I wrote is tsa, a Tcl source code analyzer. It tries to calculate source lines of code (sloc).

fickle 2.0 is a near complete rewrite of the previous one. It is much faster than before and supports many more of flex's features. New capabilities include yy_flex_debug, yylineno, and embedded comments. Other features of fickle are:

  • start states, both inclusive and exclusive (with %option stack set)
  • case sensitive and insensitive matching (-i flag, %option caseful, and %option caseless)
  • yywrap, yyrestart, yy_scan_string to manipulate input buffers
  • input, unput, yyless, and YY_INPUT to modify how fickle reads from yyin
  • change default yy prefix via -P flag

A full list of features is available from the README[L3 ].

Of course this version is fickle completely breaks specification files written against the first version of fickle. Thus the penalty of progress.

Download fickle from below:

fickle is protected by the GNU general public license.

For heavy-duty parsing needs one ought to use a syntax analyzer such as taccle.

Invoke fickle before running the program much akin to using flex:

 > tclsh fickle.tcl foo.fcl
 > tclsh foo.tcl

whereas with flex it would be:

 > flex foo.f
 > gcc -o foo lex.yy.c -lfl
 > ./foo

For comparison purposes here is a fickle specification file to match some English verbs:

 # Recognizes various English verbs in sentences.
 # This is based upon example 'ch1-02.l' from "lex & yacc" by John
 # R. Levine, Tony Mason, and Doug Brown (by O'Reilly & Associates, ISBN
 # 1-56592-000-7).  For more information on using lex and yacc, see
 # http://www.oreilly.com/catalog/lex/.
 %{
 #!/usr/bin/tclsh
 %}

 %%
 [\t ]+        # ignore whitespace
 is |
 am |
 are |
 were |
 was |
 will          puts "$yytext: is a verb"
 [a-zA-Z]+     puts "$yytext: is not a verb"
 .|\n          ECHO  ;# normal default anyway
 %%
 yylex

And the equivalent flex version:

 %{
 /*
  * this sample demonstrates (very) simple recognition:
  * a verb/not a verb.  this example derived from 'ch1-02.l'.
  */
 %}

 %%
 [\t ]+        /* ignore whitespace */
 is |
 am |
 are |
 were |
 was |
 will          { printf ("%s: is a verb\n", yytext); }
 [a-zA-Z]+     { printf ("%s: is not a verb\n", yytext); }
 .|\n          { ECHO;  /* normal default anyway */ }
 %%
 main()
 {
     yylex();
 }

Return to Jason Tang

Category Application | Category Dev. Tools