fickle -- the fast-ish lexical analyzer generator

fickle is a Tcl utility that generates Tcl code, much like flex does for C code.  From the README:

 This is a scanner generator program much like flex(1) is to C.  If you
 have no desire to author Tcl programs, particularly those that
 manipulate text, fickle is not for you.  A passing knowledge of flex
 or some other lex-like program would be useful as that fickle uses
 nearly identical syntax and commands as flex.  Two good references are
 the flex(1) man page and the O'Reilly book 'lex & yacc' by Levine,
 Mason, and Brown.

fickle differs from other Tcl scanner generators.  Whereas [yeti] dynamically generates the scanner fickle is designed to be used at compile time (and thus in theory faster).  tcLex[http://membres.lycos.fr/fbonnet/Tcl/tcLex/index.en.htm] is a C binary in contrast to fickle which is written in pure Tcl (tested under both Tcl 8.3 and 8.4).

The fickle package comes with a bunch of example files derived from ''lex & yacc''[http://www.oreilly.com/catalog/lex/].  One example I wrote is ''tsa'', a Tcl source code analyzer.  It tries to calculate source lines of code ([sloc]).

fickle 2.0 is a near complete rewrite of the previous one.  It is much faster than before and supports many more of flex's features.  New capabilities include ''yy_flex_debug'', ''yylineno'', and embedded comments.  Other features of fickle include:

   * start states, both inclusive and exclusive (with ''%option stack'' set)
   * case sensitive and insensitive matching (''-i'' flag, ''%option caseful'', and ''%option caseless'')
   * ''yywrap'', ''yyrestart'', ''yy_scan_string'' to manipulate input buffers
   * ''input'', ''unput'', ''yyless'', and ''YY_INPUT'' to modify how fickle reads from ''yyin''
   * change default yy prefix via ''-P'' flag

A full list of features is described in the ''README[http://tcl.jtang.org/fickle/README]''.

Of course this version is fickle completely breaks specification files written against the first version of fickle.  Thus this is the penalty of progress.

Download fickle from below:
   * '''fickle 2.00 at http://tcl.jtang.org/fickle/fickle-2.00.tar.gz.'''
   * '''fickle 1.00 at http://tcl.jtang.org/fickle/fickle-1.00.tar.gz.'''

fickle is protected by the GNU general public license.

For heavy-duty parsing needs one ought to use a syntax analyzer such as [taccle].

Invoke fickle before running the program much akin to using flex:

 > tclsh fickle.tcl foo.fcl
 > tclsh foo.tcl

whereas with flex it would be:

 > flex foo.f
 > gcc -o foo lex.yy.c -lfl
 > ./foo

For comparison purposes here is a fickle specification file to match some English verbs:

 # Recognizes various English verbs in sentences.
 # This is based upon example 'ch1-02.l' from "lex & yacc" by John
 # R. Levine, Tony Mason, and Doug Brown (by O'Reilly & Associates, ISBN
 # 1-56592-000-7).  For more information on using lex and yacc, see
 # http://www.oreilly.com/catalog/lex/.
 %{
 #!/usr/bin/tclsh
 %}

 %%
 [\t ]+        # ignore whitespace
 is |
 am |
 are |
 were |
 was |
 will          puts "$yytext: is a verb"
 [a-zA-Z]+     puts "$yytext: is not a verb"
 .|\n          ECHO  ;# normal default anyway
 %%
 yylex
 
And the equivalent flex version:


 %{
 /*
  * this sample demonstrates (very) simple recognition:
  * a verb/not a verb.  this example derived from 'ch1-02.l'.
  */
 %}

 %%
 [\t ]+        /* ignore whitespace */
 is |
 am |
 are |
 were |
 was |
 will          { printf ("%s: is a verb\n", yytext); }
 [a-zA-Z]+     { printf ("%s: is not a verb\n", yytext); }
 .|\n          { ECHO;  /* normal default anyway */ }
 %%
 main()
 {
     yylex();
 }
 
----

Return to [Jason Tang]

----

[Category Application] | [Category Dev. Tools]