fickle -- the fast-ish lexical analyzer generator

fickle is a Tcl utility that generates Tcl code, much like flex does for C code.  From the README:

 This is a scanner generator program much like flex(1) is to C.  If you
 have no desire to author Tcl programs, particularly those that
 manipulate text, fickle is not for you.  A passing knowledge of flex
 or some other lex-like program would be useful as that fickle uses
 nearly identical syntax and commands as flex.  Two good references are
 the flex(1) man page and the O'Reilly book 'lex & yacc' by Levine,
 Mason, and Brown.

fickle differs from other Tcl scanner generators.  Whereas [yeti] dynamically generates the scanner fickle is designed to be used at compile time (and thus in theory faster).  tcLex[http://membres.lycos.fr/fbonnet/Tcl/tcLex/index.en.htm] is a C binary in contrast to fickle which is written in pure Tcl (tested under both Tcl 8.3 and 8.4).  Like lex is to yacc, fickle is designed to co-exist with my LALR(1) parser generator [taccle].

The fickle package comes with several examples derived from ''lex & yacc''[http://www.oreilly.com/catalog/lex/].  One example I wrote is ''tsa'', a Tcl source code analyzer.  It tries to calculate source lines of code ([sloc]).

fickle '''version 2.04''' is another bug fix:
   * yyless was missing a parameter within its internal [[string match] command
   * updated README file

fickle '''version 2.03''' is a bug fix:
   * fixed problems when using both ''%option debug'' and ''%option prefix''
   * fixed an error with patterns that have a vertical bar

fickle '''version 2.02''' has the following:
   * corrects error with ''%buffersize'' option
   * reformatted code and added some more comments

fickle '''version 2.01''' had the following:
   * interactive mode (with ''-I'' flag or ''%option interactive'')
   * corrects errors in ''yy_scan_string'' and definitions containing backslashes
   * generated comments are now [TclDoc] friendly

fickle '''version 2.00''' was a near complete rewrite of the previous one.  Its features are:
   * much faster than previous versions
   * start states, both inclusive and exclusive (with ''%option stack'')
   * case sensitive and insensitive matching (''-i'' flag, ''%option caseful'', and ''%option caseless'')
   * ''yy_flex_debug'' to aid debugging (with ''-d'' flag or ''%option debug'')
   * ''yylineno'' to keep track of input line numbers (with ''%option yylineno'')
   * ''yywrap'', ''yyrestart'', ''yy_scan_string'' to manipulate input buffers
   * ''input'', ''unput'', ''yyless'', and ''YY_INPUT'' to modify how fickle reads from ''yyin''
   * change default yy prefix via ''-P'' flag

A full list of features is described in the ''README[http://tcl.jtang.org/fickle/README]''.

Download fickle from below:
   * '''fickle 2.04 at http://tcl.jtang.org/fickle/fickle-2.04.tar.gz.'''
   * '''fickle 2.03 at http://tcl.jtang.org/fickle/fickle-2.03.tar.gz.'''
   * '''fickle 2.02 at http://tcl.jtang.org/fickle/fickle-2.02.tar.gz.'''
   * '''fickle 2.01 at http://tcl.jtang.org/fickle/fickle-2.01.tar.gz.'''
   * '''fickle 2.00 at http://tcl.jtang.org/fickle/fickle-2.00.tar.gz.'''
   * '''fickle 1.00 at http://tcl.jtang.org/fickle/fickle-1.00.tar.gz.'''

fickle is protected by the GNU General Public License.  Of course, any code derived from using fickle is protected by whatever copyright you impose.

Invoke fickle before running the program much akin to using flex:

 > tclsh fickle.tcl foo.fcl
 > tclsh foo.tcl

whereas with flex it would be:

 > flex foo.f
 > gcc -o foo lex.yy.c -lfl
 > ./foo

For comparison purposes here is a fickle specification file to match some English verbs:

 # Recognizes various English verbs in sentences.
 # This is based upon example 'ch1-02.l' from "lex & yacc" by John
 # R. Levine, Tony Mason, and Doug Brown (by O'Reilly & Associates, ISBN
 # 1-56592-000-7).  For more information on using lex and yacc, see
 # http://www.oreilly.com/catalog/lex/.
 %{
 #!/usr/bin/tclsh
 %}

 %%
 [\t ]+        # ignore whitespace
 is |
 am |
 are |
 were |
 was |
 will          puts "$yytext: is a verb"
 [a-zA-Z]+     puts "$yytext: is not a verb"
 .|\n          ECHO  ;# normal default anyway
 %%
 yylex
 
And the equivalent flex version:


 %{
 /*
  * this sample demonstrates (very) simple recognition:
  * a verb/not a verb.  this example derived from 'ch1-02.l'.
  */
 %}

 %%
 [\t ]+        /* ignore whitespace */
 is |
 am |
 are |
 were |
 was |
 will          { printf ("%s: is a verb\n", yytext); }
 [a-zA-Z]+     { printf ("%s: is not a verb\n", yytext); }
 .|\n          { ECHO;  /* normal default anyway */ }
 %%
 main()
 {
     yylex();
 }
----
27sep04 [jcw] - There's a small typo, I've changed a line containing:
    "%buffersize " {
to:
    "%buffersize" {
This showed up when running "make" in the examples/ directory. 

[jt] Thanks for spotting this.  Version 2.02 includes this correction.
----

Return to [Jason Tang]

----

[Category Application] | [Category Dev. Tools]