fickle -- the fast-ish lexical analyzer generator

fickle is a Tcl utility that generates Tcl code, much like flex does for C code.  
From the README:

 This is a scanner generator program much like flex(1) is to C.  If you
 have no desire to author Tcl programs, particularly those that
 manipulate text, fickle is not for you.  A passing knowledge of flex
 or some other lex-like program would be useful as that fickle uses
 nearly identical syntax and commands as flex.  Two good references are
 the flex(1) man page and the O'Reilly book 'lex & yacc' by Levine,
 Mason, and Brown.

fickle differs from other Tcl scanner generators.  Whereas [yeti] dynamically generates the scanner 
fickle is designed to be used at compile time (and thus in theory faster).  tcLex[http://membres.lycos.fr/fbonnet/Tcl/tcLex/index.en.htm] is a C binary 
in contrast to fickle which is written in pure Tcl (tested under both Tcl 8.3 and 8.4).  
Like lex is to yacc, fickle is designed to co-exist with my LALR(1) parser generator [taccle].

The fickle package comes with several examples derived from ''lex & yacc''[http://www.oreilly.com/catalog/lex/].  One example I wrote is ''tsa'', a Tcl source code analyzer.  It tries to calculate source lines of code ([sloc]).

----
[LV] 2007 June 27
The web site tcl.jtang.org (as well as www.jtang.org) is not, at this time, accessible. I've sent emails to see if I can reconnect with Jason, to see if
this software is available from him.  However, if the reader of this page
happens to have copies to share, it might be nice to mention. 
----
A full list of features is described in the ''README[http://tcl.jtang.org/fickle/README]''.  
Check the section below for errata.

Download fickle from below:
   * '''fickle 2.04 at http://tcl.jtang.org/fickle/fickle-2.04.tar.gz.''' (also from here [http://www.ellogon.org/petasis/index.php?option=com_docman&task=cat_view&gid=34&Itemid=37] - direct link [http://www.ellogon.org/petasis/index.php?option=com_docman&task=doc_download&gid=58&Itemid=37])
   * '''fickle 2.03 at http://tcl.jtang.org/fickle/fickle-2.03.tar.gz.'''
   * '''fickle 2.02 at http://tcl.jtang.org/fickle/fickle-2.02.tar.gz.'''
   * '''fickle 2.01 at http://tcl.jtang.org/fickle/fickle-2.01.tar.gz.'''
   * '''fickle 2.00 at http://tcl.jtang.org/fickle/fickle-2.00.tar.gz.'''
   * '''fickle 1.00 at http://tcl.jtang.org/fickle/fickle-1.00.tar.gz.'''

fickle is protected by the GNU General Public License.  
Of course, any code derived from using fickle is protected by whatever copyright you impose.

Invoke fickle before running the program much akin to using flex:

 > tclsh fickle.tcl foo.fcl
 > tclsh foo.tcl

whereas with flex it would be:

 > flex foo.f
 > gcc -o foo lex.yy.c -lfl
 > ./foo

For comparison purposes here is a fickle specification file to match some English verbs:

 # Recognizes various English verbs in sentences.
 # This is based upon example 'ch1-02.l' from "lex & yacc" by John
 # R. Levine, Tony Mason, and Doug Brown (by O'Reilly & Associates, ISBN
 # 1-56592-000-7).  For more information on using lex and yacc, see
 # http://www.oreilly.com/catalog/lex/.
 %{
 #!/usr/bin/tclsh
 %}

 %%
 [\t ]+        # ignore whitespace
 is |
 am |
 are |
 were |
 was |
 will          puts "$yytext: is a verb"
 [a-zA-Z]+     puts "$yytext: is not a verb"
 .|\n          ECHO  ;# normal default anyway
 %%
 yylex
 
And the equivalent flex version:


 %{
 /*
  * this sample demonstrates (very) simple recognition:
  * a verb/not a verb.  this example derived from 'ch1-02.l'.
  */
 %}

 %%
 [\t ]+        /* ignore whitespace */
 is |
 am |
 are |
 were |
 was |
 will          { printf ("%s: is a verb\n", yytext); }
 [a-zA-Z]+     { printf ("%s: is not a verb\n", yytext); }
 .|\n          { ECHO;  /* normal default anyway */ }
 %%
 main()
 {
     yylex();
 }
----
'''[ifickle]'''

2005-06-03 [jt] I received from Detlef Groth a significant patch to fickle 2.04 to allow it to co-exist within an [[incr Tcl] program.  I have not fully reviewed the changes, but seeing his other works I have no doubt that it creates a proper scanner.  
When I have some time I will merge the changes into fickle baseline.  Meanwhile get it here:

'''ifickle 2.04 at http://tcl.jtang.org/fickle/ifickle-2.04.tcl'''

Dr. Groth was even kind enough to supply a sample [[incr Tcl] script for ifickle.  
Here it is:

 %{
 #!/usr/bin/tclsh8.4
 public variable nline 0
 public variable nword 0
 public variable nchar 0
 %}
 %buffersize 1024
 %%
 \n        { incr nline; incr nchar ; }
 [^ \t\n]+ { incr nword; incr nchar $yyleng ;}
 .         { incr nchar;}
 %%
 if {[llength $argv] == 0} {
     puts stderr "usage wc-fickle inputfile"
     exit 0
 }
 if {[catch {open [lindex $argv 0] r} yyin]} {
     puts stderr "Could not open [lindex $argv 0]"
     exit 0
 }
                                                                                 
 set sc [iwcfickle \#auto -yyin $yyin]
 $sc yylex
 puts [format "%7d %7d %7d %s" [$sc cget -nline] [$sc cget -nword] [$sc cget -nchar] [lindex $argv 0]]
 close $yyin

----
'''Versioning and Errata'''

''2005-03-11. There is an error in v2.04 on line 377 of fickle.tcl.  Thanks to Jeff S. for finding it.  The correct code is:''
    set ::${::p}in \$new_file

'''version 2.04''' is another bug fix:
   * yyless was missing a parameter within its internal [[string match] command
   * updated README file

'''version 2.03''' is a bug fix:
   * fixed problems when using both ''%option debug'' and ''%option prefix''
   * fixed an error with patterns that have a vertical bar

'''version 2.02''' has the following:
   * corrects error with ''%buffersize'' option
   * reformatted code and added some more comments

'''version 2.01''' had the following:
   * interactive mode (with ''-I'' flag or ''%option interactive'')
   * corrects errors in ''yy_scan_string'' and definitions containing backslashes
   * generated comments are now [TclDoc] friendly

'''version 2.00''' was a near complete rewrite of the previous one.  Its features are:
   * much faster than previous versions
   * start states, both inclusive and exclusive (with ''%option stack'')
   * case sensitive and insensitive matching (''-i'' flag, ''%option caseful'', and ''%option caseless'')
   * ''yy_flex_debug'' to aid debugging (with ''-d'' flag or ''%option debug'')
   * ''yylineno'' to keep track of input line numbers (with ''%option yylineno'')
   * ''yywrap'', ''yyrestart'', ''yy_scan_string'' to manipulate input buffers
   * ''input'', ''unput'', ''yyless'', and ''YY_INPUT'' to modify how fickle reads from ''yyin''
   * change default yy prefix via ''-P'' flag

----
27sep04 [jcw] - There's a small typo, I've changed a line containing:
    "%buffersize " {
to:
    "%buffersize" {
This showed up when running "make" in the examples/ directory. 

[jt] Thanks for spotting this.  Version 2.02 includes this correction.
----

Return to [Jason Tang]

----
[Category Application] | [Category Dev. Tools] | [Category String Processing]