fickle -- the fast-ish lexical analyzer generator
fickle is a Tcl utility that generates Tcl code, much like flex does for C code. From the README:
This is a scanner generator program much like flex(1) is to C. If you have no desire to author Tcl programs, particularly those that manipulate text, fickle is not for you. A passing knowledge of flex or some other lex-like program would be useful as that fickle uses nearly identical syntax and commands as flex. Two good references are the flex(1) man page and the O'Reilly book 'lex & yacc' by Levine, Mason, and Brown.
fickle differs from other Tcl scanner generators. Whereas yeti dynamically generates the scanner fickle is designed to be used at compile time (and thus in theory faster). tcLex[L1 ] is a C binary in contrast to fickle which is written in pure Tcl (tested under both Tcl 8.3 and 8.4). Like lex is to yacc, fickle is designed to co-exist with my LALR(1) parser generator taccle.
The fickle package comes with several examples derived from lex & yacc[L2 ]. One example I wrote is tsa, a Tcl source code analyzer. It tries to calculate source lines of code (sloc).
Bezoar 2015 Dec13 Links to tar.gz files are dead. However Jason has put his taccle and fickle projects in git hub fickle[L3 ]
LV 2007 June 27 The web site tcl.jtang.org (as well as www.jtang.org) is not, at this time, accessible. I've sent emails to see if I can reconnect with Jason, to see if this software is available from him. However, if the reader of this page happens to have copies to share, it might be nice to mention.
A full list of features is described in the README[L4 ]. Check the section below for errata.
Download fickle from below:
fickle is protected by the GNU General Public License. Of course, any code derived from using fickle is protected by whatever copyright you impose.
Invoke fickle before running the program much akin to using flex:
> tclsh fickle.tcl foo.fcl > tclsh foo.tcl
whereas with flex it would be:
> flex foo.f > gcc -o foo lex.yy.c -lfl > ./foo
For comparison purposes here is a fickle specification file to match some English verbs:
# Recognizes various English verbs in sentences.
# This is based upon example 'ch1-02.l' from "lex & yacc" by John
# R. Levine, Tony Mason, and Doug Brown (by O'Reilly & Associates, ISBN
# 1-56592-000-7). For more information on using lex and yacc, see
# http://www.oreilly.com/catalog/lex/.
%{
#!/usr/bin/tclsh
%}
%%
[\t ]+ # ignore whitespace
is |
am |
are |
were |
was |
will puts "$yytext: is a verb"
[a-zA-Z]+ puts "$yytext: is not a verb"
.|\n ECHO ;# normal default anyway
%%
yylexAnd the equivalent flex version:
%{
/*
* this sample demonstrates (very) simple recognition:
* a verb/not a verb. this example derived from 'ch1-02.l'.
*/
%}
%%
[\t ]+ /* ignore whitespace */
is |
am |
are |
were |
was |
will { printf ("%s: is a verb\n", yytext); }
[a-zA-Z]+ { printf ("%s: is not a verb\n", yytext); }
.|\n { ECHO; /* normal default anyway */ }
%%
main()
{
yylex();
}2005-06-03 jt I received from Detlef Groth a significant patch to fickle 2.04 to allow it to co-exist within an [incr Tcl] program. I have not fully reviewed the changes, but seeing his other works I have no doubt that it creates a proper scanner. When I have some time I will merge the changes into fickle baseline. Meanwhile get it here:
ifickle 2.04 at http://tcl.jtang.org/fickle/ifickle-2.04.tcl
Dr. Groth was even kind enough to supply a sample [incr Tcl] script for ifickle. Here it is:
%{
#!/usr/bin/tclsh8.4
public variable nline 0
public variable nword 0
public variable nchar 0
%}
%buffersize 1024
%%
\n { incr nline; incr nchar ; }
[^ \t\n]+ { incr nword; incr nchar $yyleng ;}
. { incr nchar;}
%%
if {[llength $argv] == 0} {
puts stderr "usage wc-fickle inputfile"
exit 0
}
if {[catch {open [lindex $argv 0] r} yyin]} {
puts stderr "Could not open [lindex $argv 0]"
exit 0
}
set sc [iwcfickle \#auto -yyin $yyin]
$sc yylex
puts [format "%7d %7d %7d %s" [$sc cget -nline] [$sc cget -nword] [$sc cget -nchar] [lindex $argv 0]]
close $yyinVersioning and Errata
2010-07-29. unofficial by MHN: the BEGIN procedure has not worked in my v2.04. The error could be fixed in line 522 of fickle.tcl:
eval set ::\${prefix}_state_stack \\\{\[lrange \[set ::\${prefix}_state_stack\] 0 end-1\]\\\} ;# <your bugfix note>2005-03-11. There is an error in v2.04 on line 377 of fickle.tcl. Thanks to Jeff S. for finding it. The correct code is:
set ::${::p}in \$new_fileversion 2.04 is another bug fix:
version 2.03 is a bug fix:
version 2.02 has the following:
version 2.01 had the following:
version 2.00 was a near complete rewrite of the previous one. Its features are:
27sep04 jcw - There's a small typo, I've changed a line containing:
"%buffersize " {to:
"%buffersize" {This showed up when running "make" in the examples/ directory.
jt Thanks for spotting this. Version 2.02 includes this correction.
Return to Jason Tang