Tcl/Tk 8.x extension which provides script level access to the Tcl tokenizer, as well as generic query and abstract syntax tree query and manipulation commands.
Splitting strings with embedded strings includes an example HTML scanner, and also a description of how the state matrix was generated. Implement your own version his Imaginary ad-hoc Scanner Generator Tool.