Lars H, 2010-02-03: Some months ago, while working on OpenMath things, I grew tired of the XML syntax — it just felt cluttered and clumpsy with all those </>="' characters. I wanted something more lightweight (for me as author), so where to turn, if not to Tcl? First for sketching things out, and later as a format that programs would actually process, I began to use something I dubbed TDL (for Tcl Data Language, or perhaps — in analogy with Tool Command Language and Tool Protocol Language — Tool Data Language).
First, an example to illustrate the idea. A small fragment of OpenMath is
<OMA>
<OMS cd="symocat1" name="label"/>
<OMS cd="Hopf-algebra" name="mult"/>
<OMA>
<OMS cd="list1" name="list"/>
<OMV name="a"/>
</OMA>
<OMA>
<OMS cd="list1" name="list"/>
<OMV name="b"/>
<OMV name="c"/>
</OMA>
</OMA>This is rather cluttered. The same thing as TDL could be
OMA {
/OMS symocat1 label
/OMS Hopf-algebra mult
OMA {/OMS list1 list; /OMV a}
OMA {/OMS list1 list; /OMV b; /OMV c}
}or if not so specialised
OMA {
OMS cd symocat1 name label
OMS cd Hopf-algebra name mult
OMA {OMS cd list1 name list; OMV name a}
OMA {
OMS cd list1 name list
OMV name b
OMV name c
}
}giorgio_v It reminds me of xmlgen
Lars H: Yes, there are similarities (and not only in purpose). One difference is that TDL is supposed to be data (even if treating it as code will be a convenient way of operating on it), whereas xmlgen / htmlgen rather is a way of writing scripts that construct the wanted data structure. That has implications for whether one might also want to generate it, or just write it. In addition, xmlgen / htmlgen seems to require that all tags to be used are defined as commands, whereas TDL would mostly require that only for positional argument commands.
A more trivial difference is that xmlgen / htmlgen keeps the = between attribute name and value. I wanted to get rid of that too.
JFL 2010-03-13 I had a very similar idea a few years ago, and created a script for converting XML files to "SML" files, and back. SML stood for Simple Markup Language. The script is here:
http://jf.larvoire.free.fr/progs/sml
See that file header for a full description of the syntax.
At first I thought of this SML format only as another view on XML data. Then eventually I used it as the native protocol for data transfers between programs. This proves to be very convenient because it's even easier to parse than XML, and it's MUCH easier to debug by looking at what's transmitted.
Another variation on the same theme: It's also possible to present an actual file system as a text tree like this, using the convention that directory names are always noted with a trailing /, and that the file and directory contents MUST be indented more than the name. File meta data, like date/time, can be encoded in XML/SML like attributes. Here's a script that dumps a directory tree this way:
http://jf.larvoire.free.fr/progs/show
I used that script a lot for browsing Linux /proc subdirectories. (By default it displays only the top 10 lines of each file like "head")
Both programs work on both Linux and Windows. Run them with option -? to get a command line help screen.
Enjoy!
1. A general XML element is transformed to a Tcl command with the element name as command name, and the contents as a final "body" argument. Other arguments must occur in pairs and encode the attributes in the usual key–value style. Hence
<element attr1="val1" attr2="val2"></element>
is equivalent to
element attr1 val1 attr2 val2 {}and also to
element attr1 val1 attr2 val2
since parity allows recognising the case of a missing body argument.
2. A sequence of XML elements is transformed to a sequence of commands, i.e., to a script. Hence
<element1/><element2 attr2="val2"/>
is equivalent to
element1; element2 attr2 val2
and also to
element1
element2 attr2 val23. Command names which do not fit the XML syntax for element names are used to express things other than elements and can have other syntaxes, e.g. arguments identified by position. In particular, the "/" command is used to encode character data in a sequence of elements. Thus
<OMI>3</OMI>
can be encoded as
OMI {/ 3}"/" can have any number of arguments, which are concatenated in the manner of append.
4. Elements can sometimes be given an alternative encoding, in the form of a positional command. These commands will usually have a "/" prepended to the element name. The syntax is usually that required attributes (if any) and character data contents (if any) are mandatory arguments, but no general rule defining such commands exist; each must be defined explicitly.
Example: The OM elements
<OMS cd="arith1" name="times"/>
<OMV name="x"/>
<OMI>5</OMI>could be expressed as
/OMS arith1 times
/OMV x
/OMI 5but this would be context-dependent.
Some probably think it should be a criminal offense to invent a new syntax for what is almost XML, and there are no doubt interoperability problems lurking. On the other hand, if it allows me to be more productive, then I mostly think it is a good thing.
For robust and informative handling of syntax errors, it would probably be necessary to use something like parsetcl to parse TDL, but if we're content with parsing valid TDL and throwing an error on the rest, then things are much easier. The trick is to parse TDL data by evaluating it in an empty slave interpreter.
namespace eval prettyTDL {
interp create -safe [list [namespace current]::theinterp]
theinterp hide namespace
theinterp invokehidden namespace delete ::
}The first operation on TDL code will be to prettyprint it. The central command point for prettyprinting is the prettyprint procedure, which has the call syntax
and returns the prettyprinting of the script. The supported options are:
proc prettyTDL::prettyprint {script args} {
set res ""
array set O {-indent "" -step { }}
array set O $args
theinterp eval $script
return $res
}The way the prettyprinting works is that each command appends the prettyprinted form of itself, preceded by the appropriate indentation and followed by a newline, to the local variable res in this procedure. The local array O has two entries -indent and -step which contain the current values of these parameters.
In most cases, that appending is taken care of by the unknown command in the slave interpreter, which is an alias to the following procedure in the master interpreter.
prettyTDL::theinterp alias unknown [namespace current]::prettyTDL::unknown
proc prettyTDL::unknown {name args} {
upvar 1 res res O O
switch -regexp -- $name {
{^[[:alpha:]_:][[:alnum:]_:.-]*$} {
if {[llength $args] % 2} then {
set body [lindex $args end]
set args [lreplace $args end end]
} else {
set body ""
}
append res $O(-indent) [linsert $args 0 $name]
if {[regexp {\S} $body]} then {
append res " \{\n" [
prettyprint $body {*}[array get O]\
-indent $O(-indent)$O(-step)
] $O(-indent) \}
}
append res \n
}
default {
append res $O(-indent) [linsert $args 0 $name] \n
}
}
}
proc prettyTDL::slash {args} {
upvar 1 res res O(-indent) indent
set L [list /]
for {set i 0} {$i < [llength $args]} {incr i} {
set n [string first \n [lindex $args $i]]
if {$n<0} then {
lappend L [lindex $args $i]
continue
}
if {$n>0} then {
lappend L [string range [lindex $args $i] 0 $n-1]
}
append res $indent $L { \n} \n
set L [list /]
lset args $i [string replace [lindex $args $i] 0 $n]
incr i -1
}
if {[llength $L] > 1} then {
append res $indent $L \n
}
}
prettyTDL::theinterp alias / [namespace current]::prettyTDL::slashFor example:
% prettyTDL::prettyprint {OMA {/OMS arith1 plus; /OMV a; /OMV b}}
OMA {
/OMS arith1 plus
/OMV a
/OMV b
}One could also perform various types of normalisation at this step, for example change an OMV to an /OMV.
prettyTDL::theinterp alias OMV apply {args {
upvar 1 res res O O
if {[llength $args]==2 && [lindex $args 0] eq "name"} then {
append res $O(-indent) [list /OMV [lindex $args 1]] \n
} else {
unknown OMV {*}$args
}
}}So that
% prettyTDL::prettyprint {OMA {/OMS arith1 plus; OMV name a; OMV name b}}
OMA {
/OMS arith1 plus
/OMV a
/OMV b
}In the case of OpenMath, it might be more useful to normalise from /OMV to OMV, but that is something that would rather be done at TDL-to-XML conversion.
Another useful operation is that of converting a TDL script to equivalent XML, which is what the following does — or rather, it converts a TDL script to a list of tdom data-trees. A data-tree is an encoding (using built-in Tcl data containers) of general XML elements; technically it is a list on one of the forms
(possibly there could be more), but only the first two are used here. The children is again a list (possibly empty) of data-trees. The attribute-dict is a dictionary mapping attribute names to their values, neither of which have any quoting of special characters. Similarly, the text is character data between tags, without any XML-quoting.
For XML-commands and the / command, this conversion is obvious, but for other non-XML commands some kind of XML encoding will have to be invented. The basic method is to express those as TDL:cmd elements, which must conform to the DTD fragment
<!ELEMENT TDL:arg (#PCDATA)> <!ELEMENT TDL:cmd (TDL:arg*)> <!ATTLIST TDL:cmd name CDATA #REQUIRED> <!ATTLIST TDL:arg xml:space (preserve) #FIXED 'preserve' >
In other words, the command name is made an attribute of the TDL:cmd element, while the arguments appear in sequence as the character data contents of TDL:arg elements. It may be observed that this makes the whitespace contents in TDL:arg elements highly significant.
This first block of code defines a command TDLtoXML::main which takes a TDL script as argument and returns the corresponding list of data-trees.
namespace eval TDLtoXML {
interp create -safe [list [namespace current]::theinterp]
theinterp hide namespace
theinterp invokehidden namespace delete ::
}
proc TDLtoXML::main {script} {
set res {}
theinterp eval $script
return $res
}
TDLtoXML::theinterp alias unknown [namespace current]::TDLtoXML::unknown
proc TDLtoXML::unknown {name args} {
switch -regexp -- $name {
{^[[:alpha:]_:][[:alnum:]_:.-]*$} {
set tree [list $name]
if {[llength $args] % 2} then {
lappend tree [lreplace $args end end]\
[main [lindex $args end]]
} else {
lappend tree $args {}
}
}
default {
set L {}
foreach arg $args {
lappend L [list TDL:arg {} [list [list \#text $arg]]]
}
set tree [list TDL:cmd [list name $name] $L]
}
}
uplevel 1 [list ::lappend res $tree]
}
proc TDLtoXML::slash {args} {
upvar 1 res res
foreach arg $args {
lappend res [list \#text $arg]
}
}
TDLtoXML::theinterp alias / [namespace current]::TDLtoXML::slashThis second block of code defines the TDLtoXML::xml_from_trees command, which goes the rest of the way: taking a list of data-trees as argument, and returning corresponding XML code. (One could alternatively use the appendFromList method of some tdom node object for this.) xml_from_trees also takes some options that can be used to influence indentation and the string map used for encoding characters as XML entities.
proc TDLtoXML::xml_from_trees {treeL args} {
array set Opt {
-nodesep \n
-indentstep ""
-charmap {}
}
array set Opt $args
if {![dict size $Opt(-charmap)]} then {
set Opt(-charmap) {< < > > & & ' ' \" "}
} else {
foreach char {< > & ' \"} entity {< > & ' "} {
if {![dict exists $O(-charmap) $char]} then {
dict set $O(-charmap) $char $entity
}
}
}
set res ""
foreach tree $treeL {
append res [
xml_from_treenode $tree $Opt(-nodesep) $Opt(-indentstep)\
$Opt(-charmap)
]
}
return $res
}
proc TDLtoXML::xml_from_treenode {tree sep indent map} {
switch -- [lindex $tree 0] "#text" - "#cdata" {
return [string map $map [lindex $tree 1]]
} "#comment" {
return "<!--[lindex $tree 1]-->"
} "#pi" {
return "<?[lindex $tree 1] [lindex $tree 2]?>"
} "TDL:arg" {
set sep ""
set indent ""
}
set res "<[lindex $tree 0]"
dict for {key val} [lindex $tree 1] {
append res { } $key =\" [string map $map $val] \"
}
if {[llength [lindex $tree 2]]} then {
set subsep $sep$indent
append res ">"
foreach child [lindex $tree 2] {
append res $subsep\
[xml_from_treenode $child $subsep $indent $map]
}
append res $sep "</[lindex $tree 0]>"
} else {
append res "/>"
}
return $res
}For example,
% TDLtoXML::main {OMI {/ 3}}
{OMI {} {{{#text} 3}}}
% TDLtoXML::xml_from_trees [TDLtoXML::main {OMI {/ 3}}]
<OMI>
3
</OMI>
To be continued…