pandoc-tcl-filter

NAME

Since version 0.9.0 the application is renamed to pantcl and it is now in its own repository at https://github.com/mittelmark/pantcl

Since version 0.7.0 there is as well a graphical user interface to the filters, see here GUI Manual and below for a slideshow.

pandoc-tcl-filter - filter for pandoc to embed Tcl-code within Markdown documents and show the output. Provides as well an infrastructure to create your own filters for other tools using Tcl. Code examples for embedding GraphViz's dot code, Pikchr diagrams, PIC diagrams, EQN or LaTeX equations, PlantUML diagrams, R plots and embedding the svg creator package tsvg (Thingy SVG creator) are given as well. Since version 0.7.0 there is as well a standalone mode for convering from Markdown to HTML only and a graphical user interface which does not need pandoc.

DESCRIPTION

The pandoc application is an advanced Markdown processor that can convert Markdown documents into other document formats like pdf, html or docx. As Markdown itself offers rather limited capabilities in formatting and no advanced features like references, dynamic embedding of other tools like flow chart tools or programming languages etc., the pandoc tool allows filtering of the Markdown processors before final document conversion using user written extensions. Pandoc supports per default filters using Lua. See here for the pandoc documentation about this: https://pandoc.org/lua-filters.html . Based on the code of TR on the pandoc Wiki page I implemented an initial version for a Tcl filter coded in Tcl that allows the embedding of Tcl code and evaluation of this code inside Markdown documents. This can be seen as an simpler alternative to the tmdoc::tmdoc package that is easier to embed into the pandoc universe. The tmdoc::tmdoc package in contrast can be used without using pandoc at all to convert such Tcl documents into HTML using just the Tcl packages tmdoc::tmdoc and tcllib's Markdown package. So pandoc-tcl-filter is a filter in the pandoc universe whereas tmdoc::tmdoc is a Tcl-only solution, which is independent of pandoc. Since version 0.4.0 pandoc-tcl-filter.tcl can be as well used directly to extract Markdown documentation and process it directly.

Attention! Since version 0.7.0 there is as well a graphical user interface to the filters, see here GUI Manual

LINKS

Filters

The following filters are currently available:

For more code examples and general comments about installation and usage, see the Readme.html file of the distribution.

Here an image about the infrastructure taken from the manual page :

pandoc-tcl-filter-infrastructure

Installation

All the filters mentioned above are added to a single file Tcl script that contains as well the require *rl_json* libraries for Linux and Windows. This file can be downloaded from here https://github.com/mittelmark/DGTcl/releases/download/latest/pandoc-tcl-filter.tapp

Copy this file to a folder in your PATH and make it executable. Using the following command line you can then convert, for instance, Markdown documents with embedded Tcl code or code for the other filters into an other document formats like HTML, DOCX, PDF, etc.

$ pandoc -s -o output.html --filter pandoc-tcl-filter.tapp input.md

pandoc-tcl-filter.tcl

Below is the Tcl code for the first version; the latest code can be retrieved using the links above.

#!/usr/bin/env tclsh
package require rl_json
# some useful commandline commands to know
# applying the filter:
# pandoc test.md -s --metadata title="test run with tcl filter" -o test.html --filter dgtools/filter.tcl
# inspection commands:
# pandoc -s -t json test.md > test.json
# python3 -m json.tool test.json | less
# pandoc -s -t native test.md | head # easier to read
# developing:
# cat test.json | tclsh dgtools/filter.tcl
# # read the JSON AST from stdin
set jsonData {}
while {[gets stdin line] > 0} {
   append jsonData $line
}

proc debug {jsonData} {
    puts [::rl_json::json keys $jsonData]
}

# TR example from https://wiki.tcl-lang.org/page/pandoc
# which modifies the header level
proc incrHeader {jsonData} {
    for {set i 0} {$i < [llength [::rl_json::json get $jsonData blocks]]} {incr i} {
        set blockType [::rl_json::json get $jsonData blocks $i t]
        if {$blockType eq "Header"} {
            set headerLevel [::rl_json::json get $jsonData blocks $i c 0]
            set jsonData [::rl_json::json set jsonData blocks $i c 0 [expr {$headerLevel + 1}]]
        } 
    }
    return $jsonData
}

# the code which process inline Tcl code
proc evalTclCode {jsonData} {
    # the interpreter which executes the code chunks
    interp create mdi
    mdi eval {
        set res ""
        
        rename puts puts.orig
        
        proc puts {args} {
            global res
            if {[lindex $args 0] eq "-nonewline"} {
                append res "[lindex $args 1]"
            } else {
                append res "[lindex $args 0]\n"
            }
            return ""
        }
    }
    # collect all blocks and if we have a Tcl block create and append new blocks
    set blocks ""
    for {set i 0} {$i < [llength [::rl_json::json get $jsonData blocks]]} {incr i} {
        if {$i > 0} {
            append blocks "," ;# migh get trouble if echo=false results=hide ?
        }
        set blockType [::rl_json::json get $jsonData blocks $i t]
        if {$blockType eq "CodeBlock"} {
            # content has three array elements, type, attributes and the code block code 
            set type [rl_json::json get $jsonData blocks $i c 0 1] ;#type
            set attr [rl_json::json get $jsonData blocks $i c 0 2] ;# attributes
            set a [dict create echo true results show eval true]
            if {[llength $attr] > 0} {
                foreach el $attr {
                    dict set a [lindex $el 0] [lindex $el 1]
                }
                #puts [dict keys $a]
            }
            if {[dict get $a echo]} {
                append blocks "[::rl_json::json extract $jsonData blocks $i]\n"
            }
            set cont [rl_json::json get $jsonData blocks $i c 1]
            set cblock "[::rl_json::json extract $jsonData blocks $i]"
            if {$type eq "tcl"} {
                if {[dict get $a eval]} {
                    mdi eval "set res {}"
                    set eres [mdi eval $cont]
                    set eres "[mdi eval {set res}]$eres"
                    if {[dict get $a results] eq "show"} {
                        rl_json::json set cblock c 0 1 [rl_json::json array [list string tclout]]
                        rl_json::json set cblock c 1 [rl_json::json string [regsub {\{(.+)\}} $eres "\\1"]]
                        append blocks ",[::rl_json::json extract $cblock]"
                    }
                }
            }
        } elseif {$blockType eq "Para"} {
            # check for inline Tcl commands like `tcl set x 1`
            for {set j 0} {$j < [llength [::rl_json::json get $jsonData blocks $i c]]} {incr j} {
                set type [rl_json::json get $jsonData blocks $i c $j t] ;#type
                if {$type eq "Code"} {
                    set code [rl_json::json get $jsonData blocks $i c $j c]
                    set code [lindex $code 1]
                    if {[regexp {\.?tcl } $code]} {
                        set c [regsub {^[^ ]+} $code ""]
                        set res [interp eval mdi $c]
                        set jsonData [rl_json::json set jsonData blocks $i c $j c 1 [rl_json::json string $res]]
                    }
                }
            }
            append blocks "[::rl_json::json extract $jsonData blocks $i]\n"
        } else {
            append blocks "[::rl_json::json extract $jsonData blocks $i]\n"
        }
    }
    set ret "\"blocks\":\[$blocks\]"
    append ret ",\"pandoc-api-version\":[::rl_json::json extract $jsonData pandoc-api-version]"
    append ret ",\"meta\":[::rl_json::json extract $jsonData meta]"
    return "{$ret}"
}

# give the modified document back to Pandoc again:

puts -nonewline [evalTclCode $jsonData]

Markdown example (test.md)

---
## YAML START
###  Created By    : Detlef Groth
###  Created       : Sun Aug 22 14:37:16 2021
####  Last Modified : <210823.0605> Pandoc Header
title: "The pandoc-tcl-filter written in Tcl"
shorttitle: "pandoc-tcl-filter"
author: 
- Detlef Groth
date: 2021-21-22
## YAML END
---

# Header 1 - Inline code set

X is `tcl set x 1`.

## Subheader 1.1 - Inline code time

Current time is: `tcl clock format [clock seconds] -format "%Y-%m-%d %H:%M"`

# Header 2 - code blocks

Some text, first the obvious "Hello World" example.

```{.tcl echo=true results=show}
puts "Hello World!"
```

Now an example with two puts statements;

```{.tcl echo=true results=show}
puts "Hello World!"
puts "End of TclCode!"
```

More text. Then a raw code example that is not evaluated:

```
raw code line 1
raw code line 2
```

Now some more examples that just return the last line of the code chunk:

```{.tcl}
set x 1
set x
```

Now let's use results=hide to hide the results:

```{.tcl results=hide}
set x 2
set x
```

Above there is nothing shown. Next we set eval=false and results=hide.


```{.tcl eval=false results=hide}
set x 3
set x
```

Again no output and x should be still two, let's check this:

```{.tcl}
set x
```


```{.tcl}
expr { "9" == "9.0"}
```

Does inline text works? `.tcl set x`  - not yet ...! 
But should work with the github version!


```{.tcl}
expr { "9" eq "9.0"}
```
Now an example with a function:

```{.tcl}
proc sum {arg1 arg2} {
    set x [expr {$arg1 + $arg2}];
    return $x
}

puts " The sum of 2 + 3 is: [sum 2 3]\n\n"
```

## EOF

Dot filter example

Here an example code of another filter in an embedded code chunk for using a .dot filter:

```{.dot ext=pdf echo=false width=6}
digraph G {
    size="6;9";
    margin=0.1;
    node[style=filled,fillcolor=skyblue;shape="box",width=1,fontname="Alegreya"];
    { rank=same; Rmd ; R; Md ; Pandoc; }
    Rmd -> R -> Md -> Pandoc;
    Pandoc -> HTML;
    Pandoc -> Docx;
    Pandoc -> Pdf;
    R[shape=box,style=filled,fillcolor=cornsilk,];
    Pandoc[shape=box,style=filled,fillcolor=cornsilk];
    R[label="R/Knitr"];
}
```

And here the output:

pandoc-tcl-filter-dot-example

Command line use

pandoc test.md -s --metadata title="test run with tcl filter" -o test.html \ 
   -M date="`date "%B %e, %Y %H%M"`" --filter pandoc-tcl-filter.tcl

TODO's

  • code labels
  • graphics include=true|false width=px height=px etc.
  • Tcl inline commands in the Meta block - skipped use the date option for pandoc directly as can be seen in the Makefile, see above as well in command line use
  • show use cases for other packages like chesschart or gdtcl
  • other non Tcl-code filters written in Tcl for instance:
  1. .csv for loading and displaying csv files (walk over lines and create Markdown table)
  2. .dot for displaying Graphviz dot files (done)

Graphical User Interface

Since version 0.7.0 there is as well a graphical user interface to the filters which have a graphics output. Have a look here for the manual page . This way you can easily edit your code and preview the output. Here a slideshow for some examples:

filter-view-image

History

  • 2021-08-21 - project started
  • 2021-08-23 - this wiki page created
  • 2021-08-25 - github folder within DGTcl
  • 2021-08-26 - adding svg creator for images using Thingy: a one-liner OO system
  • 2021-08-28 - adding Tcl filter infrastructure with examples for tsvg - the thingy svg creator and GraphViz's dot file processors as examples
  • 2021-11-15 - adding --help flag
  • 2021-11-23 - adding pikchr, pik, eqn filters, extending documentation
  • 2021-11-30 - adding rplot filter for embedding R code
  • 2021-12-04 - running as stand-alone application calling pandoc internally with itself as filter
  • 2021-12-19 - adding include filter, ABC music, Mermaid, PlantUML, SQLite3 application, results="asis"
  • 2022-01-09 - adding new filters for shell-scripts (filter-cmd), song books (filter-tcrd) and the tdot package (filter-tdot)
  • 2022-02-07 - adding new filter pipes with embedding Python, R and Octave scripts, standalone mode for conversion without pandoc from Markdown to HTML and graphical user interface

Discussion

Please discuss here. Suggestions, ideas are welcome.

DDG 2021-08-21: Please note, that this is a work in progress.

DDG 2021-08-25: Latest code is now on github project DGTcl

DDG 2021-11-30: After the Tcl conference I added a few more filters that are included in the standalone application, filters for Pikchr, the R statistical language and for the groff tools eqn2graph and pic2graph

DDG 2021-12-04: Since version 0.4.0 the pandoc-tcl-filter.tcl file can be as well used directly as document converter as it can calls pandoc then internally itself

TR 2021-12-09: Nice stuff. How can I make the filter output the Tcl results without having it wrapped into {.tclout} code blocks? My use case is having Tcl produce a markdown table which gets inserted inline into a markdown document. So I convert from markdown to markdown and just want Tcl to produce the markdown tables. A second pandoc run would then convert the expanded markdown into PDF or other formats. Is that possible?

DDG 2021-12-10: You mean that code block like this:

```{.tcl results="asis"}
set tab {
| Col1          | Col2          | Col3          | Col4          |
| ------------- | ------------- | ------------- | ------------- |
}
foreach i [list 1 2 3 4 5] {
    append tab "| cell $i,1   | cell $i,2 | cell $i,3 | cell $i,3 |\n"
}
set tab
```

should be rendered as Markdown table? Probably results="md" should as well work. As Tcl does not know the filetype which should be the final output. With results="tex" You could then create LaTeX tables with Tcl as well. I have already an implementation for the code above on my machine, which I could upload to git already.

TR 2021-12-10: Yes. The output of the Tcl script should be written to the document directly, without putting it into a markdown backticked code block with the {.tclout} class. I thought the {.tcl results="asis"} was for that purpose, leaving the Tcl output as is, but it still wraps the output into a code block. So, I also wonder what the "asis" is for, as this is not documented in the manual or readme. - In the end I would like to be able to do the same operations as in Rmarkdown where I can choose to produce the document and configure the output in various ways or I can produce a Tcl script and strip the other markdown off (with purl(), render() and knit()).

Oh, but I see the complexity now. Typically, we would need to produce proper json AST code for tables which is quite hard probably. But when converting from markdown to markdown anyway, we can still let the Tcl script spit out simple markdown table syntax, but then we are looking for the RawBlock AST element. I think that will be the easiest path!

DDG 2021-12-10: I have solved this issue by writing the Markdown code in a tempoary file and then calling pandoc with -t json option and extracting the relevant part of the json tree. Not very efficient but it seems to work. See here:

To test this you should try the standalone executable probably: https://github.com/mittelmark/DGTcl/suites/4624501730/artifacts/125180612

With this standalone Tcl file you should be able to do simply:

pandoc-tcl-filter.tapp table.md table.html -s --css yourstyle.css

There is no need to convert first to Markdown and then to HTML as the pandoc call creates the AST already. Hope this helps.

Oj, I see. Quite a neat 'trick'! I am testing right now, but have problems with an unmatched brace ... let's take this specific discussion to Github where it might be easier to handle and doesn't fill the Wiki with too much noise. See this issue

DDG - 2021-12-21: With version 0.5.0 you can use results="asis" to create directly Markdown code using your Tcl scripts, furthermore new filters were added for PlantUML!! (great tool, easy to install!), SQLite3 terminal application (embed and display SQL code and the resulting output in your documents), ABC music notation , Mermaid diagrams . Adding a new filter, just takes a few minutes. Furthermore it is now possible to include other Markdown files using the results="asis" type of chunk option. Details are explained in the manual page.

DDG - 2022-01-09: Adding filters for shell and scripting language scripts (Octave, R, Gnuplot, ...) - see filter-cmd.html , music lyrics with chords above - see filter-tcrd.html , and large extension of the mtex filter with many examples for LaTeX packages, such as tikz, pgf, skak, sudoku - see filter-mtex.html and a filter for the tdot package - see filter-tdot.html .

DDG - 2022-02-07: version 0.7.0 brings a graphical user interface - GUI Manual - and a standalone mode for converting directly from Markdownto HTML without using pandoc. Furthermore a pipe-filter was added which supports as well R, Python and Octave scripts. See here for the documentation: filter-pipe.html .

DDG - 2023-01-12: version 0.9.0 project was moved to its own repository at https://github.com/mittelmark/pantcl and application was renamed to pantcl.

Category Documentation
Updated 2023-01-12 12:02:05