**NAME**
''pandoc-tcl-filter'' - filter for pandoc to embed Tcl-code within Markdown documents and show the output. Provides as well an infrastructure to create your own filters for other tools using Tcl. Code examples for embedding [GraphViz]'s dot code, [Pikchr] diagrams, PIC diagrams, EQN or LaTeX equations, [R] plots and embedding the [svg] creator package [tsvg] (Thingy SVG creator) are given as well.
**DESCRIPTION**
The [pandoc] application is an advanced Markdown processor which can convert Markdown documents into other document formats like pdf, html or docx. As Markdown itself offers rather limited capabilities in formatting and no advanced features like references, dynamic embedding of other tools like flow chart tools or programming languages etc., the pandoc tool allows filtering of the Markdown processors before final document conversion using user written extensions. Pandoc supports per default filters using [Lua]. See here for the pandoc documentation about this: https://pandoc.org/lua-filters.html. Based on the code of [TR] on the [pandoc] Wiki page I implemented an initial version for a Tcl filter coded in Tcl which allows the embedding of Tcl code and evaluation of this code inside Markdown documents. This can be seen as an simpler alternative to the [tmdoc::tmdoc] package which is easier to embed into the [pandoc] universe. The [tmdoc::tmdoc] package in contrast can be used without using [pandoc] at all to convert such Tcl documents into HTML using just the Tcl packages [tmdoc::tmdoc] and [tcllib]'s [Markdown] package. So [pandoc-tcl-filter] is a filter in the pandoc universe whereas [tmdoc::tmdoc] is a Tcl only solution which is independent of pandoc.
**LINKS**
* Author: [Detlef Groth]
* Homepage: https://github.com/mittelmark/DGTcl - part of the dgw package
* Download: (download pandoc-tcl-folder only via downgit) https://downgit.github.io/#/home?url=https://github.com/mittelmark/DGTcl/tree/master/pandoc-tcl-filter
* Readme: http://htmlpreview.github.io/?https://github.com/mittelmark/DGTcl/blob/master/pandoc-tcl-filter/Readme.html
* Examples: https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/examples/example-dot.html
* Presentation at the S & T 2021: https://github.com/mittelmark/DGTcl/blob/master/pandoc-tcl-filter/doc/Groth-S-and-T-2021.pdf
* Source code: https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/pandoc-tcl-filter.tcl
* Version: 0.3.3
* License: MIT
**Filters**
The following filters are currently available:
* {.tcl} - the Tcl filter - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/examples/example-tcl.html%|%example%|%
* {.dot} - filter for [GraphViz] tools dot and neato - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/examples/example-dot.html%|%example%|%
* {.eqn} - filter for EQN equations (groff tool) - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/examples/example-pic.html%|%example%|%
* {.mtex} - filter for LaTex equations - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/examples/example-mtex.html%|%example%|%
* {.pik} - filter for [Pikchr] diagrams - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/examples/example-pik.html%|%example%|%
* {.pic} - filter for PIC diagrams (groff tool) - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/examples/example-pic.html%|%example%|%
* {.rplot} - filter for [R] plots - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/examples/example-rplot.html%|%example%|%
* {.tsvg} - filter for the [tsvg] package - https://htmlpreview.github.io/?https://raw.githubusercontent.com/mittelmark/DGTcl/master/pandoc-tcl-filter/examples/example-tsvg.html%|%example%|%
For more code examples and general comments about installation and usage, see the http://htmlpreview.github.io/?https://github.com/mittelmark/DGTcl/blob/master/pandoc-tcl-filter/Readme.html%|%Readme.html%|% file of the distribution.
**Installation**
All the filters mentioned above are added to a single file Tcl script which contains as well the require *rl_json* libraries for Linux and Windows. Ths file can be downloaded from here https://github.com/mittelmark/DGTcl/releases/download/latest/pandoc-tcl-filter.tapp
Copy this file to a folder in your PATH and make it executable. Using the following command line you can then convert for instance Markdown documents with embedded Tcl code or code for the other filters into an other document format like HTML, DOCX, PDF etc.
======
$ pandoc -s -o output.html --filter pandoc-tcl-filter.tapp input.md
======
**pandoc-tcl-filter.tcl**
Below is the Tcl code for the first version, the latest code can be retrieved using the links above.
======
#!/usr/bin/env tclsh
package require rl_json
# some useful commandline commands to know
# applying the filter:
# pandoc test.md -s --metadata title="test run with tcl filter" -o test.html --filter dgtools/filter.tcl
# inspection commands:
# pandoc -s -t json test.md > test.json
# python3 -m json.tool test.json | less
# pandoc -s -t native test.md | head # easier to read
# developing:
# cat test.json | tclsh dgtools/filter.tcl
# # read the JSON AST from stdin
set jsonData {}
while {[gets stdin line] > 0} {
append jsonData $line
}
proc debug {jsonData} {
puts [::rl_json::json keys $jsonData]
}
# TR example from https://wiki.tcl-lang.org/page/pandoc
# which modifies the header level
proc incrHeader {jsonData} {
for {set i 0} {$i < [llength [::rl_json::json get $jsonData blocks]]} {incr i} {
set blockType [::rl_json::json get $jsonData blocks $i t]
if {$blockType eq "Header"} {
set headerLevel [::rl_json::json get $jsonData blocks $i c 0]
set jsonData [::rl_json::json set jsonData blocks $i c 0 [expr {$headerLevel + 1}]]
}
}
return $jsonData
}
# the code which process inline Tcl code
proc evalTclCode {jsonData} {
# the interpreter which executes the code chunks
interp create mdi
mdi eval {
set res ""
rename puts puts.orig
proc puts {args} {
global res
if {[lindex $args 0] eq "-nonewline"} {
append res "[lindex $args 1]"
} else {
append res "[lindex $args 0]\n"
}
return ""
}
}
# collect all blocks and if we have a Tcl block create and append new blocks
set blocks ""
for {set i 0} {$i < [llength [::rl_json::json get $jsonData blocks]]} {incr i} {
if {$i > 0} {
append blocks "," ;# migh get trouble if echo=false results=hide ?
}
set blockType [::rl_json::json get $jsonData blocks $i t]
if {$blockType eq "CodeBlock"} {
# content has three array elements, type, attributes and the code block code
set type [rl_json::json get $jsonData blocks $i c 0 1] ;#type
set attr [rl_json::json get $jsonData blocks $i c 0 2] ;# attributes
set a [dict create echo true results show eval true]
if {[llength $attr] > 0} {
foreach el $attr {
dict set a [lindex $el 0] [lindex $el 1]
}
#puts [dict keys $a]
}
if {[dict get $a echo]} {
append blocks "[::rl_json::json extract $jsonData blocks $i]\n"
}
set cont [rl_json::json get $jsonData blocks $i c 1]
set cblock "[::rl_json::json extract $jsonData blocks $i]"
if {$type eq "tcl"} {
if {[dict get $a eval]} {
mdi eval "set res {}"
set eres [mdi eval $cont]
set eres "[mdi eval {set res}]$eres"
if {[dict get $a results] eq "show"} {
rl_json::json set cblock c 0 1 [rl_json::json array [list string tclout]]
rl_json::json set cblock c 1 [rl_json::json string [regsub {\{(.+)\}} $eres "\\1"]]
append blocks ",[::rl_json::json extract $cblock]"
}
}
}
} elseif {$blockType eq "Para"} {
# check for inline Tcl commands like `tcl set x 1`
for {set j 0} {$j < [llength [::rl_json::json get $jsonData blocks $i c]]} {incr j} {
set type [rl_json::json get $jsonData blocks $i c $j t] ;#type
if {$type eq "Code"} {
set code [rl_json::json get $jsonData blocks $i c $j c]
set code [lindex $code 1]
if {[regexp {\.?tcl } $code]} {
set c [regsub {^[^ ]+} $code ""]
set res [interp eval mdi $c]
set jsonData [rl_json::json set jsonData blocks $i c $j c 1 [rl_json::json string $res]]
}
}
}
append blocks "[::rl_json::json extract $jsonData blocks $i]\n"
} else {
append blocks "[::rl_json::json extract $jsonData blocks $i]\n"
}
}
set ret "\"blocks\":\[$blocks\]"
append ret ",\"pandoc-api-version\":[::rl_json::json extract $jsonData pandoc-api-version]"
append ret ",\"meta\":[::rl_json::json extract $jsonData meta]"
return "{$ret}"
}
# give the modified document back to Pandoc again:
puts -nonewline [evalTclCode $jsonData]
======
**Markdown example (test.md)**
======
---
## YAML START
### Created By : Detlef Groth
### Created : Sun Aug 22 14:37:16 2021
#### Last Modified : <210823.0605> Pandoc Header
title: "The pandoc-tcl-filter written in Tcl"
shorttitle: "pandoc-tcl-filter"
author:
- Detlef Groth
date: 2021-21-22
## YAML END
---
# Header 1 - Inline code set
X is `tcl set x 1`.
## Subheader 1.1 - Inline code time
Current time is: `tcl clock format [clock seconds] -format "%Y-%m-%d %H:%M"`
# Header 2 - code blocks
Some text, first the obvious "Hello World" example.
```{.tcl echo=true results=show}
puts "Hello World!"
```
Now an example with two puts statements;
```{.tcl echo=true results=show}
puts "Hello World!"
puts "End of TclCode!"
```
More text. Then a raw code example which is not evaluated:
```
raw code line 1
raw code line 2
```
Now some more examples which just return the last line of the code chunk:
```{.tcl}
set x 1
set x
```
Now let's use results=hide to hide the results:
```{.tcl results=hide}
set x 2
set x
```
Above there is nothing shown. Next we set eval=false and results=hide.
```{.tcl eval=false results=hide}
set x 3
set x
```
Again no output and x should be still two, let's check this:
```{.tcl}
set x
```
```{.tcl}
expr { "9" == "9.0"}
```
Does inline text works? `.tcl set x` - not yet ...!
But should work with the github version!
```{.tcl}
expr { "9" eq "9.0"}
```
Now an example with a function:
```{.tcl}
proc sum {arg1 arg2} {
set x [expr {$arg1 + $arg2}];
return $x
}
puts " The sum of 2 + 3 is: [sum 2 3]\n\n"
```
## Dot filter example
Here an example code of an other filter in an embedded code chunk for using a `.dot` filter:
======
```{.dot ext=pdf echo=false width=6}
digraph G {
size="6;9";
margin=0.1;
node[style=filled,fillcolor=skyblue;shape="box",width=1,fontname="Alegreya"];
{ rank=same; Rmd ; R; Md ; Pandoc; }
Rmd -> R -> Md -> Pandoc;
Pandoc -> HTML;
Pandoc -> Docx;
Pandoc -> Pdf;
R[shape=box,style=filled,fillcolor=cornsilk,];
Pandoc[shape=box,style=filled,fillcolor=cornsilk];
R[label="R/Knitr"];
}
```
======
And here the output:
[pandoc-tcl-filter-dot-example]
## END
======
**Command line use**
======
pandoc test.md -s --metadata title="test run with tcl filter" -o test.html \
-M date="`date "%B %e, %Y %H%M"`" --filter pandoc-tcl-filter.tcl
======
**TODO's**
* code labels
* graphics include=true|false width=px height=px etc.
* Tcl inline commands in the Meta block - skipped use the date option for pandoc directly as can be seen in the Makefile, see above as well in command line use
* show use cases for other packages like [chesschart] or [gdtcl]
* other non Tcl-code filters written in Tcl for instance:
1. .csv for loading and displaying csv files (walk over lines and create Markdown table)
2. .dot for displaying Graphviz dot files (done)
----
**History**
* 2021-08-21 - project started
* 2021-08-23 - this wiki page created
* 2021-08-25 - github folder within https://github.com/mittelmark/DGTcl%|%DGTcl%|%
* 2021-08-26 - adding [svg] creator for images using [Thingy: a one-liner OO system]
* 2021-08-28 - adding Tcl filter infrastructure with examples for tsvg - the thingy svg creator and GraphViz's dot file processors as examples
* 2021-11-15 - adding --help flag
* 2021-11-23 - adding pikchr, pik, eqn filters, extending documentation
* 2021-11-30 - adding rplot filter for embedding R code
**Discussion**
Please discuss here. Suggestions, ideas are welcome.
[DDG] 2021-08-21: Please note, that this is a work in progress.
[DDG] 2021-08-25: Latest code is now on github project https://github.com/mittelmark/DGTcl%|%DGTcl%|%
[DDG] 2021-11-30: After the Tcl conference I added a few more filters which are included in the standalone application, filters for [Pikchr], the [R] statistical language and for the groff tools eqn2graph and pic2graph
----
<<categories>> Documentation