**NAME**

''pandoc-tcl-filter.tcl'' - filter for pandoc to embed Tcl-code within Markdown documents and show the output (WIP).

**DESCRIPTION**

The [pandoc] application is an advanced Markdown processor which can convert Markdown documents into other document formats like pdf, html or docx. As Markdown itself offers rather limited capabilities in formatting and no advanced features like references, dynamic embedding of other tools like flow chart tools or programming languages etc., the pandoc tool allows filtering of the Markdown processors before final document conversion using user written extensions. Pandoc supports per default filters using [Lua]. See here for the pandoc documentation about this: https://pandoc.org/lua-filters.html. Based on the code of [TR] on the [pandoc] Wiki page I implemented an initial version for a Tcl filter coded in Tcl which allows the embedding of Tcl code and evaluation of this code inside Markdown documents. This can be seen as an simpler alternative to the [tmdoc::tmdoc] package which is easier to embed into the [pandoc] universe. The [tmdoc::tmdoc] package in contrast can be used without using [pandoc] at all to convert such Tcl documents into HTML using just the Tcl packages [tmdoc::tmdoc] and [tcllib]'s [Markdown] package. So [pandoc-tcl-filter] is a filter in the pandoc universe whereas [tmdoc::tmdoc] is a Tcl only solution which is independent of pandoc. 

**pandoc-tcl-filter.tcl**

Here is the Tcl code:

======
#!/usr/bin/env tclsh
package require rl_json
# some useful commandline commands to know
# applying the filter:
# pandoc test.md -s --metadata title="test run with tcl filter" -o test.html --filter dgtools/filter.tcl
# inspection commands:
# pandoc -s -t json test.md > test.json
# python3 -m json.tool test.json | less
# pandoc -s -t native test.md | head # easier to read
# developing:
# cat test.json | tclsh dgtools/filter.tcl
# # read the JSON AST from stdin
set jsonData {}
while {[gets stdin line] > 0} {
   append jsonData $line
}

proc debug {jsonData} {
    puts [::rl_json::json keys $jsonData]
}

# TR example from https://wiki.tcl-lang.org/page/pandoc
# which modifies the header level
proc incrHeader {jsonData} {
    for {set i 0} {$i < [llength [::rl_json::json get $jsonData blocks]]} {incr i} {
        set blockType [::rl_json::json get $jsonData blocks $i t]
        if {$blockType eq "Header"} {
            set headerLevel [::rl_json::json get $jsonData blocks $i c 0]
            set jsonData [::rl_json::json set jsonData blocks $i c 0 [expr {$headerLevel + 1}]]
        } 
    }
    return $jsonData
}

# the code which process inline Tcl code
proc evalTclCode {jsonData} {
    # the interpreter which executes the code chunks
    interp create mdi
    mdi eval {
        set res ""
        
        rename puts puts.orig
        
        proc puts {args} {
            global res
            if {[lindex $args 0] eq "-nonewline"} {
                append res "[lindex $args 1]"
            } else {
                append res "[lindex $args 0]\n"
            }
            return ""
        }
    }
    # collect all blocks and if we have a Tcl block create and append new blocks
    set blocks ""
    for {set i 0} {$i < [llength [::rl_json::json get $jsonData blocks]]} {incr i} {
        if {$i > 0} {
            append blocks "," ;# migh get trouble if echo=false results=hide ?
        }
        set blockType [::rl_json::json get $jsonData blocks $i t]
        if {$blockType eq "CodeBlock"} {
            # content has three array elements, type, attributes and the code block code 
            set type [rl_json::json get $jsonData blocks $i c 0 1] ;#type
            set attr [rl_json::json get $jsonData blocks $i c 0 2] ;# attributes
            set a [dict create echo true results show eval true]
            if {[llength $attr] > 0} {
                foreach el $attr {
                    dict set a [lindex $el 0] [lindex $el 1]
                }
                #puts [dict keys $a]
            }
            if {[dict get $a echo]} {
                append blocks "[::rl_json::json extract $jsonData blocks $i]\n"
            }
            set cont [rl_json::json get $jsonData blocks $i c 1]
            set cblock "[::rl_json::json extract $jsonData blocks $i]"
            if {$type eq "tcl"} {
                if {[dict get $a eval]} {
                    mdi eval "set res {}"
                    set eres [mdi eval $cont]
                    set eres "[mdi eval {set res}]$eres"
                    if {[dict get $a results] eq "show"} {
                        rl_json::json set cblock c 0 1 [rl_json::json array [list string tclout]]
                        rl_json::json set cblock c 1 [rl_json::json string [regsub {\{(.+)\}} $eres "\\1"]]
                        append blocks ",[::rl_json::json extract $cblock]"
                    }
                }
            }
        } elseif {$blockType eq "Para"} {
            # check for inline Tcl commands like `tcl set x 1`
            for {set j 0} {$j < [llength [::rl_json::json get $jsonData blocks $i c]]} {incr j} {
                set type [rl_json::json get $jsonData blocks $i c $j t] ;#type
                if {$type eq "Code"} {
                    set code [rl_json::json get $jsonData blocks $i c $j c]
                    set code [lindex $code 1]
                    if {[regexp {\.?tcl } $code]} {
                        set c [regsub {^[^ ]+} $code ""]
                        set res [interp eval mdi $c]
                        set jsonData [rl_json::json set jsonData blocks $i c $j c 1 [rl_json::json string $res]]
                    }
                }
            }
            append blocks "[::rl_json::json extract $jsonData blocks $i]\n"
        } else {
            append blocks "[::rl_json::json extract $jsonData blocks $i]\n"
        }
    }
    set ret "\"blocks\":\[$blocks\]"
    append ret ",\"pandoc-api-version\":[::rl_json::json extract $jsonData pandoc-api-version]"
    append ret ",\"meta\":[::rl_json::json extract $jsonData meta]"
    return "{$ret}"
}

# give the modified document back to Pandoc again:

puts -nonewline [evalTclCode $jsonData]

======

**Markdown example (test.md)**

======
---
## YAML START
###  Created By    : Detlef Groth
###  Created       : Sun Aug 22 14:37:16 2021
####  Last Modified : <210823.0605> Pandoc Header
title: "The pandoc-tcl-filter written in Tcl"
shorttitle: "pandoc-tcl-filter"
author: 
- Detlef Groth
date: 2021-21-22
## YAML END
---

# Header 1 - Inline code set

X is `tcl set x 1`.

## Subheader 1.1 - Inline code time

Current time is: `tcl clock format [clock seconds] -format "%Y-%m-%d %H:%M"`

# Header 2 - code blocks

Some text, first the obvious "Hello World" example.

```{.tcl echo=true results=show}
puts "Hello World!"
```

Now an example with two puts statements;

```{.tcl echo=true results=show}
puts "Hello World!"
puts "End of TclCode!"
```

More text. Then a raw code example which is not evaluated:

```
raw code line 1
raw code line 2
```

Now some more examples which just return the last line of the code chunk:

```{.tcl}
set x 1
set x
```

Now let's use results=hide to hide the results:

```{.tcl results=hide}
set x 2
set x
```

Above there is nothing shown. Next we set eval=false and results=hide.


```{.tcl eval=false results=hide}
set x 3
set x
```

Again no output and x should be still two, let's check this:

```{.tcl}
set x
```


```{.tcl}
expr { "9" == "9.0"}
```

Does inline text works? `.tcl set x`  - not yet ...!


```{.tcl}
expr { "9" eq "9.0"}
```
Now an example with a function:

```{.tcl}
proc sum {arg1 arg2} {
    set x [expr {$arg1 + $arg2}];
    return $x
}

puts " The sum of 2 + 3 is: [sum 2 3]\n\n"
```

## END

======

**Command line commands**

======
pandoc test.md -s --metadata title="test run with tcl filter" -o test.html --filter pandoc-tcl-filter.tcl
======
**TODO's**

    * graphics
    * Tcl inline commands in the Meta block
    * show use cases for other packages like [chesschart] or [gdtcl]

----

'''Discussion:'''

Please discuss here ...

[DDG] 2021-08-21: Please not, that this is a work in progress.

<<categories>> Documentation