request new command lstride

The following code is used:

proc lib_ME__copy_what {whatRef old new} {
  upvar $whatRef whatDEF
  global attributeDEF ARG_TYPE_ATTRIBUTE RET_TYPE_ATTRIBUTE ARG_DEFAULT

  set mapL  [list $old $new]

  array set attributeDEF \
    [concat {*}[lmap {k v} [array get attributeDEF $old,*]        {list [string map $mapL $k] $v} ]]
  array set ARG_TYPE_ATTRIBUTE \
    [concat {*}[lmap {k v} [array get ARG_TYPE_ATTRIBUTE $old,*]  {list [string map $mapL $k] $v} ]]
  array set RET_TYPE_ATTRIBUTE \
    [concat {*}[lmap {k v} [array get RET_TYPE_ATTRIBUTE $old,*]  {list [string map $mapL $k] $v} ]]
  array set ARG_DEFAULT \
    [concat {*}[lmap {k v} [array get ARG_DEFAULT $old,*]         {list [string map $mapL $k] $v} ]]
  set whatDEF($new) $whatDEF($old)
}

the syntax I recommend is

proc lib_ME__copy_what {whatRef old new} {
  upvar $whatRef whatDEF
  global attributeDEF ARG_TYPE_ATTRIBUTE RET_TYPE_ATTRIBUTE ARG_DEFAULT

  set mapL  [list $old $new]

  array set attributeDEF       [lmap {k v} [array get attributeDEF $old,*]        {lstride [string map $mapL $k] $v} ]
  array set ARG_TYPE_ATTRIBUTE [lmap {k v} [array get ARG_TYPE_ATTRIBUTE $old,*]  {lstride [string map $mapL $k] $v} ]
  array set RET_TYPE_ATTRIBUTE [lmap {k v} [array get RET_TYPE_ATTRIBUTE $old,*]  {lstride [string map $mapL $k] $v} ]
  array set ARG_DEFAULT        [lmap {k v} [array get ARG_DEFAULT $old,*]         {lstride [string map $mapL $k] $v} ]
  set whatDEF($new) $whatDEF($old)
}

The new command lstride has the same effect as {*} this mean create a flat list

more easy example

lappend x 1 2 3
=> 1 2 3

lappend a {*}[list 1 2 3]
=> 1 2 3

lappend b [lstride 1 2 3]
=> 1 2 3

lappend c [list 1 2 3]
=> {1 2 3}

arjen - 2024-05-17 11:28:46

The operation {*} does not create a flat list, but instead splits up the list in separate elements:

  • set a {*}[list 1 2 3]

wrong # args: should be "set varName ?newValue?"

results in an error, because the set command would get four arguments in stead of two.

Also, "stride" is usually concerned with stepping through a list with a certain step size. So in my opinion the name is misleading. I thought you were proposing a command to make a subselection of a list like:

  • lstride [list 1 2 3 4 5 6] 2

1 3 5

(and possibly more options ;))


Andreas Otto - 17 may 2024 - 14:25

the name flat list is just a wording used by me for a list being automatic expandet on a command-line. an other wording would be auto-expand-list

JMN - 2024-05-18

The lmap command, like any other in Tcl, is free to interpret its arguments as it wishes - and in this case it chooses to evaluate the script and append each result unflattened to a result list.

There is no command you can put inside the script argument to change the way it behaves in that regard, and it wouldn't make sense for Tcl to try to do that from within an argument anyway.

The functionality you seem to want is something I have wanted too - but the solution would involve something like a flag to lmap or an entirely separate command that interprets the result differently.

To use arbitrary variable names in the script argument whilst also allowing access to the current variables in the calling context gets a little complex though.

The following function 'captures' variable values in the calling context (they are available to the script but writes won't get written back to the original vars in the calling context)

I agree that it would be nice to have an lmap like function that returns a flat list - but not that it's something that can/should be done in the way you describe.

Hopefully someone can come up with a Tcl or C version nicer than this somewhat convoluted example - which is unlikely to be performant.

    proc lflatmap {varnames list script} {
        set result [list]
        set values [list]
        foreach v $varnames {
            lappend values "\$$v"
        }
        # -- --- ---
        #capture - use uplevel 1 or namespace eval depending on context
        set capture [uplevel 1 {
            apply { varnames  {
                set capturevars [dict create]
                set capturearrs [dict create]
                foreach fullv $varnames {
                    set  v [namespace tail $fullv]
                    upvar 1 $v var
                    if {[info exists var]} {
                        if {(![array exists var])} {
                            dict set capturevars $v $var
                        } else {
                            dict set capturearrs capturedarray_$v [array get var]
                        }
                    } else {
                        #A variable can show in the results for 'info vars' but still not 'exist'. e.g a 'variable x' declaration in the namespace where the variable has never been set
                    }
                }                        
                return [dict create vars $capturevars arrs $capturearrs]
            } } [info vars]  
        } ]
        # -- --- ---
        set cvars [dict get $capture vars]
        set carrs [dict get $capture arrs]
        foreach $varnames $list {
            lappend result {*}[apply\
                [list\
                    [concat $varnames [dict keys $cvars] [dict keys $carrs]]\
                    [string map [list %script% [list $script]] {
                        foreach arrayalias [info vars capturedarray_*] {
                            set realname [string range $arrayalias [string first _ $arrayalias]+1 end]
                            array set $realname [set $arrayalias][unset arrayalias]
                        } 
                        return [eval %script%]
                    }]\
                ] {*}[subst $values] {*}[dict values $cvars] {*}[dict values $carrs]]
        }
        return $result
    }

The version above is a form of closure. This is pretty inefficient as it copies a lot of data around especially if called from a context with a lot of variables such as the global namespace which has large arrays such as ::env

The following version links to the vars in the calling context, is simpler and much faster - but can write back to variables in the calling context. Whether that aspect is an advantage or not might depend on your view of functional programming concepts - but the speed advantage for the link version vs the capture version seems compelling. Speaking of functional programming - the use of the name 'flatmap' here is probably not ideal, as that has a different meaning in the fp world.

    proc lmapflat {varnames list script} {
        set result [list]
        set values [list]
        foreach v $varnames {
            lappend values "\$$v"
        }
        set linkvars [uplevel 1 [list info vars]]
        set nscaller [uplevel 1 [list namespace current]]

        set apply_script ""
        foreach vname $linkvars {
            append apply_script [string map [list %vname% $vname]\
             {upvar 2 %vname% %vname%}\
            ] \n
        }
        append apply_script $script \n
        
        #puts "--> $apply_script"
        foreach $varnames $list {
            lappend result {*}[apply\
                [list\
                    $varnames\
                    $apply_script\
                    $nscaller\
                ]  {*}[subst $values]\
            ]
        }
        return $result
    } 

Simpler still is the following.

It has side effects in that the vars in varnames leak into the calling context, which I don't like, but lmap does that anyway.

    proc lmapflat2 {varnames list script} {
        set result [list]
        set varlist [list]
        foreach varname $varnames {
            upvar 1 $varname var_$varname ;#ensure no collisions with vars in this proc
            lappend varlist  var_$varname
        }
        foreach $varlist $list {
            lappend result [uplevel 1 $script]
        }
        return [concat {*}$result]
    }

Finally there is the simplest option - which is close to what was originally being done in the code at the top - just put a trivial wrapper around it.

    proc lmapflat {varnames list script} {
        concat {*}[uplevel 1 [list lmap $varnames $list $script]]
    }

The earlier more complex options are more inline with functional program thinking in terms of reducing side effects - but often at a high cost in performance.

Ideally list operations in Tcl should be fast (which the current builtins are reasonably so I think) - as they tend to be called often in inner loops. For me - even the last wrapping option above is a layer too much in some situations - and I'd prefer a built in fast way to do it.

When I saw that lsearch had a -stride option - I tried things like:

    lsearch -all -inline -stride 2 {k1 v1 k2 v2 k3 v3} *
    k1 v1 k2 v2 k3 v3  ;# unexpected

I hope some of the above noise is useful to someone anyway.. and I'd like to see some more fast operations for unstructured lists, as in my experience they tend to crop up naturally every now and then.