'''or''':   Everything is a list of strings
   '''or''':   How to get rid of the [{*}]
----

On the page [Everything is a string] the discussion at some moment
turns to the topic '''Is everything a list?'''. [LES] on April 27,
2004: says there:

  [[..]] the more I use Tcl, the more I am convinced that everything
  is a list. And that's one of the best features in Tcl.

Since this questions is worth further exploration i set up this page.
In recent explorations of Tcl i got convinced that the [EIAS] point of
view is not only a shortcoming, but it could be even a great benefit
to consider that [Everything is a list] of strings.

First I want to discuss the execution model of Tcl. Fact is, that the
[Endekalogue] (or [Dodekalogue] as of Tcl 8.5) strives to, but is not
sufficient to understand how Tcl works, since it only describes
syntactic conventions.  Fact is also, that already in the [*dekalog]s
we learn implicitely about lists.  Now, how does Tcl its work:

    * A Tcl script is a string, that is firstly split into a list of commandlines, which are evaluated from first to last, unless one of them alters this execution order.

    * Each commandline is split into a list of words which are subject to substitutions.

    * The first word on the resulting list of words is looked up in the list of [command]s which, when found, is applied to the rest of the list of words.  Or in a different terminology: the rest of the list of words on the commandline are given as arguments to the first which is a [command].

We already can see, by virtue of this paraphrasing, that lists are at
the core of the Tcl language.

Now to the data nature of lists in Tcl. Some selected citations from
the [EIAS] page:

    * [KJN] 2004-11-04: The endekalogue does not define lists - it leaves it to commands to decide whether to interpret an argument as a list (and, implicitly, to define what is meant by a list).

    * [RHS] 05Nov2004: [[..]] you want you be able to say ''"this is a list"'' and ''"this is not a list"''. In Tcl, there's no such concept. You can only say "I want to treat this like a list" and "I want to treat this like some-other-thing-that-isn't-a-list".

    * [NEM] 25July05: [[..]] what is at the heart of this debate is not [[..]], but rather a recognition that the notion of a "type" of a value is extrinsic to the value itself.

In fact, it is not until the command receives its arguments as a list
of strings, that the type of each is interpretated either as string,
list, integer or float.

To stay with strings and lists consider the following, slightly
modified, example from [shimmering]:

======
# set x {a  b c}
a  b c
# puts $x
==> a  b c

======

`x` is set to a string, and `puts` expects a string.  However if we:

======
# lappend x d
a b c d
# puts $x
==> a b c d
# set x
a b c d
======

then the string stored in `x` is converted into a list of strings,
visible by the loss of the double space between the first two
elements. `puts` expects a string, so the list in `x` is converted
back to a string on the fly on invocation. `x` is not altered by this
conversion as shown on the following line.

----
'''How to get rid of the [{*}] expansion prefix.'''

'''or''': How to benefit most from the notion that
[Everything is a list] (of strings).

Since ever, in Tcl list handling had to be done somewhat quirky.  Up
to Tcl 8.4 this was expressed by the need of using the [eval] command
to unwrap lists on a commandline, which leeds one straight to
[Quoting hell].  In Tcl 8.5 the syntax of the Tcl language has been
changed, introducing the expansion prefix [{*}] which can be put in
front of any word to provide this [{expand}]ing.  This prefix breaks
implicitely with rule [[10]] of the [Endekalogue], which states that
each character is processed exactly once, and introduced the new era
of the [Dodekalogue].

In the following I suggest considering re-verting the Tcl syntax to a
- however modified - Endekalogue, and changing semantic instead of
syntax.  Maybe as a goal for [The Mystical Tcl 9.0]. Back to the
roots!

Let's revisit the execution model.  While, rule [[6]] (or [[7]] in
8.5) talks about command substitution, which is the term for function
calls in the Tcl language, it does not explicitely state that every
command returns exactly zero or one result.  More precisely, a command
returns either an empty string, or a string.

This is a gross asymmetry in the Tcl language: commands take lists of
arguments but can only return strings!

'''What if''' we loose this restriction, and state:

    * every command returns a list of words.  Note that this includes the posibility to return an empty list or a list with only one word

    * command substitution expands the invoking commandline with the list of words returned by the command

    * [return] returns the list of words in its commandlines to the invoking stack frame

(Note that this loose formulation can be expressed very precisely in
terms used by the [*dekalog], without the need to mention the command
[return] explicitely.)

While this should maintain almost complete backwards compatibility for
simple scripts, it gives a lot of benefits some of which shall be
enumerated later, first some explorations.

You can emulate the proposed behaviour, by prepending any [[bracket]]
expression with the [{*]} expansion prefix, and enclosing the
arguments of any return statements in a `[list ..]` construction
(without the {*}), if there are more then one return value.

======
# proc valueOf x {return [list $x]}
# set x {a  b c}
a  b c
# valueOf $x
{a  b c}
# valueOf {*}[set x]
wrong # args: should be "valueOf x"
======

Of course! any command can now return a number of values, and `valueOf`
only accepts only one.  We rewrite `valueOf` as variadic function,
using the tautology `[list {*}$args]` = `$args`.

======
# proc valueOf args {return $args}
# valueOf {*}[set x]
a b c
# remember!
# valueOf $x
{a  b c}
======

This example shows already the deep implications of the approach:

    * We can switch easily between list and string representation of a variable by using either `[set x]` or `$x`.

    * The idiom: use `set x` instead of `return $x`, will break if x contains a list: the former returns a list, the later returns a list with one element (which is a list).

Why does this benefit us?

    * Some of us have stumbled over the imposibility to return more then one value from a [proc], e.g. a status and a value.  We have work around either by setting variables in the callers stack frame, which is a 'bad thing' (tm) or by returning lists, which had to be broken up by [eval] (or [{*}].

    * The natural [Functional Programming] style of Tcl can be exploited to any extent without changing the syntax.

    * The simplear, cleaner syntax of the Tcl language.

    * Variadic functions can be exploited more naturally

An example, stressing the [Functional Programming] style:

======
namespace import ::tcl::mathop::*
proc map {prefix args} {
    set r {}
    foreach e $args {lappend r [uplevel $prefix $e]}
    set r
}
if {& [map {file exists} [map {file join / etc} passwd shadow group]]} {
   puts "We seem to be on a unix box with shadow passwords"
}
======
Which seems fairly more natural than:
======
if {& {*}[map {file exists} {*}[map {file join / etc} passwd shadow group]]} {
...
======
Hint for people not used to [Functional Programming]: try to read the
idiom from right to left: a list of names gets converted into a list
of filesystem paths gets converted into a list of flags, which are all
'and'ed together.


A typical "old style" implementation for comparision:

======
set files {}
foreach file {passwd shadow group} {
    lappend files [file join / etc $file]
}
set flag 0
foreach file $files {
    set flag [expr {$flag & [file exists $file]}]
}
if {$flag} {$flag} {
   puts "We seem to be on a unix box with shadow passwords"
}
======
----

Note: While playing with this, I found some gotcha's in the use
of [{*}]:

======
# proc valueOf args {return $args}
# set x {a  b c}
# set y "{*}[valueOf $x]"
{*}{a  b c}
# set y "{*}$x"
{*}a  b c
======

I would have expected the result to be `a b c` and `a b c`
respectively, since e.g. first `{*}$x` should be substituted by
`a  b c`, and then the '"' quotes stripped off.  When reading the
[Dodekalogue] I realize, that the {*} expansion occurs only at the
start of a word, which explains the shown behaviour.

Note that the proposed semantic change to [[bracket]] expansion would
have to deal with a sensible interpretation of [subst]ituting a list
of words returned by a [[bracket]] expression into a string.  I guess
the most simple one would be to use the string representation of the
list.
----

The extensive use of variadic functions as well as introducing (return)
values for loops would further even more the
[Simplification of the Tcl language] and make scripts more
understandable.  See [Simplification of the Tcl language] for the
respective discussion.


----
!!!!!!
%|[CategoryDiscussion]|[CategoryConcept]|[CategorySyntax]|%
!!!!!!