Virtualizing vars and arrays

This page presents some thoughts on a mechanism which is not currently available in Tcl -JCW

The unknown command is a powerful mechanism in Tcl to take over when an undefined command is called. It can seamlessly introduce shell-like "exec" calls in an interactive Tcl environment, and lazy package loading through auto_index, to name a few useful cases.

The question is: could a similar approach be used for variables and arrays?

Here's an example of what I have in mind: suppose you want to implement something which looks, walks, and quacks like an array, but isn't. One such use would be for arrays which are persistent, and not (yet) fully loaded in memory. Another would be to make remote data appear to be local.

Traces go a long way, as the "global shared arrays" of Tequila illustrate, but they only work with complete copies in memory for now.

The "trace add variable <blah> array <cmd>" which is available in recent Tcl versions seems to be yet another step in this direction. But from a brief test, I can't seem to get at what array request was being made, let alone taking over its function.

A good example is "array names foo bar*". It would be nice if it were possible to catch that call for array foo, and return a list which matches the intended result, without actually having to set up the array with all the necessary name-value pairs.

How about extending the current trace syntax as follows: the code

   trace add variable foo array trigger
   set x [array names foo bar*]

leads to a call to trigger of the form:

   trigger foo {names bar*} array

And the return of this call is then used to set the value of "x".

Or maybe some variation - if the above breaks existing code.

Is this worth discussing further? Are there other ways to virtualize data?

OhyesohyesIwantit NOW.

That is, I see a lot of opportunities for data-on-demand.

This is, among other things, a nice fit for Tcl's event-based concurrency, channels, and congeniality toward laziness. IT as an industry (discipline, craft, ...) still has a long way to go in distribution of computations, and this sort of virtualization can only help.

Pondering a bit more, I see that I was mixing up two different issues:

• intercepting all array activity to virtualize it
• fake the existence of vars/arrays, as "unknown" does for commands

The first is more important to me -jcw

Good distinction.