Version 160 of tclOO missing features for Itcl

Updated 2007-06-10 14:05:34 by APW

APW 2007-06-06 Some time ago I started efforts to reimplement incr Tcl on top of tclOO [L1 ]. I started based on tclOO, using only the API functions of tclOO and a lot of additonal code in C. With that approach I had about 90% of the test suite of incr Tcl running successfully, but I saw, after all that, I was only using a small part of tclOO code.

My second approach was implementing a TMOS-like [L2 ] extension using a lot of code from the XOTcl reference implementation and again achieved about 90% of the test suite of incr Tcl running successfully. I felt that the code was too big and too complicated, partially because of having all these XOTcl features running. Additionally there were some features of incr Tcl which would have needed a lot of redesign of that code to pass the incr Tcl test suite.

So I decided to come back to the tclOO implementation, thinking that it is, after all, the best starting point. This time I changed/extended the tclOO code directly so as to be able to use as much as possible of that code, because I had the feeling it has a lot of good implemented features, which I did like to use. I also did like the implementation of the dispatch and class/object creation mechanism much more than the one in the TMOS implementation.

I started implementing extensions to tclOO (directly in tclOO, as the API is not enough) to get a lot of features of incr Tcl to run. This includes a lot of code implemented in pure-Tcl on top of tclOO. Doing that, I learned a lot more details and "specialities" about incr Tcl.

Currently I have an implementation which is successfully running about 70% of the incr Tcl test suite.

Having now a better feeling about what is needed for incr Tcl, I will start to write down the missing tclOO features and my solutions for them, in the hope that there will be some comments (especially from dkf). These comments could also include suggestions on how to solve the problem in a different way, without having to modify tclOO.

In the following, solution means that is my current (not always optimal) solution for the problem.

In the code examples I am using a meta syntax, not the incr Tcl or tclOO syntax, to make the code shorter!

Here starts the list; I will add topics as they come up:


what
Namespace names for incr Tcl
problem
incr Tcl expects that the namespace of a class is the same as the class name
solution
Add an option -withnamespace or a method createwithnamespace

DKF: This is actually a serious problem, since the current design of the system specifically decouples the namespace and the class (i.e., object) name.


what
calling of methods without my in front of the method name
problem
incr Tcl resolves command calls first within the class hierarchy before using the normal Tcl rules
solution
Add a namespace unknown command as a dispatcher for incr Tcl methods

DKF: Good solution.

MJ: Won't this lead to problems if you define commands with the same names as incr Tcl commands in the global namespace? In that case namespace unknown will not trigger.


what
access to class variables
problem
incr Tcl has 3 protection levels which guard the access to class variables in the class hierachy.
solution
I use the apply command for constructing the body parts for the class methods (and [constructor]s and [destructor]s) after the complete class has been parsed and all classes in the hierarchy are known. There is an additional parameter, self, added automatically to the arguments of the method, and the variables of all the classes in the hierarchy are mapped using namespace upvar. Variables in incr Tcl have to live in the namespace where they are defined as there might be the same variable name in different classes in the hierarchy even with different protection (private, protected and public). In generating the namespace upvar commands the private variables of inherited classes are skipped.

DKF: implement the self variable by prepending each method body with:

 set self [self]

what
incr Tcl rules for calling/traversing the constructors and destructors
problem
The rules for traversing the constructors/destructors in incr Tcl are very "special" and cannot be mapped to the model in tclOO. Example:
  class foo {
      constructor {args} { puts foo }
  }
  class bar {
      constructor {args} { puts bar }
  }
  class foobar {
      superclass foo bar
      constructor {args} { puts foobar }
  }
  class geek {
      constructor {args} { puts geek }
  }
  class mongrel {
      superclass foobar geek
      constructor {args} { puts mongrel }
  }

  output: geek bar foo foobar mongrel

In other words incr Tcl walks through the list of superclasses starting at the end based on the definition order. It looks for inherited classes up to the root class and unrolls the built stack. Then for the next class in inheritance, the same is done.

solution
New command: "invokeixtclconstructors" similar to the [next] command in tclOO (to be done: add details here)

DKF: The [next] command, if put first, causes the constructors to be walked in the desired order.

 % package req TclOO; namespace path oo
 % class create foo { constructor {} {next; puts foo} }
 ::foo
 % class create bar { constructor {} {next; puts bar} }
 ::bar
 %  class create foobar { superclass foo bar; constructor {} {next; puts foobar}}
 ::foobar
 % class create geek {constructor {} {next; puts geek}}
 ::geek
 % class create mongrel {superclass foobar geek; constructor {} {next; puts mongrel}}
 ::mongrel
 % [mongrel new] destroy
 geek
 bar
 foo
 foobar
 mongrel

DKF: Discussion seems to indicate that there is a deeper problem here, but not one I understand yet.

APW: I need info class superclasses ... on a lot of places for incr Tcl and it returns the class names not in the order the have been defined, so I am calling

  class create mongrel {superclass geek foobar; constructor {} {next; puts mongrel}}

to get the order as expected in incr Tcl, for the incr Tcl input:

  ixtcl::class mongrel { inherit foobar geek; constructor {} {puts mongrel}}

what
incr Tcl special init code script in constructors
problem
In incr Tcl constructors it is possible to define an init script, which is invoked directly before the code in the constructor, but after the object is instantiated. Example:
  The script after the arguments and before the body is the "constructor init code"

  class foo {
      constructor {args} { puts foo }
  }
  class bar {
      constructor {args} { puts bar }
  }
  class foobar {
      superclass foo bar
      constructor {args} {
         foo::constructor ; # this is the init code which calls the constructor of class foo
      } { puts foobar }
  }
  class geek {
      constructor {args} { puts geek }
  }
  class mongrel {
      superclass foobar geek
      constructor {args} {
        foobar::constructor
      } { puts mongrel }
  }

  output: foo bar foobar geek mongrel
solution
Add the init code script in building the apply code for the constructors

DKF: Definitely sounds like you need a custom constructor command. It might even be easier to not use TclOO constructors at all and instead define your own scheme using multiple methods (this is what you'd have to do if doing XOTcl on top of TclOO).


what
incr Tcl command ixtcl::body
problem
In incr Tcl, there are several ways to partially define a method: protection + name, protection + name + params, protection + name + params + body. That method can be redefined later on, but the params list for the method - if defined in the class definition - has to be compatible.
solution
Add additional information about method params and status to each method definition in the tclOO internal structures. There is an additional problem in that tclOO cannot handle partially defined methods, the solution to which is adding this functionality.

DKF: I thought about having abstract methods - early drafts had it in - but in the end I took it out. I reasoned that it would be easier to implement such things on top of a basic core than it would be to figure out how to do it directly.


what
incr Tcl info command
problem
Getting the original body of a method when using apply internally
solution
Save the original body in the tclOO internal structures of a method. In general I am generating a method info within each class. That is like a class proc in incr Tcl syntax. It is something like a singleton method and it can be called without an object being instantiated. info first trys the incr Tcl specific commands, and if that does not work, it forwards to the ::info command.

DKF: I think this sort of thing should be fairly simple to hang off the metadata mechanism.


what
incr Tcl singelton methods
problem
In incr Tcl it is possible to have class methods which are like singletons. They can be called uisng the class or an instantiated object. Nevertheless, if they have protection level protected they can be only called from within the class namespace. That will not work with the current tclOOCall.c implementation, as it just calls unknown and also if that is a user defined method, there is not enough information available for specific diagnostics.
solution
Add some code in AddMethodToCallChain, to do some additional checks and use a class specific call back to report the appropriate error.

DKF: This is like Java's class methods? If so, try this:

  • Put them as procedures in the class's namespace (you won't be treating the class as an object from the incr Tcl view)
  • Use namespace path to splice in (but be aware that there is already an existing path)

what
incr Tcl method calls and protection violation errors
problem
In incr Tcl it is necessary to generate specific error messages when a protection violation is detected. That will not work with the current tclOOCall.c implementation.
solution
Add some code in AddMethodToCallChain, to use a class-specific call back in that case, which can be set via a setprotectioncallback command, to report the appropriate error.

DKF: I'm thinking that this would perhaps be better better done through a custom method type. One of the key things I did with TclOO is to provide a general method implementation system, allowing many different implementations to be used within a single framework. As far as I'm aware, this is quite different to how Tcl-based OO systems have worked in the past. Maybe I'm wrong (it depends on whether the protection applies to the method implementation or to the interface) but in that case I'd need to understand exactly what is going on.

As a side note, it has been my policy in TclOO to make non-callable methods invisible (except through my) as opposed to throwing errors when you try to call them.

APW 2007-06-10 - Throwing errors where calling non-callable methods is exactly what incr Tcl expects. :;)


what
Calling incr Tcl methods
problem
In incr Tcl when calling a method it is expected that the method runs in the class namespace with the variables of the class and all inherited (and accessible) variables to be mapped into that namespace temporarly for the call.
solution
In creating classes allow a parameter namespace for the AllocObject method, which is NULL for the "normal" tclOO system and is != NULL for generating classes for incr Tcl (that needs also changes for "class create"). This namespace is used later on in InvokeProcedureMethod if != NULL, which fits for Itcl calling methods and also to allow the class to have the same namespace as the class name (see also what: Namespace names for Itcl)
  if (tobjPtr->selfClsPtr->thisPtr->namespacePtr != NULL) {
     /* use the classes namespace if it exists for example for Itcl */
     namespacePtr = tobjPtr->selfClsPtr->thisPtr->namespacePtr;
  } else {
     namespacePtr = tobjPtr->namespacePtr;
  }

The variables of the different classes are mapped (if accessable) using the "namespace upvar" command (that code is inserted in the body of a method at the very beginning), when the class is completely parsed. The path for that "namespace upvar" command is handled using the self variable in the method parameters (see above). This allows the methods to be compiled once only for performance on the very first call to a class method.

DKF: This feels very much like something best achieved through custom method types. All this stuff can be controlled easily at that level.

APW 2007-06-10 - But then I will also need an own InvokeProcedureMethod function, so most of the method handling is the done without tclOO :-( .



APW 2007-06-09 - @DKF: after spending some thoughts on your answers I have the feeling there will not much I can use directly from tclOO. Using custom method types, I will need my own InvokeProcedureMethod and then I can directly pass all the needed data via clientData to my InvokeProcedureMethod. So there will be no further need to use the metadata mechanism, as that would be more overhead to call it. For the info stuff I will nevertheless need my own code, as it is so different to tclOO info. For constructors and destructors - as you suggested - I will need my own code. To handle protection violation - as you suggested - I will need my own code. So what I can use from tclOO would mostly be creation, administration and deletion of classes and objects. I cannot use tclOOCall.c, because I need additional information (and the special code to provide in there the checking of protection violation), I will not need tclOODefineCmds.c, as I directly call the tclOO API functions, I will not need tclOOInfo.c (see above). That was not my intention! I would like to use much more of it! I like the implementation of tclOOCall.c, but at least for Itcl (maybe also for others) it is too limited at the moment. Do you have any better ideas?


APW 2007-06-10 - @DKF: what about the following: you call the following code in tclOOCall.c in function AddMethodToCallChain only if there is no other callback function in the object (oPtr) of that function.

    /*
  • Enforce real private method handling here. We will skip adding this
  • method IF
  • 1) we are not allowing private methods, AND
  • 2) this is a private method, AND
  • 3) this is a class method, AND
  • 4) this method was not declared by the class of the current object.
     *
  • This does mean that only classes really handle private methods. This
  • should be sufficient for incr Tcl support though.
     */

    if (!(contextPtr->flags & PRIVATE_METHOD)
            && (mPtr->flags & PRIVATE_METHOD)
            && (mPtr->declaringClassPtr != NULL)
            && (mPtr->declaringClassPtr != contextPtr->oPtr->selfCls)) {
        return;
    }

and you add a functionality to register a callback on a per object basis which either returns NULL, which is the case of "return;" above or you get back a tmPtr which is either the same as before calling the callback function or a method which could for example - as in Itcl needed - handle protection violation as needed. That method in tmPtr must be a registered method for the object/the classes of the object.


APW I would feel happy if native english speakers could correct the wording and the grammar above where necessary.

escargo - I think it might be clearer if instead of mixing bullets and definition lists, the text that is currently part of the bullet was at the beginning of the text after problem (maybe in bold face or italic to set it off). That's too radical a chance for me to make without checking for other opinions first. I think the list would also take up less space on the page.

APW You are right, I am also not happy with the layout, but it was the best, which I was able to produce. If someone knows better please go ahead. Have changed a little bit, but don't know a method to get the text behind "problem:" instead of the next line.

escargo - I stuck in some horizontal rules to make the problem - solution pairs more obvious, but I think the result looks much better.

APW Yes indeed!

jcw - I changed the last example, maybe you like that better - else, just restore it, please :)

APW - Thanks jcw, I have added another tag, maybe that's a better solution to what I had in mind.


[ Category Discussion | Category Itcl | Category Object Orientation ]