Log of chat from Mon 4 Aug 2003
jyl OK its 1:45 my time, I'll give it till 2:45 my time, afterwards I have to do donuts
dkf OK
jyl OK so... lets first agree on the problem: object internal reps disappear when
an object it used as part of some string and no other ref to the object exists, right?
jyl s/it used/is used/
jyl is that a correct description or is the problem also composed of other sub-problems
dkf You mean as in: set str "foo [expr {1+2}] bar"
dkf where the numeric rep is lost
jyl yes dkf thats one example
jyl or set obj [makeamagicobject]
jyl set str "$magicobject"
jyl x-ed
jyl set str "$obj foo foo foo"
jyl set obj blah
jyl now, the object in $obj is inaccessible and irretrievable even though it should be retrievable?
jyl dkf?
dkf In the following case, the rep is *not* lost:
set magicobj [makemagicobj]
set str "$magicobj"
unset magicobj
as Tcl is smart enough to just discard those quotes, but combining the object with
anything via string concatenation will lose the rep.
jyl thats why I said "$obj foo foo foo"
dkf In the [set str "$magicobj foo foo foo"] case, the rep is currently *definitely* lost
(excluding the UNICODE string internal rep, which we can ignore for this discussion as
it is a transparent rep anyway.)
dkf Now, if we use [list] instead, we can retain the rep.
jyl OK.. sorry but cannot resist: I think where I want to go is to replace the string built
in object type with a special kind of list that just "looks like a string"
dkf The essence of TIP#126 is that the internal rep would be retrievable (with a suitable
[string range] command, for example) from $str
jyl of course this has problems such as what happens if someone modifies the part of the
"string" whose component is an object... mind boggling
<davidw> lost where? magicobj or str?
jyl I have read #126 briefly but I need to understand better
dkf Which in turn means, contrary to the current situation, that the internal rep would need
to be stored in $str in the first place
jyl yes, dkf
jyl sorry davidw not ignoring you, but u'll need to provide better context....
jyl dkf sounds like we're agreeing about a kind of segmented string-list combination
dkf Something like that.
jyl itd be a humungous job, certainly not for 8.5
dkf My idea is that a string rep should have an ordered list of non-overlapping ranges
that are associated with objects.
jyl in order to get started we need a 9.0 tag in the CVS so we can hack
dkf The TIP never claims to be for 8.5
<davidw> jyl: when you say the internal rep gets lost, are you talking about in the
original 'magicobj' variable, or lost in the 'str' variable?
jyl would this solve shimmering *entirely*?
jyl davidw the original magicobj
dkf davidw: Try this instead: set str "[makemagicobj] foo foo foo"
<davidw> ah, ok, that's what I thought, just wanted to be positive
jyl davidw TEA seems to say the internal rep should be reconstructable from the stringrep
but noone is doing that in practice
dkf It won't solve shimmering entirely, but it will make it much rarer indeed.
jyl dkf thx for the better example
jyl dkf OK lets turn to the parts of shimmering that arent covered... I'd like a *complete*
approach if possible
dkf All core objects are reconstructable right now.
jyl dkf I meant extensions
dkf (well, given the right context; many of them can't be directly converted to.)
jyl what parts of shimmering (with example, if possible, please) arent addressed by
the segmented string-list approach?
dkf The Tcl_Obj structure should have been called Tcl_Value, except that was already taken
jyl Tcl_ObjValue
dkf The main part of shimmering that isn't addressed is this:
set foo [makeMagicObj]; llength $foo
<davidw> err... what's the way to printf something as an unsigned value?
jyl OK step through what happens there
jyl %ud
<davidw> do you have to cast it or ... what's the flag
dkf If something fails that, then it's not generally a valid object name.
dkf s/valid/nice to use/
jyl or more succintly the example should be
jyl llength [makemagicobj]
dkf And we should never have called the things objects. Everyone has (misplaced)
expectations about objects, and only the core itself views them that way.
<davidw> oh right...thanks
dkf jyl: But that would discard the object anyway.
jyl dkf yep OK so I dont understand the way in which llength loses
dkf The main other thing needed is an "inhibit type conversion" flag in Tcl_ObjType
dkf The way in which llength loses is that it does an internal-rep smash on the object.
jyl so llength is an example of the general problem of wanting to have two or more
valid objtypes that are also diffrent from string
jyl in this case magic-object and list
jyl lindex would do a similar job on this value, correct?
dkf yes
dkf and yes again
jyl or [expr .. ] and so on... there are many examples
jyl all of this is based on the (often incorrect) assumption that the original intrep
can be reconstructed from the stringrep which can be constructed (and will be the same)
from any special objtype
dkf Most examples involve list (and theoretically dict, though that doesn't like singletons)
jyl s/and will be the same/& - also often an incorrect assumption/
dkf Most reasonable object formats are probably like this: [a-z]+\d+
jyl well expr (convert to number), using as an array, and ... hmm.. vaguely gesturing
at space, many others ( ) could do similar things
dkf which is distinct from the other sources of trouble: ints and floats
(strings are dealt with by #126 )
jyl well... ints and floats --> strings are handled by #126 but is the original intrep
saved if we do <magicalobj> -> int -> string?
dkf It'd be nuts to use an object rep like "0"
dkf Note that intrep smash only happens when the object is successfully converted to another type.
dkf (Anywhere where that is not true in the core is a bug.)
jyl OK I getcha... people have to be perverse and make their stringrep for the object be like "17"
dkf exactly
jyl we still have to worry about construction... what happens if a string looks like the
stringrep of "magicalobject23", is it a real reference?
dkf I've no intention of babysitting someone
jyl s/is it/does it become/
dkf "magicalobject23" isn't a reference unless it came from [makemagicalobj]
jyl how would you even know to go look for the real reference.. I say no to that...
yes I agree with what you wrote about having to come originally from [makemagicobject]
jyl so:
jyl concat magic alob ject17 ==> (string) "magicalobject17" does not make a
reference to the 17th magical object
jyl and for {set i 0} {$i < $lastmagicalobject} {incr i} {set str "magicalobject$i"}
does not make references to all magicalobjects created so far
dkf Depends. It is possible to build a weakhashmap-alike to go from names to objects,
but not required. That's up to the extension.
dkf (by weakhashmap, I mean a hashtable that doesn't retain references to the objects,
just pointers, and which loses the entry for an object when the object is destroyed)
jyl dkf if the weakhashmap is intended, the extension would have to provide an EXPLICIT
operation to take a stringrep and convert it to a ref
jyl just by consing the name you dont automatically get a reference, right?
dkf The main complexity is what to do if an entry is deleted when the weakhashmap is being searched
jenglish You should.
dkf The only safe way to convert a pure string back to a reference is by a table search, of course.
jyl dkf that is up to the extension what to do about consistency issues like that
jenglish [join [list "magica" "lobje" "ct17"] ""] should be utterly
indistinguishable from "magicalobject17".
miguel jyl: is this a counterexample to your claim/expectation?
[mig@mini mig]$ tclsh
% open /tmp/m
file3
% string length [read [join [list fi le 3] {}]]
392
jyl jenglish dkf and I are going in a different direction...
dkf The main thing is if the reference ever really goes, the table won't preserve it.
jenglish I see that... not sure if I like that direction though.
jyl miguel and jenglish thats because I wrote the lookup code that goes from the name
to the object and it does the lookup inside "read" if it gets a string object
instead of a channel object
dkf The weakhashmap idea is how to add Tcl-style introspection. It's just quite a
bit more complex to implement.
dkf It's quite possible that if #126 was implemented, another TIP would be done to
provide canned weakhashmap support...
jyl the porblem with what jenglish and miguel are asking for is that currently its
up to each objtype implementation... do we want to provide a general facility that
will do it for all objtypes? something worth considering, at the least
jyl dkf again we're thinking alike
jyl Id rather solve the whole issue, dkf, i.e. enhance TIP#126 to do it all
jyl makes it much more likely to be accepted in the TCT imho
dkf I find I think even more like kbk
jyl dkf explain?
dkf You've not seen us talking together at the conference
jyl completing each others' sentences and such?
jyl ...anyways...
dkf I think I'd like to leave #126 (and related stuff) for the moment.
It feels like something to work on in a BK fork...
dkf jyl: Yes, we were doing that.
jyl I think a new objtype operation to ask an objtype to take a string and
convert it to a valid tcl_obj with this objtype is what we're looking for
jenglish Hrm... I think I disagree with the entire rationale behind TIP 126.
I _don't_ think we should encourage the kind of thing that TclBlend does.
jyl objtypes would then have to age objects much more carefully since they can
"come back from the dead" anytime... memory mgmt issues loom
jyl jenglish and e4graph ...
dkf the memory issues are definitely why I would want to try this in a branch.
jyl jenglish very soon I'm going to jump the gun and make it easy to write such
extensions, by releasing a library called genobject that abstracts the guts
of what these extensions do and makes it easy
dkf that and the fact that it stamps its merry way through a large fraction of the core.
jyl I think by forcing the shimmering issue out in the open like that is the
only way to get it resolved
<davidw> yay!
<davidw> go jyl!
jenglish You want to encourage MORE buggy extensions?
jyl jenglish define buggy
dkf At least it would be standardizing the bugginess!
jyl dkf I will never again be bitten by releases such as 8.4.3
jenglish Handle semantics are a well-defined, time-honored way of representing
stateful objects in Tcl. If you do things that way, intrep preservation
becomes an efficiency issue only, not a corectness issue.
dkf Hmph. Since I reckon that the intrep of an object is the exclusive preserve
of that object's type, I think that the hack that broke with 8.4.3 was a
poor hack anyway.
dkf OTOH, I strongly suspect that the 8.4.3 problems could be prevented with the
"don't convert me" flag.
jyl jenglish I agree with you that hanging stuff off of ptr2 is only one possible
approach.. another would be to map the address of the Tcl_Obj --> intrep...
but currently there's no way to get a callback when a Tcl_Obj reaches refcount 0...
if that were possible then yes, Id agree completely with you...
however the current state of things with Tcl_Obj makes such uses of ptr2 necessary
jyl dkf are you saying the extensions which exposed the 8.4.3 problems are
jyl "a poor hack"? or that the change in 8.4.3 was "a poor hack"
(dangerous smile curling around the corners of my mouth )
dkf I'm saying that they are doing something deeply non-kosher
jyl .. but necessary?
dkf I think I'll reserve judgement on the necessity of that kind of thing.
jyl dkf tell me how to make an extension that wants to recycle its own intrep
when there are no more Tcl_Obj references
dkf It's definitely not a good idea on the core not using ptr2 for anything;
several types do in-fact use it (including the list, dict and subcommand types)
dkf In theory, it's easy.
dkf Obvious.
jyl Oh and also wants to use the object as a functor
dkf Only problem is that it is very susceptible to intrep loss
(frex, it can't be used as a command name)
jyl dkf thats exactly the problem solved by genobj and feather
jyl and independently by tclblend
dkf Hardly. AFAICR, feather had the intrep smash problem.
jyl dkf yes, if you stick a feather object in a list, its gone
jyl s/stick in a list/treat as a list/
dkf The easiest way to fix this is a flag in Tcl_ObjType, but that can't be done in 8.*
jyl same with tclblend, e4graph and everyone else
dkf (structure and field-positions are fixed by both the core and extension code.)
jyl why/how would that fix things? lets say we have a flag set on the
objtype saying dont convert me... OK then what?
jyl what happens when you *do* try?
jyl forget 8.* we're only discussig 9.* here
dkf If the flag was set, the core would notice and refuse to convert the type,
instead of converting it to command-name type.
jyl whoa... so refuse to convert to type foo for whatever target foo,
instead do the conversion to command type? why?!
jyl anyways this was fun, its 2:45 my time... time's up
dkf Eh? That's backward
dkf It's already type foo. It just won't be converted to any core type instead.
jyl OK so I misunderstood... so you want to convert a type foo --> list,
it refuses and thats that, or it refuses and converts to command-name?
patthoyts wanders in.
dkf It refuses and that's that.
jyl OK agreed... gets a Tcl error, I suppose
dkf (won't go to list *or* command-name.)
dkf Depends on the situation. In some cases, there might be a sensible fallback instead.
jyl Instead of a flag, Id like to allow conversion to command names but disallow list... possible?
dkf (for command-name, you just don't cache the command info. It's not a big deal.
For lists, you can build a singleton list around it.)
dkf In theory, could have several flags to describe what to do.
jyl that is, instead of only asking the target type to convert an object TO it
I'd first like to ask the source type to convert *itself* to the target type.
This could return one of three results: a converted object, a flat-out refusal,
or a defer-to-target-type
jyl If we get a defer, then only call the target type's convert-to method
dkf There's obviously things still to resolve in this area. Now you get back to donuts!
jyl yeah thx Donal... that was mucho fun
jyl pls record all this if you can, cut-n-paste to the wiki or something