Log of chat from Mon 4 Aug 2003
----
 jyl	OK its 1:45 my time, I'll give it till 2:45 my time, afterwards I have to do donuts
 dkf	OK
 jyl	OK so... lets first agree on the problem: object internal reps disappear when 
        an object it used as part of some string and no other ref to the object exists, right?
 jyl	s/it used/is used/
 jyl	is that a correct description or is the problem also composed of other sub-problems
 dkf	You mean as in: set str "foo [expr {1+2}] bar"
 dkf	where the numeric rep is lost
 jyl	yes dkf thats one example
 jyl	or set obj [makeamagicobject]
 jyl	set str "$magicobject"
 jyl	x-ed
 jyl	set str "$obj foo foo foo"
 jyl	set obj blah
 jyl	now, the object in $obj is inaccessible and irretrievable even though it should be retrievable?
 jyl	dkf?
 dkf	 In the following case, the rep is *not* lost:
	   set magicobj [makemagicobj]
	   set str "$magicobj"
	   unset magicobj
	as Tcl is smart enough to just discard those quotes, but combining the object with 
        anything via string concatenation will lose the rep.
 jyl	thats why I said "$obj foo foo foo"
 dkf	In the [set str "$magicobj foo foo foo"] case, the rep is currently *definitely* lost 
        (excluding the UNICODE string internal rep, which we can ignore for this discussion as 
        it is a transparent rep anyway.)
 dkf	Now, if we use [list] instead, we can retain the rep.
 jyl	OK.. sorry but cannot resist: I think where I want to go is to replace the string built
        in object type with a special kind of list that just "looks like a string"
 dkf	The essence of TIP#126 is that the internal rep would be retrievable (with a suitable 
        [string range] command, for example) from $str
 jyl	of course this has problems such as what happens if someone modifies the part of the 
        "string" whose component is an object... mind boggling 
 <davidw>	lost where? magicobj or str?
 jyl	I have read #126 briefly but I need to understand better
 dkf	Which in turn means, contrary to the current situation, that the internal rep would need 
        to be stored in $str in the first place 
 jyl	yes, dkf
 jyl	sorry davidw not ignoring you, but u'll need to provide better context....
 jyl	dkf sounds like we're agreeing about a kind of segmented string-list combination
 dkf	Something like that.
 jyl	itd be a humungous job, certainly not for 8.5
 dkf	My idea is that a string rep should have an ordered list of non-overlapping ranges 
        that are associated with objects.
 jyl	in order to get started we need a 9.0 tag in the CVS so we can hack
 dkf	The TIP never claims to be for 8.5
 <davidw>	jyl: when you say the internal rep gets lost, are you talking about in the 
        original 'magicobj' variable, or lost in the 'str' variable?
 jyl	would this solve shimmering *entirely*?
 jyl	davidw the original magicobj
 dkf	davidw: Try this instead: set str "[makemagicobj] foo foo foo"
 <davidw>	ah, ok, that's what I thought, just wanted to be positive
 jyl	davidw TEA seems to say the internal rep should be reconstructable from the stringrep 
        but noone is doing that in practice
 dkf	It won't solve shimmering entirely, but it will make it much rarer indeed.
 jyl	dkf thx for the better example
 jyl	dkf OK lets turn to the parts of shimmering that arent covered... I'd like a *complete* 
        approach if possible
 dkf	All core objects are reconstructable right now.
 jyl	dkf I meant extensions
 dkf	(well, given the right context; many of them can't be directly converted to.)
 jyl	what parts of shimmering (with example, if possible, please) arent addressed by 
        the segmented string-list approach?
 dkf	The Tcl_Obj structure should have been called Tcl_Value, except that was already taken 
 jyl	Tcl_ObjValue
 dkf	The main part of shimmering that isn't addressed is this: 
        set foo [makeMagicObj]; llength $foo
 <davidw>	err... what's the way to printf something as an unsigned value?
 jyl	OK step through what happens there
 jyl	%ud
 <davidw>	do you have to cast it or ... what's the flag
 dkf	If something fails that, then it's not generally a valid object name.
 dkf	s/valid/nice to use/
 jyl	or more succintly the example should be
 jyl	llength [makemagicobj]
 dkf	And we should never have called the things objects.  Everyone has (misplaced) 
        expectations about objects, and only the core itself views them that way.
 <davidw>	oh right...thanks
 dkf	jyl: But that would discard the object anyway.
 jyl	dkf yep OK so I dont understand the way in which llength loses
 dkf	The main other thing needed is an "inhibit type conversion" flag in Tcl_ObjType
 dkf	The way in which llength loses is that it does an internal-rep smash on the object.
 jyl	so llength is an example of the general problem of wanting to have two or more 
        valid objtypes that are also diffrent from string
 jyl	in this case magic-object and list
 jyl	lindex would do a similar job on this value, correct?
 dkf	yes
 dkf	and yes again
 jyl	or [expr .. ] and so on... there are many examples
 jyl	all of this is based on the (often incorrect) assumption that the original intrep 
        can be reconstructed from the stringrep which can be constructed (and will be the same) 
        from any special objtype
 dkf	Most examples involve list (and theoretically dict, though that doesn't like singletons)
 jyl	s/and will be the same/& - also often an incorrect assumption/
 dkf	Most reasonable object formats are probably like this: [a-z]+\d+
 jyl	well expr (convert to number), using as an array, and ... hmm.. vaguely gesturing 
        at space, many others (  ) could do similar things
 dkf	which is distinct from the other sources of trouble: ints and floats 
        (strings are dealt with by #126 )
 jyl	well... ints and floats --> strings are handled by #126 but is the original intrep 
        saved if we do <magicalobj> -> int -> string?
 dkf	It'd be nuts to use an object rep like "0" 
 dkf	Note that intrep smash only happens when the object is successfully converted to another type.
 dkf	(Anywhere where that is not true in the core is a bug.)
 jyl	OK I getcha... people have to be perverse and make their stringrep for the object be like "17"
 dkf	exactly
 jyl	we still have to worry about construction... what happens if a string looks like the 
        stringrep of "magicalobject23", is it a real reference?
 dkf	I've no intention of babysitting someone 
 jyl	s/is it/does it become/
 dkf	"magicalobject23" isn't a reference unless it came from [makemagicalobj]
 jyl	how would you even know to go look for the real reference.. I say no to that... 
        yes I agree with what you wrote about having to come originally from [makemagicobject]
 jyl	so:
 jyl	concat magic alob ject17 ==> (string) "magicalobject17" does not make a 
        reference to the 17th magical object
 jyl	and for {set i 0} {$i < $lastmagicalobject} {incr i} {set str "magicalobject$i"} 
        does not make references to all magicalobjects created so far
 dkf	Depends.  It is possible to build a weakhashmap-alike to go from names to objects, 
        but not required. That's up to the extension.
 dkf	(by weakhashmap, I mean a hashtable that doesn't retain references to the objects, 
        just pointers, and which loses the entry for an object when the object is destroyed)
 jyl	dkf if the weakhashmap is intended,  the extension would have to provide an EXPLICIT 
        operation to take a stringrep and convert it to a ref
 jyl	just by consing the name you dont automatically get a reference, right?
 dkf	The main complexity is what to do if an entry is deleted when the weakhashmap is being searched 
 jenglish	You should.
 dkf	The only safe way to convert a pure string back to a reference is by a table search, of course.
 jyl	dkf that is up to the extension what to do about consistency issues like that
 jenglish	[join [list "magica" "lobje" "ct17"] ""] should be utterly 
                indistinguishable from "magicalobject17".
 miguel	 jyl: is this a counterexample to your claim/expectation?
	[mig@mini mig]$ tclsh
	% open /tmp/m
	file3
	% string length [read [join [list fi le 3] {}]]
	392
 jyl	jenglish dkf and I are going in a different direction...
 dkf	The main thing is if the reference ever really goes, the table won't preserve it.
 jenglish	I see that... not sure if I like that direction though.
 jyl	miguel and jenglish thats because I wrote the lookup code that goes from the name 
        to the object and it does the lookup inside "read" if it gets a string object 
        instead of a channel object
 dkf	The weakhashmap idea is how to add Tcl-style introspection. It's just quite a 
        bit more complex to implement.
 dkf	It's quite possible that if #126 was implemented, another TIP would be done to 
        provide canned weakhashmap support...
 jyl	the porblem with what jenglish and miguel are asking for is that currently its 
        up to each objtype implementation... do we want to provide a general facility that 
        will do it for all objtypes? something worth considering, at the least
 jyl	dkf again we're thinking alike
 jyl	Id rather solve the whole issue, dkf, i.e. enhance TIP#126 to do it all
 jyl	makes it much more likely to be accepted in the TCT imho
 dkf	I find I think even more like kbk 
 jyl	dkf explain? 
 dkf	You've not seen us talking together at the conference 
 jyl	completing each others' sentences and such? 
 jyl	...anyways...
 dkf	I think I'd like to leave #126 (and related stuff) for the moment.  
        It feels like something to work on in a BK fork...
 dkf	jyl: Yes, we were doing that.
 jyl	I think a new objtype operation to ask an objtype to take a string and 
        convert it to a valid tcl_obj with this objtype is what we're looking for
 jenglish	Hrm... I think I disagree with the entire rationale behind TIP 126.  
                I _don't_ think we should encourage the kind of thing that TclBlend does.
 jyl	objtypes would then have to age objects much more carefully since they can 
        "come back from the dead" anytime... memory mgmt issues loom
 jyl	jenglish and e4graph ...
 dkf	the memory issues are definitely why I would want to try this in a branch.
 jyl	jenglish very soon I'm going to jump the gun and make it easy to write such 
        extensions, by releasing a library called genobject that abstracts the guts 
        of what these extensions do and makes it easy
 dkf	that and the fact that it stamps its merry way through a large fraction of the core.
 jyl	I think by forcing the shimmering issue out in the open like that is the 
        only way to get it resolved
 <davidw>	yay!
 <davidw>	go jyl!
 jenglish	You want to encourage MORE buggy extensions?
 jyl	jenglish define buggy
 dkf	At least it would be standardizing the bugginess! 
 jyl	dkf I will never again be bitten by releases such as 8.4.3 
 jenglish	Handle semantics are a well-defined, time-honored way of representing 
                stateful objects in Tcl.  If you do things that way, intrep preservation 
                becomes an efficiency issue only, not a corectness issue.
 dkf	Hmph.  Since I reckon that the intrep of an object is the exclusive preserve 
        of that object's type, I think that the hack that broke with 8.4.3 was a 
        poor hack anyway.
 dkf	OTOH, I strongly suspect that the 8.4.3 problems could be prevented with the 
        "don't convert me" flag.
 jyl	jenglish I agree with you that hanging stuff off of ptr2 is only one possible 
        approach.. another would be to map the address of the Tcl_Obj --> intrep... 
        but currently there's no way to get a callback when a Tcl_Obj reaches refcount 0... 
        if that were possible then yes, Id agree completely with you... 
        however the current state of things with Tcl_Obj makes such uses of ptr2 necessary
 jyl	dkf are you saying the extensions which exposed the 8.4.3 problems are
 jyl	"a poor hack"? or that the change in 8.4.3 was "a poor hack" 
        (dangerous smile curling around the corners of my mouth  )
 dkf	I'm saying that they are doing something deeply non-kosher
 jyl	.. but necessary?
 dkf	I think I'll reserve judgement on the necessity of that kind of thing.
 jyl	dkf tell me how to make an extension that wants to recycle its own intrep 
        when there are no more Tcl_Obj references
 dkf	It's definitely not a good idea on the core not using ptr2 for anything; 
        several types do in-fact use it (including the list, dict and subcommand types)
 dkf	In theory, it's easy.
 dkf	Obvious.
 jyl	Oh and also wants to use the object as a functor
 dkf	Only problem is that it is very susceptible to intrep loss 
        (frex, it can't be used as a command name)
 jyl	dkf thats exactly the problem solved by genobj and feather
 jyl	and independently by tclblend
 dkf	Hardly.  AFAICR, feather had the intrep smash problem.
 jyl	dkf yes, if you stick a feather object in a list, its gone
 jyl	s/stick in a list/treat as a list/
 dkf	The easiest way to fix this is a flag in Tcl_ObjType, but that can't be done in 8.*
 jyl	same with tclblend, e4graph and everyone else
 dkf	(structure and field-positions are fixed by both the core and extension code.)
 jyl	why/how would that fix things? lets say we have a flag set on the 
        objtype saying dont convert me... OK then what?
 jyl	what happens when you *do* try?
 jyl	forget 8.* we're only discussig 9.* here
 dkf	If the flag was set, the core would notice and refuse to convert the type, 
        instead of converting it to command-name type.
 jyl	whoa... so refuse to convert to type foo for whatever target foo, 
        instead do the conversion to command type? why?!
 jyl	anyways this was fun, its 2:45 my time... time's up
 dkf	Eh?  That's backward
 dkf	It's already type foo.  It just won't be converted to any core type instead.
 jyl	OK so I misunderstood... so you want to convert a type foo --> list, 
        it refuses and thats that, or it refuses and converts to command-name?
	patthoyts wanders in.
 dkf	It refuses and that's that.
 jyl	OK agreed... gets a Tcl error, I suppose
 dkf	(won't go to list *or* command-name.)
 dkf	Depends on the situation. In some cases, there might be a sensible fallback instead.
 jyl	Instead of a flag, Id like to allow conversion to command names but disallow list... possible?
 dkf	(for command-name, you just don't cache the command info. It's not a big deal. 
        For lists, you can build a singleton list around it.)
 dkf	In theory, could have several flags to describe what to do.
 jyl	that is, instead of only asking the target type to convert an object TO it 
        I'd first like to ask the source type to convert *itself* to the target type. 
        This could return one of three results: a converted object, a flat-out refusal, 
        or a defer-to-target-type
 jyl	If we get a defer, then only call the target type's convert-to method
 dkf	There's obviously things still to resolve in this area.    Now you get back to donuts!
 jyl	yeah thx Donal... that was mucho fun 
 jyl	pls record all this if you can, cut-n-paste to the wiki or something
----
[Category Suggestions]