Page started by [Theo Verelst] Apart from having been called a fruitcake once (har har, what a drag), I find most things going on here serious, so maybe the weight of this page can be carried on something called as light a 'wiki', considering it is a about a programming language which to some degree i salso better of with this consideration (read also: Verelst is pissed about a lot of things, and thinks he can affect the world by contentwise discussions, even when the are rants). Seriously. Long ago, when computers started to get a physical place in the world, memories, and communication rates, and computational resources were small. Yet, neworked, usefull, multitasking machines, with serious price tag, of course, were made which could do most things modern computers can do: multiprocess, communicate over various channels, run databases, run os-es with powerfull commands, be entertaining, etc. By progress almost solemnly coming from the advance in electrical engineering, discs got faster and bigger, as did processors, networks and main memories. Then computers started to get more popular outside the heavy business and research area, probably because chip and board manufacturers, and the design community, found ways and used them to get the price down, and because technology evolved to make then do fun enough things, especially to add some graphics to the machines. When graphics started to get in the picture, memory and a lot of other things got strained. I remember vividly the advent of X on some (at the time university) computers, and the quadric growth of required frame buffer memory as a function of the image size. That happens to have come from the same subject tcl was related to: chip design, where large screens were usefull to represent the content of a chip. A bit later, personal computing became different from lets say apples and trs80's and derived architectures based on the same microprocessors (or quality improvements such as the beeb) when the processors, memory and boards got into the lets say Mega ballgame, several megaherz clock frequency, almost a megapixal on the screen, a megabyte per second networks, and several tens of megabytes of harddiscs. No mpeg yet, but more fun: nice images, graphical user interfaces, lot more powerfull OS then on most micros. And a lot of programming: to get all those menus, pictures, dialog boxes, windows and application interfaces up and running from about a meg of core memory (early PC's, Atari ST, of course the classic Mac and later ones). More professionally speaking, quickly forgetting gks and such, one had Unix, Vms, ibm stuff (I guess, didn't use it), and what as interestingly been named X. With a lot of libraries to handle large frame buffers of all variations, lightweight windows, dots, lines and rectange based drawing, blitting, windows handlers and what more. And suddenly: neat little 'hello world'-like programs taking no more than possibly even a few tens of kilobytes of stdio library (assuming that is statically linked) and one programming complexity step higher started to have a ground level of a megabyte just for simple executables. Which is not funny with an OS based on little programs for many seperate tasks (call then reentrant, parallel, communicating objects), so disc and memory sizes became a major issue. And a lot of programming was concerned with making good use of resources, such a screen redraw [algorithms], which make up for a lot of application programming unless you have a good way to shield that of. At least on the systems running that graphics 'shell', memory managements was mostly to some degree advanced: dynamic allocation OS supported, and the hardware would support paging methods. Still, showing 5 big pictures on a screen forces you to somewhere store the image buffer data for all of them at times, unless you can programwise 'construct' the content, or unless you can assume they will never be completely visible all at the same time. On the PC side of things, 'window redraw' was one of the complex issues to program. Currently, main memories usually are big enough to store a number of full size frame buffers easily enough, which makes all that a lot easier, potentially. ---- I was recently playing with the graphical programming tool [bwise], simple but usefull [Database routines for in core] and [tclhttpd] combined in a single XP running wish84, about which I'll do a page [Web server with bwise graphical programming and database] when I find the opportunity. That setup, made from whatisit, maybe a few hundred kilobytes of combined scripts recorded about ''8 megabytes'' in the task management window. I do have to mention bwise can take pictures on its canvas, and the database has the capability to display records which refer to pictures, which I used to comment on a number of recent digital photographs. I made the pictures automatically smaller to fit in the dbase presentation window, and then again smaller to be shown as a little block on the bwise canvas, but the original size would be about a meg square in high quality jpeg (half a megabyte), which has to get read in before the resizing (at least as I did it now). I just checked: the executables of wish84 are very small, and the basic dll's for tc and tk together about 1.8 Mb. Starting wish or tkcon makes the system (XP sp 1 in this case, I'll try linux, too) report about 5 megabytes memory use. I think it would be a lot better if some memory could be given back to the system, I remember having usd 16 megabytes at some point, especially after putting a about 1000 square pixel image on the canvas.. Not that that is necessarily problematic, but it is a good illustation of not very sparing memory policies in action. ''[DKF] 26-8-2003 - Some versions of the memory allocator do hand memory back to the OS. It's not trivial to do in general though since the OS only allocates in terms of whole memory pages, not individual bytes. Some time I must find some time to make the general version of the [Tcl_Obj] hand-back allocator work properly...'' ---- [JMN] 2003-08-26 From what I can tell, there seems to be an attitude in the TCL community with regards to TCLs apparent retention of memory, that it doesn't really matter - or that it's too hard to give the memory back to the OS and besides other scripting langs such as Perl have the same sorts of issues... or something like that. Maybe these really are valid reasons and it is the OS fault more than TCL - but all I know is that when I run lots of TCL scripts there comes a point where applications crash and misbehave and it's all very frustrating because I know all that memory can't really be being used and I wish the whole 'high-watermark' style of holding onto memory would just disappear. I vaguely recall seeing something about there being the possibility of choosing different memory allocators for use with TCL - but as someone who doesn't delve into the C side of TCL I wouldn't know how useful this is. Then again - right now my browser alone appears to be using over 195MB of ram - so nowdays I couldn't feel comfortable just picking on TCL as a ram hog - it's apparently just the way of things. Ok.. so while typing the above, DKF has disproved my sweeping statements about 'attitude' of the TCL community to the situation ;) Good to see that at least someone cares.. even if improvements in the area are a long way off. ---- [DKF] 26-8-2003: A bit more about why we don't hand memory back to the OS. The key thing is that it is hard to implement well due to things like fragmentation. Determining whether a memory block is free (you ''really'' don't want to deallocate something still in use) is quite expensive too. Given that for most apps (not all, I'll accept), the max amount of memory used is fairly closely related to the average, it's not a big enough win given how complex it is to debug and tune. I also think that there is a libc that comes with this sort of support built-in. I think it might be the glibc used on recent Linuxes, but couldn't swear to it. I also suspect that Windows can also do that sort of thing for you, but the last time this discussion happened on tcl-core I didn't pay much attention. Perhaps all you need to do is analyze your own apps for memory leaks (scripts can leak even if the core doesn't!) and switch to a suitable version of the core and a suitable libc/OS. We're quite careful with core memory leaks, and Tcl is much better than Tk in this particular area. ---- [TV] Without getting into the whole discussion above at this point, what I remember from older good enough unix systems with respect to memory management, you'd have malloc and free of course, the one gives a chunk of memory which is guaranteed to be contiguous. and free gives back that whole chunck to the OS to reuse for whatever, I mean after free is open to multi gb/s blit blast of rewriting for all the os cares. The idea of the contiguousnes of course being that that is easy to program on. The use would be to make sure you get enough of a big chunck to do sensible processing on without running the machine into direct virtual memory problems, so you'd preferably allocate a significant but not too large fraction of the main memory, which you are sure you are going to make good use of. The paging mechanisms on such machines, completely unlike the early PC like machines can in principle allow the system to fragment the whole memory, for instance into pages or in to segments, which then can be mapped to actual physical memory pages by the pager, in principle in any order and permutation, so that for instance on page basis you'd have complete freedom to swap in all pages you want anywhere in the actual memory banks, without the programmer knowing it (unless something goes wrong), and given you may want to take programming habbits (page with programmer page number adjacencly will most likely be needed soon) into account. Hardware units would do page owner management (out of process detection) and virtual page handling, so when one access various pages not all too often, not much speed penalty is paid. Malloc-ing more often makes the OS give virtual memory ranges anywhere in the theoretical range, with no foreknowable place in physical memory, though probably at malloc time, all involved pages wouldn't start being swapped out to disc. So as long as all programs don't use more than physical memory, nothing gets swapped or slow, and paging with reordering can be transparent to the programmer, so that he doesn't need to think in a limited heap/stack and/or relocating way. Completely unlike a simplistic model I progammed in, too, where you'd have a process with heap and stack in opposing direction, and malloc somewhere in free space, limited in size to whatever memory fragmentation dictates. Then one at startup (the biggest program first) takes a big piece of linear memory, and hopes for the best. I know the pentium has a page/segment handling capability with somewhat variable page size, though I must admit I don't recall/know what it can do exactly, apart from basic process memory space protection at least used in Linux. Let alone that I'm aware of what the various not(mega)not(hard) OS version do in these departments, except I do know decent program startup planning makes a lot of difference in process swap times, and that Linux, too, eats memory and doesn't seem to return much for the various screen management programs I've worked with. And for instance redhat 8 doesn't exactly remind me of a one floppy profi os in that department. Then again a heavy pricetag mini didn't support all too many high quality X servers, either. I remember doing a extra malloc layer for *small* memory chunks, to prevent memory block allocation data being larger than the memory block sizez or being near that size. That is also relevant on unix. [DKF]: Some libc implementations have calls to allow for tuning of malloc in this way. We don't bother in Tcl because the cost of maintaining platform-specific hacks (these things tend to vary even between UNIXes) is rather high; if someone wants to squeeze things for a particular deployment, they can always embed Tcl in an app that sets those things up. ---- [elfring] ''26 Aug 2003'' Can any links to [memory allocation] libraries help? [http://groups.google.com/groups?seekm=40ed1d8f.0301131025.79f84b41%40posting.google.com] Will the library "Hoard Memory Allocator" [http://www.cs.utexas.edu/users/emery/hoard/] be used for Tcl in the near future? ''[DKF]: Not by default, though you can link it in yourself. It fails in three ways: 1) [LGPL] license, 2) written in [C++], 3) does not even approach the style guidelines. First problem is not necessarily a show-stopper and certainly lets you use it as a replacement malloc(), but the second and third problems are major and cannot be resolved in short-order without significant effort. Do you want to do a comparative analysis with the memory allocator that was contributed to Tcl from the [AOLserver] people?'' Is the following software in use to improve TCL code quality? 1. http://www.garret.ru/~knizhnik/jlint/ReadMe.htm#introduction ''([DKF]: Mostly a Java tool. The code that handles C is not likely to be useful if the Tcl style-guide is followed.)'' 2. http://www.splint.org/, http://sourceforge.net/projects/splint/ 3. http://www.kessler.de/prd/gimpel/p_pclint.htm ''([DKF]: Unable to assess without translation DE->EN.)'' 4. http://www.lambdacs.com/debugger/USENIX/Debugger_USENIX_2003.html ''([DKF]: Technique unlikely to scale to significant programs.)'' 5. http://debugger.netbeans.org/ ''([DKF]: Java debugger. Does not appear to be relevant)'' 6. http://www.eclipse.org/ ''([DKF]: Which bit did you have in mind?)'' 7. http://www.gnu.org/search/fsd-search.py?q=debug ''([DKF]: Please indicate which of the search results are relevant.)'' 8. http://sourceforge.net/tracker/?func=detail&aid=667010&group_id=1&atid=350001 ''([DKF]: That page is a rehash of your link list which you have posted here.)'' [DKF]: A general suggestion. Given how many links above are unspecific or just plain irrelevant, why not thin them out a bit so that they are properly meaningful? Better yet, why not post some links to tools that will help with memory debugging? None of the links above (with the possible exception of the ones to pclint - which I can't assess - and Eclipse - which is an unhelpful wall of links) are relevant to the topic of this page (as opposed to general debugging.) ---- [TV] Some interesting links, though I had in mind the general subject which is not so much debugging or 'black box' [analysis], but as always it is better to know how the whole game works, that is how do the average and other types of computers deal with memory ([allocation], [process binding] and [limit checking], [virtual addressing] and [page mapping], [page size] options and [overhead]) and what the various [operating systems] which support tcl/tk have to offer, or limit one with. [Lint] is an analyzer from the unix time, which should be fine, the hoard package is for [distributed computing] which is not explicitly supported in tcl, though bare sockets get you there fine. The [gnu debugger] (the other day I found out it now lives also in [cygwin] under the name of '[insight]') which has a tk ui is very powerfull tool in the hands of the serious developer which allows assembly level (lower one cannot go, usually, microprogrammability, though not a bad idea for these issues, is out since the death of some PDP's that could and probably some exceptions) debugging, mixed with C source code, and stack tracing of nested functions calls with the common unix DB like tracing and breakpointing possibilities. It does deal with multitasking and I've at least used interupt driven sound programs with it, and allows remote (socket based) operation, so you can unburden the machine running the program under test from it, letting it largely unaffected by the test (debug) program, apart from some tcl/ip networking overhead. I'm sure I did not really bring up a quality problem primarily, though that could be, its just straight thinking: I load a program, have so many libraries which I need and normally therefore must load in core, read so much data, which expands to a certain amount of core data, and then the little sum seems often way on the unnecessarily high side, and at times rediculously so. When things start running, it gets even further, because then the dynamics kick in, meaning memory, even though it is not used insipidly, gets fragmented, leaving a lot of unuseable pieces in the main memory so that effectively things clog up. Which at least doesn't happen much under tcl in my experience. Obviously, certain branches of software/informatics land are driven or led astray if you like by all kinds of not even secondary considerations, and it often is just too appearent they don't realy know what they are talking about, yet ask for a lot of air time, attention, and empathy, and if you're unlucky to work for them, sucking up, without good contentwise reason. [DKF]: It's probably easier to talk specific cases. There's relatively little flab in the core, but lots of cacheing. You can eliminate many of those caches (and tune down the default sizes of a number of other structures) but you won't necessarily gain, especially when you factor in Time costs as well. ---- [elfring] ''27 Aug 2003'' I assume that the package "[TclTest]" is heavily used. A lot of [test cases] are automatically executed and checked. But I think that the usual developer [experience] is that some [leaks] are left and new [problems] can creep into the software. The search for [programming errors] need not be done manually all the time. - Let tools do the job for us. I pointed to some [free tools] like the [lint] variants that perform [code analysis] and [flaw detection]. I hope that TCL [implementations] will be available for that in the future. Java classes can be called by the package "[TclJava]". I asked the [SourceForge] administration to add them to the [debugging] toolset. * Are you sure that the scanner [nessus] [http://www.nessus.org/intro.html] does not detect [holes] in TCL [servers]? ;-) * How much can we control the [memory management strategies] of the diverse libraries? It is very hard. [DKF]: You seem to be under the impression that a tool can do a better job of finding flaws than a person. This is bullshit. A tool may apply a simplistic rule to loads of stuff, but it does not provide an easier way of determining whether the code is actually correct unless the specification of the code is '''utterly''' nailed down. But nobody doing such work uses Tcl, Java or C (except as a stage inside a compiler for some other language.) Those tools also cost a fortune because the only people who really need them are the safety-critical-software-systems people, and they have the budget. They also have the training in writing formal mathematical specifications and in specification refinement and retrenchment. This is a ''very'' specialised area. In the mean time, no tool can check the spec against the code. Professional coverage and memory analysis tools help (I've nothing against free tools per-se, but if they don't let me track down new problems they're a waste of my time), but they don't work well with sophisticated cases (which are fairly common in the Tcl core which has been heavily worked on for years.) And then you have to worry about whether the spec itself is correct. It is known that the test suite is incomplete and in some cases arguably wrong. But what tool could generate a correct case? Generating a case that the current implementation says is correct is pretty easy to do, but ass-backwards. And what has this got to do with the subject of the page? BTW, Nessus is irrelevant to this discussion. It tests applications. Not scripting languages. (It's current attacks detect 3 flaws related to Tcl-based things, none of which are inherent in the language but instead in the way that the language was used.) '''STAY ON TOPIC!''' ---- [TV] The whole idea of using tcl and in another sense tk is to NOT have to deal much with the memory allocation problem, which does not lead to large processes and much memory use by tcl and tk. The malloc game, for those unfamiliar, goes as follows: #include char *basepointer; basepointer = (char *) malloc(1000); for (i=0; i<1000; i++) basepointer+i = (char) 0; In C code, that is. It can also appear in assembly code, and other languages, but the name and widespread use to my knowledge comes from C/Unix, and it is fundamental in Unix, Linux, and I guess basically the same in windows, though that I'm not intimately aware of. I like using the cygwin unix/posix abstraction layer for ease of use... The first line makes sure the function is taken with the right argument size, for instance an integer, then a 'pointer' is declared to point to characters (bytes) and the have name basepointer, which we set to the return value of the malloc (memory allocate) function. After it has returned, the malloc function has either returned a NULL pointer, meaning memory didn't get allocated, or a pointer which can be used to find the lowest memory address of a contiguous piece of memory, guaranteed of at least the requested size. memory 0 basepointer basepointer+1000 end of memory ----------------------------------------------------------------------------------------------------------------------- | | m1 | | ----------------------------------------------------------------------------------------------------------------------- The basepointer contains an integer number between zero and the number of bytes in the memory -1 -1000, and points to the ' base' of the allocated piece of memory m1. The operating system, which has to do with the malloc function, it usually doesn't exist as an independent library function, remembers it gave the piece of memory m1 to our process which did the malloc call in some of its internal tables, of course also costing some memory. The computer hardware, possibly the pentium chip, risc processor or what else deals with memory protection in your computer (in very old ones that is nothing) must also be informed that when your process receives processor time, that is the time slicer which gives us the nice semi-parallel tasks on all modern OS-es, it has been given access to our m1 memory chunk, and usually all other processes must e denied access, and generate the familiar 'segmentation violation' when they would access m1. All this must also take into account that virtual addressing doesn't necessarily treat the whole main and especially the virtual memory as a linear list of addresses, although in practice this is the most convenient for the usual setup of a core memory of chips with memory banks, and a threefold virtual extension by swapping pieces of memory to and from a fast reachable piece of the harddisc. The function free can be used to give back the piece of memory to be reused at will by the operating system: free(*base_pointer); After which the memory location pointed to by base_pointer and the subsequent 999 can not be read from or written to anymore, which then generates an error on machines with memory protection. We could also repeatedly allocate pieces of memory, by repeatedly calling malloc: for (i=0; i