Software creation with Tcl/Tk Is primarily an essay by Fricio Rocha on how to go about writing software for Tcl and Tk, and what tools to look into.
Fabricio Rocha 2009-06-13:
Maybe you want to keep track of your expenses in a way that GnuCash can't help you. Maybe you have an apparently great idea for something that could be done on your PC but it seems that nobody thought the same before. You can imagine the interface, the dialogs. You can even imagine how nice your program would be, what other people would think of it, what about a website...
In other words: you want to create a new software. You know a little bit of Tcl/Tk and you feel it is the best tool for moving your ideas from your mind to reality. But... how to do it? Where to start? Is there a library that could help? How to use it? How to organize your code? Can you draw your own icons or would you like to find some for free? How could the interface look better? Which procedures to write first? How can you take advantage of some Tcl peculiarities? How can you leave room for future versions, how to avoid a crash if your program reaches something that you don't want to implement immediately?
There is a bunch of questions that fill your head when you feel the bite of the programming bug, but it is not so easy to find answers for them, and even professional programmers sometimes find themselves in doubts. There is a good number of coding standards guides, but much less information about software architecture and design, and the documents in this category are mostly driven to company-sized development, big databases, etc; not to the lone programmer which wants to create a little desktop or mobile application.
I propose this page in the hope that more experienced Tclers can teach and discuss with newbies and intermediate users which practices they use when creating new apps, what must be avoided, how to create good interfaces, how to keep focus on the important things, and so on. I admit I have some big struggles with some of these topics, and I believe that lots of newbies in programming find themselves doing the same questions. It should not become a coding guide like how-to-name-your-variables, nor a bunch of copied-and-pasted procs; but a more generic guide on how to go from an idea to a running software -- something which might be applied to most programming languages, but here with some special Tcl/Tk sauce. Far from being the definite cake recipe; but a good way to start.
So, when the light shines on your head, what do you do first? How to improve the idea? Do you prefer to start coding right after, or look for a similar software before trying?
Fabricio Rocha 2009-06-14: When a new idea comes to my mind, I like to write it down like explaining to myself what the program should do, which should be the highlights and special features in a user's view, etc. Then the concept seems to get more organized naturally, and soon some more ideas and implementation details come. I don't know if this is really a good thing, because I feel I spend lots of time writing and planning before actually doing code; but there is a lot of people which says that planning is never lost time. I also like to search for a piece of software which does what I am trying to create; if there is, I don't see much sense in redoing the same thing.
LV: Another useful tip is not to start solving the problem before you understand the problem. I often find developers (including myself) begin coding before they have taken the step of writing down a vision of what is needed. Figure out how to express the real problem, who is experiencing the problem, and then start prototyping simple solutions. If you can end up with a simple solution that takes care of 80% of the problem, that might be good enough.
Fabricio Rocha 2009-06-29: AMG posted the following text under the Development Tips section, then I moved it to the 'What to consider..." section, now it is here as a very reasonable design technique:''
AMG: When developing in C, I start by designing my data structures. Once I'm happy with the structure layout and the relationships between the structures, I make a list of functions (names and arguments) that will manipulate these structures. Smart data, dumb code. When I put enough thought into my data structure design, I find that the rest of the work is comparatively easy. This advice should hold for Tcl as well; I just haven't had a huge amount of experience with using Tcl at this level. One thing I do sometimes is to store application runtime data in an SQLite :memory: database; this greatly reduces the amount of work needed to do complex queries and updates. If the queries aren't too complex, and if the data doesn't have too many "foreign keys", dicts and arrays work too. (I use arrays when I need the ability to treat separate elements as separate variables, e.g. with trace, upvar, and the like.)
When developing in Tcl, I either find that my application is too simple to worry about proper design procedures, or that the structure of the Tcl code is largely dictated by the C code I previously developed. So I make sure to carefully design the interfaces to the Tcl commands I implement in C. I like to make these interfaces very low-level, then wrap them using Tcl code to make them palatable for everyday use. This minimizes the amount of "glue logic" that I have to write in C, since that task is much better handled by Tcl. Then if I have a nice interface (be it a Tcl wrapper, or maybe just C code that does a simple task and needs no wrapper), my Tcl application follows from that.
AMG, regarding his lack of experience with large Tcl applications: I should clarify. My small Tcl applications would be huge if written in C. I have written large Tcl applications, but only when I used a sloppy design that ballooned my code size beyond all reason. When I am careful, I find that I stay under (usually well under) 1000 lines of Tcl. I guess the lesson here is that if your Tcl program is growing long, stop and check for repeated patterns that can be "factored out"--- Tcl, like LISP, is extremely amenable to this. Also check if a lot of lines are spent on interfacing with inconvenient data structures; put these first on your list of things to fix. And don't be afraid to try revolutionary approaches, once you've made a backup. :^) If your code is becoming a burden to develop and maintain, try to refactor/simplify/streamline it so that it becomes fun and exciting to work with, and then you will be more receptive to inspiration and more capable to act on it. So... here's an idea to help you hone your programming skills. Find (or write!) some crappy-but-working code, then challenge yourself to cut its code size in half while making it more readable, flexible, efficient, and functional. Document your experiences on this wiki. :^) The next time around, you'll be better prepared to write your code "right" the first time. (File this paragraph under "refactoring")
Fabricio Rocha 2009-06-28: In the general concept, for most (if not all) programming languages, the use of global variables is something to avoid. But in Tcl/Tk one will find that sometimes globals are not only encouraged but also needed. Keith Vetter placed some interesting discussion about that in the page Variable versus a Global Array some years ago. In the design of a certain application, it is often simpler and recommended that a big global array is used for storing things like user preferences and application/widgets state -- and it can be decently organized as if it is a multidimensional array, just by using key names with a separator character (",", ".", for example), which is something valid for Tcl.
Fabricio Rocha 2009-12-06: Using namespaces is also an approach for keeping in a certain and private space all the useful and important data which must be accessed by many parts of an application. It seems to be less messy than using a global array. But I still have to understand in which situation it is better to use namespaces or global arrays for storing this data.
By the way, all the Tk widgets are created in the global scope (I'd prefer to say they are created in the "root namespace"), and some of them work linked to global variables as the simplest way to get and set the data shown on them. The radiobutton groups and the ttk::checkbutton are the major example of that, in my opinion. As previously said, you can create an array whose elements are the variables linked to those widgets (these elements can be even part of "the big central array"), something that would work fine for smaller apps. Bigger applications might have different "widget-values arrays" for each of its windows, dialogs and panels, and these arrays might be stored in separate namespaces, which can (or could) keep all the information of the corresponding module of the application. The thing to remember is that the "-variable" option for those widgets should, in most cases, receive the fully qualified name of these arrays and their elements, such as -variable ::Users::Wdgs(accountframe.opts).
Bigger applications (and small ones as well) might also benefit from an idea I saw somewhere here in the wiki (sorry I can't find the page anymore): use a global array for storing widgets paths to which you will have to refer in different procedures. Do it once when the widget is created (with something like set Widgets(UserDialog.entryAdmin) ttk::entry .dialogusers.frmleft.entAdmin), use the value stored in the array in your procedures, and if someday you decide to put a labelframe there you will not need to change all references to the widget in your code.
Is it useful to create interface mockups? On paper, on a prototyping software, in Tcl/Tk itself? What must be observed in the interface design?
Fabricio Rocha 2009-06-13: Paper prototyping is the first thing I use when thinking about an interface. This technique can be very complex, but in my simple version I just get a notepad, a pencil, draw an empty window and start drawing widgets and writing tiny letters on them. Sometimes I draw some different versions of the same window or dialog and actually "seeing" it, even with all the imperfections of the drawing, really helps a lot. I know there is a good number of people who uses Tcl/Tk itself for prototyping, and then it becomes a matter of adding backstage functionality later; but I'm still not at this stage. I really miss a visual interface designer for Tk (Visual Tcl is quite outdated); I have already used Glade a lot for creating interface mockups which later could be used in the program. I have also tried some free prototyping software (the Pencil project [L1 ] looked very interesting) but I still could not find a convinceable one.
Developers are almost always encouraged to adhere to interface standards for their specific platforms, unless they really believe that their ideas for an interface are a differential or better than those standards (and they really know what they are doing). While those standards are a kind of limit to creativity, they generally ensure that your software will be rapidly understood by new users, and so they will allow the user to concentrate in what the software does, not in how to make it do it.
Tcl/Tk is notoriously cross-platform and, in spite of all the efforts to make it look native in all the supported platforms, it does not always look native, and this makes things harder for the developers, specially when they are not writing software which will run on only one platform. Also, (Fabricio Rocha - AFAIK) there is not something like a Tk Interfaces Guide in the shape of the GNOME Human Interface Guidelines and others.
This also makes it hard to make suggestions on "do's and don't"s in interface design with Tk, specially because the subject always seem to create a lot of controversy; but I (Fabricio Rocha) would try these:
peterc 2009-06-15: Yep. Try to restrict your UI to 640x460 (leaving 20 pixels for window decoration). There are very small monitors out there, projectors, standard definition televisions being used as monitors, and cards using default settings. As a user, there's nothing worse than being unable to click an OK button or get to UI controls.
In general, your windows shouldn't need to go beyond 640x460 anyway. If they do, it means your interface is too cluttered. Human beings don't deal well with a large number of elements in front of them at once. There are studies that suggest 7 is optimum, 9-11 about the most before it starts to look all too complex.
Luckily, user interfaces have been around long enough that there are a number of well-explored strategies to reduce UI complexity without forcing you to leave options out:
I do agree with the comment about Tab notebooks in Preference dialogs (anything involving forms) above. They're useful and highly appropriate when there aren't many pages, but, switching to a tab-less notebook controlled by a treeview or listbox is a good idea if your notebook grows past 5-6 pages.
LV: It would be great if someone coded up some small coherent examples of these "best practice" solutions.
Try to find if there is something like the software you want to create? Try to find libraries which might be used?
LV: There are many people who prefer writing something from scratch rather than reuse someone else's solution. That may be why, in the Tcl community, there are many object oriented solutions available (itcl vs xotcl vs snit vs ...). There are certainly other examples as well. The tension between beginning with unfamilar software (that many times has minimal support) versus starting from scratch to write a package with which you will then be intimately familar is a common one. Rediscovering bugs, portability issues, etc. are additional efforts that one takes on when beginning from scratch.
Where to find useful code? How will the use of third-party libraries affect your coding? Do you have to tell the authors? How to distribute libraries with your software?
Fabricio Rocha 15 Jun 2009 - Personal opinion here: there is A LOT of Tcl/Tk code spread over the whole internet and this is not so evident for Tcl newbies. Difficulty increases because a large portion of this code was made many years ago for much older Tcl versions, and it is mostly unusable nowadays.
The most interesting sources for Tcl/Tk code and libraries are (please add to the list):
Fabricio Rocha - 14-Jun-2009 - As many programming languages, Tcl/Tk does not really need anything more than a text editor, and the most seasoned Tclers seem to be very happy with Vi or emacs. I use Komodo Edit because of its autocompletion tips, which are really helpful when you don't have a pocket reference guide. An up-to-date GUI design tool, as I said previously, would be greatly appreciated (I am trying to create one, but it's still too far from being eligible for discussion). Debugging could be useful as well. I have tried to use RamDebugger but it didn't work. But so far the simple use of puts for checking variables in strategic code locations has been sufficient.
LV I know that I like eskil as a file diff, tkinspect to introspect running tk programs, frink as well as TDK's tclchecker - these are tools that I frequently use during development.
Split the code in multiple files... or not? Which directories should be created to hold what? What size is the limit for a single file? Is it better to embed images in code or keep them in files?
LV: If during development I realize that I have some code that with a little effort could be useful in the future, I consider seriously splitting the code off into a package.
Fabricio Rocha 2009-06-14: For those not so familiar with Tcl, packages are much like C's include or Pascal/Delphi uses, but in a somewhat different way (actually, the source command is closer to C's "include"). Packages can be used in a number of interesting ways, and one that I find fascinating is that you can create a simple API to your software by putting part of it in a package which can then be used by plug-ins or other applications - something nice if you want your application to be extensible. The Tcl community seems to favour packages instead of source commands, and I hope someone explains better when to use one or another; but I tend to believe that you would source a file when it is simply a part of your program that, in the sake of organization, was written in a separate file; and you would make that part a package if it is intended to be a reusable bunch of procedures and data structures.
Copy procedure names on paper? Run again whenever a new procedure is added?
AMG: Test suites. Make a test suite, then run it after every edit you make. Your test suite doesn't always have to be formal, but it certainly helps. If your test suite takes an annoyingly long time to execute, then you're probably cramming too much functionality into your program. :^) Seriously, if you can find a way to compartmentalize your program, you can develop and test its parts in isolation. The downside, of course, is that you have to worry about integrating the parts into a whole. But that's a much better problem to have than the problem of maintaining spaghetti code. If you define the interfaces between your compartmentalized modules, you will be able to mix-and-match. Specifically, you will be able to take code that was designed to work in your application and instead plug it into your test harness.
Speaking of test suites: If you're really serious about getting a particular program segment right, you will want to test the test suite itself. Make a backup of your code, then introduce bugs into your working copy. Go line-by-line, expression-by-expression, condition-by-condition. Change < to <=, $x to $y, etc. Make sure that your test suite catches every error you add to the code. If you can break your code without your test suite catching it, add more test cases until the problem becomes visible. MAJOR BENEFIT: Your new test cases may pinpoint bugs you would otherwise have never known about. And once you're satisfied that your test suite has comprehensive coverage, don't forget to restore your code from backup. :^)
There are some pages in the Wiki which go into details about programming style in the sense of making the code more readable and easy to maintain:
Tips for writing quality software
Tk coding styles and philosophies
Which kind of documentation is needed and expected by users? How detailed must it be? Write these docs during development or just before the initial release? How to know if a website is needed, and when to create it?
Fabricio Rocha 15 Jun 2009 - IMHO documentation is the biggest problem in the free/open-source software world: usually it is too geeky and hard to understand, made for developers only, or it is too shallow and do not really explain what someone would need to know for using the software. I think that writing for developers is a priority in the creation of a new software if you want other developers to get interested to help, and this comes naturally if you have the habit of writing down your ideas before and during development. But it is adviseable to write for users as well before doing any releases -- after all, developers are primarily users, and if they know which features your program has (or should have) it will be easier for them to know what to do for helping.
Fabricio Rocha 20096-06-15: Documentation has its traditions in the software world. One of them is the pure-text README file included in the application's root directory (under Windows it is usually named "Readme.txt"). I believe that even though the majority of users never even know there is such file (and prefer to search something about the program in a Google search), it is good practice to create a README file with a one-or-two-paragraphs description of what the program does, which are the requirements for using it; quick instructions on installation and running; the copyright disclaimer and your contacts. And it is important to maintain the README file, reviewing it at each new release of the software.
There are other files which are traditional (and even required sometimes; as in the Free Software Foundation rules), like INSTALL and COPYRIGHT. But the most important thing to consider is the "user help", the file(s) with detailed instructions for using the program. Unices, Windows and MacOS have distinct "help systems", if we can call them so: in Unix/Linux the man pages are still a tradition; Windows uses CHM files (and LV can hopefully tell what is it for MacOS). I don't know any tool which would help someone to maintain all those file formats at once in cross-platform development.
LV: Unfortunately, I have never gotten a version of MacOS X that will run on the antique Macs that I have around the house. It would probably be better to ask Jeff Hobbes, Jean-Claude Wippler, Steve Landers, Daniel Steffen, etc.
One tool that at least helps in some degree is the doctools software that is part of the tcllib source distribution. One can use a specialized markup, then use doctools to generate html, Unix man pages, wiki pages, and some other formats that I'm forgetting.
So I (Fabricio Rocha) would consider that the best option -- and a tendence -- is to create the program's documentation in one or more simple HTML pages, as almost every computing platform has an internet browser and the format allows graphics and easy navigation through hyperlinks. Most desktop applications have (or should have) a "Help" menu which should be the door to this documentation, and there are at least two ways to provide that door. One approach is to create a menu item which when clicked will cause the system's default browser open the documentation homepage. Second approach is to embed in your software a "help reader" (A Hypertext Help System, by Keith Vetter, is worth a look). The help files should be packed with the application (or at least complete packs should be a download option), as not every user might be connected to Internet while using the software. You can simply create a directory under the application's root for holding the HTML pages and graphics; but some OSes, especially the Unix-based ones, might have some standard documentation directories which ideally should be used.
LV: I know that as a developer, one of the frustrating things for me is to be working on Unix in Tcl and try to use the man command to find information about an extension and not find it. Also, I know that at home I generally don't have an internet connection, so as Fabricio says, the occasional program which attempts to use the internet to provide help info is indeed a frustrating thing.