Future developments

Disheartened? This intrigues me. After all, few of us have the foresight to really know what is going to be the way of the future. I for one don't see that improving the tools used by an elite few (and we are, let's face it, a minority) should take precedence over creating tools that could be more acceptable to the programming community at large. Fewer and fewer people use command lines any more, and Emacs is still often only for the cognoscenti, wonderful as it is. Many of my colleagues write all their scientific papers in Microsoft Word; if I was to try to convince them to use literate programming tools, I would have no chance with an Emacs/TeX combination. I might have a small chance with a Word for Windows package; who knows, using the coming abilities of the OLE 2.0 (Object Linking Environment) one could create a hypertext-style environment that might be much more attractive than anything currently seen under UNIX/TeX/Emacs (at least for now, we'll see what Sun can offer in terms of objects and such in the future). OK, so the argument is that maybe there are too few of us to chase all of these goals at once. But then, maybe there are too few of us to yet come to a consensus on what is best. We will see.

To be sure, but then I was not thinking about the next few years so much as I was thinking about the next few months, or perhaps a year or so. It seems unlikely there will be but one "way of the future", and in any case I do not expect that literate programming will ever be more than a minority pursuit (see below).

The urgency of spreading literate programming to the programming community at large escapes me. For one thing, one would have to foster better writing skills while one was at it. This pessimism on my part may be parochial: some of the courses I teach here are courses "with a substantial writing component", as it is put hereabouts. The University wisely requires that all its undergraduates take such courses in addition to the basic English composition course; judging by the overall quality of the work I receive, the undergraduates are much in need of practice and assistance. At that, the students I see are mostly Liberal Arts students, and 3rd and 4th year ones to boot.

No doubt there is a significant non-overlap of the set of programmers possessed of decent writing skills with the set of those prepared to cope with TeX or LaTeX. Your colleagues may well be instances to the point. But then, one of the improvements I would hope for in the existing tools would be a better interface to TeX or LaTeX. Surely we can reach some agreement among us about what is desirable along these lines? The other thing needed to bridge the gap, so it seems to me, is a decent online help system for TeX or LaTeX, the which could be extended to include Web commands. Embedded command formatting would not be so difficult to deal with were one not constantly forgetting which command sequence does what. A main advantage of the GUI-based word processors is that they reduce the need for having the formatting manual (the TeXbook, in my case) by one's side.

As for the gap between the text editors embedded the GUI-based word processors with Emacs, surely the various X windows versions of Emacs can serve as a bridge? Having posed this, I should note that I have never actually used Epoch or Lucid Emacs. The Emacs port to OS/2 2.0 is what I have to work with, and a Presentation Manager version of that will not soon be available, alas. [Emacs Version 19 will be a precondition for such, but one grows weary waiting for that...] Again, though, it seems to me that the major advantage of the GUI-based editors is the menu/dialog box system, which reduces the load on one's memory. Emacs could surely use some of that! (The Emacs 18 help system is clumsy at best; again, I mostly find myself reaching for the printed manual.)

But I suppose this is mostly by-the-by; even with such improvements as these, I doubt you would be able to wean your colleagues away from what they are using to consider tools that require Emacs plus TeX, or some such combination. But then I wonder, what is so terribly wrong with this state-of-affairs? Is it that you must use or maintain code that they write? Or that they expect to use code that you write? Trying to use literate programming tools on a multi-programmer project when none of the others are, will, I agree, pose a significant problem.

I am unfamiliar with OLE 2.0. But I wonder what the impact of hypertext features would be on tangling and weaving. The hypertext model is, after all, a general graph. The compiler and the formatter, however, expect linear text in a fairly rigid order. What one could do, I suspect, is to include the hypertext links in the web source to make navigation among the parts of it easier when writing, but leave the underlying text in the proper linear order, then have the tangle and weave processors strip the links out before handing the results on to the compiler and formatter.

I think that there are enough of us to see what is better, if not what is best. We all realize, for example, that user configurable pretty-printing would be desirable in CWEB , FWEB, and the like. And anyone who has stared at the code for CWEB, FWEB, and SpiderWeb, should realize that the existing code is not readily extended. Perhaps the first thing that should be done with CWEB 3.0, when it is released, is to rewrite it in C++, with appropriate care taken to do a proper "object-oriented" job of it.

All the more reasons why all students should be introduced to literate programming and to TeX as early as possible! At present they see a total disconnect between "verbal" courses and "mathematical" courses (and activities and occupations). They also see precious little connection between computing and other technical courses (e.g., calculus). It should all be a seamless web. Maybe it's parochial optimism, but I have the impression that when you force them to try, most students produce much better written work than their usual output.

I find the same situation with my students, although I teach mostly CS students. We also have "writing emphasis" courses (I am trying to develop one in literate programming, even as we speak). One of the things I have found with literate programming, though, is that a poorly written literate program seems to be easier to maintain than an average quality non-literate one. Admittedly, all my evidence is anecdotal, and I have yet to see a really horrible literate program (perhaps once I have got a bunch of students trying it?), but maintenance is such an important part (perhaps the most important part) of CS, that literate programming seems worth pursuing for that reason alone.

Agreed, although I think that "agreement...about what is desirable..." is one of those things that never happens, for various reasons. Corporations want standards, but they want everyone else to accept their standard rather than accept someone else's (NeXT being a notable exception) -- witness the current PC-database-standard wars between Microsoft and Borland (and the various companies behind each one), not to mention the upcoming operating system war. Regarding "embedded command formatting," I have noticed that everyone who uses WinWord around here runs it with all of the formatting symbols (paragraph marks, dots for spaces, etc.) displayed. If they don't, WinWord tends to do strange things when they delete invisible "characters." Is this WYSIWYG or embedded commands?

One problem that surfaces with WYSIWYG editors, but not text-editor/embedded-command systems is one of tools. Most WYSIWYG editors have undocumented (usually binary) file formats, so we can't rely on the many text-based tools that have been developed (e.g. most everything in UNIX). We have to rely on the manufacturers to supply new tools, or work out the file format (ugh!) and make our own. One of the things I like about the various WEB systems (and TeX as well) is that they are maintained by people who actually use them on a regular basis, which is not true for (I would venture to say) most commercial software.

I work with MS-Windows rather than OS/2, but I echo Paul's wish for a Presentation Manager (Windows in my case) version of Emacs. I have tried the Micro Emacs Windows port, and while it's really good, there are a few things (e.g., not highlighting mouse-selected text) that make it difficult to use with any efficiency.

Emacs plus TeX, of course, gives you more control than does a WYSIWYG word processor, but then again, most people don't seem to want that kind of control. I reckon Paul's right, here, too.

I kind of take exception to this statement (in a nice way, of course). While I haven't looked at the code for FWEB, I have worked with both CWEB and Spidery WEB, and was quite astonished at how easy they were to understand and modify. I had to fix several bugs before I could CWEB working on my PC (all fixed now in the forthcoming 3.0 release), and have made several major extensions to my copy of Spidery WEB. All of these were surprisingly easy, and did not break the existing code.

Sorry, have I missed a NeXT announcement of some sort? Last time I looked, NeXT was hardly an exception: Display PostScript rather than standard X11, Objective C rather than standard C++. True, they're porting this non-standard software to a standard CPU (i486), but not to one of the standard OSs, DOS, Windows, OS/2, Linux, but as a new, non-standard OS. Or do you mean that NeXT is an exception to "corporations want standards"? That I would believe.

What I meant was that, rather than invent their own "standards," the NeXT people looked around and said, "What's already out there that we can adopt as a standard?" They decided (rightly or wrongly, possibly both) on Unix as the operating system, PostScript as the display language, and Objective C as a systems language. There's no way of knowing for sure but I believe their reasons were something like:

1. Unix because, let's face it, there isn't any other operating system that is implemented on a large number of different platforms.

2. Display Postscript so that they could use the same graphics language for printer and screen output. X-Windows is lovely, but it would probably be a lot tougher to build an X11 printer than it was to build a PostScript video display.

3. At the time the NeXT came out, Objective C and C++ were pretty much running neck and neck as regards which language was C's "legitimate object-oriented heir." C++'s current popularity seems to be as much from media attention as anything else, but the choice was not clear cut at the time. NeXT flipped a coin, and backed the wrong horse, although the decision to go with an object-oriented descendant of C accurately predicted the current trend. [Actually, Objective-C still has its adherents. Baby duck syndrome aside, whether you prefer Objective-C or C++ depends mostly on whether you come from the Smalltalk or the Simula school of object-oriented programming, the former advocating type checking at run-time, the latter, at compile time. Most of us have strong opinions on this issue, but this forum really isn't the place to get involved in that religious war -- we have enough of our own.]

The main point I was railing at was the NIH (Not Invented Here) problem that is rampant in our industry. I chose NeXT as an example because all of their choices involved "standards" developed elsewhere that were already (somewhat) widely in use. I apologize for not making myself clearer the first time.

Lee is a bit more optimistic about the file formats of WYSIWYG editors. We have worked on conversion programs from various editor formats to TeX (or LaTeX). It is not correct to call them undocumented; the documentation is often out of date (as most seems to be) or more often just nearly impossible to find. His sense was correct, but the biggest problem we found was that the owner RESERVES THE RIGHT TO CHANGE IT AT ANY TIME WITH NO NOTICE. Indeed, the worst case implied by such a statement has been done and only after you install the new version do you learn that the binary format has changed. Have you ever used WORD to wander around between DOS, MACs, and NeXTs? Even using RTF you don't have a panacea of portability. Knuth said one of his design goals for TeX was that a real "archive" would be possible. TeX written more than 10 years ago still runs. (It has been extended but not really changed.)

I long ago quit trying the evangelist business. I don't try to convince people that they should use TeX instead of a WP. Most of them do a more than adequate job on memoranda, ... Human nature still dominates and it is rare that I accept statements of the nature that "(any) WP does a good job of formatting a document with mathematics (or a large number of other difficult items)." Granted a lot of people say nobody complained ... The majority of the customers at McDonalds don't complain but they also don't try to convince others that it is gourmet food ... My compliments to Lee, Paul, and others for a really good set of comments.

Care to explain NeXTMail? And what about X? I get the impression that NeXT Corp. would not add X to NeXTStep even if you programmed it up for them. What they have is better, so what already exists is irrelevant! Even if they could work well together. I wonder how standard their Unix implementation is too.

1. Don't know about NeXTMail. It seems to handle plain old Internet mail just fine (unfortunately our NeXT is in NJ and I am in TX, so I can't check out any of this -- please take this and the following comments as my impressions rather than gospel truth).

2. I was under the impression that NeXTstep did support X (I think I read it in the documentation somewhere). Maybe not.

3. Their implementation of Unix is as standard as any I have seen. In fact, when I ftp something from the net, I usually try to build it on the NeXT first, because it almost always builds correctly the first time. I occasionally have problems on other Unix machines. [Relevant side issue: Now that C is officially standardized, I find it interesting that none of the cc compilers on any of the Unix systems I have accounts on conforms to the standard.]

4. I didn't claim the NeXT people were perfect -- just that they did a few major things right.

Well, I like this idea! A couple of suggestions, if I may. Firstly, you might want to look at Chapter 9 ("Mathematical Writing") of Knuth's "Literate Programming", if you have not already done so, for some hints about how students at Stanford got on with literate programming. I presume you are considering one or both of Knuth's book and Wayne Sewell's "Weaving a Program" as texts. I am tempted to suggest that if you feel that you can only use one, then Knuth is the one to try, even though not all of it is about literate programming. It seems to me that you are more apt to get them thinking about style and organization of their writing this way.

It seems to me that one ought to play it straight here, namely to try to get the students to follow the conventions of expository writing as best they can, allowing for the differences between a literate program and other types of exposition writing. They should give a proper introduction explaining to the reader what is coming, try to follow a coherent narrative organization, and so on. You are, after all trying to help them improve their writing, albeit in a unusual genre of exposition.

One interesting question about style is this: should they try to write a proper conclusion to their program summing up what it does and what they think is most important among the ideas for the program they have previously discussed? The examples of literate programs we have from Knuth do not have a conclusion. More generally, it seems to me that in literate programs written in noweb or FunnelWeb, as they allow a more free form style, one may be able to use a model of expository writing closer to the traditional models, than with those written in CWEB , FWEB, or a Spidery Web. A proper conclusion may or may not be useful, but one perhaps should try it to see.

In general, it seems to me, that since this is meant to be a writing course, the students should include more by way of commentary in what they write than they might in a literate program written for use, and you should insist on good organization and writing style. The problem of expository structure in a literate program raises a second issue. Though they are certainly an improvement on manuscript and typewriting, conventional text editors, including Emacs, do not offer that much by way of support for crafting expository organization nor do they provide much help for refining the ideas to be presented. For that sort of thing, one needs an "outliner". Unfortunately, the only ones that I know about are for MS-DOS or the Macintosh. Indeed, the only two I have tried are both DOS programs, viz. MaxThink and Kamas.

I have used MaxThink for some years now in writing my lecture notes, and am quite pleased with it. Now I find myself trying to use it for literate programming as well. The idea is to think through the ideas for the program or library modules, writing these in as topics in the outline, adding source as appropriate as text attached to a topic, then use the structuring features of MaxThink to work out an organization for the web source, and finally have MaxThink write it out in a suitable form to dump it into Emacs to finish the job. Pity that time constraints have prevented me thus far from completing this process for any of the things I am working on.

As for Kamas (which is a DOS shareware product), I have only recently downloaded it from one of the mirrors of the simtel20 archive (e.g. ftp.uu.net, ~ftp/systems/ibmpc/msdos/simtel20/editor/kamas25.zip), and have only played with it a bit, so I can't say much about it, save that it seems to offer an interesting alternate approach to MaxThink, and also seems to have some of the more useful features of MaxThink, though not, by any means all. I should note that I also downloaded PC-Outline while I was at it (this is, curiously, in the txtutl subdirectory of the simtel20 archive---pco334.zip), but it seems to me that PC-Outline lacks most of the amenities of MaxThink and Kamas, and is decidedly less useful as a consequence. Yet even PC-Outline is clearly more useful than the best Emacs package, namely "allout.el", and never mind the equally inferior outlining components of Word and WordPerfect. (I just downloaded web-mode.el, and notice that it too offers some outlining capabilities, but no more, so it seems to me than "allout.el". To be sure "web-mode.el" looks to be well thought out and written; Bart Childs and Mike Motl have done a nice job here.)

Having said this, I should note that I took the trouble to investigate alternatives to MaxThink so as to be able to present alternatives to the students in the "writing component course" I am teaching this term, thinking that use of such a tool could help them with the problems of expository organization that I saw in many of their papers. My presentation of this idea to the class some weeks ago, however, seems to have gone over like a lead balloon. I was pleased to discover, though, through a conference with a student in that class several days later, that her paper, which was unusually well organized, had been written with the aid of the outline capability in MS-Word. Primitive though that is, she had made good use of it in structuring her exposition.

I am also moved to note a remark that Neil Larson, the designer of MaxThink, included in the user manual, namely: "Word processing focuses on WYSIWYG---what you see is what you get", but "MaxThink focuses on WYTIWYG---what you think is what you get". Later on in the manual, after remarking on the work it would take to convert MaxThink from a text-mode DOS program to MS-Windows, he explains his decision not to undertake this thusly: "After all this, the intellectual processes for better thinking, writing, and planning are not improved in the slightest. Summary: While Windows may be fashionable, it does not extend MaxThink's capabilities!" (Apologies, sort of, to the WYSIWYG fans, but this is a point worth pondering in that connection, especially since Leslie Lamport makes a similar point, though with rather less justification, early on in his LaTeX book.)

In any event, I am not actually recommending that one require students to lay hands on a good outliner, but merely trying to make a few points about appropriate tools for expository writing in general, and literate programming in particular. But I do commend Lee Wittenberg's idea to others reading this list who teach CS students, and I would encourage Lee to post information about the design for his course to the mailing list. (I have more things to say about other matters of interest that Lee Wittenberg raises in his thoughtful comments, but I will do this in a subsequent post.)