My experience with literate programming

Used to happen to me all the time. Then, for my systems programming seminar class this semester, our professor. insisted that we use noweb for all our programs. The first three programs (!) which were: Generating efficient Mips assembly code from the switch statement, a Threads package, and Multiple precision long division, all worked upon compilation. That never happens. Ever. I attribute this success story to literate programming entirely. In my opinion, it's a lot harder to get 10-20 lines of code wrong if you write a paragraph describing exactly what they are supposed to do, and if those 10-20 lines are some logical chunk of your program that the language would prevent you from separating out.

In addition, when your program is done, you have 30-65 pages of TeX output ready to hand in (for this class, anyhow)... no write-up! Being a big TeX advocate, [[noweb]] has been like a dream come true. Recently I started switching to |CWEB|, but I am not sure I like the output. With [[noweb]], I could have complete control over the sectioning of the LaTeX document, but CWEB has its own ideas about paragraphs and sections and tables of contents that don't really make me happy.

Here's a question for all you literate programming fans out there who are dying to respond to some literate programming related post: Is there a literate programming system (for Unix) that pretty prints your C and allows you to do things like |CWEB| does with your output, yet still gives you control over the sectioning like [[noweb]]? I liked noweb because all it allowed me to do was to break my code into chunks and document it on the fly, but I like the way the code in |CWEB| looks better. What's my tool of choice?

Hope this can at least provide some thread of interest for readers of this group. Anyone have failure stories about literate programming? I want to hear them (so I don't repeat your mistakes. I plan to write every piece of non-trivial code from now on in some literate programming system.

This has been on my non-urgent wish list for some time, together with a couple of other CWEAVE variants. While I probably won't get do it on the short term, here is the simple idea that should do the trick: add just one code to the (C)WEB language that signals the end of a section without starting a new one, but rather gets you into a similar mode as you had in limbo (i.e., not in any section, but talking directly to TeX). That would give you all the opportunity you need for sectioning (in the document sense, rather than the WEB sense) and other non-program related stuff. For a really pleasant result you might want some additional modifications, like changing the support macros that are currently geared towards the ``WEB-section as basic structuring element'' philosophy, and adding hooks to allow the WEB-section to be numbered subsidiary to the larger document sections. One step further would be to modify CWEAVE into an eqn-style filter that just prettyprints code sections, while passing everything else through unchanged (anybody volunteer to do this?

CWEB & LaTeX. Seriously, in the current state of affair you need some knowledge of (La)TeX programming. CWEB actually provides a five-level-hierarchy for sections. Chunks (started with `@ ') may not necessarily marked with numbers, it's possible to do without. Of course, then one has to tag the refinement definitions with numbers.

CWEB.sty provides an internal (programmers') interface that gives access to this document structure. The current default implementation produces the result as known from the plain TeX macros. For somebody halfway fluent in LaTeX coding, it's easy to set up a different rendering by writing a style option. Actually, one of the reasons that a LaTeX 2e CWEB document class hasn't been released yet is my wish to write such a package as an example. (The 2nd reason is that I don't use CWEB currently. PS: Of course, one can do the same in plain TeX. The problem you mention isn't a CWEB problem, it's a missing feature in Knuth's macros.

CWEB.sty has all the code already. `Just' redefine the appropriate macros from the protected level (\CWEB ...), there are enough hooks.

You can also try my c2CWEB program. It will prettyprint ordinary C code, and in the usual case you just need to insert a few specially formatted C comments (something like /*@@*/ etc.). It uses a modified CWEAVE to print the source code together with a kind of preprocessor which translates the C code into a CWEB input file. (I am currently working on a different package, thus it is not adapted to Leeuwen's CWEBx which I prefer).

A "Spidery noweb" toolkit for building noweb filters to prettyprint code chunks has been on my "to do" list for a long time. Once I can find the time (a major problem, as we all know), I intend to build at least a prototype and distribute it. However, until that time comes, I am only a tentative volunteer. If anyone else wants the job (and has the time), I will be glad to pass along my ideas.

As long as people are sharing their literate programming experiences, I wanted to pass along mine. It isn't yet what I consider a resounding success story, but it's gradually getting there. A couple of months ago, I stumbled across literate programming. I don't remember if this was on the net, in a library, or a magazine article, but shortly I began voraciously reading whatever I found. It looked very useful, given that I and another engineer share responsibility for a large software project. I found Knuth's book and the literate programming FAQ, located some FTP directories, etc.

I wasn't familiar with TeX, and didn't know an easy way to get TeX output printed on any of the printers where I work, so I hunted for tools that could work without TeX. I was also concerned about the cryptic appearance of the web files I was able to find.

Simultaneously I began tinkering with my own tools. Initially I decided to create my source documents in Microsoft Word (a decision I have since reversed). One of the problems with that approach was that tangling required a Word macro, which couldn't be put in a makefile. The upshot of that effort was that I was able to get a bunch of code running for a demo, and show my boss and coworkers a very pretty collection of Word documents, and people had the unusual experience of actually understanding the code.

Then it came time to merge the code with somebody else's code, maintained on another computer, and what ended up happening was that after extracting all the .C and .H files to their computer, we never bothered updating the Word documents as we worked on the code. No tragedy ensued from this, but I concluded that I should use plain ASCII files for my source documents. Some reasons for preferring an ASCII source document are (1) much quicker and easier to edit, (2) much smaller, (3) works well with standard programming tools like AWK and GREP, (4) no mystery Microsoft formatting, (5) version control systems like ASCII files.

I also began reading up on Postscript, since we have a very pretty Postscript laser printer at work. I found a program called Ghostscript that allowed me to print Postscript files on my 24-pin dot matrix printer at home; they actually look pretty good, albeit slow.

I had initially wanted to avoid using any specialized language to control tangling and weaving, but I ended up with a language of suitable simplicity. It has only nine '@' commands. I don't even need a cheat sheet. So the tools I ended up with have the following pleasant features: the source document is plain ASCII, the TANGLE and WEAVE processes can both be put in a makefile, and the output is a Postscript file suitable for printing at home or at work. I was able to update all my Word files, and convert them to the new system, in about a day.

This probably sounds like a pretty round-about path just to avoid TeX. But it was highly instructive, and I saved a lot of hard disk space and hassle by avoiding TeX and dvips. The tools I ended up with are quite compact, since all the really hard work is done by either a Postscript printer or by Ghostscript.

I will briefly comment on this. First, thanks for posting something on topic. I have been literately programming now for a couple or three years. I started by dabbling with cnoweb and a simple example program, then moved on to more serious literate programming systems (and I do know the difference). I have been using both noweb and FWEB recently, and have had an ongoing discussion with Norman Ramsey over literate programming and (of all things) Fortran. I find that I much prefer the degree of control over formatting given by noweb. However, it's much easier to handle the output of the Fortran source using fweb. It's a quandary for me as to which tool I will focus on. However, for now, it will be FWEB until I can formulate a post-processor for noweb output that will clean up the Fortran source into some compiler-acceptable form. My programs aren't perfect; but they get the job done. And, I can go back to them after some time and pick up the pieces. That says quite a bit about the paradigm.

*** Success: I wrote a device driver some time ago using noweb. The final document included not only the C code for the driver, but the shell scripts needed to install and uninstall it, an example program and the Makefile to build the driver and format the document. [Actually, there were two documents, a README describing how to extract the Makefile and use it and the literate program document itself...] There were few choices I could make over the use of the literate programming tool since I wasn't confined to a single programming language. The success came when I was reviewing the document for typos and whatnot (after the driver had been written and tested) -- I found that the mechanism for protecting critical sections could be done better, so I changed the <<sleep>> and <<wakeup>> chunks, and I also found, as a result of seeing the program uncluttered by device-driver housekeeping, a small but significant bug. Oh, and of course, the nicely formatted document provides detailed and accurate documentation for both the driver software and the hardware (the hardware documentation proved especially elusive when I was writing the driver in the first place...)

** Failure: I started writing a Tcl/C++ program using literate programming, but it rapidly fell apart when I realized that I didn't have a good design to pin the document structure to. Every time I wanted to modify the design, I had to more-or-less re-write the literate program document to reflect the new design. I went through two design phases like that before I quit using literate programming. That was the prototype, the real thing, if it ever gets written, will have a proper design and it will be well-worth using literate programming, now I know what the structure of the document should be. Moral: get the design right first.

What sort of tools would have helped this process of discovering a good design? How did your choice of languages (Tcl/C++) affect the design space you wanted to/were able to explore? Discovering a good design is the hardest, the most frustrating and the most enjoyable part of programming for me. I try to manage this activity by writing design notes along with code being developed. When the design changes radically I clone a new directory and make changes there. But this has not been very satisfactory. How do other people manage/document design explorations?

Most all literate programming uses I have seen are for finished programs but it seems to me that it can be useful during program development as well. I would like to be able to document various design branches taken, rejected or explored, extract a final document, extract a historical perspective, etc. Any thoughts on how to go about this?

A related question is how to manage literate programming in a multi-person project. Typically people produce design documents, using TeX or some word processor, that frequently get out of sync with the related code. Again, the problem is managing the program development process (and I don't mean use of RCS or some such source code control system).

What attracted me to literate programming was the very enthusiastic comments at the start of Knuth's writings on the subject --- programming is like writing a book. I take this to mean that the programmer writes drafts, outlines, etc. Throws them away when they don't work and starts again. Finding the right design is about finding the right document structure. I can't say that this is how I write programs (I don't write many) but listening to this newsgroup I get the feeling that people use literate programming as a way to reorganize their traditional programming methods. Does anyone use this very literal (and slow?) approach to literate programming? You presumably have to be a very good writer and programmer.

I find literate programming quite useful during program development, as well. While in the "noodling around" phase, I generally write explanations in a "stream of consciousness" fashion (as ideas occur to me), inserting code chunks whenever I describe something that should be code. When I have got this rough draft finished, I weave (but never tangle or compile) it, and use these thoughts as a basis for a "real" design. Usually, I find that I use most of the code chunks (and their associated explanations), but the final web has them in a completely different order (geared to the reader), and often in a totally different program structure, as well.

The greatest benefit of literate programming is that a programmer's thoughts no longer disappear into thin air once the program is written; they are preserved in the web. The maintenance programmers who come after have these thoughts as a foundation for future work. These thoughts, as you say, should include "branches taken, rejected or explored". This saves later programmers from following trails known to be false.

I tend to believe that I work that way. In any event, I have always been a "slow and steady" programmer, even before literate programming. As I mentioned in earlier messages, I often treat webs as drafts and completely reorganize or rewrite them. I also suspect that a great many of the literate programmers out there do pretty much the same thing. I guess we use "programming talk" rather than "writing talk" because we're thinking more about programming than writing (the final result has to work, after all, not just look pretty).