Literate programming and modularity

From:	Kirk Saranathan
Date:	04 May 2000

I thought that I would raise about an aspect of literate programming others may find interesting. While this topic may be been discussed already, I haven't found it in any of the literate programming archives I have searched. Modularity in literate programming. I have noticed that my code when tangled usually consists of a big chunk of code which calls a few functions (most of which I haven't written, thus the big chunk). My concern is that how do I develop reusable code in that big chunk. I usually have all my chunks somehow specific to that program. I feel literate programming is to [blame?] for this. Literate programming encourages you to say what your do to achieve the goal of the program your are writing. So you usually don't end up saying <<push [[i]] into stack x>> and as a result have program specific code joined to what could be reusable (pushing to stack for example). It takes a little ESP to predict what should be a function and what should be a chunk; you only concentrate on doing one thing at a time and sometimes overlook that another chunk has done or will do something similar, in which it should be in a function. All comments are welcome. Is this sort of stuff covered in Knuth's web book?

From:	Bob Apthorpe
Date:	04 May 2000

Admittedly I have only written a couple scripts using literate programming techniques but the apparent lack of modularity bothered me too. Looking at the way I code, I find that I create subroutines for two reasons:

1) to reuse common fragments of code
2) to organize my thoughts

In literate programming, the chunk takes the place of the subroutine in case #2. Since most of my applications are less than 1000 lines of code, I usually don't have a lot of code reuse. Needless to say, my tangled code looks like a sprawling hack.

Organizing ones thoughts with a language's subroutine feature instead of the literate program chunk leads to what I will call "incidental modularity". That is, the code looks modular even if it wasn't designed to be. Literate programming just keeps you honest. If you didn't start with a modular design, literate programming will make that apparent. In some ways that's a good thing because it forces a developer to explicitly specify the interfaces to his code. This makes life difficult for all us slackers who like a coding style that makes us look smarter and more organized than we really are.

Anyway, the way out of this mess is to start out with a good design or at least a good design method. My coworkers have had a lot of success designing with UML. Though it's primarily an object-oriented design method, I am sure UML can make procedural design easier too. Once the initial design is finished and the interfaces have been specified, literate programming should fill in the gaps to produce understandable and reusable code.

From:	Lee Wittenberg
Date:	27 May 2000

Bob Apthorpe writes: Admittedly I have only written a couple scripts using literate programming techniques but the apparent lack of modularity bothered me too. Looking at the way I code, I find that I create subroutines for two reasons:

1) to reuse common fragments of code
2) to organize my thoughts
In literate programming, the chunk takes the place of the subroutine in case #2. Since most of my applications are less than 1000 lines of code, I usually don't have a lot of code reuse. Needless to say, my tangled code looks like a sprawling hack.

In the early days of programming, programmers made a distinction between "open" and "closed" subroutines. I quote from the first programming textbook ever (by Wilkes, Wheeler and Gill, 1951):

The simplest form of subroutine consists of a sequence of orders [read "instruction" for "order"] which can be executed as it stands into a program. When the last order of the subroutine has been executed the machine proceeds to execute the order in the program which immediately follows. This type of subroutine is called an "open" subroutine. ... A "closed" subroutine is ... designed so that when its task is finished it returns control to the master routine at a point immediately following that from which it was called in.

I like to think of my chunks as open subroutines, an idea whose time has returned. As to your tangled code looking like "like a sprawling hack", that's a good thing, as it encourages you to rely primarily on the woven code (and the original web). It's important to remember why Knuth made the original WEB (and CWEB) produce deliberately unreadable tangled code (hence the name "tangle"). Nothing of vital importance, here, of course, as this is primarily a test, but interesting (I think) nonetheless.

From:	Henk Vink
Date:	04 May 2000

Kirk Saranathan writes: I thought that I would raise about an aspect of literate programming others may find interesting. While this topic may be been discussed already, I haven't found it in any of the literate programming archives I have searched.

In raising this issue of modularity, you have thought about some of the things that crossed my mind as well. By the way, what flavor of literate programming are you using?

Modularity in literate programming. I have noticed that my code when tangled usually consists of a big chunk of code which calls a few functions (most of which I haven't written, thus the big chunk). My concern is that how do I develop reusable code in that big chunk. I usually have all my chunks somehow specific to that program. I feel literate programming is to [blame?] for this. Literate programming encourages you to say what your do to achieve the goal of the program your are writing. So you usually don't end up saying <<push [[i]] into stack x>> and as a result have program specific code joined to what could be reusable (pushing to stack for example). It takes a little ESP to predict what should be a function and what should be a chunk; you only concentrate on doing one thing at a time and sometimes overlook that another chunk has done or will do something similar, in which it should be in a function.

This is quite similar to what I though at first, but I think you have to approach this in another way. There is no way to say what should be a function and what should be a chunk in literate programming, if there were a way literate programming is entering in the domain of the programming language itself, and it was never designed to do that! What should be a chunk is part of your ideas of how your program grows, what should be a function is not so much dictated by literate programming but by the way you should program in your language (disregarding the extra LP-functionality). In this an overlap is created by extension of the used language with LP-functionality, so now you have more than one way to do it, (to use a Perl-slogan). Resolving this overlap would either mean readjusting the used language which is out of the question, or adapting that is limiting the LP-functionality (also not recommended).

The books do not go into this question, but approach it from the positive, the way literate programming was intended. It was designed to give extra functionality and documentation in a way that a program can grow to its final result. This is to be achieved by repeated interaction of documentation and program-code, so the way to judge whether your program is good or not, is to look at the documentation read it and see whether it is clear, complete and understandable for the current version of your program. If you change the code, you have to adjust the documentation-part of that chunk as well. You choose a chunk on different criteria, it has to fit what is described. While a function has to achieve a certain functionality. These may or may not coincide, you may have to find your way to find what works best, your programs get a bit of personality and you have to explore a bit of your creativity to see what solution fits the specific decision you have to make. But an advise would be to use functions as you would have done not using literate programming and split of a chunk if the description becomes to complicated.

All comments are welcome. Is this sort of stuff covered in Knuth's web book?

About modularity in files, there is also more than one way to do it. I am in favor to have a functionality in literate programs that permits you to use a WEB-include in a WEB-file and also to have a single WEB-file generate several TANGLED-sources. This will also create an overlap, but if a project consists of several files why not keep them together so that you can describe them together as well. The original WEB for PASCAL did not have this functionality, and this I think is a lack. Why not have a WEB-file that contains an intended (Turbo) Pascal unit or include file, a simple Pascal source file that uses the unit, a script or standard initialization file and a makefile to show the relation of the different components?

If literate programming is to document, why not document it all from within a single WEB-file that functions as a project-description. Well, in CWEB it is already possible but for the make- and script-files. But if you start on this road, you will find another overlap, what to call a project and what to use as includes, but here also it can be resolved by keeping each project documentation clear, complete and understandable. So if your company always uses some standard chunks that may not be changed, they can be put in an include-file, but you can also choose to have standard code in a unit or object-file that gets linked on source-level, and have that documented in the program-description, or if you want to keep your programs complete so that you can interchange them with other people, you may have to choose for the option of actually copying everything that is non-standard within your WEB-file, so that it will not get lost and you will not have a WEB-file dependant on other WEB-file(s).

From:	Kirk Saranathan
Date:	22 May 2000

Henk Vink writes: In raising this issue of modularity, you have thought about some of the things that crossed my mind as well. By the way, what flavor of literate programming are you using?

These days I use noweb with C/C++. I started with Knuth's CWEB but it was too complicated for what it did.

I have recently acquired a job that expects me to write c++. I find that c++ forces a structure to the code (using classes) which makes literate programming tools much less useful. What goes into a chunk usually goes into a member function or a class definition so chunks are replaced by structure provided in the programming language. This topic has been visited in the literate programming archives but its something that has just now come up in practice for me. Of course when writing code that does complicated things (mathematical for example) its nice to have typeset docs paired with code chunks. Having said all this I do find literate programs much easier to read/debug (especially in C) when revisiting them.

True. Compilers also find optimizing many little files easier than one big one. Why make the reader look at a makefile that is external to the source to understand the component structure of the source? Describe the structure in the web file. Putting everything in the web file is better.