Literate Testing

Designing better APIs by focusing on use-cases, explanation, justification, and tested code samples.

Introduction

It’s well established that designing good, usable APIs is important. The Literate Testing approach helps you create such APIs, with better documentation and - crucially - automated tests. Instead of starting with an API and its implementation, you generate it through a dialogue between documentation, use-case tests and implementation sketches.

Literate

means you design by writing prose. You tell a story, you explain and justify. This focuses your mind on your users and their use-cases.

Testing

means you don't just promise, you prove with automated tests.

Literate Testing

weaves the two together. You liberally sprinkle your prose with code samples, which are either directly executable tests, or automatically cited from such tests. This adds precision, forcing you to think things through. And later on, when citations change, they indicate where surrounding prose might need an update too.

The end result are APIs based on use-cases, with automated tests and comprehensible, precise and maintainable documentation.

Specific Benefits

In particular, literate testing uncovers:

• Inconsistent or poor naming – Its easy to live with a poor name in a test - but hard to explain it.

• Overly complex design – You, the designer, have the design fully present in your mind. It often does not really seem complex when you write the tests. But try writing down all the details of why it is how it is: it will quickly make you wish it was simpler. And remember: your story should be convincing.

• Need for simplified APIs atop lower-level APIs – If you have to explain the same setup code over and over again, you’ll see the need for a simplified API for typical setups.

• Feature creep – Features and options you don’t really need, but thought you might just add quickly, become more expensive because every feature now incurs the full cost of documentation, test, design and implementation.

And it ensures:

• Full documentation – When you release, you always release a fully documented product. What good is a feature no one knows about?

• Working examples – People learn best from examples. They’ll scan documentation for example code, read it, and only resort to reading the surrounding text when the code isn’t clear to them by itself. They’ll often copy/paste this example code directly into their own projects and expected it to work. Examples are a crucial part of a user’s early experience with your product.

Tools

Here’s a collection of tools that support literate testing for a bunch of languages. Mostly, though, it’s not the tool that matters, but the approach.

• Java: JCite (my own creation)
• Python: doctest
• Haskell: literate haskell
• Shell Scripts: tush, shell-doctest, and maybe docbash (which I wrote for pbranch)
• Erlang: updoc

Examples

First, let me show you some results of literate testing in action in two of my own projects.

Abacus Formula Compiler:

• Focus on simplest use case; crucial that examples work
• Use-cases as drivers
• Complex topic, simple API

pbranch extension for Mercurial:

• Use-cases as drivers
• Complex topic, complex setup

Python offers a plethora of examples. However, I have no idea if the APIs in question where written with docs and tests first. In any case, I like the following:

• csv module
• hashlib module

Background

The name, Literate Testing, derives from Donald Knuth’s Literate Programming. Knuth wanted program code to be embedded in a coherent exposition of its workings, focused on readers, not compilers.

Literate testing is similar, but for test code, not implementation code. I think that the benefit of literacy is much higher on essential use-case tests and APIs than on implementation code, except for very intricate algorithms.

Depending on the structure and verbosity of the test implementation language, citing of fragments of test code can be more useful than having directly executable test code in the documentation. It keeps the documentation to the point and leaves the tests in their accustomed surroundings, like JUnit, with full IDE support preserved.

What It’s Not

Sadly, the terminology in this field is far from clear. The ideas I outlined above have been floating around under the following headings:

• Literate Testing, obviously (Google search)
• Documentation Driven Development (Google search)
• Documentation Driven Testing (Google search)

However, there are similar yet crucially different approaches going by the same names. These are mostly from the Behaviour Driven Development school. They focus on readable tests, rather than tests embedded in readable prose. So they lack explanation and justification.

Links

• How To Design A Good API and Why it Matters (Joshua Bloch; also on video)
• Wikipedia on doctest
• Results of earlier Google search for literate testing (by me)
• Benefits of citing test sources (by me)