Literate Testing

Designing better APIs by focusing on use-cases, explanation, justification, and tested code samples.

Introduction

It’s well established that designing good, usable APIs is important. The Literate Testing approach helps you create such APIs, with better documentation and – crucially – automated tests. Instead of starting with an API and its implementation, you generate it through a dialogue between documentation, use-case tests and implementation sketches.

Literate
means you design by writing prose. You tell a story, you explain and justify. This focuses your mind on your users and their use-cases.
Testing
means you don’t just promise, you prove with automated tests.
Literate Testing
weaves the two together. You liberally sprinkle your prose with code samples, which are either directly executable tests, or automatically cited from such tests. This adds precision, forcing you to think things through. And later on, when citations change, they indicate where surrounding prose might need an update too.

The end result are APIs based on use-cases, with automated tests and comprehensible, precise and maintainable documentation.

Specific Benefits

In particular, literate testing uncovers:

Inconsistent or poor naming
Its easy to live with a poor name in a test – but hard to explain it.
Overly complex design
You, the designer, have the design fully present in your mind. It often does not really seem complex when you write the tests. But try writing down all the details of why it is how it is: it will quickly make you wish it was simpler. And remember: your story should be convincing.
Need for simplified APIs atop lower-level APIs
If you have to explain the same setup code over and over again, you’ll see the need for a simplified API for typical setups.
Feature creep
Features and options you don’t really need, but thought you might just add quickly, become more expensive because every feature now incurs the full cost of documentation, test, design and implementation.

And it ensures:

Full documentation
When you release, you always release a fully documented product. What good is a feature no one knows about?
Working examples
People learn best from examples. They’ll scan documentation for example code, read it, and only resort to reading the surrounding text when the code isn’t clear to them by itself. They’ll often copy/paste this example code directly into their own projects and expected it to work. Examples are a crucial part of a user’s early experience with your product.

Tools

Here’s a collection of tools that support literate testing for a bunch of languages. Mostly, though, it’s not the tool that matters, but the approach.

Examples

First, let me show you some results of literate testing in action in two of my own projects.

Abacus Formula Compiler:

pbranch extension for Mercurial:

Python offers a plethora of examples. However, I have no idea if the APIs in question where written with docs and tests first. In any case, I like the following:

Background

The name, Literate Testing, derives from Donald Knuth’s Literate Programming. Knuth wanted program code to be embedded in a coherent exposition of its workings, focused on readers, not compilers.

Literate testing is similar, but for test code, not implementation code. I think that the benefit of literacy is much higher on essential use-case tests and APIs than on implementation code, except for very intricate algorithms.

Depending on the structure and verbosity of the test implementation language, citing of fragments of test code can be more useful than having directly executable test code in the documentation. It keeps the documentation to the point and leaves the tests in their accustomed surroundings, like JUnit, with full IDE support preserved.

What It’s Not

Sadly, the terminology in this field is far from clear. The ideas I outlined above have been floating around under the following headings:

However, there are similar yet crucially different approaches going by the same names. These are mostly from the Behaviour Driven Development school. They focus on readable tests, rather than tests embedded in readable prose. So they lack explanation and justification.

Links