Photo

Literate Testing

Designing better APIs by focusing on use-cases, explanation, justification, and tested code samples.

Introduction

It’s well established that designing good, usable APIs is important. The Literate Testing approach helps you create such APIs, with better documentation and - crucially - automated tests. Instead of starting with an API and its implementation, you generate it through a dialogue between documentation, use-case tests and implementation sketches.

Literate
means you design by writing prose. You tell a story, you explain and justify. This focuses your mind on your users and their use-cases.
Testing
means you don't just promise, you prove with automated tests.
Literate Testing
weaves the two together. You liberally sprinkle your prose with code samples, which are either directly executable tests, or automatically cited from such tests. This adds precision, forcing you to think things through. And later on, when citations change, they indicate where surrounding prose might need an update too.

The end result are APIs based on use-cases, with automated tests and comprehensible, precise and maintainable documentation.

Specific Benefits

In particular, literate testing uncovers:

And it ensures:

Tools

Here’s a collection of tools that support literate testing for a bunch of languages. Mostly, though, it’s not the tool that matters, but the approach.

Examples

First, let me show you some results of literate testing in action in two of my own projects.

Abacus Formula Compiler:

pbranch extension for Mercurial:

Python offers a plethora of examples. However, I have no idea if the APIs in question where written with docs and tests first. In any case, I like the following:

Background

The name, Literate Testing, derives from Donald Knuth’s Literate Programming. Knuth wanted program code to be embedded in a coherent exposition of its workings, focused on readers, not compilers.

Literate testing is similar, but for test code, not implementation code. I think that the benefit of literacy is much higher on essential use-case tests and APIs than on implementation code, except for very intricate algorithms.

Depending on the structure and verbosity of the test implementation language, citing of fragments of test code can be more useful than having directly executable test code in the documentation. It keeps the documentation to the point and leaves the tests in their accustomed surroundings, like JUnit, with full IDE support preserved.

What It’s Not

Sadly, the terminology in this field is far from clear. The ideas I outlined above have been floating around under the following headings:

However, there are similar yet crucially different approaches going by the same names. These are mostly from the Behaviour Driven Development school. They focus on readable tests, rather than tests embedded in readable prose. So they lack explanation and justification.