Test Driven Development

#The Problem In order for an open source project to benefit optimally from modifications from multiple developers, we need to reduce as much as possible the following common problems:

• people are unsure exactly what the existing code does or even what it's supposed to do ("the specification") because there is no explicit spec.
• even when there is a spec, there's no automatic way to see if the code actually meets the spec.
• code for new features or even bug fixes breaks existing functionality (introduces new bugs). As more people contribute more changes, more things get broken ("bit rot").
• a developer rewrites a large piece of code on the grounds that it's hard to understand or has bugs, but in practice the new code is not perfect (or perfectly documented), so this exchanges one set of bugs for a new set of bugs.

#The Solution All of these problems arise from the unavoidable tendency of each developer to focus on one problem (e.g. his exciting new feature) in order to implement it, forgetting many other detailed requirements in the code. Without protections, this leads to bit rot. This can only be balanced by a safety net of comprehensive automated tests that will always keep in mind all the detailed requirements, so the developer can focus on his task.

We will seek to reduce these problems through a test-driven development process:

• new features and bug fixes should first be proposed as a detailed specification that defines the desired behavior. Above all this should include concrete examples that illustrate clearly what should (and should not) happen. These examples will directly furnish test cases. Developers and savvy users should discuss the spec, refine the test cases, and contribute new test cases, until they converge to agreement.
• the spec of test cases should then be turned into automated tests that catch possible categories of misbehavior. The new tests should be run on the existing code.
• Based on lessons from this process, developers should discuss how to implement. They should try especially to come up with test cases that show that a particular implementation won't work. Ideally they should split up "lead implementer" and "lead test writer" roles (though both people can contribute to both tasks), since it's hard for one person to do both simultaneously.
• implementation and testing should proceed in small incremental cycles that each demonstrably give us one improvement while assuring us that nothing was broken. Each cycle should not be considered completed unless there is a comprehensive set of tests that cover possible misbehaviors.
• Refactoring should be considered when there is duplication of code or a clearly more efficient, simpler, or more general implementation of the specification. It should only be undertaken when there is a clear agreement about specification, and a comprehensive test suite that covers that specification; otherwise there will be no way to know what the refactoring might have broken. Developers should only refactor if they agree that it is needed, either because the existing design simply can't accommodate a desired behavior, or a decisive improvement in simplicity or performance is possible. Refactoring can be highly beneficial, but this significant effort should only be undertaken when developers agree it will in fact yield a better payoff than other possible work.

Current Status

Admittedly, it's hard to do this process alone. (One of its key benefits is that it's a social process). Given that this project has very much started out as a one-man project, I certainly can't say I've been doing this all along. Here's where we are today:

• from the beginning I started with a large set of automated functional tests (spnet/test.py), admittedly in the suboptimal form of "one honking big test function". They test progressively building up an spnet database from zero (as opposed to unit tests that each test an isolated test case). These were worth their weight in gold, and saved my butt on many, many occasions. But this hardly represents a comprehensive "spec" of test cases. WARNING: this is a destructive test; it drops the current spnet database and should never be run if your database contains data you don't want to lose.

• recently we switched to running tests under the popular test framework nosetests, and began adding unit tests.

• As part of David Ketcheson's work on incoming, I've written a suite of unit tests covering indexing of incoming posts. David and I are working together in a sort of TDD "pair programming" collaboration, which has been great. I'd like to work this way in the future.

• For a new developer wanting to work on a specific improvement, we should probably work together in this same way, i.e. start by trying to come up with test cases for both the desired new behavior and for the "current spec". This can be a quick process and has immediate benefits both for clarifying possible ways to implement the new feature, and for testing it and possible bit rot.

• we now need to write tons of unit tests covering all aspects of existing functionality, starting with components that we want to modify.

Basic Guidelines

• tests are written in separate test modules (i.e. not in the same file as functional code). Any file named test_SOMETHING.py will be scanned for test functions, which should also be named test_SOMETHING().
• if such a module contains a setup() function, it will be called for initialization prior to each test.
• For details, see the nose docs.
• For testing Google+ indexing, it is very useful to create a Google+ Page. Such a "page" is equivalent in G+ to a new user ID: you can write new test posts as that user ID, without worrying that any of your regular "Circles" will get bombarded with them (because the G+ page has a different user ID). You can then test spnet code's ability to properly index those posts.

Manual (interface) testing

We do not currently have a system for automated testing of web browser interfaces. So for the moment we're stuck with loading a test database with spnet data, and manually viewing the web pages and user interfaces.

• get a copy of the latest spnet database
• load it into mongodb on your test platform using mongorestore --drop
• run python -i web.py and point your web browser at http://localhost:8000 to test different views and interfaces.