Occasionally I hear someone say that the Clojure community is against testing. I understand the sources of this – Rich Hickey (creator of Clojure) has spoken about testing in a number of talks and I think some of those comments have been misconstrued based on pithy tweet versions of more interesting longer thoughts. I’d like to give my own opinions on testing, what I think Rich has said, and what I observe in the Clojure community.
Tests have cost
I think tests are highly valuable. I don’t know how I could write high quality code without writing tests. However, while tests are valuable, there are downsides to testing that are not examined often enough.
Writing tests takes time and there is an opportunity cost where that time could be spent doing something else. That doesn’t mean you shouldn’t write tests, it just means that a) you should consider whether the test you’re writing is valuable and b) you should get as much bang for the buck as possible. For example, testing getters and setters in your Java beans so your coverage report is 100% is a waste of your time.
Example based testing can (at times) also have a high maintenance cost. Some of this can be laid at the fault of languages or programming styles that require substantial mocking frameworks to test the functionality of the app. Perhaps more importantly, maintaing large test code bases can sometimes be harder than maintaining the actual app itself. Are the tests valuable enough to justify that maintenance? You should be asking yourself this question and either thinning parts that don’t matter or improving your test code to minimize the cost.
Tests are not design
TDD (test-driven design) is a technique that promises to lead you towards a solution using only tests to guide you on your way, by throwing increasingly rich tests at your code base and refactoring the code to meet new challenges while maintaining the answers to prior challenges.
TDD is clearly a tool that some people find useful. Personally, I have found TDD to be a poor way for me to explore a new problem with an unknown solution. TDD tends to usually be either too slow or too fast for my purposes. Usually what I really need is not more tests, but more thinking. Pen and paper or a whiteboard are often a better fit for thinking than tests.
As I said, clearly TDD is a tool some people find useful, however I see it as just one of many strategies to explore a solution space.
Things Rich said :)
In Simple Made Easy, Rich talks (~16:00) about bugs, noting that every bug you see in the field a) passed the type checker and b) passed the test suite. He also talks with disdain about “guard rail programming” and asks the question “Do guard rails guide you where you want to go?”
I think the point being made here is not that guard rails (tests, type checker) are bad, but that guard rails are not the ideal tool for navigation (design). The other point is that when bugs are found, all of our guard rails have failed – what we are left with is thinking, and thinking is dependent on understanding the problem, and understanding is easier with simple (non-intertwined) programs. Rich goes on to make the point that ignoring complexity (deferring thinking) makes development faster in the beginning but hurts you enormously later on. In other words, developing an app is a marathon, not a sprint.
Fogus asked Rich about TDD in his interview and again there Rich does not say that tests or TDD are bad, but rather that for him, thinking is more valuable than writing tests. In particular he makes the point: “A bad design with a complete test suite is still a bad design.”
Clojure community observations
What I see in the community is a wide diversity of techniques to testing, certainly wider than Java and perhaps even wider than Ruby. Off the top of my head I see people exploring:
- REPL testing
- clojure.test – classic example based testing
- Speclj – BDD style tests
- Midje – a test framework supporting both top-down and bottom-up solution development
- tools.trace – maybe more debugging than testing?
- test.generative – generative testing
- Simulant – simulation system testing
- Simple-check – QuickCheck style property testing
- core.typed – optional type checking as a library
- core.contracts – contracts library
- Sterling – specifications inspired by Eiffel’s contracts
I see all of these actively being used. In going back to my comments above about testing cost, there is a tremendous amount of value in testing your code at the REPL. Not all of those checks need to make it into a permanent automated testing suite. If they do make it in, you should strongly consider test strategies beyond example-based tests.
Techniques like generative, simulation, or specification tests are testing styles that can provide greater coverage than you can hand-roll in example or BDD tests and they tend to be less fragile as well. They may have a higher up-front cost but so far in my experience they have a lower maintenance cost and much higher coverage, thus yielding greater overall value.
In summary, my own opinion is that the Clojure community is interested in tests that maximize long-term value (because Rich has set that tone). This has led the community to different areas than other communities. Rich has also argued that high quality design is based not on your testing process but on the quality of thought and preparation in understanding the problem to be solved and creating solutions made of non-intertwined pieces.