Clojure and testing

13

Occasionally I hear someone say that the Clojure community is against testing. I understand the sources of this – Rich Hickey (creator of Clojure) has spoken about testing in a number of talks and I think some of those comments have been misconstrued based on pithy tweet versions of more interesting longer thoughts. I’d like to give my own opinions on testing, what I think Rich has said, and what I observe in the Clojure community.

Tests have cost

I think tests are highly valuable. I don’t know how I could write high quality code without writing tests. However, while tests are valuable, there are downsides to testing that are not examined often enough.

Writing tests takes time and there is an opportunity cost where that time could be spent doing something else. That doesn’t mean you shouldn’t write tests, it just means that a) you should consider whether the test you’re writing is valuable and b) you should get as much bang for the buck as possible. For example, testing getters and setters in your Java beans so your coverage report is 100% is a waste of your time.

Example based testing can (at times) also have a high maintenance cost. Some of this can be laid at the fault of languages or programming styles that require substantial mocking frameworks to test the functionality of the app. Perhaps more importantly, maintaing large test code bases can sometimes be harder than maintaining the actual app itself. Are the tests valuable enough to justify that maintenance? You should be asking yourself this question and either thinning parts that don’t matter or improving your test code to minimize the cost.

Tests are not design

TDD (test-driven design) is a technique that promises to lead you towards a solution using only tests to guide you on your way, by throwing increasingly rich tests at your code base and refactoring the code to meet new challenges while maintaining the answers to prior challenges.

TDD is clearly a tool that some people find useful. Personally, I have found TDD to be a poor way for me to explore a new problem with an unknown solution. TDD tends to usually be either too slow or too fast for my purposes. Usually what I really need is not more tests, but more thinking. Pen and paper or a whiteboard are often a better fit for thinking than tests.

As I said, clearly TDD is a tool some people find useful, however I see it as just one of many strategies to explore a solution space.

Things Rich said :)

In Simple Made Easy, Rich talks (~16:00) about bugs, noting that every bug you see in the field a) passed the type checker and b) passed the test suite. He also talks with disdain about “guard rail programming” and asks the question “Do guard rails guide you where you want to go?”

I think the point being made here is not that guard rails (tests, type checker) are bad, but that guard rails are not the ideal tool for navigation (design). The other point is that when bugs are found, all of our guard rails have failed – what we are left with is thinking, and thinking is dependent on understanding the problem, and understanding is easier with simple (non-intertwined) programs. Rich goes on to make the point that ignoring complexity (deferring thinking) makes development faster in the beginning but hurts you enormously later on. In other words, developing an app is a marathon, not a sprint.

Fogus asked Rich about TDD in his interview and again there Rich does not say that tests or TDD are bad, but rather that for him, thinking is more valuable than writing tests. In particular he makes the point: “A bad design with a complete test suite is still a bad design.”

Clojure community observations

What I see in the community is a wide diversity of techniques to testing, certainly wider than Java and perhaps even wider than Ruby. Off the top of my head I see people exploring:

I see all of these actively being used. In going back to my comments above about testing cost, there is a tremendous amount of value in testing your code at the REPL. Not all of those checks need to make it into a permanent automated testing suite. If they do make it in, you should strongly consider test strategies beyond example-based tests.

Techniques like generative, simulation, or specification tests are testing styles that can provide greater coverage than you can hand-roll in example or BDD tests and they tend to be less fragile as well. They may have a higher up-front cost but so far in my experience they have a lower maintenance cost and much higher coverage, thus yielding greater overall value.

In summary, my own opinion is that the Clojure community is interested in tests that maximize long-term value (because Rich has set that tone). This has led the community to different areas than other communities. Rich has also argued that high quality design is based not on your testing process but on the quality of thought and preparation in understanding the problem to be solved and creating solutions made of non-intertwined pieces.

Comments

13 Responses to “Clojure and testing”
  1. I don’t see that TDD means not to think. I agree it may be practiced w/o thinking. For me TDD frees my mind from the nearly endless details, so that I can focus on the problem being solved.

  2. Apropos of this post, I would encourage the development community (particularly programmers and testers) to consider the difference between writing tests (we would say writing checks) and testing. These two posts may shed some light on some of the controversy by highlighting what checks can and cannot do. http://www.satisfice.com/blog/archives/856, http://www.satisfice.com/blog/archives/638.

    I would also suggest that developing a product is neither a sprint nor a marathon. Both similes suggest that we know that there’s a finish line, and that we know where it is at the outset. To me, development is more like going at a tangled ball of string where we can’t tell how many threads are in the ball, whether we’ll end up with one long piece or several shorter pieces, or whether we’ll decide that some bits are too tangly and too short to bother messing with.

    —Michael B.

  3. Alex says:

    @James: I did not say that TDD means not to think. I encourage more awareness and thinking regardless of what practice you follow. If TDD is useful to you, please use it – I respect that.

    @Michael: Agreed on both points.

  4. I view TDD a bit differently than most people. I’m big on whiteboarding and pen-and-paper myself (in fact, my colleagues even joke that if I’m at the gates of heaven and am asked to justify my getting in, I’d probably ask for a marker and a whiteboard).

    I tend to try things out a bit, and then write a test and test-drive the same thing again. I test-drive because my tests describe my requirements, and then I write necessary and sufficient code to fulfill/meet these requirements. I find that code written test-first sometimes lends itself to certain forms of refactoring. I don’t have any concrete examples today, sorry bout that, but I’d say this is similar to “your code talks to you”. I then use code coverage to figure out if there’s by chance any code that I may have forgotten to test-drive. Such “uncovered” code is either simple stuff like getter-setter (can be ignored), or some stale requirement (remove the code), or some undocumented requirement (create a story card and test-drive the code).

    Today, I’m a coach. When I have to talk to experienced devs about writing tests, I tell them that these are not “tests” about them, and instead these are expressions of intent in an executable form that they place for the next dev to make use of.

  5. Alex says:

    @Sriram: When you say “expressions of intent in an executable form”, that sounds very much to me like the definition of contracts or specifications. Have you tried using them to annotate your code?

  6. Ryan says:

    For me, by far the single highest benefit of testing is to automatically run my code. I don’t have to test the correctness of every function, but I do have to have tests that *run* every function.

  7. Chris Freeman says:

    Concerning Examples:
    High maintenance costs are definitely a smell that you should refactor your tests. Usually, someone’s added some accidental complexity. This can happen with example-based tests, but avoiding it is fairly simple. And example-based tests are hugely important for discovering edge-cases and interpretations of your problem which you hadn’t yet considered.

    Concerning TDD:
    In TDD, design doesn’t magically happen. The TDD developer MUST spend time thinking about design at the beginning and at _every_ Refactor step, and too many people seem to miss this part.

    When I’m thinking about the problem, I’m not thinking (much) about the solution. Conflating the two tends to anchor me to a single solution (bad) and anchor me to a particular understanding of the problem (worse).

  8. Paul Stadig says:

    An important point that was missed is that every bug in production also got past your thinking. I say that not to disagree with what’s been said, but to point out that this form of argumentation can cut both ways.

    There is also implicit in the argument “we need more thinking and less TDD” that thinking is over and against testing, but it’s already been said that writing tests isn’t and shouldn’t be a mindless process. That may not be the intent of the argument, but that is what people have caught from it.

    One thing that was not mentioned that I have found useful in the past is assertions. I found it particularly useful implementing a data structure and using assertions to verify invariants before and after operations. Perhaps assertions don’t really fall into the “testing” category. It may be that they aren’t as useful for other things as they are for implementing data structures, but they are a tool that can be used when appropriate.

    I wish more people could have the perspective that all these things are tools, and just because one tool works well in one situation doesn’t mean that it should be used in every situation.

  9. Luc Prefontaine says:

    Hi agree with Rich,

    for me thinking about the design and coding state is far more productive than writing tests.
    I usualy get someone else to write unit tests after the code is stable in my view, mainly to get others
    on board and then have these rail guards in so latecomers can work on the code with some
    relative confidence.

    I can’t use tdd, it cuts me off potential avenues has I am toying with tthe code. To me it’s
    central to work from the core outward, not the reverse. Even guessing the API upfront can lead
    you in a dead end.

    I think that there’s an explanation for this behavior. I worked on many async/concurrent/real time
    apps. In these kinds of applications, unit tests and other rail guards are of little use.
    Mainly because it’s almost impossible to recreate real life conditions that lead to a production bug.

    So what’s left ? Well, design and code reviews. It may take a day before you actually fix a hole
    in the code, you need a significant number of hours to figure out were the code leaks and how you
    can fix it. Even if you have to change only three lines of code. And then you get this reviewed by
    someone else….

    So after a number of years, you focus on safe design choices to minimize these bugs
    rather than trying to create some tests to detect them somehow.

    Frankly, the first approach to me pays much more in the long run than any battery of tests
    written afterward. It looks sounder to me to avoid pitfalls at design/coding time than falling
    in sand traps and then trying to get out of them. Toying with the code these days in the REPL
    just boosts my productivity in this regard. I can tests limits faster and decide how to make
    things more resilient w/o writing tests at this stage.

    Tests have some value after a first internal release and not all tests have the same value.
    It’s a better investment to choose which tests to create after you defined how a service or library
    behaves than doing this before hand. Most of the time, integrated tests with mocking up or
    full scale simulations are more valuabke than unit tests but you cannot write these before having layered
    a number of core services to support them.

    If you spend a lot of time writing small scope tests, you will take more time to reach this stage.

    From Rich’s bio, I think he reached the same conclusions, being exposed to similar problems.
    I did not ever think of creating a language to make these apps simpler to write but it seems a logical
    follow up to the same initial conclusions.

    More and more applications are less and less linear since software is now pervasive in many
    aspects of our human life. Time to market and resilience are becoming important issues.
    Questionning test strategies is absolutely required as any other aspect
    in creating software systems in this regard.

    Luc P.

  10. I think Hickey also mentions in that “Simple Made Easy” talk how we focus too much on the construction of software and not enough on the artifacts. I’m probably botching what he means by “artifact”, but I consider automated tests valuable artifacts.

    – They’re the most trusted form of documentation one has about a library or system.

    – They give future developers insight on the usage and how to most easily provide the requisite environment.

    – They act as a canary. In other words, if the tests are difficult to develop, then chances are there is something wrong with the design (poor abstractions, too many side-effects, etc.)

    – They can help define a performance baseline. If a test starts taking noticeably longer (or faster) to complete, then should be easy to track down which change was responsible.

    Anyway, good article. No single testing strategy fits all environments, as I think you try to convey, but it’s always good to discuss.

  11. bob says:

    TDD has finally kind of ebbed in places Ive worked and that is somewhat of a relief. On projects where 100% coverage was a stated goal, tests were meaningless copy and paste and frequently broke as the system changed creating a huge amount of maintenance and very little value.

    A system design becomes a self-fulfilling prophecy of sorts, tests tend to only reinforce and reflect the expectations that the system already has, and those expectations ARE the future bugs.

    I personally have never seen a unit test reveal a production bug, and I doubt I ever will.

  12. Avdi Grimm says:

    I find that what my code needs is more thinking and less scribbling on a whiteboard.

    …that’s not actually what I find, but hopefully you take my point. You talk about thinking and drawing pictures as if they are complementary actions, and thinking and writing tests as if they are somehow not. Personally, I find writing tests to be just as conducive to thought as drawing pictures, although in different ways. Writing a test often brings up practical objections to a design that are easy to gloss over when scribbling on a board. They are both useful tools for design.

  13. Alexis says:

    Like James I’m not sure I agree with the TDD comments. I do think the important theme here is a good one. Testing, like everything else we do, has a time and place. We need it in order to accomplish our overall goals, but too much, as well as too little will greatly impede our overall objectives.