Coverage != testing

8

This is in response to Mario’s quote on Eric’s blog mentioned by Eric in a comment on Mike’s blog (highly and unfairly excerpted here):

If we always start with failing tests before writing code that adds new capability to an application, it is inevitable that tests we write will cover classes with trivial complexity.

I think the danger with this line of thinking is that it implies that

  coverage == testing

But I would strongly state the opposite and the justification as:
  coverage != testing whereas
  coverage == use and
  testing == correct use (by way of assertion checking)

and there is a world of difference between just using the ATM and verifying it actually gave you $60. :)

Or as I’ve seen elsewhere and don’t have the reference right now: Testing verifies your code, coverage verifies your testing, but you can’t say that coverage tests your code.

Comments

8 Responses to “Coverage != testing”
  1. Eric Burke says:

    So…what are your thoughts about code generators that automatically generate getter/setter tests just to satisfy code coverage requirements?

  2. Hi, I do use a code generator every day. Unit tests are generated for getters/setters and it makes a lots of sense if you extend them (not that common)… That being said, unless there is an integration pattern somewhere around (more common), I do not see real use.
    Enjoy your day

  3. Alex says:

    I don’t have any obsession with 100% coverage, so I don’t think I would take the time even to run a code generator for getter/setter tests. Generally, if I write getters/setters with interesting code in them, I test them. If I later add code to a getter/setter to make it interesting, I add tests at that point as needed. For objects with equals/hashCode I typically write tests for those which often has a side effect of testing setters at least.

  4. Julio Santos says:

    You use a strawman argument in my opinion. First you’re responding to someone who’s advocating test first. You everything you do is test first, you don’t need to worry about coverage, you will have it by simple definition of doing tests first.

    I must admit I’m guilty of being a 100%-coverage advocate, and trying hard to get there (98% on my current project). In part because I know I make mistakes and in part because 90% of my nastiest bugs are in the un-covered few percent.

    More importantly to your point, and why it’s a strawman argument, is that I have never seen anyone writing tests that don’t test anything (or the wrong thing) just for he sake of coverage. No one is writing “assert_equal 3, calc.add(1,1)” just so calc gets coverage. So yes, in practice coverage == testing.

  5. Alex says:

    @Julio, points fairly made, although I disagree.

    I think the main point is that simply calling a method (coverage) does not verify anything about whether it did the right thing. And there are lots of ways for indirect coverage (from testing another method) to execute a method without actually testing it. This can give you a false sense of security that coverage == testing. In the case of test-first, you’re right that theoretically every method should have an associated test. However, the overwhelming majority of developers do not work test-first.

    The comments were made in the context of testing “simple” getters and setters, and in that case I think I could make an argument that writing a bunch of tests like:

    public void testGetName() {
      Person p = new Person();
      p.setName("SpongeBob");
      assertEquals("SpongeBob", p.getName());
    }
    

    for auto-generated get/set methods is testing for the sake of coverage. Now, if that setter is doing a bunch of name validation or the getter is actually fronting a cache or talking to a database or constructing the name from multiple fields or something, then by all means test it.

    My experience has been that 75-80% coverage is where the rate of return falls off considerably. That is to say, coverage beyond that point is valuable, but is not valuable enough to justify the effort. Note, that that target is an average. For really important or critical or complex stuff, I would strive for something much higher. And for less important code I would shoot for something lower.

  6. Scott Bale says:

    I want to add my 2 cents here, as I’ve been thinking about this topic again as I’ve been retroactively adding tests to others’ existing code.

    This has basically been said already above, but I’m impressed once again that, when doing “test after” like I’m doing this week, and when tackling a codebase that is not easily testable, I find myself writing long and complicated test fixture setups just to enable a single test with a single assert. The side effect is that all that setup covers a lot of code, but does **not** test it. This is a little scary to me, that even if someone with good intentions tries to retroactively add tests to legacy code like I am doing, it’s almost unavoidable that the resulting coverage stats will be skewed due to the aforementioned complexity in test fixture setup. As Julio stated, and I heartily agree, with TDD in theory you can have much more confidence in raw coverage statistics, since every line of production code should be backed by at least one test. But if you find yourself in a situation like I’m in I’m not sure what can be done other than to be very skeptical of test coverage stats at all. And, as Kent Beck says, to write the tests you wish you had.

    re: code generators, in response to Eric and Alex, I hadn’t really thought about it before, but I would be in favor of generating tests for simple getters/setters if for no other reason than they would be great, obvious placesetters for future tests if and when (as Alex said) the getters or setters had to change to have more complex behavior. I don’t feel any particular need to have n-percent code coverage, but I am in favor of doing anything that eases other developers’ future enhancement/maintenance of a codebase I work on.

Trackbacks

Check out what others are saying about this post...
  1. [...] Coverage != testing- I couldn’t agree more, man. [...]

  2. [...] I was reading Howard’s post today on his crisis of faith with respect to unit testing and it launched an interesting conversation with a colleague. I’m a unit testing advocate. But I’m not a big fan of test-first or test-driven development (TDD). And I’m not a believer in 100% coverage. [...]