Distributed testing frameworks


I’m doing a little research right now on distributed testing frameworks and thought I could blog as I go. You may be asking, “But Alex, you work at Terracotta, making a distributed computing product, surely you have a distributed testing framework?” And sure enough we do have one. Well, actually 2 or 3, all with their pros and cons.

But I have some new needs and ideas and I’m doing a little poking to see if it makes sense to extend an existing external framework rather than an existing internal framework or writing something new. I’m also wondering whether integrating an existing framework with Terracotta would be a useful thing for a lot of others as well. In particular, using Terracotta can make it a lot easier to coordinate test actions and phases and collect results.

Here are some good meta pages I’ve found that seem to list every project that came to the top of my mind plus a lot of others:

I don’t really want to bore you with the detailed requirements I have but things I would expect to get out of a distributed test framework would include the ability to start up multiple nodes, walk those nodes through a series of phases (not just a single load), start/stop nodes as part of the test, use some kind of scripting for some or all of the actions taken in each phase, and collect and aggregate results.

I’d really like to write the scripts in something simpler than Java like Groovy, Jython, Ruby, Javascript, etc. If you mention XML, you have to go sit in the corner and wear a funny hat until you apologize.

Ones that I find to be leading choices at the moment:

  • JMeter – this was the first thing to came to mind and something I’ve used in the past with success. I think it would probably hit 90% of my needs. The scripts are a little clunky for my taste but certainly usable. Plus, you can plug in bean shell scripts, so I think I could do something reasonable.
  • The Grinder – seems to have advanced quite a bit since the last time I looked at it. Scripts are in Jython which makes them both flexible and readable, which I like a lot. I’ve worked with people that used this successfully and liked it.

Ones that I looked at but am not currently considering (nothing negative implied here, just not a good match for my reqs):

  • Testare – seems to focus on J2EE testing and container testing
  • Java Test Runner – doesn’t seem like a good match, doesn’t seem to be a community around it

Am I missing anything stupidly obvious and important for consideration? I’d be happy to use something that was written or scripted in a language other than Java as long as I could launch Java stuff efficiently. Any distributed JRuby test framework I should be aware of?

Searching just on my top finds so far yielded a shoot-out report of JMeter, The Grinder, and Mercurcy LoadRunner. He ultimately picked The Grinder and liked some of the same things I like about it. My own general impressions of Mercury tools in general is that they are expensive and some of the lowest quality software I’ve ever used (on any dimension), which doesn’t seem good for tools intended to improve your own quality.


3 Responses to “Distributed testing frameworks”
  1. Hey – Just came across your post via dzone.com. At my current position I inherited a suite of JMeter test cases and a harness for running and generating reports from its raw data. I’m now in the process of moving over to The Grinder v3 to take advantage of the ability to script the tests and re-use elements to build larger tests.

    JMeter is great for getting a test running quickly, but for long term maintainability, I think the Grinder is an excellent choice between those two.


Check out what others are saying about this post...
  1. [...] I had the occasion recently to evaluate the Grinder for a test system I was building. I was really pleasantly surprised by everything that I found. The Grinder has a fairly clean aesthetic that is hard to quantify but makes getting started a pleasant experience. What I found the most enjoyable about it was the use of Jython to script the actual test activity. Python has a reputation for clean, beautiful code and I found that it was actually fun to hack on the test scripts (once I refreshed my Pythonese a bit). [...]

  2. [...] As you might expect, this first day was mostly about setting up my computer. Alex and I did talk about what I would work on first, which is the distributed testing framework that is in development and a set of test use cases centered around Terracotta’s distributed cache functionality. [...]