Tuesday, August 07, 2007

just enough design

So in recent weeks the number of test steps in our Selenium-based framework has quadrupled, from about 1000 to about 4000. The time it takes to run these tests is starting to cause significant pain.

So I set out to spike a little harness that would run sets of tests concurrently. (Note: it is *so* nice to have fork() available, after years of working in Windows environments.)

Trying to achieve the Simplest Test Harness That Could Possibly Work, I started out running all of the tests in the same test environment. Unfortunately, some of the tests change the state of the test environment, which causes other tests running concurrently to fail.

I started separating the state-changing tests (which may not be run concurrently) from the other tests (which may be run safely).

My colleague objected to this. Strongly. And he's right. He framed his argument in terms of old-school "software engineering" principles, but the argument is sound in terms of agile mantras like "make it work; make it good; make it fast". Segregating magic tests from non-magic tests will eventually cause failure. Or madness. Or both.

My next spike, I'm going to clone the test environment and run sets of tests in their own test environments. I think that's the *next*-simplest thing that could work, and it will get us a long way down the road-- until we have to design some more.

1 comment:

Smart Monkey Tester said...

I presume that you're bringing your AUT's machine to its knees, not the machine upon which the tests are running? Otherwise, you could farm out the browser work to other machines, separating by sequential tests, if any.