Saturday, December 24, 2005

But it works on *my* machine....

Welcome to the first post.

This almost was an article, but the ending was "we threw it all away". And who wants to hear that?


Introduction

The following code consists of four unit tests for a class called "Range" written in C#. Range accepts two arguments, a begin date and an end date. From that it calculates a range of dates. The unit tests are intended to show whether a particular date is within that range or not.

Range dateRange = new Range(DateTime.Now, DateTime.Now.AddDays(3));
Assert.IsFalse(dateRange.Includes(DateTime.Now.AddDays(-1)),"Yesterday should not be in the date range");
Assert.IsTrue(dateRange.Includes(DateTime.Now),"Today should be included in the date range");
Assert.IsTrue(dateRange.Includes(DateTime.Now.AddDays(3)),"The end of the range should exist in the date range");
Assert.IsFalse(dateRange.Includes(DateTime.Now.AddDays(4)),"4 days past the start date should not exist in the date range");
}

It is worth noting that this is real code, written by a .NET expert, to test a real application.

One of the tests has a bug. Under certain circumstances, it fails consistently. Can you see the bug? If you don’t see the bug, can you figure out the circumstances under which these tests could get into trouble?

Here’s the second clue: the third tests fails under certain circumstances. Got it yet?

Here’s the third clue: the third test passes in the local development environment but fails on the build machine. Does that tell you anything?

At this point, you know as much as I did when I diagnosed the problem.

But It Works On My Machine

Even in the most agile of environments, I still occasionally hear: “but it works on my machine!” Unfortunately, in this environment, I didn’t have access to Visual Studio, thus no access to the debugger, or any of the wonderful tools available with an MSDN license. But that’s OK—I don’t know C# anyway. Besides, I like low-tech solutions to high-tech problems. A trusty text editor is all anyone needs to solve this problem.

The Good, The Bad…

We’re setting up a 3-day range:

Range dateRange = new Range(DateTime.Now, DateTime.Now.AddDays(3));

What do the three passing tests have in common?

Assert.IsFalse(dateRange.Includes(DateTime.Now.AddDays(1)
Assert.IsTrue(dateRange.Includes(DateTime.Now)
Assert.IsFalse(dateRange.Includes(DateTime.Now.AddDays(4)

The first and third are clearly representative of equivalence classes within and beyond the range. But that second one is a boundary condition just like our third test, and it always passes, where the third test sometimes fails.

What do the second and third tests have in common?

…And The Ugly

Our two boundary tests with common elements are:

Assert.IsTrue(dateRange.Includes(DateTime.Now)
Assert.IsTrue(dateRange.Includes(DateTime.Now.AddDays(3)

The first one passes, the second one fails. Remember how we set up the initial condition?

Range dateRange = new Range(DateTime.Now, DateTime.Now.AddDays(3));

Do a little thought experiment:

We set up the initial conditions to test against: DateTime.Now and DateTime.Now.AddDays(3). Then a little bit later, we set up the conditions for the test itself.

(Do you have the answer now?)

The .NET “DateTime.Now” function is really precise. I don’t know how precise (I don’t have a debugger, remember?), but what would happen if it were precise to, say, a minute? So set up the test conditions at 12:00—that gives us a range from noon today until noon three days from now.

For the second test, our date in question would be 12:01 today, which is on the early side of our three-day range.

But…

(now you have it, right?)

…for our third test, we set up a DateTime value that is 12:01 three days from now, which is outside the range we set up at the beginning; and the test fails

Of course, the .NET DateTime.Now function is a lot more precise than one minute. In fact, it is just precise enough that our third test will pass in a really fast environment—like a local development environment—and fail in a slower environment, like a build machine that has other claims on the CPU.

Tomorrow Is Another Day

So what do we do with our fragile set of Range unit tests?

One option would be to assign the first instance of DateTime.Now to a variable and use it throughout all of the tests, so that we use a consistent value both to set up the test conditions and to run the tests. But that seems like an awful lot of work, for some little unit tests.

I agree with something Brian Marick said once: I have an “increasing inability to tell the difference between acceptance and unit tests”. In the end, we simply took a note and scrapped these tests as unit tests. They smelled bad.


3 comments:

Michael said...

A little research shows that the DateTime object is precise to milliseconds, so the wonder is that the test in question ever worked.

This is a fine example of several things, including:

- the value of the tests being inseparable from the quality of the tests

- the phenomenon of developer-written tests that are subject to the same kinds of errors as developer-written code

- that all oracles are heuristic, and open to failure

Nice post.

---Michael B.

ade said...

I've had the same sort of bugs happen to me on projects. Usually these tests work as long as you don't run them late at night, near the end of the month or near the end of the year. Eventually I realised the root cause of these intermittent failures.

The fundamental problem with these tests is that they're using the concept of 'now'. As such they're neither deterministic nor repeatable. Instead the tests should specify actual dates (like 2005-12-27) that don't change every time you run the tests. Nowadays when I see tests that don't isolate themselves from sources of randomness (like the external environment, random number generators and of course the system clock) I grow suspicious.

Antony Marcano said...

Nice post Chris. I started writing a comment but it kinda turned into a blog post...

I included some practical examples of how to make the test more robust.

Hope you find it useful.