This almost was an article, but the ending was "we threw it all away". And who wants to hear that?
Introduction
The following code consists of four unit tests for a class called "Range" written in C#. Range accepts two arguments, a begin date and an end date. From that it calculates a range of dates. The unit tests are intended to show whether a particular date is within that range or not.
Range
Assert.IsFalse(dateRange.Includes(DateTime.Now.AddDays(-1)),"Yesterday should not be in the date range");
Assert.IsTrue(dateRange.Includes(DateTime.Now),"Today should be included in the date range");
Assert.IsTrue(dateRange.Includes(DateTime.Now.AddDays(3)),"The end of the range should exist in the date range");
Assert.IsFalse(dateRange.Includes(DateTime.Now.AddDays(4)),"4 days past the start date should not exist in the date range");
}
It is worth noting that this is real code, written by a .NET expert, to test a real application.
One of the tests has a bug. Under certain circumstances, it fails consistently. Can you see the bug? If you don’t see the bug, can you figure out the circumstances under which these tests could get into trouble?
Here’s the second clue: the third tests fails under certain circumstances. Got it yet?
Here’s the third clue: the third test passes in the local development environment but fails on the build machine. Does that tell you anything?
At this point, you know as much as I did when I diagnosed the problem.
But It Works On My Machine
Even in the most agile of environments, I still occasionally hear: “but it works on my machine!” Unfortunately, in this environment, I didn’t have access to Visual Studio, thus no access to the debugger, or any of the wonderful tools available with an MSDN license. But that’s OK—I don’t know C# anyway. Besides, I like low-tech solutions to high-tech problems. A trusty text editor is all anyone needs to solve this problem.
The Good, The Bad…
We’re setting up a 3-day range:
Range
What do the three passing tests have in common?
Assert.IsFalse(dateRange.Includes(DateTime.Now.AddDays(1)
Assert.IsTrue(dateRange.Includes(DateTime.Now)
Assert.IsFalse(dateRange.Includes(DateTime.Now.AddDays(4)
The first and third are clearly representative of equivalence classes within and beyond the range. But that second one is a boundary condition just like our third test, and it always passes, where the third test sometimes fails.
What do the second and third tests have in common?
…And The Ugly
Our two boundary tests with common elements are:
Assert.IsTrue(dateRange.Includes(DateTime.Now)
Assert.IsTrue(dateRange.Includes(DateTime.Now.AddDays(3)
The first one passes, the second one fails. Remember how we set up the initial condition?
Range
Do a little thought experiment:
We set up the initial conditions to test against: DateTime.Now and DateTime.Now.AddDays(3). Then a little bit later, we set up the conditions for the test itself.
(Do you have the answer now?)
The .NET “DateTime.Now” function is really precise. I don’t know how precise (I don’t have a debugger, remember?), but what would happen if it were precise to, say, a minute? So set up the test conditions at 12:00—that gives us a range from noon today until noon three days from now.
For the second test, our date in question would be 12:01 today, which is on the early side of our three-day range.
But…
(now you have it, right?)
…for our third test, we set up a DateTime value that is 12:01 three days from now, which is outside the range we set up at the beginning; and the test fails
Of course, the .NET DateTime.Now function is a lot more precise than one minute. In fact, it is just precise enough that our third test will pass in a really fast environment—like a local development environment—and fail in a slower environment, like a build machine that has other claims on the CPU.
Tomorrow Is Another Day
So what do we do with our fragile set of Range unit tests?
One option would be to assign the first instance of DateTime.Now to a variable and use it throughout all of the tests, so that we use a consistent value both to set up the test conditions and to run the tests. But that seems like an awful lot of work, for some little unit tests.
I agree with something Brian Marick said once: I have an “increasing inability to tell the difference between acceptance and unit tests”. In the end, we simply took a note and scrapped these tests as unit tests. They smelled bad.