Wednesday, December 28, 2005

My wish for 2006

Well, one of my wishes, anyway...

I'd like to see someone take Watir and attach it to a Ruby on Rails back end that would hold test data, test-running instructions, test results, and metadata like requirements or defects linked to particular tests. Picture something like Mercury TestDirector+WinRunner, but with Rails+Watir.

Jonathan Kohl agrees with me that it would be possible, but it would take some real effort to prove that such a thing would work. I'm not sure I'm bold enough to start such a project myself.

Tuesday, December 27, 2005

What part of "$&" don't you understand?

Perl has some awfully handy stuff, even when it's in Ruby.

The Pragmatic Programmers start their Regular Expression chapter in the Pickaxe by discussing $& (the last regular expression match), and $', and $` (the strings immediately before and after the last RE match). Awfully convenient stuff, borrowed directly from Perl. At the end of the chapter they apologize:

"Having said all this, we have to 'fess up. Andy and Dave normally use the $-variables rather than worrying about MatchData objects. For everyday use, they just end up being more convenient. Sometimes we just can't help being pragmatic."

I wrote a little Ruby utility script recently to handle a bookkeeping chore, that does some screen-scraping and Excel-file-reading. About forty lines. $& figured prominently.

That's not so offensive, is it?

Saturday, December 24, 2005

But it works on *my* machine....

Welcome to the first post.

This almost was an article, but the ending was "we threw it all away". And who wants to hear that?


Introduction

The following code consists of four unit tests for a class called "Range" written in C#. Range accepts two arguments, a begin date and an end date. From that it calculates a range of dates. The unit tests are intended to show whether a particular date is within that range or not.

Range dateRange = new Range(DateTime.Now, DateTime.Now.AddDays(3));
Assert.IsFalse(dateRange.Includes(DateTime.Now.AddDays(-1)),"Yesterday should not be in the date range");
Assert.IsTrue(dateRange.Includes(DateTime.Now),"Today should be included in the date range");
Assert.IsTrue(dateRange.Includes(DateTime.Now.AddDays(3)),"The end of the range should exist in the date range");
Assert.IsFalse(dateRange.Includes(DateTime.Now.AddDays(4)),"4 days past the start date should not exist in the date range");
}

It is worth noting that this is real code, written by a .NET expert, to test a real application.

One of the tests has a bug. Under certain circumstances, it fails consistently. Can you see the bug? If you don’t see the bug, can you figure out the circumstances under which these tests could get into trouble?

Here’s the second clue: the third tests fails under certain circumstances. Got it yet?

Here’s the third clue: the third test passes in the local development environment but fails on the build machine. Does that tell you anything?

At this point, you know as much as I did when I diagnosed the problem.

But It Works On My Machine

Even in the most agile of environments, I still occasionally hear: “but it works on my machine!” Unfortunately, in this environment, I didn’t have access to Visual Studio, thus no access to the debugger, or any of the wonderful tools available with an MSDN license. But that’s OK—I don’t know C# anyway. Besides, I like low-tech solutions to high-tech problems. A trusty text editor is all anyone needs to solve this problem.

The Good, The Bad…

We’re setting up a 3-day range:

Range dateRange = new Range(DateTime.Now, DateTime.Now.AddDays(3));

What do the three passing tests have in common?

Assert.IsFalse(dateRange.Includes(DateTime.Now.AddDays(1)
Assert.IsTrue(dateRange.Includes(DateTime.Now)
Assert.IsFalse(dateRange.Includes(DateTime.Now.AddDays(4)

The first and third are clearly representative of equivalence classes within and beyond the range. But that second one is a boundary condition just like our third test, and it always passes, where the third test sometimes fails.

What do the second and third tests have in common?

…And The Ugly

Our two boundary tests with common elements are:

Assert.IsTrue(dateRange.Includes(DateTime.Now)
Assert.IsTrue(dateRange.Includes(DateTime.Now.AddDays(3)

The first one passes, the second one fails. Remember how we set up the initial condition?

Range dateRange = new Range(DateTime.Now, DateTime.Now.AddDays(3));

Do a little thought experiment:

We set up the initial conditions to test against: DateTime.Now and DateTime.Now.AddDays(3). Then a little bit later, we set up the conditions for the test itself.

(Do you have the answer now?)

The .NET “DateTime.Now” function is really precise. I don’t know how precise (I don’t have a debugger, remember?), but what would happen if it were precise to, say, a minute? So set up the test conditions at 12:00—that gives us a range from noon today until noon three days from now.

For the second test, our date in question would be 12:01 today, which is on the early side of our three-day range.

But…

(now you have it, right?)

…for our third test, we set up a DateTime value that is 12:01 three days from now, which is outside the range we set up at the beginning; and the test fails

Of course, the .NET DateTime.Now function is a lot more precise than one minute. In fact, it is just precise enough that our third test will pass in a really fast environment—like a local development environment—and fail in a slower environment, like a build machine that has other claims on the CPU.

Tomorrow Is Another Day

So what do we do with our fragile set of Range unit tests?

One option would be to assign the first instance of DateTime.Now to a variable and use it throughout all of the tests, so that we use a consistent value both to set up the test conditions and to run the tests. But that seems like an awful lot of work, for some little unit tests.

I agree with something Brian Marick said once: I have an “increasing inability to tell the difference between acceptance and unit tests”. In the end, we simply took a note and scrapped these tests as unit tests. They smelled bad.