Tuesday, May 23, 2006

Kevin Lawrence is still right

Back in March 2005 Kevin Lawrence wrote this great article for Better Software called "Grow Your Test Harness Naturally"

He's got a link to the PDF on his blog, luckily.

It's one of those articles that keeps being cited by various people one runs into here and there. I read it over and over until I Got It.

The basic premise is: don't write the framework first. Write some tests, then extract the common code from those tests, write more tests, abstract common code, write more tests, refactor, repeat. The refactoring generates the framework.

The thing is, I had never had enough code complex enough, running against an app stable enough, to really and truly build a framework this way.

Until now.

I have a large number things to do with Watir.

First I automated each path through the code individually. I did about 10 of these.
Each of my paths had to set up a certain set of rules, so I abstracted a method "make_rules"
Each of my paths had to set some data before doing a final click, so I abstracted a method for that.
At this point my 1o individual scripts had shrunk to about 5 lines of code anyway, so I put them all in one file.
But I still have to handle various variations in the path through the code, so I wrote a controller.rb file that orchestrates all of the making of rules, the data handling and final clicking.

Now I'm adding new paths through the GUI. Adding each new path means rejiggering my existing libraries to accomodate the new path. But since my libraries were generated as a consequence of actual code, they're naturally in tune with what I want to do. Updating them to handle the first new path through the code takes some thought, but the next 10 times I need to go down that path all of the methods are already in place, and the repetition becomes trivial.

The framework is reliably going through the GUI about 80 times now, every time subtly different than every other time.

The thing is, I don't think I could have *designed* this framework. For one thing, I didn't have enough information when I started writing to anticipate all of the paths I would need. For another, even though my framework isn't very complex, it's still large enough that I doubt that I could have held the whole architecture in my head in order to write it in advance.

I can't believe how a) robust and b) readable this framework is turning out to be. I still have some mistakes to clean up, and I have a couple more paths through the code to add, but I'm so pleased to finally find out for myself that Kevin Lawrence is still right.

Wednesday, May 03, 2006

Internet Explorer Basic Authorization vs iFrame is just wrong and stupid

UPDATE October 2009: it seems Angrez has added support for Basic Auth to a fork and it should hit the main Watir distro RSN. News will be here. Good things come...

So I wanted to bypass the 401 Authorization Required screen that IE kicks up without resorting to actually having to manipulate the popup. So I hacked Watir's goto() method, which works great:

def goto(url)
@ie.navigate(url,nil,nil,nil,"Authorization: Basic AbdXyz46bG1ubw==\n")
sleep 0.2
return @down_load_time

I know I can get away with this because the user/password on the test systems are never going to change, and even if they do, it's an easy script change.

The trouble is that the main page has an iFrame, and for some reason, when called from a script, stupid IE won't use Basic Authorization credentials to request the iFrame.

So now I'm getting the authorization popup when IE tries to load the iFrame contents. The main page is loading fine.

Deeply frustrating.

And stupid.

Tuesday, May 02, 2006

Two things about performance testing

A couple of things to stick in your back pocket:

1) Think geometrically, not arithmetically. Think about orders of magnitude.

10 is the same as 20 is the same as 50.
2000 is the same as 5000 is the same as 7000.
100,000 is the same as 200,000 is the same as 300,000.

The interesting information comes between the orders of magnitude: is 10,000 faster or slower than 10? is 1000 faster or slower than 100,000?

2) Think about precision and accuracy.

It is precise to say "10 records per second on this OS on this hardware on this network".
It is accurate to say "a rate of between 1 and 10 records per second on any OS on any reasonable hardware".
It is accurate to say "10 records per second running locally; 1 record per second over the network; 10 seconds per record over VPN".

Accuracy is critical in the general case; precision is critical in the specific case.