Tuesday, May 02, 2006

Two things about performance testing

A couple of things to stick in your back pocket:

1) Think geometrically, not arithmetically. Think about orders of magnitude.

10 is the same as 20 is the same as 50.
2000 is the same as 5000 is the same as 7000.
100,000 is the same as 200,000 is the same as 300,000.

The interesting information comes between the orders of magnitude: is 10,000 faster or slower than 10? is 1000 faster or slower than 100,000?

2) Think about precision and accuracy.

It is precise to say "10 records per second on this OS on this hardware on this network".
It is accurate to say "a rate of between 1 and 10 records per second on any OS on any reasonable hardware".
It is accurate to say "10 records per second running locally; 1 record per second over the network; 10 seconds per record over VPN".

Accuracy is critical in the general case; precision is critical in the specific case.

1 comment:

Pradeep Soundararajan said...

hmm ! could you explain it in detail ?