Alan just posted this: http://angryweasel.com/blog/?p=325
This is the nicest test example I've seen in a long time, and I think it bears a little more analysis.
If I were in charge of the development of this app, I would make automated testing happen on 3 levels.
First, there would have to be some sort of test for generating a single random number. Which seems easy enough, but at this level, you really have to understand exactly what a call to rand() (or whatever) on your particular OS is going to do. I once wrote a script in Perl (praise Allah just a toy, not production code) that returned perfectly random numbers on OSX but returned exactly the same values every time on Windows. You'd better test that your call to rand() really returns random numbers, regardless of the OS it is running on.
At a higher level, you'd want to do exactly what Alan talks about in looping 100,000 times (although 100K seems like overkill to me). This is what I think of as an "integration test". You're going to have some method or possibly even an API call or REST endpoint like "generate_five_random_numbers_and_add_the_results". You're going to want to exercise that wrapper enough to convince yourself that the numbers are right and that the math is right.
For some time now I've been making my living writing automated UI/browser tests along with doing ET. Here's my take on a UI test for Alan's app:
Open the page
Check that text "Total" exists.
Check that the 6 textboxes exist.
Click the Roll button.
Check that the state of the page changes (however that happens).
Check that there are values in each textbox (even this might be too fancy, depending on the app and the test framework).
Also getting fancy, you could check that the 6 textboxes exist in the correct order. (Selenium's "glob:" feature makes this pretty painless.)
Doing math in UI tests isn't very smart. Doing data comparison over multiple runs in UI tests isn't very smart. Do that stuff at lower levels, where you can take advantage of programming power and speed. UI tests are a shallow layer where you simply check that the user has all the stuff they need to get the job done.
This is the nicest test example I've seen in a long time, and I think it bears a little more analysis.
If I were in charge of the development of this app, I would make automated testing happen on 3 levels.
First, there would have to be some sort of test for generating a single random number. Which seems easy enough, but at this level, you really have to understand exactly what a call to rand() (or whatever) on your particular OS is going to do. I once wrote a script in Perl (praise Allah just a toy, not production code) that returned perfectly random numbers on OSX but returned exactly the same values every time on Windows. You'd better test that your call to rand() really returns random numbers, regardless of the OS it is running on.
At a higher level, you'd want to do exactly what Alan talks about in looping 100,000 times (although 100K seems like overkill to me). This is what I think of as an "integration test". You're going to have some method or possibly even an API call or REST endpoint like "generate_five_random_numbers_and_add_the_results". You're going to want to exercise that wrapper enough to convince yourself that the numbers are right and that the math is right.
For some time now I've been making my living writing automated UI/browser tests along with doing ET. Here's my take on a UI test for Alan's app:
Open the page
Check that text "Total" exists.
Check that the 6 textboxes exist.
Click the Roll button.
Check that the state of the page changes (however that happens).
Check that there are values in each textbox (even this might be too fancy, depending on the app and the test framework).
Also getting fancy, you could check that the 6 textboxes exist in the correct order. (Selenium's "glob:" feature makes this pretty painless.)
Doing math in UI tests isn't very smart. Doing data comparison over multiple runs in UI tests isn't very smart. Do that stuff at lower levels, where you can take advantage of programming power and speed. UI tests are a shallow layer where you simply check that the user has all the stuff they need to get the job done.