Static tests and dynamic tests

A typical test expects a particular set of data to exist, right?

Say I have some software that accepts a chunk of XML and does a search for people in a database by last name and first name. It might accept

<requestroot xmlns="http://foo">
<lastname xmlns="" > MCMAHON < /lastname>
<firstname xmlns="">CHRIS</firstname>
</requestroot>


and it might return

<responseroot xmlns="http://foo">
<user xmlns="">CHRIS MCMAHON</user>
</responseroot>

so I could write a test that does something like


SEND

<requestroot xmlns="http://foo">
<lastname xmlns="">MCMAHON</lastname>
<firstname xmlns="">CHRIS</firstname>
</requestroot>

RECEIVE AND PARSE

<responseroot xmlns="http://foo">
<user xmlns="">CHRIS MCMAHON</user>
</responseroot>

That is a static test, and that's how unit tests work. More than likely, the returned value CHRIS MCMAHON isn't in a database at all; it's returned by a mock database object.

#######################################################

But if I'm doing system testing, I'd rather talk to a real database. And I don't want to rely on the fact that CHRIS MCMAHON will always be in that database. Furthermore, I want my tests to do some of the work for me, and go exploring on their own, to be dynamic. So let's make system tests in Ruby that do system-test things instead of unit-test things. This test will generate its own expected results, and we can loop as many times as want, generating new expected results as long as the computer keeps running...

First I want to to be able to generate some search criteria:

def random_text( length )
output = ""
length.times do
select_array = ('A'..'Z').to_a
select_array << '%'
select_array << '%'
output <<>
end
output
end

And generate some query data for the test:

fstnm = random_text(1)
lstnm = random_text(2)

Then I want to hit the database directly and get some results:

tq = @c.run("SELECT * FROM namestable WHERE fstnm LIKE \'#{fstnm}%\' and lstnm LIKE \'#{lstnm}%\'”)

result = tq.fetch_all

I'll probably manipulate "result" to figure out exactly what I have. Then I want to take the same query and run it through the code I'm testing:

str = <<END_OF_XML
<requestroot xmlns="http://foo">
<lastname xmlns="" >#{lstnm} < /lastname>
<firstname xmlns="">#{fstnm}</firstname>
</requestroot>
END_OF_XML

SEND #{str}

So a test might generate search criteria

lstnm = "JO%"
fstnm = "Q%"

"result" (properly parsed) might then contain

QUENTIN JONES
QXBRT JOHNS
QUIGLEY JOHNSON
QUIMBY X JOVANIVICH

I think it's quite reasonable that my SUT might assume a Q to be followed by a U, and might assume no middle names in the database. Therefore the software might return a set of data for parsing:


<responseroot xmlns="http://foo">
<user xmlns="">QUENTIN JONES</user>
<user xmlns="">QUIGLEY JOHNSON</user>
</responseroot>

Which of course would be a bug, since neither QXBRT nor QUIMBY are in the search results. The test would assert the four users from the database, and fail because the software would have only returned two users.

Seriously, could anyone invent "QXBRT"?

#####################################################################

Thanks to Harry Robinson of Google, who's traveled this path before . When I was starting this sort of work, he was gracious enough to lend his opinion. During the course of the conversation, he dropped this little gem which I've been saving:

To go with a military analogy, static tests are like
battlements: they are fairly cheap to build and maintain,
and they can help keep you from losing ground you have
already won. Generated tests are like tanks or ground
troops: they need more thought in their design, and you need
them if you want to win new territory, but they work best
when they are on the move.