Monday, January 23, 2006

Teaching the Scripting For Testers class

SQE (the stickyminds people) have announced the schedule for the STAREast conference, and I'll be teaching the Scripting for Testers tutorial on May 16, almost certainly with my colleague Dave Hoover.

I'm really looking forward to working with Dave. Not only is he a better programmer than me, but we share an interest in teaching, and in working with (and in being) beginners on the way to being experts.

Brian Marick
and Bret Pettichord started the class in 2003, and this tutorial was a major contributor to the existence of the Watir test framework, to which I've contributed a bit of work. This tutorial will be the first since 2004 not taught by Bret (and the first ever not taught by one of the two of them, as far as any of us know).

It's a good class, and some great people have been a part of teaching it in the past. I'm hoping that the class will take on a life of its own from now on. Enough people have taught it, and enough people have taken it, that I think the course has enough momentum by itself to survive without needing sponsorship and evangelism. In fact, I'm hoping that it will evolve into a "scripting jam session" under certain circumstances-- watch this space...

All of the course materials are open source, so anyone is free to teach this class anywhere they like. Unfortunately, the materials are not exactly self-explanatory, so it helps to have been associated with a class once if you intend to teach it later.

So if you're coming to STAREast, consider signing up for the Scripting for Testers tutorial. We intend to provide thrills, chills, and spills for the discerning beginner.

Friday, January 13, 2006

Static tests and dynamic tests

A typical test expects a particular set of data to exist, right?

Say I have some software that accepts a chunk of XML and does a search for people in a database by last name and first name. It might accept

<requestroot xmlns="http://foo">
<lastname xmlns="" > MCMAHON < /lastname>
<firstname xmlns="">CHRIS</firstname>
</requestroot>


and it might return

<responseroot xmlns="http://foo">
<user xmlns="">CHRIS MCMAHON</user>
</responseroot>

so I could write a test that does something like


SEND

<requestroot xmlns="http://foo">
<lastname xmlns="">MCMAHON</lastname>
<firstname xmlns="">CHRIS</firstname>
</requestroot>

RECEIVE AND PARSE

<responseroot xmlns="http://foo">
<user xmlns="">CHRIS MCMAHON</user>
</responseroot>

That is a static test, and that's how unit tests work. More than likely, the returned value CHRIS MCMAHON isn't in a database at all; it's returned by a mock database object.

#######################################################

But if I'm doing system testing, I'd rather talk to a real database. And I don't want to rely on the fact that CHRIS MCMAHON will always be in that database. Furthermore, I want my tests to do some of the work for me, and go exploring on their own, to be dynamic. So let's make system tests in Ruby that do system-test things instead of unit-test things. This test will generate its own expected results, and we can loop as many times as want, generating new expected results as long as the computer keeps running...

First I want to to be able to generate some search criteria:

def random_text( length )
output = ""
length.times do
select_array = ('A'..'Z').to_a
select_array << '%'
select_array << '%'
output <<>
end
output
end

And generate some query data for the test:

fstnm = random_text(1)
lstnm = random_text(2)

Then I want to hit the database directly and get some results:

tq = @c.run("SELECT * FROM namestable WHERE fstnm LIKE \'#{fstnm}%\' and lstnm LIKE \'#{lstnm}%\'”)

result = tq.fetch_all

I'll probably manipulate "result" to figure out exactly what I have. Then I want to take the same query and run it through the code I'm testing:

str = <<
END_OF_XML
<requestroot xmlns="http://foo">
<lastname xmlns="" >#{lstnm} < /lastname>
<firstname xmlns="">#{fstnm}</firstname>
</requestroot>
END_OF_XML

SEND #{str}

So a test might generate search criteria

lstnm = "JO%"
fstnm = "Q%"

"result" (properly parsed) might then contain

QUENTIN JONES
QXBRT JOHNS
QUIGLEY JOHNSON
QUIMBY X JOVANIVICH

I think it's quite reasonable that my SUT might assume a Q to be followed by a U, and might assume no middle names in the database. Therefore the software might return a set of data for parsing:


<responseroot xmlns="http://foo">
<user xmlns="">QUENTIN JONES</user>
<user xmlns="">QUIGLEY JOHNSON</user>
</responseroot>

Which of course would be a bug, since neither QXBRT nor QUIMBY are in the search results. The test would assert the four users from the database, and fail because the software would have only returned two users.

Seriously, could anyone invent "QXBRT"?

#####################################################################

Thanks to Harry Robinson of Google, who's traveled this path before . When I was starting this sort of work, he was gracious enough to lend his opinion. During the course of the conversation, he dropped this little gem which I've been saving:

To go with a military analogy, static tests are like
battlements: they are fairly cheap to build and maintain,
and they can help keep you from losing ground you have
already won. Generated tests are like tanks or ground
troops: they need more thought in their design, and you need
them if you want to win new territory, but they work best
when they are on the move.

Thursday, January 05, 2006

A software tester should always have the means and opportunity to read the source code for the software being tested. I've had a number of jobs as a tester, and on some them, getting to read the source code was quite an ordeal.

I'm currently working on an integration project. We had a new Business Analyst join the team, and on his first week, one of the developers gave him a guided tour of the most relevant parts of the source code for one of the applications we are integrating. I tagged along. I had surfed some of this code, but I am not a Java expert by any means, and having the design and implementation explained by one of the developers that wrote the code was a great help in understanding how and why to test the integration of that application.

Imagine that. A BA and a tester learning about the code from a happy developer. I like my job.