Saturday, July 18, 2009

a year of STPedia

This month marks one year of the column Matt Heusser and I co-write every month for Software Test and Performance magazine, "STPedia". It was great to mark our anniversary in a double-length column on "Agile Testing in Practice" in July's newly redesigned print version of ST&P and to help out with the launch of ST&P's redesigned portal, stpcollaborative. ST&P are changing things around for the better.

The format of the column is that we have a paragraph or two introducing the topic for the month, then we define dictionary-style a set of terms related to the topic, then usually we draw some sort of conclusion from how the terms are related. ST&P has a 6-month editorial calendar, so at this point we've treated many subjects twice. And really, how much is there to say about source control? From here on out we're going to range a little more widely in our choice of topic.

A little Inside Baseball, for those who are interested in how the writing gets done: we have no set process. Some of the time one of us will write the whole draft for the other to edit. For instance, September's column is mostly Matt, and October (at this point at least) is mostly me. Other times one of us will start the column, run out of steam, and the other will finish it. I remember one column in particular where I just could not come up with a conclusion that made sense. Matt rewrote the last half so that it all made sense. Sometimes one of us will write some of the definitions, and the other will handle what's left over. Matt has a wider range of expertise than me, so he likely wrote the really technical stuff. If you read the column month to month, you probably get about 50% of each of us over time.

The fact that we've been at this a year with no end in sight speaks to our mutual respect. Matt is a bright, honest, motivated, dedicated guy. And fun to work with. I think we still have a lot to say, I hope you'll consider reading our stuff.

Saturday, July 11, 2009

against kanban

I've been somewhat alarmed by the enthusiasm with which the agile community has embraced Lean manufacturing principles, especially kanban. While I'm sure that kanban is effective in some circumstances, I am also convinced that it exposes software teams to real risk of impaired function.

Like many of my qa/tester colleagues who were working in the field in the 90s, I used to be a real process-head. I was part of projects that enthusiastically implemented Six Sigma, CMM, ISO9000, as well as other approaches like RUP, spiral development, etc. etc. We thought then that if only we could get the process right, we could crank out great software just like factories crank out cars and engineers crank out bridges.

Six Sigma and ISO9000 were lifted straight from manufacturing. CMM was the product of a lot of analysts who looked at some properties of a selected set of software projects.

Then the Agile movement came along, and with it new processes, the most significant of which turned out to be XP and Scrum together. These were revolutionary, because they were software development processes actually created from scratch by people who were very very good at creating software. Obviously, because these processes were created by smart people creating software, these processes map very well to how we do excellent software development.

We can argue about the details: is it better to have stable velocity or increasing velocity? should we estimate using points or using ideal hours? Regardless of these details, if your team measures velocity and uses velocity to plan iterations, and if your team always does pair programming, test-driven development, and continuous integration, then you will either release excellent software every iteration; or you will quickly expose and face the reasons why not.

Kanban doesn't make it necessary to face your inadequacies. I have two examples:

At one point my dev team had a chance to interview a self-appointed "kanban expert" on the subject of kanban and Lean. At this point the team had successfully shipped something like 25 of the last 27 iterations, so releasing iterations to production was routine, we had the whole process pretty well dialed in. And we had some killer questions for the kanban expert.

Under this questioning, it turned out that the kanban expert's team had been using a Scrum/XP process, but they blew iteration after iteration after iteration. From his answers to our questions, it seemed that his team lacked the skill and discipline to be consistently successful iteration after iteration. So one iteration, when they knew they'd blown it one more time, they simply stopped doing iterations or planning to release features. This is apparently when this person attached "kanban expert" to his resume.

Before we started releasing every iteration, my team had been using a kanban-like process. We would identify a set of features, then branch our trunk code for each feature under development. When a feature was finished, we would merge the feature branch back into the trunk code. This way we always had features being developed, and we could release trunk safely at any time.

We had a lot of very bright developers on the team. Some of them were maybe a little too bright. With no timebox pressure, we had people doing a lot of gold-plating, making architecture changes that had no relation to the feature being developed, making (cool!) experiments in the underlying code, and generally making merging not very fun.

Furthermore, it was an enormous headache for the marketing people. Without timeboxes or estimates, it was difficult to know when a particular feature would be available, which made it almost impossible to do effective marketing for the product. When we switched to planning iterations based on our velocity and our backlog, we made the marketing people very, very happy.

The fundamental difference between software development and manufacturing or engineering is that in software development we do not manipulate physical objects. We manipulate concepts, and algorithms, and process. Processes designed to optimize the handling of physical objects simply do not map to the work we do when we create software. I'm sympathetic to those who see estimation and iteration planning as waste, but I think the risk inherent in not having a mechanism to expose our inadequacies is too much risk to take, even for the best of teams. We have no warehouses to fill up or assembly lines to halt, we have only the contents of our heads and of our code repository. For software development, the only truly useful metric is: can you deliver running tested features on a consistent schedule? Iteration timeboxing exposes that metric in the most obvious way possible.

What we find in both literature and anecdote about the application of processes taken from manufacturing and engineering, puzzlingly, is that for every team for which the process is successful and helpful, there are other teams who implement the same process only to meet with failure.

What I suspect is that ANY process or methodology (that is not actively destructive), applied to a skilled, disciplined, high-functioning, motivated team, will succeed, regardless of the process. Likewise, any process applied to a low-functioning team will likely fail.

With an XP/Scrum sort of iterative process, we will know very soon exactly how and why we are failing. This seems to be not true of kanban or of any other software development process taken from the manufacturing arena.

Tuesday, July 07, 2009

validate order with regular expressions

Today I wrote some UI tests to validate that certain text appears on pages in a particular order. Since I've done this now in two different jobs using two different tools, I thought it might be of interest to others.

There are a couple of reasons you might want to do this. For example, a page might have a list of records sorted by certain aspects of the record. Here's a crude example:

|name |date|description|
|chris|july|tester |
|tracy|june|gardener |

One set of critical tests validates that name/date/description appear in the right order; that chris/july/tester appears in the right order; and that tracy/june/gardener appears in the right order.

Another critical test is that chris appears above tracy in the display.

The first challenge for the tester is to identify and be able to address the smallest part of the page that contains the text under consideration. Every tool does this in a slightly different way, but Firebug is your friend here.

The next challenge is to make sure that your particular framework knows how to interpret regular expressions. I have seen this in place in Selenium-RC and SWAT, and I have rolled my own in Watir. (But for all I know Watir might have this kind of thing automatically by now. It's been some time since I've used Watir on a real project.)

Now, to check the horizontal order, you construct a test that looks something like

|assert_text|id=text_container|chris.+july.+tester |
|assert_text|id=text_container|tracy.+june.+gardener |

To test that chris appears before tracy on the page:


Regular expressions are for matching text. The first set of tests is fairly straightforward. The "." operator says "match any character". The "+" operator says "match 1 or more instances of whatever". This means that text like "namedate" would cause the test to fail (which is exactly what we want) but text like "name foo date" would pass (which might not be what we want but it's still a decent test, we could write a fancier regex if that happened to be critical).

The second test is a little trickier. The trouble is that by default "." does not match any newline characters. And since chris has to appear above tracy in the page, there will always be a newline involved. Most if not all languages have a way to tell the "." operator to match a newline, but some of them are really awkward. (I'm looking at you, .NET.)

So instead we make a little workaround: the \W means "match anything that's not a word" and "\w" means "match anything that is a word". "Word" is defined as any letters plus any numbers plus "_", but it doesn't matter. [\d\D] (for digits) would have worked as well, since the point is that one of those expressions will always match anything we encounter between one interesting string ("chris") and the other interesting string ("tracy").