Chris McMahon's Blog

Weekend Testing - Wikipedia Test Event Apr 6

2013-04-02T12:07:00.001-06:00

Wikipedia is improving the new user experience.

Wikipedia is developing a new way to create accounts on Wikipedia and a new experience for new users when logging in.

Weekend Testing Americas has an online test event on the first Saturday of every month 10AM-1PM Pacific time (17:00-20:00 UTC).

On April 6 WTA will be testing account creation, login, and the new user experience for Wikipedia . Justin Rohrman will facilitate.

If you'd like to join this Saturday:

Send a message on Skype to weekendtestersamericas
Join #wikimedia-dev on freenode on IRC
Read over the Test Plan
Test!

(If you can't attend the Weekend Testing session but you are interested in QA for Wikipedia, we have public QA events ranging from bug triage to Cucumber development every week.)

Who I am and where I am, March 2013

2013-03-26T18:22:00.000-06:00

I am the QA Lead for the Wikimedia Foundation I test Wikipedia. My one year anniversary was in January. I do exploratory testing and I do browser test automation with my colleague Željko Filipin. I have a strong interest right now in community testing of all kinds.

I'll be talking about testing Wikipedia at some conferences soon:

GTAC April 23-24, NYC
Telerik Test Summit May 3-4, Austin TX
Wikimedia Hackathon May 24-26, Amsterdam
Agile2013 (pending acceptance) Aug 4-10, Nashville TN

I used to write about software a lot: SearchSoftwareQuality.com (warning: registration wall), StickyMinds.com and a couple of articles for PragPub. I wrote a chapter for Beautiful Testing.

For many years I have been telecommuting from Durango CO where I created and hosted the Writing About Testing peer conference. WAT is unlikely to happen again.

I am on my way out of Durango, but I will continue to telecommute. In the coming months you might also find me near:

Santa Barbara CA
Tucson AZ
San Francisco CA
Southern Utah
Ithaca NY
Boulder CO
Dulce NM

I'm @chris_mcmahon on Twitter, christopher.mcmahon on all the Google services, chrs_mcmhn on Skype, although I rarely use Skype. I am on a lot of Wikimedia IRC channels every day. I am not on LinkedIn or Facebook.

A community test event: mojibake and meaning

2013-01-26T13:11:00.000-07:00

UPDATE 28 January the official announcement on the WMF blog.

The Wikimedia Foundation is developing new editing software for Wikipedia. For the first time in history we intend to have a native rich text WYSIWYG editor for the Mediawiki engine that does not require users to know wiki markup syntax in order to create and edit articles on Wikipedia.

Right now the Foundation needs help testing this editor, and we invite you to participate.

We have a suite of automated tests for the editor that is capable of checking the contents of existing articles, loading them into VisualEditor, saving the article, and checking that no information has been lost in the round-trip saving operation. We are confident that the editor is robust enough to load and save any given content on the English Wikipedia.

What is not clear is whether VisualEditor is robust enough to preserve information after that information has been altered: additions, changes, deletions, other manipulations of the content of Wikipedia pages may or may not cause unintentional alterations to the contents of pages upon saving. We just don't know.

Of particular interest is manipulation involving non-Latin characters. Various language scripts and other aspects of the greater Unicode character set should be handled correctly by the VisualEditor, and we want to find out if that is true.

This sort of testing of course is best done by real human beings. Ultimately this is a test not only that VisualEditor preserves altered text correctly, but that VisualEditor preserves *meaning* correctly.

This testing should be of interest not only to those who will eventually want to edit Wikipedia in their native language, but also to those interested in accessibility, internationalization and localization, maintainability and scalability as well.

For the week starting 28 January 2013, the Foundation is providing extraordinary support for people testing non-Latin character handling in VisualEditor. We will be available on IRC, on mail lists, on social media, etc. fielding questions and facilitating discussion of the behavior of VisualEditor handling of non-Latin characters. It would be a fine thing if you could help us test VisualEditor.

We have created a test plan for the VisualEditor test event.

We plan to have more community test events in the future every month for Wikimedia Foundation development projects. Coming up are sessions for "Echo", our new notification system for editors of Wikipedia, improvements to our Article Feedback system for critiquing Wikipedia articles, and more. If you are interested in doing exploratory testing for Wikipedia, we encourage you to join our new Features Testing Group, the central repository for information about these ongoing testing exercises.

Open browser test automation at WMF

2012-12-16T13:47:00.000-07:00

Almost a year ago I started working as QA Lead for the Wikimedia Foundation. Among other things, I had a mandate to create an automated browser testing practice at WMF.

From the start I wanted this project to be a world class, completely open, reference implementation of such a project, using the best and most modern tools and practices I could find. I wanted this to be a project that anyone could read, anyone could run, and to which anyone could contribute. I wanted this to be an industry standard implementation of a well-designed, well-implemented, working browser test automation project.

Around 2006 my career had veered off into a test automation approach that, while valid and useful in certain circumstances, would be inappropriate for the WMF project. And in the years since 2006, the tools and practices that were immature at the time had grown into mature, stable, powerful projects of their own.

I set out to educate myself about the details of the cutting edge of browser test automation in every language. I visited Austin TX twice and San Francisco several times in the past year to discuss approaches to the project in person with experts on the subject.

WMF hired Željko Filipin in October 2012 specifically for the browser test automation project, and just this week we opened the curtain and turned on the lights. The WMF browser test automation project is in Ruby, using

* page_object gem
* watir-webdriver
* selenium-webdriver
* Cucumber
* RSpec
* rake
* Jenkins integration

The initial announcement is here.
A first pass at technical documentation is here, including instructions to run the tests locally if you want to see them in action on your own machine.
Some community concerns are here.
The main code base is managed in gerrit but there is a more accessible read-only mirror on github.
Our Jenkins instance is running on the Cloudbees service using Sauce Labs hosts, and the current test results are visible here.
If you would like to follow the project or contribute to it yourself, we are starting a community group you can join. Feel free to add yourself to the page here to follow the project or contribute to it.

Many people helped me along the way to making this project what it is. Hopefully I haven't left out anyone, feel free to remind me if I did!

Jeff Morgan for creating the page-object gem and for answering tons of my questions
Jari Bakken, especially for this presentation about webdriver in Ruby.
Brahma Ghosh
Charley Baker
Alister Scott and his Watirmelon blog
Marlena Compton and Matt Brandt for discussing their experience at Mozilla WebQA with me.
Bret Pettichord for hosting the Test Automation Bazaar
Jim Holmes of Telerik for hosting the Telerik Test Summit

And especially Željko for actually building the actual project! (And for kicking my ass along the way, I still have a lot to learn.)

Wikimedia Foundation hiring QA staff

2012-06-01T09:43:00.001-06:00

The Wikimedia Foundation currently has two open positions for QA staff. One is for a QA Engineer and the other for a Volunteer QA Coordinator.

I want to point out what a unique opportunity this is.

Before I was hired at WMF four months ago as "QA Lead", there had never been anyone working on the Wikipedias and related projects whose only focus was testing and quality. There was no UI test automation, there was no program of community testing.   Actually, there still isn't. That's where these new staff come in. No exaggeration: this is an opportunity to create from scratch the quality and testing practices for one of the great achievements in human history. But WMF only has about 100 staff, and only about half of those are technical. At the moment, there are about 50 regular contributors to WMF software (and millions and millions of users!), so this QA staff will be outnumbered and outgunned. So we need help, both from automation and from a community of people interested in testing.

WMF has two priorities this year for QA and testing: one is to implement some browser-level test automation. The other is to involve both the testing community and the Wikipedia community in more testing activities.

As QA Lead, I figured it was my job to lay some groundwork for these activities. So I did a lot of research on the current state of browser automation (I'd not been following recent developments there for some time) and built a demonstration and example of what I think the best available browser test "stack" should be.   (Note: there is some bad code in there right now, for a purpose) I think Ruby is the way to go for this, and this is my justification for that. I am really looking forward to working with our new QA Engineer to expand these tests, bring in community contributions, hook it up to Jenkins, run it against our brand new test environment, etc. etc. My little spike is only an example, and I look forward to having my mind changed about how it will ultimately be of use.

The other thing I've done is to begin having "test events" where WMF invites outside groups to help test some aspect of the Wikipedia software. On May 5 2012 we teamed up with Weekend Testing to validate the new frequent deployment scheme when we began updating all the Wikipedia software every two weeks. On June 9 we'll be teaming up with Openhatch.org for a "shakedown cruise" of the new Wikipedia Article Feedback system that will be rolled out to all of Wikipedia in just a few weeks.   I'm looking forward to working with the new Volunteer QA Coordinator to expand community testing both in partnership with other organizations and within the Wikipedia community itself.

As for me, to the extent that I am a "Lead", I tend to lead from behind. I'll be working with new and evolving WMF projects to bring testing and quality work where it can be the most valuable. I'll be working with the test automation communities to improve development practices through testing. I'll be looking for cool stuff out in the world for Wikipedia to make use of. It's a brilliant future.

Join the Wikipedia/OpenHatch.org test event 9 June

2012-05-29T11:52:00.002-06:00

Wikipedia allows users to leave feedback on each article. Experienced Wikipedians analyze this feedback in myriad different ways to improve the Wikipedia user experience and to improve the encyclopedia itself.

The Wikimedia Foundation has been creating a new Article Feedback system, and on Saturday 9 June from 10AM-noon Pacific time, WMF invites testers and anyone else interested to participate in a "shakedown cruise" to test a near-final version of the new Article Feedback system before the system is rolled out to all of Wikipedia.

Following on May's successful test event with Weekend Testing Americas, WMF is teaming up with the Open Source fans at OpenHatch.org for this event. I am hoping that having experienced exploratory testers plus people interested in improving Wikipedia articles will be a killer combination of expertise and interest to shake out any final issues in this critical aspect of the Wikipedia experience.

Note: anyone who shows up on 9 June will have access to some of the Article Feedback features, but if you create an account on Wikipedia no later than 4 June and also make at least ten edits to Wikipedia, your account will be "autoconfirmed" and you will have access to many more Article Feedback features than a casual user would. (Feel free to edit for instance the Wikipedia article on your home town, or an article on software testing, or dice games, or if you're shy, just update your User page and your Talk page a few times, those edits count toward the ten also.)

This is the announcement and there is an optional sign up sheet. The Article Feedback Test Plan has all the details

Testing (automation or whatever) 101: ask a good question.

2012-05-14T20:40:00.002-06:00

I tried to do A, and I really don't understand the response I got, X. Does this make sense?

I know it should be possible to do A, but I tried it and X happened. What sort of conditions would cause that?

I tried to do A, and X, Y, and Z happened. X makes sense, but I don't understand Y, what's going on here?

It doesn't really matter whether you're asking about automation or any other kind of testing. The tricky part is that before asking the question, you had better be pretty familiar with A, and you had better be able to report X, Y, and Z in a reasonable way.

I have a corollary, and I have a (counter) example.

I have seen any number of people in the software world complain about testers who submit bad bug reports. I'm sure it's true, I've seen the evidence, and it boggles my mind. A good bug report will explain A and explain X, and a great bug report will phrase the issue in terms of a question.

Not long ago I got an email from someone asking about a little script I wrote some time ago. He asked me to give it to him. I have not replied. I was astonished. For one thing, a cursory google search would turn up the 30 lines of code in question. But even worse than that: why don't you WRITE IT YOURSELF?

It's quite possible my script no longer works. It's quite possible that there are better ways to accomplish what the script does than what I wrote. But I absolutely refuse to copy'n'paste 30 lines of code in an email response.

Eric Raymond (if you don't know that name, google it) wrote an essay a long time ago How To Ask Questions The Smart Way. I'm guessing that many readers of my blog are not familiar with it. This is a travesty.

NB: the last time I pointed a software tester to Raymond's essay, I was accused of misanthropy and worse. Testing might be dead.

Testing Summit at Telerik

2012-05-06T21:01:00.001-06:00

I attended the Test Summit peer conference this weekend at the invitation of Jim Holmes of Telerik.

It was outstanding, as such peer conferences tend to be, and I and others will be posting a lot of information as a result of what went on there.

But I want to talk about the conference itself.

Software testing has a long history of peer conferences, starting (to the best of my knowledge) with the Los Altos Workshop on Software Testing (LAWST). Bret Pettichord borrowed the LAWST format for his Austin Workshop on Test Automation (AWTA) in the mid-2000s, and I borrowed from AWST for my Writing About Testing (WAT) conferences in 2010 and 2011.   I think other examples exist. The format has gotten looser over the years. LOTS looser, as we find that motivated participants are pretty good at self-organization.

As far as I am aware, the Telerik Test Summit (TTS?) is the first such software testing peer conference created and sponsored by a commercial company.   I think this is important, and I think TTS has important implications.

As a general rule, I will not endorse or support commercial companies, even the ones I work for. I don't even link from blog posts to my own writing for articles that are behind a registration-wall or pay-wall. Now that I work for the Wikimedia Foundation, I have even more motivation to be as impartial as I possibly can.

And yet I accepted the invitation from Telerik and Jim easily. I'd like to explain that.

I knew everyone in the room at TTS, and many of the attendees are good friends. I had never met Jim Holmes.

But I knew his work. Jim created the CodeMash conference, which is notable not only because of its radical growth in popularity over the last several years, but also because of its reputation for friendliness and hospitality.   I'm not involved in CodeMash and I've never attended it, but I know a lot of people who have, and Jim has a stellar reputation among my friends and acquaintances.

I didn't go to TTS because Telerik invited me; I went because Jim Holmes has an unimpeachable reputation for integrity and hospitality.

We were not pitched. We were not marketed to. There were no Telerik tools being discussed in the room. We were given a tour of the Telerik office in Austin; some attendees chose to pair up with Telerik staff to see what they were working on, some did not.   I consider this just hospitality.

TTS should be a lesson to other companies. Many of the best people working in software will give away their best ideas and even their best work freely. But the moment such people suspect they are being used or co-opted for some purpose other than trying to move the cause forward, we will turn on you with a vengeance you can not imagine.

So my advice to any companies who might want to sponsor their own peer conferences in the future: first, get yourself a Jim Holmes; second, listen to what he has to say; then, do what he tells you. That's probably harder than you imagine.

Weekend Testing for Wikipedia May 5

2012-05-01T12:32:00.000-06:00

On Saturday May 5 at 10:00 AM Pacific Time the Weekend Testing Americas will be investigating the new release of Mediawiki on Wikimedia Foundation sites before WMF rolls out the new version to all of Wikipedia.

I am really excited about this project, I do hope you will consider joining in.

Details of how to join are on the official Weekend Testing site.
1. Add “weekendtestersamericas” to your Skype contacts if you haven’t already. 
2. Fifteen minutes prior to the start of the session, please message “weekendtestingamericas” and ask to be added to the chat session. Once we see you, we will add you to the session.

The test plan is here

See you on Saturday!

conf report: Test Automation Bazaar

2012-03-26T19:55:00.001-06:00

I went to the Test Automation Bazaar because one of the (many) things I want to do at Wikimedia Foundation is to start a browser test automation project, open to the greater software testing community, in support of Wikipedia and related projects. I have been out of the loop on this front (forgive the mixed metaphor) for some time, and I particularly wanted to learn about two things:

* what an attractive, modern, well-designed browser test automation framework looks like
* page objects and their use

I got what I came for. Particular thanks to Brahma Ghosh and Jeff Morgan/Cheezy for the great discussion. I am putting on the white belt for this project, but at the same time, I have years of valuable UI test design experience to bring to bear.

But beyond that, TAB was full of friends, many of whom I had never met.

Kudos to Bret Pettichord for making it happen, I wish we'd had more time to chat, I've known Bret longer than anyone else there, through some weird times back in the day. Tiffany Fodor, whom I met once about three years ago, Rick Hower likewise. Željko Filipin, Alister Scott, Marek Jastrzebski, whom I've known for years and never met in person. And the amazing Jari Bakken, who I met for the first time in San Francisco a few weeks ago. Charley Baker was sorely missed by many.

The kindness and generosity of people in the Ruby community is just remarkable.

And there was ukulele playing. It was my fault.

deja vu: code, culture, and QA

2012-02-26T13:17:00.000-07:00

Some years ago I had the privilege of making some suggestions for Brian Marick's book Everyday Scripting based on the first article I ever wrote for Better Software magazine. That article appeared in 2004, and I just recently ran into a similar situation at work.

Wikipedia is localized for well over 100 languages. I had only been working at Wikimedia Foundation a couple of weeks when I heard that discrepancies between the localized message files from version to version could cause problems when upgrading. I didn't know what kind of problems, but since we're upgrading all the Wikipedia wikis to version 1.19, that sounded like sort of a big deal, so I followed up.

It turns out that changes to the localization files are essentially undocumented, no tools exist to monitor such changes, and we simply did not know anything about discrepancies in those files. So I decided it would be useful to look into that.

You can find the Wikipedia localization files for version 1.19 here and for version 1.18 here if you want to follow along.

Since there are well over 100 files in each directory and each file has 1000s of lines, checking for discrepancies manually is impossible. From one of the senior people on the Wikimedia dev staff I got a few examples of certain places in these files where discrepancies would cause big problems. (See technical note at the end.) Although I've cleaned the code up quite a bit (one-off scripts don't have to be DRY, right?) here's what I did to cite discrepancies for one of the examples:

In a directory called 'mediawiki' I have one directory 'lang118' and another 'lang119'. In those directories are all of the Messages*.php files for each version. What I want to do is read each file in each version, identify the contents of the $namespaceNames array, and compare those contents for every file in each directory.

path119 = 'mediawiki/lang119/'
path118 = 'mediawiki/lang118/'

r119namespaceNames_array = []
r118namespaceNames_array = []

def get_values ( path, array_name  ) 
  Dir.foreach(path) do |name|
  unless File.directory?("#{path}#{name}")
    text = File.read("#{path}#{name}")
    text.scan(/namespaceNames.+?\)/m)
    array_name << name + $~.to_s
         end #unless
  end #do
end

get_values(path119, r119namespaceNames_array)
get_values(path118, r118namespaceNames_array)

mismatch = r119namespaceNames_array - r118namespaceNames_array
disc= mismatch.length.to_s
puts "number of files with discrepancies in $namespaceNames array is #{disc}"

mismatch.each do |string|
  file = string.split(".php")
  puts file[0]
end

This script runs from the directory above 'mediawiki'. It defines the paths to where the localization files live, and defines two arrays to hold the values to be compared. For each directory it calls the 'get_values' method, and puts the name of the file and the contents of the $namespaceNames array of that file into the appropriate array. Subtracting one array from the other yields a set of all mismatches, and with that the script knows how many files have mismatches, and what the names of those files are.

Reading this script should be fairly straightforward for anyone who knows a little bit of Ruby. Note a few things, though:

* 'unless' is equivalent to 'if not', and the script needs to not check directories, only files
* File.read is the same as Perl's "slurp", it puts the entire contents of the file into the variable 'text'
* the 'scan' method takes a regular expression for an argument. Here the regular expression is saying "give me all the text that begins with the string 'namespaceNames' and ends with the string ')'. I had forgotten that '.+' is 'greedy', and will match past the terminating string, so doing '.+?' prevents that, thanks Charley Baker for the reminder. The 'm' at the end of the regex tells it to match multiple lines, which is necessary because each value of the $namespaceNames array is on a single line and I want to match all of them in one fell swoop.

The output from this script looks like

number of files with discrepancies in $namespaceNames array is 16
MessagesEn_ca
MessagesEn_rtl
MessagesFrp
MessagesIg
MessagesMk
MessagesMzn
MessagesNb
MessagesNds_nl
MessagesNo
MessagesOr
MessagesOs
MessagesQug
MessagesSa
MessagesSr_ec
MessagesWar
MessagesYue

At this point it made sense to just look at the problem files with my eyeballs and see what was in their $namespaceNames arrays. With a little help from diff(), that's what I did. I reported the discrepancies I found on a public mail list for Wikimedia tech issues.

A couple of interesting things happened because of that. Again, keep in mind that I am a total n00b with these systems. While I have a little more information now, I had no idea of what the consequences of such discrepancies would be.

I got an answer on the mail list from a senior Wikimedia dev person who analyzed the discrepancies I reported and said in effect "everything's fine, we are good to upgrade based on these examples". And while there are several other areas in these localization files that could cause issues, my example demonstrates that the technical risk for upgrading to 1.19 seems low.

But then some days later in a a conversation on IRC, a different senior Wikimedia dev person said in effect "whoa, whoa, whoa, if we release these changes without at least some review from the language communities affected, we are going to be in for big trouble".

As I write this I do not know if the localization files for Wikipedia will be upgraded next week or not; that decision is not in my hands. However, I am immensely pleased that as a total n00b I was able to provide true concrete examples of the data in question to inform that decision.

I decided to write about this for a number of reasons:

To my mind, nothing in this story has anything to do with "testing". For some time now I have been saying that "QA is not evil", and to me, this was an exercise in pure Software Quality Assurance. Since my official title at Wikimedia is "QA Lead", this makes me happier than you would imagine.

One of the great neglected areas of software projects is the state of the actual data in applications, be it held in files or databases or whatever. One of the most important skills QA/testing people can bring to bear on a software project is the ability to isolate critical chunks of data from enormous data stores. That was true when I wrote "Is Your Haystack Missing a Needle" in 2004, it was true when Brian published "Everyday Scripting" in 2007 and it remains true today. If as a QA/testing person you don't know how to read a bunch of files and do regular expressions (and for that matter do SQL queries too), you owe it to yourself and to your projects to learn. (Frankly, I hadn't done this kind of thing in a long, long time, and it felt great to get back on that horse.)

Finally, I wrote this because all of the data and all of the conversations we had were completely open and public. I could give you a link to the email thread where I published the detailed discrepancies and got the reply, I could publish a link to the IRC log where people discussed the cultural risks of upgrading the localization files. The only reason I don't is because they're not germane to the story. I so enjoy working in an open culture.

Technical notes:

My original script checked for discrepancies among four arrays: $namespaceNames, $namespaceAliases, $magicWords, and $specialPageAliases. The $magicWords array was trickier, and I had to do this:

text = File.read("#{@@path118}#{name}")
text.scan(/magicWords.+?\);/m)
if $~.to_s.length > 0
array = $~.to_s
array_no_space = array.gsub(/\s+/,"")
@@nsn118magicWords_array << name + array_no_space

For one thing, $magicWords is an array-of-arrays, so I check for a terminating string of ');' instead of just ')'. For another thing, some of the files didn't contain the $magicWords array. For another thing, I found some random differences in whitespace between versions for many many files, so I eliminated all the whitespace in the strings in question by doing 'array.gsub(/\s+/,"")'. The comparison only became valid once those things happened.

Who I Am and Where I Am, early 2012

2012-02-04T13:57:00.000-07:00

I've been pretty quiet in recent times, but that's going to change somewhat in 2012, so I thought I'd write this(*) to catch up.

As of last week, I am the QA Lead for the Wikimedia Foundation. My job there will be to create, codify, and execute the software testing and quality assurance regimes for the software that powers Wikipedia and its associated properties.

I've worked some other interesting places, among them Thoughtworks and Socialtext. I like open source and wikis. I have been a dedicated telecommuter/remote worker since 2006. Depending on when you read this, I'm in either Santa Fe NM or Durango CO, or somewhere else.

I have written about software a lot. Most of my writing in recent times has been for SearchSoftwareQuality.com (warning: registration wall), but I've also written a lot for StickyMinds.com and a couple of articles for PragPub. I wrote a chapter for Beautiful Testing.

I created the Writing About Testing peer conference and associated mail list and wiki. I like to think WAT has had some influence on the testing/QA field over the last couple of years. I've given presentations at a couple of Agile conferences, once at PNSQC, a couple of AWTAs, and some smaller peer conferences from time to time. I attended two GTACs.

I've been using browser automation tools (Selenium and Watir) since they existed. I know a lot about their history and something about how to use them well. As a programmer I am slow and simple. What programming I do is usually in Ruby.

I play fretless electric bass guitar with a couple of jazz bands, but outside of the WAT conference I'm more well known for playing irreverent tunes on a cheap green ukulele at software conferences. I'm a pretty good musician.

I'm @chris_mcmahon on Twitter, christopher.mcmahon at gmail and gplus. I ignore LinkedIn and I don't have a Facebook page, don't try to reach me there.

* props to Warren Ellis, from whom I stole the idea of this post.

Just Fix It

2011-12-04T12:18:00.002-07:00

On the writing-about-testing mail list recently was a discussion of defect tracking. Given a good enough code base and a mature dev team, I think defect tracking is mostly unnecessary, and it's worth talking about why that is.

Some time ago there was a popular meme in the agile testing community that goes "Just Fix It", but I haven't heard it mentioned in some time, and I think it's worth reviving the discussion. The idea behind Just Fix It is to bypass the overhead of creating a defect report, having those defect reports go through some sort of triage process, and only then addressing the problems themselves represented by the defect reports. You save a lot of time and overhead if you Just Fix It.

For some time now I have specialized in testing at the UI, a combination of functional testing and UX work. In my experience, in a good code base, important defects found at the UI level are almost always what I think of as "last mile" issues, where the underlying code has changed in some way but the hook for that code into the UI has been mangled or overlooked. These are cases where unit tests are almost certainly passing, but the app is broken anyway. Some examples:

Explanatory text has disappeared or no longer describes function accurately.
A widget that used to function no longer does. For example, a Submit button no longer makes anything happen.
A call to some underlying function is no longer correct. For example, a Search function that used to return results no longer does.

While a Just Fix It culture is not necessarily agile, examples of Just Fix It are easier to describe in a typical agile situation.

Small Team

Many agile teams share a single space, making communication easy and instantaneous. In such a situation, a conversation like this might happen:

Tester: "Hey, the froobnozzle stopped froobing, anyone know anything about that?"
Dev: "Wow, I didn't realize my last commit would break the froob function, I'll Just Fix It."

This is a canonical example of what Lisa Crispin calls the "whole team approach", where testers, devs, and everyone else is working on the same stories at the same time in the same place simultaneously.

And if it's appropriate, there's no real reason a tester couldn't Just Fix It themselves.

Large Team

But some teams are too large for a conversation like this to be practical. Assume a collocated team with a really big story board with dozens of story cards all being moved around a lot. Say a tester finds an issue with the froobnozzle.

Tester: grabs a red sticky note and writes brief description of froobnozzle problem. Puts sticky note on froobnozzle story card
...minutes later...
Dev: whoa, a red card on my story, better Just Fix It.

Distributed Team

Distributed teams tend to have really sophisticated issue-tracking systems in place, where stories are represented in software of some sort, where they can be assigned, have their status changed, etc. etc. If a distributed team is small enough, a tester will know that Joe is working on the froobnozzle story, so:

Tester to Joe on IM: "hi Joe, I think you might have just broken the froobnozzle."
Joe the Dev: "whoa, good catch, I'll Just Fix It."

Large Distributed Team

On a large distributed team, identifying who might be in a position to Just Fix It can be complicated. One strategy is to read the commit logs upon identifying a defect to see who or what may have caused the problem. Another strategy might be to review all the stories in play to discover who might be working on the froobnozzle this iteration.

But sometimes these sorts of approaches are too complicated or take too much time. One pattern I have seen on several occasions in large distributed teams is to designate a knowledgeable person on the dev staff, or possibly a Scrum Master type, to represent the whole dev team for questions about behavior or function. I have seen this role called the Face, and the Ninja, and the Disturbed.

Tester: "hi Face, I just discovered the that the froobnozzle got broken within the last day or so."
Face: "whoa, let me check that for you"
Face: "good catch, Joe broke that two commits ago, he's Just Fixing It"

Defect Found in Production

A customer probably reported it. The fix is deployed to production within minutes or maybe hours of the report. (Again, a good code base allows this.)

I worry that too often "root cause analysis" is a synonym for "blame". Defects in production are almost certainly a process problem, and the place to address process problems are in retrospectives or similar conversations.

Besides, if your team is releasing so many defects to production that you have to track them, you have bigger problems.

Won't Fix

True story: just this week I was refactoring some Selenium tests and discovered a bug. This was in a part of the application that is not exposed to customers, it is only for internal users employed by my company. The bug was that attempting to enter a duplicate record causes an unhandled exception and the user is presented with an ugly stack trace. This was an old bug, and was not part of the work of the current iteration.

I work on a large distributed team. As I noted above, we have a sophisticated issue-tracking system in place. All of the work we do is documented and tracked in this system. We have no designated defect-tracking system, just a single monolithic sophisticated issue-tracking software application.

Upon finding the bug, I had a conversation with the dev who knows about that part of the code. We agreed that this was a no-harm-no-foul situation, no data corrupted, minimal inconvenience to the user, no customer exposure. We agreed that Just Fixing It right this minute wasn't very important.

So I created a new issue in the issue tracking system and assigned it to the dev who knows about that part of the code. This issue has the same visibility and status as every other issue in the system. My bug report issue will be treated the same as every other issue in the system, included in the backlog, and prioritized to be worked on with every other issue in the backlog.

I don't even really think of that particular issue as a defect. It's just a description of the state of a part of the application, some work that we might choose to do at some point. I'm sure we'll Just Fix It pretty soon.

a selenesse trick

2011-09-30T10:44:00.001-06:00

Selenesse is the mashup between Fitnesse and Selenium I helped work on some time ago. I keep encountering this pattern in my selenesse tests, so I thought I'd share it...

Every once in a while a test needs some sort of persistent quasi-random bit of data. In the example below I'm adding a new unique "location" and then checking that my new location appears selected in a selectbox. This is also a neat trick for testing search, or anywhere else you need to add a bit of data to the system and then retrieve that exact bit of data from somewhere else in the system.

| note | eval javascript inline FTW! |
| type; | location_name | javascript{RN='TestLocation' + Math.floor(Math.random()*9999999999);while (String(RN).length < 14) { RN= RN+'0';}} |
| note | instantiate a variable and assign the new value to it in a single line FTW! |

| $LOCATION= | getValue | location_name |

| note | click a thingie to add the new data to the system |
| click | location_submit_button ||
| waitForTextNotPresent | Adding ||
| note | check that the newly added data appears as the selected entry in a selectbox FTW! |

more UI test design (once more from Alan Page)

2011-09-08T07:12:00.000-06:00

Before it gets lost in history, I want to riff off Alan Page once again, who made some excellent points. But as someone who has been designing and creating GUI (browser) tests for a long time, I'd like to address some of those and also point out some bits of ROI that Alan missed.

Making UI tests not fragile is design work. It requires an understanding of the architecture of the application. In the case of a web app, the UI test designer is really required to understand things like how AJAX works, standards for identifying elements on pages, how the state of any particular page may or may not change, how user-visible messages are managed in the app, etc. etc. Without this sort of deep understanding of software architecture, UI tests are bound to be fragile.

I've said before that UI tests should be a shallow layer that tests only presentation, and that rarely tests logic, math, or any underlying manipulation of data. If tests are designed in this way, then they will be robust and maintainable over the life of the app.

UI test design is a skill. Designing such tests is no harder or easier than any other activity that requires skill and understanding. The tools with which I am familiar provide enough power to create reasonable, robust, maintainable tests.

Finally, I think Alan is missing one aspect of automated UI tests that I find the most valuable of all.

From green-screen mainframe systems to bleeding-edge web applications, in my experience every software system suffers from one particular sort of error that is always extremely difficult to see when testing manually: when something goes missing.

A search that used to return results no longer does. A widget that used to be on the page no longer is. A bit of text critical to the user's work goes missing.

Actions that cause errors are easy to find when testing manually, as are errors of presentation. Elements and functions that used to exist but no longer do are difficult to find: it is not easy for a human being to see the absence of a thing, but such errors stick out like sore thumbs in an automated UI test suite.

In my experience, this is one of the most valuable aspects of automated UI testing, and one of the best reasons to invest in UI test automation. The absence of a thing in the UI is simply not detectable with unit tests or with integration tests. That critical bit of function that doesn't manage to cross the last interface to the UI is only detectable at the UI itself, and automated UI tests are very, very good at detecting errors where something has gone missing.

Automated Test Design (riffing/ripping off Alan Page)

2011-08-30T18:12:00.001-06:00

Alan just posted this: http://angryweasel.com/blog/?p=325

This is the nicest test example I've seen in a long time, and I think it bears a little more analysis.

If I were in charge of the development of this app, I would make automated testing happen on 3 levels.

First, there would have to be some sort of test for generating a single random number. Which seems easy enough, but at this level, you really have to understand exactly what a call to rand() (or whatever) on your particular OS is going to do. I once wrote a script in Perl (praise Allah just a toy, not production code) that returned perfectly random numbers on OSX but returned exactly the same values every time on Windows. You'd better test that your call to rand() really returns random numbers, regardless of the OS it is running on.

At a higher level, you'd want to do exactly what Alan talks about in looping 100,000 times (although 100K seems like overkill to me). This is what I think of as an "integration test". You're going to have some method or possibly even an API call or REST endpoint like "generate_five_random_numbers_and_add_the_results". You're going to want to exercise that wrapper enough to convince yourself that the numbers are right and that the math is right.

For some time now I've been making my living writing automated UI/browser tests along with doing ET. Here's my take on a UI test for Alan's app:

Open the page
Check that text "Total" exists.
Check that the 6 textboxes exist.
Click the Roll button.
Check that the state of the page changes (however that happens).
Check that there are values in each textbox (even this might be too fancy, depending on the app and the test framework).

Also getting fancy, you could check that the 6 textboxes exist in the correct order. (Selenium's "glob:" feature makes this pretty painless.)

Doing math in UI tests isn't very smart. Doing data comparison over multiple runs in UI tests isn't very smart. Do that stuff at lower levels, where you can take advantage of programming power and speed. UI tests are a shallow layer where you simply check that the user has all the stuff they need to get the job done.

where ideas come from

2010-12-20T09:48:00.004-07:00

Today stickyminds published my article with expanded descriptions of the "10 Frontiers for Software Testing" that I suggested as starting points for those interested in attending the second Writing About Testing conference.

Since I announced the CFP for the first WAT conference in October 2009, I have published several dozen articles on software and software testing. (I actually lost count: it is well over thirty but fewer than fifty individual pieces.)

My friend Charley Baker asked me recently where I get the ideas for so many articles.   It is an interesting question, and worth answering:

The most important source of ideas is simply everyday work. As I go about doing my job, it happens fairly often that a situation crops up that I think would be of general interest to the community of software testers and developers. So I write it down and I make it public. Articles about bugs, bug reports, test design, architecture, workflow, telecommuting, frameworks, war stories all come from noticing the details of the everyday work.

Here is the story of the very first software article I ever published: I have been following Brian Marick's work for a long time now. Brian used to be the editor of Better Software magazine, and he would occasionally solicit articles for the magazine on his blog. In March 2004 Brian asked for submissions for a piece along the lines of "add(ing) a scripting language to her manual testing portfolio.". In particular, I recall that Brian wanted an article suitable for beginners with an example of a testing problem that could only be solved by scripting a little program.

I had written book reviews before but I had never published a piece about software. I had just encountered a situation at work that was a perfect example of what Brian wanted. I was working for a company that was switching from shipping whole custom-built servers to shipping installation CDs for COTS hardware. The installation CDs contained more than 4000 files. The switch was a little bumpy, and at one point we very nearly shipped an installation CD missing 4 critical files of the 4000.   I had been teaching myself Perl (so I was a beginner myself), and I wrote a little script in Perl to compare recursively the contents of large directories, so that it would be easy to see if some few files had gone missing. I described what I had done, Brian published it in Better Software, and in one of the highlights of my career as a writer, that article (with me as a character!) became the basis of the first example in Brian's book Everyday Scripting with Ruby. (Get the book: it will make you a better coder, no matter your level of skill.) The article was titled "Is Your Haystack Missing a Needle"

Another source of ideas for software articles comes from having some Very Large Idea that evolves over a long time. At Bret Pettichord's Austin Workshop on Test Automation in 2007, in a moment of inspiration, I gave a five-minute lightning talk demonstrating an example of using the artistic language of critical theory (in particular, New Criticism) to evaluate the quality of a piece of software. The talk got an enthusiastic reaction from the people in the room, mixed with some skepticism as I recall.   It struck me at the time as being an odd idea, but the more I considered it, the more it made sense. I wrote a long paper on the subject and submitted the paper to the CAST 2008 conference, but it was rejected. I published it on my blog, and I still refer to it now and then. My thinking on the subject has matured and expanded since then, so if you'd like to see the latest example, look at PragPub magazine for November of this year. In 2008 I was a lonely voice on the subject. Today I have colleagues, it is nice to see others considering critical theory applied to software as well.

Finally, every once in a while, I manage to do something really unusual, something that will actually change peoples' minds about how they go about their work. In 2006 I was working for Thoughtworks on an EAI project. Our code base had great unit test coverage and integration test coverage, and as the QA guy, I was not finding defects in what we were creating. But we had to interact with a legacy database, and we were often surprised by unusual or corrupt historical data. I made it my business to expose as much of that bad data as I could. I wrote a little Ruby script that would do quasi-random queries in the database, request the same data from the API we were building, and compare the results, running within an infinite loop. I found a significant number of issues in this way, where the API we were building failed to handle data we never expected to find in the database. To my knowledge, no one had ever published anything describing a situation like this.

So I wrote a draft of an article on the subject and submitted it to Brian at Better Software. Nearly all of my articles have been published with only minor editorial changes, but this draft was a hot mess. Any reasonable editor would have rejected it outright. What Brian did instead was to dissect the piece, pull out the essential concepts, and make diagrams showing what I had failed to describe well. He sent me some diagrams, I made some corrections, he sent me some more diagrams. Once the diagrams were correct, I re-wrote the piece from scratch as a description of Brian's diagrams. I've always thought he should have had co-author credit for that piece. It was called "Old School Meets New Wave" and it had some really goofy artwork, a photo of a skinny punk kid with a pink mohawk overlaid on a black-and-white fifties dude with a fedora.

It ended up being one of the best articles of my career. Some time later a tester named Paul Carvalho told me that he had created and gotten funded a testing effort at his company based on the concepts in that article. Sometimes writing really can change the world. It has happened to me a couple of times since then, but that article was the first time I knew I had made a difference to someone else by writing about software. (Paul, if you read this, I hope I didn't garble your story, it was a long time ago we had that conversation.)

From about 1998 until the middle of the decade, the field of software testing and software development experienced any number of radical shifts, with the increased value for the role of a tester because of Y2K testing, the rise of open source, the rise of the agile movement, the rise of dynamic programming languages, and more. But by late 2009 my own sense was that the public discourse on software testing in particular had become stale and outdated. I started the writing-about-testing mail list and the WAT conference in an attempt to encourage new voices and new ideas in the public discourse on software testing. A little over a year later, I think we have had some influence. Since the first WAT conference, Alan Page, Matt Heusser, and others have begun calling for some examination of what the future of software testing holds.

New ideas in our field come from three places. They come from beginners who stumble upon some beautifully simple idea and are moved to tell the world about what they have done.   They come from people who think about the work on a really grand scale over a long period of time and build a body of work to support that grand idea. And they come from people who truly make a breakthrough of some sort and are moved to explain that breakthrough to everyone.

So Charley, that is where my ideas come from.

(UPDATED: fixed garbled links)

Writing About Testing participants

2010-12-10T12:33:00.006-07:00

I took a poll of those interested so far in attending the second Writing About Testing peer conference May 13 and 14 and found that nine people are very seriously considering attending. This is what they are thinking of discussing:

Lisa Crispin (CO) new and emerging approaches since the publication of Agile Testing
Alan Page (WA) a new approach to test design, also personas for tester career roles
Marlena Compton (Australia) ongoing research in visualization of software project data
Dawn Cannan (NC) executable specifications within larger testing projects
Sylvia Killinen (CO) practicing software craftsmanship
Elena Yatzeck (IL) implementing DSLs for use by non-programmers
Shmuel Gershon (Israel) diverse approaches to writing about testing using personas
Charley Baker (CO) large-scale, Enterprise automation systems, open source
Marisa Burt (CO) EAI in Enterprise systems

UPDATED:
Zeger Van Hese (Belgium) critical theory, etc.
Frank Cohen (CA) handling AJAX and Flex
Markus Gärtner (Germany) ATDD success stories, test estimation
Joey McAllister (CO) stickyminds.com
There is one spot open. The (fairly arbitrary) deadline for submissions is January 1.

Anyone who would like to join the writing-about-testing mail list may submit a request.

UI test smells: if() and for() and files

2010-11-30T15:36:00.000-07:00

I read with interest Matt Archer's blog post entitled "How test automation with Selenium or Watir can fail"

He shows a couple of examples that, while perfectly valid, are poor sorts of tests to execute at the UI level. Conveniently, Mr. Archer's tests are so well documented that it is possible to point to exactly to where the smells come in.

The test in the first example describes a UI element that has only two possible states: either "two-man lift" or "one-man lift", depending on the weight of an item. In a well-designed test for a well-designed UI, it should be possible for the test to validate that only those two states are available to the user of the UI, and then stop.

But Mr. Archer's test actually reaches out to the system under test in order to open a file whose contents may be variable or even arbitrary, iterates over the contents of the file, and attempts to check that the proper value is displayed in the UI based on the contents of the file. Mr. Archer himself notes a number of problems with this approach, but he fails to note the biggest problem: this test is expensive. The contents of the file could be large, the contents of the file could be corrupt, and since each entry generates only one of two states in the UI, all but two checks made by this test will be unnecessary.

Mr. Archer goes on to note a number of ways that this test is fragile. I suggest that the cases involving bad data in the source file are excellent charters for exploratory testing, but a poor idea for an automated test. An automated UI test that simply checks that the two states are available without reference to any outside test data is perfectly adequate for this situation.

In his second example, Mr. Archer once again iterates over a large number of source records in order to achieve very little. Again, exercising the same UI elements 100 times using different source data is wasteful, since all the UI test should be doing is checking that the UI functions correctly. However, there is an interesting twist in Mr. Archer's second example that he fails to notice.   If Mr. Archer were to randomly pull a single source record from his list of 100 records for each run of the test, he would get the coverage that he seems to desire for his 100 records over the course of many hundreds of individual runs of the test. I took a similar approach in a test harness I once built for an API test framework, and I described that work in the June 2006 issue of Better Software magazine, in a piece called "Old School Meets New Wave".

Both of Mr. Archer's examples violate two well-known UI test design principles. The first principle is called the "Testing Pyramid for Web Apps".   As far as I can tell, this pyramid was invented simultaneously and independently by both Jason Huggins (inventor of Selenium) and by Mike Cohn. (Jason's pyramid image is around the web, but I nabbed it from here.)

Any test that reaches out to the file system or to a database is going to belong to the middle tier of business-logic functional tests. And even then, most practitioners would probably use a mock rather than an actual file, depending on context. While it is not always possible for UI tests to avoid the business-logic tier completely, it should be the case that UI tests are in fact focused on testing *only* the UI. Loops and conditionals in UI tests are an indication that something is being tested that is not just part of the UI. Business logic tests should to the greatest extent possible be executed "behind the GUI". From years of experience I can say with authority that UI tests that exercise business logic become both expensive to run and expensive to maintain, if they are maintainable at all.

The other principle violated by these examples is that highest-level tests should never have loops or conditionals. The well-known test harness Fitnesse does not allow loops or conditionals in its own UI. Whatever loops or conditionals may be required by the tests represented in the Fitnesse UI must be coded as part of the fixture for each test. For a detailed discussion of this design pattern, see this post by Gojko Adzic: "How to implement loops in Fitnesse test fixtures"

close reading/critical thinking

2010-11-21T09:43:00.000-07:00

The last Weekend Testers (Australia/New Zealand) was brilliant. Let me urge you to read Marlena Compton's report and the transcript of the session.

This sort of practical implementation of critical theory is long overdue in the testing community, and the WTANZ crew did a great job of using a well-known theoretical tool to analyze and dissect some real problems in some real work.

Compare what WTANZ did with Zeger Van Hese's recent demonstration of deconstruction.

This sort of work, bringing reputable and sophisticated critical theory to bear on actual testing and QA activity, is a field wide open, barely explored, and long overdue.

May we see much more of it soon.

more on certs, more numbers

2010-11-16T19:08:00.000-07:00

I noticed (thanks Twitterverse) that there was an interview with Rex Black over on the UTest blog. In that interview he reveals a very interesting number:

"...the ISTQB has issued over 160,000 certifications in the last ten years."

Using the numbers from my previous post: if we assume that there are about 3,000,000 software testers in the world right now, and if we issued 160,000 certifications right now, that would mean about 5 certifications for every 100 software testers.

I would be willing to bet that there were about the same number of testers ten years ago: Y2K was just over and the value of dedicated testers had been shown.   But as Alan Page and others have noted, there is a lot of turnover, a lot of churn, among those practicing software testing.

So my numbers start to get a little sketchy here, I don't have anything to back them, so consider this a thought experiment: as noted above, let's say that there were about 3 million testers a decade ago and there are still 3 million testers today. Let's say half of today's testers have started since 2000.   This gives us a field of 4.5 million testers who could have acquired a certification in the last decade. This makes for about 3 certified testers for every 100 possible certifications.

I think it is an excellent bet that a significant fraction of those 160,000 certifications were issued in the UK, Australia, and New Zealand.   Just to make it even, call it about 1/3, put 60,000 certs in those regions, leaving 100,000 for the rest of the world. That brings us down to about 2 certs per 100 testers.

But that still seems high to me. I might have missed something. Regardless, it still looks like a pretty small market, and I'd bet the market has been shrinking a lot with the rise of agile adoption and the economic downturn.

an object of interest

2010-11-11T18:01:00.003-07:00

I bought this recently at a guerilla art show:

Here it is hanging in my office:

The poster caught my eye because I've loved the Alice books all my life and I re-read them often. I am especially fond of the Tenniel illustrations, and the one for Jabberwocky is a favorite.

The poster also caught my eye because of the odd and interesting typeface. The story behind that typeface is fascinating. I asked the artist to send me that story in email so that I could have it written down:

The story is this:
Just south of where I grew up (near Green Bay, WI) is the Hamilton Wood Type Museum. A while back, I visited armed with a few sheets of printmaking paper with the goal of printing some or all of the Jabberwocky poem from some original wood type. During the course of the 19th and 20th century Hamilton made wood type for advertisements and headlines and circuses and had gone on to accumulate wood type from other manufacturers who had given in to market pressures or the eventual obsolescence of the letterpress industry. What remained when I visited were cases and cases of cabinets full of uncategorized type.. roughly 1000 different type faces and sizes. I spent a better part of a day just finding a type face I liked, using a rather capricious method to determine "the one": the style of the lower case 'g'. I estimate the type to have been produced in the early 20th century, probably for about 20 years, if that. It is an obscure, unnamed type face. I set the type but realized that by choosing according to lower case 'g', I had picked a case that only let me set 3 lines of text. This was all that was left of this type in existence. So, I printed the top three lines first on a large flat bed cylinder press called a Vandercook 325G (incidentally I have the exact same model press in my shop here), disassembled the text and composed the 4th line. When I returned to Colorado, I replicated the illustration from Through the Looking Glass and then added that to the print.
That's the story.
Enjoy
Dan

Zeger Van Hese is a Belgian software tester who, like me, is interested in critical theory and what application critical theory might have to the work of creating software. The other day he mentioned in passing a seminal work by Walter Benjamin, The Work of Art in the Age of Mechanical Reproduction that I had not read in many years.

In the light of Benjamin's work, my poster is a strange object indeed. While it was created in a process of mechanical reproduction, it was created only once. The means to create it are lost in an anonymous bin in an obscure warehouse somewhere in Wisconsin. And even if someone were dedicated enough to find that one particular bin, not even enough of this particular wood type exists to even print all four lines from Jabberwocky.

My poster would have been a strange item even for 1936, when Benjamin wrote about mechanical reproduction. But to have such a thing on my wall in 2010 is, for me, astonishing.

XP, Scrum, Lean, and QA

2010-11-06T10:37:00.002-06:00

Before I do this, two things up front: for one thing, I am a crappy programmer. I read code far better than I write it, and I read non-OO code better than I read OO code. Also, I am writing as someone who knows a lot about Quality Assurance and testing, and very little about the hands-on day-to-day work of modern programming. So here goes:

As a QA guy, I know this: long before Scrum and XP and the Agile Manifesto, people working in Computer Science and software development knew three things about software quality: having an institutional process for code review always improves software quality; having automated unit tests (for some value of "unit") with code coverage improves software quality; and having frequent, fast builds in a useful feedback loop improves software quality. Sorry, I don't have references handy, I read most of this stuff about a decade ago. Maybe some heavy CS people could leave references in the comments.

The XP practices simply institutionalize those three well-known practices, and for the time, dialed them up to 11. Pair programming is just a very efficient way to accomplish code review. TDD is just a very efficient way to accomplish unit test coverage. CI is just a very efficient way to get a fast feedback loop for the build.

There is nothing magical about these practices, and I have worked on agile teams that don't do pair programming but do awesome code review. I have worked on agile teams whose microtest suite runs heavily to integration tests instead of traditional unit tests. I have worked on agile teams with a dedicated build guy. I started my career working in an incredibly well-designed COBOL code base. No objects in sight. Had I known then what I know now about test automation, I could have written an awesome unit/integration test framework for that system. The XP practices themselves are not sacred. The principles behind those practices are.

But the XP practices themselves are just a small piece of having a successful agile team. In musical terms, these are the equivalent of knowing scales and chords, just basic technical practices necessary to get along in the business. Of course they are not necessary: The Ramones and Tom Petty have only a basic grasp of the technical aspects of music, but they cranked out some monster hits. Put any of those guys in a jazz jam session or a symphony orchestra and they would be completely lost. There is some nasty software out in the world that makes a lot of money.

I like Scrum, for a number of reasons. For one thing, it has an aesthetic appeal. The concepts of developing, then releasing, then retrospective speaks to me strongly, not least because they map closely to the ideas from the performing arts of practice, perform, rehearse.

I also like Scrum because of its emphasis on human interaction rather than institutional process. Scrum is lightweight by design, and leaves much room for people to act as people with other people. Scrum favors mature, intelligent adults.

Finally, I like Scrum because it is a process derived directly from the actual practice of creating software. It is described in plain English and it relies on no special concepts. It was crafted out of whole cloth by good developers in a tough spot.

I dislike Lean/kanban for the same reasons. As a mature adult, I resent having any of my activities identified as "waste". I resent not having the end of an iteration to celebrate. I resent being treated as a station in a production chain.

Unlike Scrum, the Lean principles were not derived from the actual work of software development. They came from automobile manufacturing, and were overlaid on software development in what I consider to be a poor fit. Putting on my QA hat again, there are two other popular software development methodologies that came directly from manufacturing, and the state of those methodologies is instructive. One of them is ISO9000. The fatal flaw of ISO9000 is that once a process is in place, it becomes very difficult and expensive to change that process. This is fine in manufacturing, but it is death to a reasonable software development process. The other methodology from manufacturing is Six Sigma. Six Sigma is very expensive, and while it might yield information valuable to managers, it provides no benefit to those actually doing the day-to-day work of software development. I am not aware of any manufacturing processes shown conclusively to improve the hands-on work of software development.

XP and Scrum are not nearly enough to guarantee a successful software project. For a comparable situation, just because a band has a rehearsal schedule and some gigs does not guarantee that they will be international superstars. Brian Marick at one point talked a lot about four principles that also increase the chance of a successful software project: skill and discipline, ease and joy. I won't explain those here, interested readers can find that work themselves.

But beyond even skill, discipline, ease and joy, a successful software project requires that we as creators of the software reach out and interact with the world in a way that changes the lives of those who encounter our software. It is an awesome power. In some cases, we can make our users' lives a living hell. But it's a lot more fun to make everyone happy.

ignoring certification; with numbers

2010-10-29T10:47:00.004-06:00

All of the questions about tester certification were answered many years ago. They exist and they cannot be made to unexist. The only remaining question on the subject is: how many tester certifications can be sold? And the answer to that question doesn't matter to anybody except the people selling the certifications.

A while ago on the writing-about-testing mail list we did a little exercise to come up with some back-of-the-napkin estimates about the number of software testers in the world. We used US Department of Labor data and also some other public information about software employment worldwide. We also had access to some privileged information about magazine subscriptions. In addition, a number of us have done serious work in social networking, and we have some analytical tools from that work to help estimate. Using all of that, we came up with a pretty consistent estimate that there are probably around 300,000 software testers in the US, and maybe 3 million in the whole world.

That is a pretty small market in which to sell tester certifications.

Elisabeth Hendrickson recently did a fascinating analysis of QA/testing job ads. According to her data, it is a good bet that 80% of the people doing modern software testing work in the US have programming skills of one sort or another.

Jason Huggins of Sauce Labs has been tracking job ads that mention browser automation tools. Jason notes a remarkable recent increase in the demand for Selenium skill. You can see the trend for some popular automation tools at indeed.com. The QTP vs. Selenium trend in job ads is fascinating, but looking closely, this graph indicates a general across-the-board increase in demand for technical skills in traditional UI-based software testing.

Finally, sorry I don't have a link handy, but I have seen a number of reports of a radical increase in the rate of adoption of agile practices among US companies of every size and description. And the agile whole-team approach to software development makes dedicated, siloed traditional V&V test departments irrelevant.

The existing tester certifications simply do not apply to this sort of work. Certification is becoming more and more useless to US testers, and to their employers as well.

I feel like I am pretty plugged in to the world-wide tester community and the world-wide agile community, and anecdotal evidence suggests that indeed, the market for tester certification in the US is very small. Again, this is anecdotal evidence, but the hot spots for certification seem to be the UK and Australia/New Zealand, possibly areas of Southeast Asia, possibly areas of Eastern Europe. Once more with the anecdotal evidence, but I would suggest that in political climates that favor a high degree of regulation of business practices, certification will be more popular.

So if we eliminate from our worldwide tester population of 3 million the majority of US testers and a significant fraction of the rest of the world as potential buyers of a tester certification, that leaves a pretty tiny market for tester certification.

I think we can say with some confidence that professional tester certification can safely be ignored by the vast majority of software testers. That said, if you are required to get a certification, or if you just want to get a certification, go ahead and do it. It won't hurt you, and at the very least, you'll learn how software was tested in 1996.

I think we can also safely say that any supposed controversy surrounding tester certification is overblown and can also be ignored.

Which suggests one more interesting question: if the supposed controversy over certification really is as trivial as these statistics indicate, then why does so much of the testosphere spend so much time agonizing over it?

I have a cynical answer to that, but I'll keep it to myself.

Update: made the links nice

Call for Participation: Second Writing About Testing peer conference

2010-10-25T16:16:00.002-06:00

The Second Writing About Testing Conference: Frontiers for Software Testing

I am pleased to announce the call for papers for the Second Writing About Testing Conference to be held May 13 and 14 2011 in Durango Colorado.

For more information about the original conference and the Writing About Testing mail list please see:
http://chrismcmahonsblog.blogspot.com/2010/02/writing-about-testing-listconf-update.html
http://chrismcmahonsblog.blogspot.com/2009/10/cfp-peer-conference-writing-about.html

Writing About Testing is a peer conference for those interested in influencing the public discourse on software testing and software development by writing and speaking on those subjects. The discussion revolves around blogging, giving presentations at conferences and user groups, and writing for established media outlets, both for pay and for other reasons. There will very likely be representatives from established media outlets attending. Having software writers and publishers talking to each other face to face is a unique aspect of WAT.

For the first WAT conference we asked only that participants be interested and have had already written about software. The second WAT will be a little different.

For the second WAT we ask that applicants propose a talk (informal is fine, no slides required) of 30-45 minutes about some subject critical to their work as a tester or developer, or designer or business analyst, but which is not generally recognized as being part of such work. A list of suggestions for such talks is below. The purpose of this is to expand the practice of software beyond the current artificial boundaries of recognized software activity.

To that end, presentations on these subjects are not welcome, unless the presentation has obviously unusual aspects:

test heuristics/mnemonics
exploratory testing
classic test automation (record-and-play, automation pyramid, etc. Information on unusual approaches to test automation is welcome.)
Scrum/Lean/Context-Driven/whatever. In general, anything involving capital letters is not welcome.
certification (for or against)

In a nutshell: don't bore the other attendees with stuff that has been discussed to death for years. This conference is to discuss frontiers.

Given that the presentations will be on unusual subjects, there will be a minimal requirement for having published previous work. Any publicly available source of writing, for instance a blog, would qualify an attendee. Applicants with no publicly available writing at all will not be considered. Attendees are encouraged to write with ambition and daring after the conference ends.

Attendance will be limited to about fifteen people. There will be a nominal fee of $50.00 per person to help cover room rental, and lunch will be provided. A discounted rate at a convenient bed and breakfast hotel is available.

Durango has a lot to offer visitors. Conference attendees may wish to arrive early or stay late to take advantage of the nearby attractions: beautiful mountains to the north and east, desert sandstone canyons to the west and south. The steam train from Durango to Silverton and back is a fantastic experience, as is soaking in the hot springs nearby. Within a short drive are Mesa Verde National Park, Canyons of the Ancients National Monument, Monument Valley and the Navajo Reservation, Great Sand Dunes National Park, etc. Local opportunities for hiking, biking, and boating abound.

There are direct flights to Durango from Denver and from Phoenix. Many attendees will likely come from the Denver area, so carpooling from there may be possible.

To submit a proposal, either send a message to me at christopher dot mcmahon on gmail, or join the writing-about-testing mail list at http://groups.google.com/group/writing-about-testing and submit your proposal there.

The deadline for submissions is Jan 1.
Invitations to the conference to be sent Feb 1.
The conference itself will be May 13/14.

Here are some possible frontier subjects for presentations:

System Administration

There is a surge of interest in recent times in a concept called "DevOps". DevOps proposes an alliance among software developers and system administrators in order to create the best possible production environment experiences. Testers need to be a part of that conversation.

Data Visualization

Not only our applications, but the whole world around us generates incomprehensible amounts of data, and the only way to make sense of it all is to render that data in a visual or tactile fashion. Testers need to understand these technologies in the service of their work.

Frameworks/Mashups

Good test automation today happens at every level. A single framework may exercise the user interface, call REST or SOAP APIs, and reach into a database, all in the course of a single test suite. Myriad tools for such testing exist, and knowing how to get such tools to talk to each other for a particular purpose is becoming a critical skill for testers.

User Experience

Great strides have happened in user experience work in the last few years, and there are exciting advances on the horizon. Testers have largely ignored the conversation happening among user experience experts.

Web Design

New javascript libraries like jquery are making things possible in browsers that were unimaginable just a few years ago. Flex, Flash, and Adobe Air are making huge inroads into application UI design. HTML5 is on the horizon and looming. Testers need to know what is happening in this arena, and largely do not.

Web Services (REST/SOAP)

Twitter, Facebook, and the bleeding edge of web applications are no longer about the UI. Today it is all about the APIs, and the third party applications that use those APIs to bring killer experiences to users. Testers need to know how web services and APIs work.

Environments/VMs/Cloud Computing

Managing test environments has always been challenging. New cloud computing services in some ways make such work more challenging, but the reward is a vastly simplified process for provisioning test environments. This work needs public exposure.

Agile Methods

Agile methods work, but even today, no one knows exactly why. The explanations we have are frequently facile and often abused. Testers could be the ones to provide the well-considered explanations for the effectiveness of agile methods.

Process Work/Quality Assurance

QA has a bad reputation in the testing community that it does not deserve. I have said before on stickyminds.com and in Beautiful Testing that QA is not evil, that it is work that still needs doing, and that often testers are in a good position to provide quality assurance. Bring back real discussion about Quality Assurance.

Aesthetics/Artistic Performance

There is a wealth of knowledge available from disciplines within the Liberal Arts that apply directly to software development. Testers can help bring that knowledge over to the world of software development.