Chris McMahon's Blog

Just Fix It

2011-12-04T12:18:00.002-07:00

On the writing-about-testing mail list recently was a discussion of defect tracking. Given a good enough code base and a mature dev team, I think defect tracking is mostly unnecessary, and it's worth talking about why that is.

Some time ago there was a popular meme in the agile testing community that goes "Just Fix It", but I haven't heard it mentioned in some time, and I think it's worth reviving the discussion. The idea behind Just Fix It is to bypass the overhead of creating a defect report, having those defect reports go through some sort of triage process, and only then addressing the problems themselves represented by the defect reports. You save a lot of time and overhead if you Just Fix It.

For some time now I have specialized in testing at the UI, a combination of functional testing and UX work. In my experience, in a good code base, important defects found at the UI level are almost always what I think of as "last mile" issues, where the underlying code has changed in some way but the hook for that code into the UI has been mangled or overlooked. These are cases where unit tests are almost certainly passing, but the app is broken anyway. Some examples:

Explanatory text has disappeared or no longer describes function accurately.
A widget that used to function no longer does. For example, a Submit button no longer makes anything happen.
A call to some underlying function is no longer correct. For example, a Search function that used to return results no longer does.

While a Just Fix It culture is not necessarily agile, examples of Just Fix It are easier to describe in a typical agile situation.

Small Team

Many agile teams share a single space, making communication easy and instantaneous. In such a situation, a conversation like this might happen:

Tester: "Hey, the froobnozzle stopped froobing, anyone know anything about that?"
Dev: "Wow, I didn't realize my last commit would break the froob function, I'll Just Fix It."

This is a canonical example of what Lisa Crispin calls the "whole team approach", where testers, devs, and everyone else is working on the same stories at the same time in the same place simultaneously.

And if it's appropriate, there's no real reason a tester couldn't Just Fix It themselves.

Large Team

But some teams are too large for a conversation like this to be practical. Assume a collocated team with a really big story board with dozens of story cards all being moved around a lot. Say a tester finds an issue with the froobnozzle.

Tester: grabs a red sticky note and writes brief description of froobnozzle problem. Puts sticky note on froobnozzle story card
...minutes later...
Dev: whoa, a red card on my story, better Just Fix It.

Distributed Team

Distributed teams tend to have really sophisticated issue-tracking systems in place, where stories are represented in software of some sort, where they can be assigned, have their status changed, etc. etc. If a distributed team is small enough, a tester will know that Joe is working on the froobnozzle story, so:

Tester to Joe on IM: "hi Joe, I think you might have just broken the froobnozzle."
Joe the Dev: "whoa, good catch, I'll Just Fix It."

Large Distributed Team

On a large distributed team, identifying who might be in a position to Just Fix It can be complicated. One strategy is to read the commit logs upon identifying a defect to see who or what may have caused the problem. Another strategy might be to review all the stories in play to discover who might be working on the froobnozzle this iteration.

But sometimes these sorts of approaches are too complicated or take too much time. One pattern I have seen on several occasions in large distributed teams is to designate a knowledgeable person on the dev staff, or possibly a Scrum Master type, to represent the whole dev team for questions about behavior or function. I have seen this role called the Face, and the Ninja, and the Disturbed.

Tester: "hi Face, I just discovered the that the froobnozzle got broken within the last day or so."
Face: "whoa, let me check that for you"
Face: "good catch, Joe broke that two commits ago, he's Just Fixing It"

Defect Found in Production

A customer probably reported it. The fix is deployed to production within minutes or maybe hours of the report. (Again, a good code base allows this.)

I worry that too often "root cause analysis" is a synonym for "blame". Defects in production are almost certainly a process problem, and the place to address process problems are in retrospectives or similar conversations.

Besides, if your team is releasing so many defects to production that you have to track them, you have bigger problems.

Won't Fix

True story: just this week I was refactoring some Selenium tests and discovered a bug. This was in a part of the application that is not exposed to customers, it is only for internal users employed by my company. The bug was that attempting to enter a duplicate record causes an unhandled exception and the user is presented with an ugly stack trace. This was an old bug, and was not part of the work of the current iteration.

I work on a large distributed team. As I noted above, we have a sophisticated issue-tracking system in place. All of the work we do is documented and tracked in this system. We have no designated defect-tracking system, just a single monolithic sophisticated issue-tracking software application.

Upon finding the bug, I had a conversation with the dev who knows about that part of the code. We agreed that this was a no-harm-no-foul situation, no data corrupted, minimal inconvenience to the user, no customer exposure. We agreed that Just Fixing It right this minute wasn't very important.

So I created a new issue in the issue tracking system and assigned it to the dev who knows about that part of the code. This issue has the same visibility and status as every other issue in the system. My bug report issue will be treated the same as every other issue in the system, included in the backlog, and prioritized to be worked on with every other issue in the backlog.

I don't even really think of that particular issue as a defect. It's just a description of the state of a part of the application, some work that we might choose to do at some point. I'm sure we'll Just Fix It pretty soon.

a selenesse trick

2011-09-30T10:44:00.001-06:00

Selenesse is the mashup between Fitnesse and Selenium I helped work on some time ago. I keep encountering this pattern in my selenesse tests, so I thought I'd share it...

Every once in a while a test needs some sort of persistent quasi-random bit of data. In the example below I'm adding a new unique "location" and then checking that my new location appears selected in a selectbox. This is also a neat trick for testing search, or anywhere else you need to add a bit of data to the system and then retrieve that exact bit of data from somewhere else in the system.

| note | eval javascript inline FTW! |
| type; | location_name | javascript{RN='TestLocation' + Math.floor(Math.random()*9999999999);while (String(RN).length < 14) { RN= RN+'0';}} |
| note | instantiate a variable and assign the new value to it in a single line FTW! |

| $LOCATION= | getValue | location_name |

| note | click a thingie to add the new data to the system |
| click | location_submit_button ||
| waitForTextNotPresent | Adding ||
| note | check that the newly added data appears as the selected entry in a selectbox FTW! |

more UI test design (once more from Alan Page)

2011-09-08T07:12:00.000-06:00

Before it gets lost in history, I want to riff off Alan Page once again, who made some excellent points. But as someone who has been designing and creating GUI (browser) tests for a long time, I'd like to address some of those and also point out some bits of ROI that Alan missed.

Making UI tests not fragile is design work. It requires an understanding of the architecture of the application. In the case of a web app, the UI test designer is really required to understand things like how AJAX works, standards for identifying elements on pages, how the state of any particular page may or may not change, how user-visible messages are managed in the app, etc. etc. Without this sort of deep understanding of software architecture, UI tests are bound to be fragile.

I've said before that UI tests should be a shallow layer that tests only presentation, and that rarely tests logic, math, or any underlying manipulation of data. If tests are designed in this way, then they will be robust and maintainable over the life of the app.

UI test design is a skill. Designing such tests is no harder or easier than any other activity that requires skill and understanding. The tools with which I am familiar provide enough power to create reasonable, robust, maintainable tests.

Finally, I think Alan is missing one aspect of automated UI tests that I find the most valuable of all.

From green-screen mainframe systems to bleeding-edge web applications, in my experience every software system suffers from one particular sort of error that is always extremely difficult to see when testing manually: when something goes missing.

A search that used to return results no longer does. A widget that used to be on the page no longer is. A bit of text critical to the user's work goes missing.

Actions that cause errors are easy to find when testing manually, as are errors of presentation. Elements and functions that used to exist but no longer do are difficult to find: it is not easy for a human being to see the absence of a thing, but such errors stick out like sore thumbs in an automated UI test suite.

In my experience, this is one of the most valuable aspects of automated UI testing, and one of the best reasons to invest in UI test automation. The absence of a thing in the UI is simply not detectable with unit tests or with integration tests. That critical bit of function that doesn't manage to cross the last interface to the UI is only detectable at the UI itself, and automated UI tests are very, very good at detecting errors where something has gone missing.

Automated Test Design (riffing/ripping off Alan Page)

2011-08-30T18:12:00.001-06:00

Alan just posted this: http://angryweasel.com/blog/?p=325

This is the nicest test example I've seen in a long time, and I think it bears a little more analysis.

If I were in charge of the development of this app, I would make automated testing happen on 3 levels.

First, there would have to be some sort of test for generating a single random number. Which seems easy enough, but at this level, you really have to understand exactly what a call to rand() (or whatever) on your particular OS is going to do. I once wrote a script in Perl (praise Allah just a toy, not production code) that returned perfectly random numbers on OSX but returned exactly the same values every time on Windows. You'd better test that your call to rand() really returns random numbers, regardless of the OS it is running on.

At a higher level, you'd want to do exactly what Alan talks about in looping 100,000 times (although 100K seems like overkill to me). This is what I think of as an "integration test". You're going to have some method or possibly even an API call or REST endpoint like "generate_five_random_numbers_and_add_the_results". You're going to want to exercise that wrapper enough to convince yourself that the numbers are right and that the math is right.

For some time now I've been making my living writing automated UI/browser tests along with doing ET. Here's my take on a UI test for Alan's app:

Open the page
Check that text "Total" exists.
Check that the 6 textboxes exist.
Click the Roll button.
Check that the state of the page changes (however that happens).
Check that there are values in each textbox (even this might be too fancy, depending on the app and the test framework).

Also getting fancy, you could check that the 6 textboxes exist in the correct order. (Selenium's "glob:" feature makes this pretty painless.)

Doing math in UI tests isn't very smart. Doing data comparison over multiple runs in UI tests isn't very smart. Do that stuff at lower levels, where you can take advantage of programming power and speed. UI tests are a shallow layer where you simply check that the user has all the stuff they need to get the job done.

where ideas come from

2010-12-20T09:48:00.004-07:00

Today stickyminds published my article with expanded descriptions of the "10 Frontiers for Software Testing" that I suggested as starting points for those interested in attending the second Writing About Testing conference.

Since I announced the CFP for the first WAT conference in October 2009, I have published several dozen articles on software and software testing. (I actually lost count: it is well over thirty but fewer than fifty individual pieces.)

My friend Charley Baker asked me recently where I get the ideas for so many articles.   It is an interesting question, and worth answering:

The most important source of ideas is simply everyday work. As I go about doing my job, it happens fairly often that a situation crops up that I think would be of general interest to the community of software testers and developers. So I write it down and I make it public. Articles about bugs, bug reports, test design, architecture, workflow, telecommuting, frameworks, war stories all come from noticing the details of the everyday work.

Here is the story of the very first software article I ever published: I have been following Brian Marick's work for a long time now. Brian used to be the editor of Better Software magazine, and he would occasionally solicit articles for the magazine on his blog. In March 2004 Brian asked for submissions for a piece along the lines of "add(ing) a scripting language to her manual testing portfolio.". In particular, I recall that Brian wanted an article suitable for beginners with an example of a testing problem that could only be solved by scripting a little program.

I had written book reviews before but I had never published a piece about software. I had just encountered a situation at work that was a perfect example of what Brian wanted. I was working for a company that was switching from shipping whole custom-built servers to shipping installation CDs for COTS hardware. The installation CDs contained more than 4000 files. The switch was a little bumpy, and at one point we very nearly shipped an installation CD missing 4 critical files of the 4000.   I had been teaching myself Perl (so I was a beginner myself), and I wrote a little script in Perl to compare recursively the contents of large directories, so that it would be easy to see if some few files had gone missing. I described what I had done, Brian published it in Better Software, and in one of the highlights of my career as a writer, that article (with me as a character!) became the basis of the first example in Brian's book Everyday Scripting with Ruby. (Get the book: it will make you a better coder, no matter your level of skill.) The article was titled "Is Your Haystack Missing a Needle"

Another source of ideas for software articles comes from having some Very Large Idea that evolves over a long time. At Bret Pettichord's Austin Workshop on Test Automation in 2007, in a moment of inspiration, I gave a five-minute lightning talk demonstrating an example of using the artistic language of critical theory (in particular, New Criticism) to evaluate the quality of a piece of software. The talk got an enthusiastic reaction from the people in the room, mixed with some skepticism as I recall.   It struck me at the time as being an odd idea, but the more I considered it, the more it made sense. I wrote a long paper on the subject and submitted the paper to the CAST 2008 conference, but it was rejected. I published it on my blog, and I still refer to it now and then. My thinking on the subject has matured and expanded since then, so if you'd like to see the latest example, look at PragPub magazine for November of this year. In 2008 I was a lonely voice on the subject. Today I have colleagues, it is nice to see others considering critical theory applied to software as well.

Finally, every once in a while, I manage to do something really unusual, something that will actually change peoples' minds about how they go about their work. In 2006 I was working for Thoughtworks on an EAI project. Our code base had great unit test coverage and integration test coverage, and as the QA guy, I was not finding defects in what we were creating. But we had to interact with a legacy database, and we were often surprised by unusual or corrupt historical data. I made it my business to expose as much of that bad data as I could. I wrote a little Ruby script that would do quasi-random queries in the database, request the same data from the API we were building, and compare the results, running within an infinite loop. I found a significant number of issues in this way, where the API we were building failed to handle data we never expected to find in the database. To my knowledge, no one had ever published anything describing a situation like this.

So I wrote a draft of an article on the subject and submitted it to Brian at Better Software. Nearly all of my articles have been published with only minor editorial changes, but this draft was a hot mess. Any reasonable editor would have rejected it outright. What Brian did instead was to dissect the piece, pull out the essential concepts, and make diagrams showing what I had failed to describe well. He sent me some diagrams, I made some corrections, he sent me some more diagrams. Once the diagrams were correct, I re-wrote the piece from scratch as a description of Brian's diagrams. I've always thought he should have had co-author credit for that piece. It was called "Old School Meets New Wave" and it had some really goofy artwork, a photo of a skinny punk kid with a pink mohawk overlaid on a black-and-white fifties dude with a fedora.

It ended up being one of the best articles of my career. Some time later a tester named Paul Carvalho told me that he had created and gotten funded a testing effort at his company based on the concepts in that article. Sometimes writing really can change the world. It has happened to me a couple of times since then, but that article was the first time I knew I had made a difference to someone else by writing about software. (Paul, if you read this, I hope I didn't garble your story, it was a long time ago we had that conversation.)

From about 1998 until the middle of the decade, the field of software testing and software development experienced any number of radical shifts, with the increased value for the role of a tester because of Y2K testing, the rise of open source, the rise of the agile movement, the rise of dynamic programming languages, and more. But by late 2009 my own sense was that the public discourse on software testing in particular had become stale and outdated. I started the writing-about-testing mail list and the WAT conference in an attempt to encourage new voices and new ideas in the public discourse on software testing. A little over a year later, I think we have had some influence. Since the first WAT conference, Alan Page, Matt Heusser, and others have begun calling for some examination of what the future of software testing holds.

New ideas in our field come from three places. They come from beginners who stumble upon some beautifully simple idea and are moved to tell the world about what they have done.   They come from people who think about the work on a really grand scale over a long period of time and build a body of work to support that grand idea. And they come from people who truly make a breakthrough of some sort and are moved to explain that breakthrough to everyone.

So Charley, that is where my ideas come from.

(UPDATED: fixed garbled links)

Writing About Testing participants

2010-12-10T12:33:00.006-07:00

I took a poll of those interested so far in attending the second Writing About Testing peer conference May 13 and 14 and found that nine people are very seriously considering attending. This is what they are thinking of discussing:

Lisa Crispin (CO) new and emerging approaches since the publication of Agile Testing
Alan Page (WA) a new approach to test design, also personas for tester career roles
Marlena Compton (Australia) ongoing research in visualization of software project data
Dawn Cannan (NC) executable specifications within larger testing projects
Sylvia Killinen (CO) practicing software craftsmanship
Elena Yatzeck (IL) implementing DSLs for use by non-programmers
Shmuel Gershon (Israel) diverse approaches to writing about testing using personas
Charley Baker (CO) large-scale, Enterprise automation systems, open source
Marisa Burt (CO) EAI in Enterprise systems

UPDATED:
Zeger Van Hese (Belgium) critical theory, etc.
Frank Cohen (CA) handling AJAX and Flex
Markus Gärtner (Germany) ATDD success stories, test estimation
Joey McAllister (CO) stickyminds.com
There is one spot open. The (fairly arbitrary) deadline for submissions is January 1.

Anyone who would like to join the writing-about-testing mail list may submit a request.

UI test smells: if() and for() and files

2010-11-30T15:36:00.000-07:00

I read with interest Matt Archer's blog post entitled "How test automation with Selenium or Watir can fail"

He shows a couple of examples that, while perfectly valid, are poor sorts of tests to execute at the UI level. Conveniently, Mr. Archer's tests are so well documented that it is possible to point to exactly to where the smells come in.

The test in the first example describes a UI element that has only two possible states: either "two-man lift" or "one-man lift", depending on the weight of an item. In a well-designed test for a well-designed UI, it should be possible for the test to validate that only those two states are available to the user of the UI, and then stop.

But Mr. Archer's test actually reaches out to the system under test in order to open a file whose contents may be variable or even arbitrary, iterates over the contents of the file, and attempts to check that the proper value is displayed in the UI based on the contents of the file. Mr. Archer himself notes a number of problems with this approach, but he fails to note the biggest problem: this test is expensive. The contents of the file could be large, the contents of the file could be corrupt, and since each entry generates only one of two states in the UI, all but two checks made by this test will be unnecessary.

Mr. Archer goes on to note a number of ways that this test is fragile. I suggest that the cases involving bad data in the source file are excellent charters for exploratory testing, but a poor idea for an automated test. An automated UI test that simply checks that the two states are available without reference to any outside test data is perfectly adequate for this situation.

In his second example, Mr. Archer once again iterates over a large number of source records in order to achieve very little. Again, exercising the same UI elements 100 times using different source data is wasteful, since all the UI test should be doing is checking that the UI functions correctly. However, there is an interesting twist in Mr. Archer's second example that he fails to notice.   If Mr. Archer were to randomly pull a single source record from his list of 100 records for each run of the test, he would get the coverage that he seems to desire for his 100 records over the course of many hundreds of individual runs of the test. I took a similar approach in a test harness I once built for an API test framework, and I described that work in the June 2006 issue of Better Software magazine, in a piece called "Old School Meets New Wave".

Both of Mr. Archer's examples violate two well-known UI test design principles. The first principle is called the "Testing Pyramid for Web Apps".   As far as I can tell, this pyramid was invented simultaneously and independently by both Jason Huggins (inventor of Selenium) and by Mike Cohn. (Jason's pyramid image is around the web, but I nabbed it from here.)

Any test that reaches out to the file system or to a database is going to belong to the middle tier of business-logic functional tests. And even then, most practitioners would probably use a mock rather than an actual file, depending on context. While it is not always possible for UI tests to avoid the business-logic tier completely, it should be the case that UI tests are in fact focused on testing *only* the UI. Loops and conditionals in UI tests are an indication that something is being tested that is not just part of the UI. Business logic tests should to the greatest extent possible be executed "behind the GUI". From years of experience I can say with authority that UI tests that exercise business logic become both expensive to run and expensive to maintain, if they are maintainable at all.

The other principle violated by these examples is that highest-level tests should never have loops or conditionals. The well-known test harness Fitnesse does not allow loops or conditionals in its own UI. Whatever loops or conditionals may be required by the tests represented in the Fitnesse UI must be coded as part of the fixture for each test. For a detailed discussion of this design pattern, see this post by Gojko Adzic: "How to implement loops in Fitnesse test fixtures"

close reading/critical thinking

2010-11-21T09:43:00.000-07:00

The last Weekend Testers (Australia/New Zealand) was brilliant. Let me urge you to read Marlena Compton's report and the transcript of the session.

This sort of practical implementation of critical theory is long overdue in the testing community, and the WTANZ crew did a great job of using a well-known theoretical tool to analyze and dissect some real problems in some real work.

Compare what WTANZ did with Zeger Van Hese's recent demonstration of deconstruction.

This sort of work, bringing reputable and sophisticated critical theory to bear on actual testing and QA activity, is a field wide open, barely explored, and long overdue.

May we see much more of it soon.

more on certs, more numbers

2010-11-16T19:08:00.000-07:00

I noticed (thanks Twitterverse) that there was an interview with Rex Black over on the UTest blog. In that interview he reveals a very interesting number:

"...the ISTQB has issued over 160,000 certifications in the last ten years."

Using the numbers from my previous post: if we assume that there are about 3,000,000 software testers in the world right now, and if we issued 160,000 certifications right now, that would mean about 5 certifications for every 100 software testers.

I would be willing to bet that there were about the same number of testers ten years ago: Y2K was just over and the value of dedicated testers had been shown.   But as Alan Page and others have noted, there is a lot of turnover, a lot of churn, among those practicing software testing.

So my numbers start to get a little sketchy here, I don't have anything to back them, so consider this a thought experiment: as noted above, let's say that there were about 3 million testers a decade ago and there are still 3 million testers today. Let's say half of today's testers have started since 2000.   This gives us a field of 4.5 million testers who could have acquired a certification in the last decade. This makes for about 3 certified testers for every 100 possible certifications.

I think it is an excellent bet that a significant fraction of those 160,000 certifications were issued in the UK, Australia, and New Zealand.   Just to make it even, call it about 1/3, put 60,000 certs in those regions, leaving 100,000 for the rest of the world. That brings us down to about 2 certs per 100 testers.

But that still seems high to me. I might have missed something. Regardless, it still looks like a pretty small market, and I'd bet the market has been shrinking a lot with the rise of agile adoption and the economic downturn.

an object of interest

2010-11-11T18:01:00.003-07:00

I bought this recently at a guerilla art show:

Here it is hanging in my office:

The poster caught my eye because I've loved the Alice books all my life and I re-read them often. I am especially fond of the Tenniel illustrations, and the one for Jabberwocky is a favorite.

The poster also caught my eye because of the odd and interesting typeface. The story behind that typeface is fascinating. I asked the artist to send me that story in email so that I could have it written down:

The story is this:
Just south of where I grew up (near Green Bay, WI) is the Hamilton Wood Type Museum. A while back, I visited armed with a few sheets of printmaking paper with the goal of printing some or all of the Jabberwocky poem from some original wood type. During the course of the 19th and 20th century Hamilton made wood type for advertisements and headlines and circuses and had gone on to accumulate wood type from other manufacturers who had given in to market pressures or the eventual obsolescence of the letterpress industry. What remained when I visited were cases and cases of cabinets full of uncategorized type.. roughly 1000 different type faces and sizes. I spent a better part of a day just finding a type face I liked, using a rather capricious method to determine "the one": the style of the lower case 'g'. I estimate the type to have been produced in the early 20th century, probably for about 20 years, if that. It is an obscure, unnamed type face. I set the type but realized that by choosing according to lower case 'g', I had picked a case that only let me set 3 lines of text. This was all that was left of this type in existence. So, I printed the top three lines first on a large flat bed cylinder press called a Vandercook 325G (incidentally I have the exact same model press in my shop here), disassembled the text and composed the 4th line. When I returned to Colorado, I replicated the illustration from Through the Looking Glass and then added that to the print.
That's the story.
Enjoy
Dan

Zeger Van Hese is a Belgian software tester who, like me, is interested in critical theory and what application critical theory might have to the work of creating software. The other day he mentioned in passing a seminal work by Walter Benjamin, The Work of Art in the Age of Mechanical Reproduction that I had not read in many years.

In the light of Benjamin's work, my poster is a strange object indeed. While it was created in a process of mechanical reproduction, it was created only once. The means to create it are lost in an anonymous bin in an obscure warehouse somewhere in Wisconsin. And even if someone were dedicated enough to find that one particular bin, not even enough of this particular wood type exists to even print all four lines from Jabberwocky.

My poster would have been a strange item even for 1936, when Benjamin wrote about mechanical reproduction. But to have such a thing on my wall in 2010 is, for me, astonishing.

XP, Scrum, Lean, and QA

2010-11-06T10:37:00.002-06:00

Before I do this, two things up front: for one thing, I am a crappy programmer. I read code far better than I write it, and I read non-OO code better than I read OO code. Also, I am writing as someone who knows a lot about Quality Assurance and testing, and very little about the hands-on day-to-day work of modern programming. So here goes:

As a QA guy, I know this: long before Scrum and XP and the Agile Manifesto, people working in Computer Science and software development knew three things about software quality: having an institutional process for code review always improves software quality; having automated unit tests (for some value of "unit") with code coverage improves software quality; and having frequent, fast builds in a useful feedback loop improves software quality. Sorry, I don't have references handy, I read most of this stuff about a decade ago. Maybe some heavy CS people could leave references in the comments.

The XP practices simply institutionalize those three well-known practices, and for the time, dialed them up to 11. Pair programming is just a very efficient way to accomplish code review. TDD is just a very efficient way to accomplish unit test coverage. CI is just a very efficient way to get a fast feedback loop for the build.

There is nothing magical about these practices, and I have worked on agile teams that don't do pair programming but do awesome code review. I have worked on agile teams whose microtest suite runs heavily to integration tests instead of traditional unit tests. I have worked on agile teams with a dedicated build guy. I started my career working in an incredibly well-designed COBOL code base. No objects in sight. Had I known then what I know now about test automation, I could have written an awesome unit/integration test framework for that system. The XP practices themselves are not sacred. The principles behind those practices are.

But the XP practices themselves are just a small piece of having a successful agile team. In musical terms, these are the equivalent of knowing scales and chords, just basic technical practices necessary to get along in the business. Of course they are not necessary: The Ramones and Tom Petty have only a basic grasp of the technical aspects of music, but they cranked out some monster hits. Put any of those guys in a jazz jam session or a symphony orchestra and they would be completely lost. There is some nasty software out in the world that makes a lot of money.

I like Scrum, for a number of reasons. For one thing, it has an aesthetic appeal. The concepts of developing, then releasing, then retrospective speaks to me strongly, not least because they map closely to the ideas from the performing arts of practice, perform, rehearse.

I also like Scrum because of its emphasis on human interaction rather than institutional process. Scrum is lightweight by design, and leaves much room for people to act as people with other people. Scrum favors mature, intelligent adults.

Finally, I like Scrum because it is a process derived directly from the actual practice of creating software. It is described in plain English and it relies on no special concepts. It was crafted out of whole cloth by good developers in a tough spot.

I dislike Lean/kanban for the same reasons. As a mature adult, I resent having any of my activities identified as "waste". I resent not having the end of an iteration to celebrate. I resent being treated as a station in a production chain.

Unlike Scrum, the Lean principles were not derived from the actual work of software development. They came from automobile manufacturing, and were overlaid on software development in what I consider to be a poor fit. Putting on my QA hat again, there are two other popular software development methodologies that came directly from manufacturing, and the state of those methodologies is instructive. One of them is ISO9000. The fatal flaw of ISO9000 is that once a process is in place, it becomes very difficult and expensive to change that process. This is fine in manufacturing, but it is death to a reasonable software development process. The other methodology from manufacturing is Six Sigma. Six Sigma is very expensive, and while it might yield information valuable to managers, it provides no benefit to those actually doing the day-to-day work of software development. I am not aware of any manufacturing processes shown conclusively to improve the hands-on work of software development.

XP and Scrum are not nearly enough to guarantee a successful software project. For a comparable situation, just because a band has a rehearsal schedule and some gigs does not guarantee that they will be international superstars. Brian Marick at one point talked a lot about four principles that also increase the chance of a successful software project: skill and discipline, ease and joy. I won't explain those here, interested readers can find that work themselves.

But beyond even skill, discipline, ease and joy, a successful software project requires that we as creators of the software reach out and interact with the world in a way that changes the lives of those who encounter our software. It is an awesome power. In some cases, we can make our users' lives a living hell. But it's a lot more fun to make everyone happy.

ignoring certification; with numbers

2010-10-29T10:47:00.004-06:00

All of the questions about tester certification were answered many years ago. They exist and they cannot be made to unexist. The only remaining question on the subject is: how many tester certifications can be sold? And the answer to that question doesn't matter to anybody except the people selling the certifications.

A while ago on the writing-about-testing mail list we did a little exercise to come up with some back-of-the-napkin estimates about the number of software testers in the world. We used US Department of Labor data and also some other public information about software employment worldwide. We also had access to some privileged information about magazine subscriptions. In addition, a number of us have done serious work in social networking, and we have some analytical tools from that work to help estimate. Using all of that, we came up with a pretty consistent estimate that there are probably around 300,000 software testers in the US, and maybe 3 million in the whole world.

That is a pretty small market in which to sell tester certifications.

Elisabeth Hendrickson recently did a fascinating analysis of QA/testing job ads. According to her data, it is a good bet that 80% of the people doing modern software testing work in the US have programming skills of one sort or another.

Jason Huggins of Sauce Labs has been tracking job ads that mention browser automation tools. Jason notes a remarkable recent increase in the demand for Selenium skill. You can see the trend for some popular automation tools at indeed.com. The QTP vs. Selenium trend in job ads is fascinating, but looking closely, this graph indicates a general across-the-board increase in demand for technical skills in traditional UI-based software testing.

Finally, sorry I don't have a link handy, but I have seen a number of reports of a radical increase in the rate of adoption of agile practices among US companies of every size and description. And the agile whole-team approach to software development makes dedicated, siloed traditional V&V test departments irrelevant.

The existing tester certifications simply do not apply to this sort of work. Certification is becoming more and more useless to US testers, and to their employers as well.

I feel like I am pretty plugged in to the world-wide tester community and the world-wide agile community, and anecdotal evidence suggests that indeed, the market for tester certification in the US is very small. Again, this is anecdotal evidence, but the hot spots for certification seem to be the UK and Australia/New Zealand, possibly areas of Southeast Asia, possibly areas of Eastern Europe. Once more with the anecdotal evidence, but I would suggest that in political climates that favor a high degree of regulation of business practices, certification will be more popular.

So if we eliminate from our worldwide tester population of 3 million the majority of US testers and a significant fraction of the rest of the world as potential buyers of a tester certification, that leaves a pretty tiny market for tester certification.

I think we can say with some confidence that professional tester certification can safely be ignored by the vast majority of software testers. That said, if you are required to get a certification, or if you just want to get a certification, go ahead and do it. It won't hurt you, and at the very least, you'll learn how software was tested in 1996.

I think we can also safely say that any supposed controversy surrounding tester certification is overblown and can also be ignored.

Which suggests one more interesting question: if the supposed controversy over certification really is as trivial as these statistics indicate, then why does so much of the testosphere spend so much time agonizing over it?

I have a cynical answer to that, but I'll keep it to myself.

Update: made the links nice

Call for Participation: Second Writing About Testing peer conference

2010-10-25T16:16:00.002-06:00

The Second Writing About Testing Conference: Frontiers for Software Testing

I am pleased to announce the call for papers for the Second Writing About Testing Conference to be held May 13 and 14 2011 in Durango Colorado.

For more information about the original conference and the Writing About Testing mail list please see:
http://chrismcmahonsblog.blogspot.com/2010/02/writing-about-testing-listconf-update.html
http://chrismcmahonsblog.blogspot.com/2009/10/cfp-peer-conference-writing-about.html

Writing About Testing is a peer conference for those interested in influencing the public discourse on software testing and software development by writing and speaking on those subjects. The discussion revolves around blogging, giving presentations at conferences and user groups, and writing for established media outlets, both for pay and for other reasons. There will very likely be representatives from established media outlets attending. Having software writers and publishers talking to each other face to face is a unique aspect of WAT.

For the first WAT conference we asked only that participants be interested and have had already written about software. The second WAT will be a little different.

For the second WAT we ask that applicants propose a talk (informal is fine, no slides required) of 30-45 minutes about some subject critical to their work as a tester or developer, or designer or business analyst, but which is not generally recognized as being part of such work. A list of suggestions for such talks is below. The purpose of this is to expand the practice of software beyond the current artificial boundaries of recognized software activity.

To that end, presentations on these subjects are not welcome, unless the presentation has obviously unusual aspects:

test heuristics/mnemonics
exploratory testing
classic test automation (record-and-play, automation pyramid, etc. Information on unusual approaches to test automation is welcome.)
Scrum/Lean/Context-Driven/whatever. In general, anything involving capital letters is not welcome.
certification (for or against)

In a nutshell: don't bore the other attendees with stuff that has been discussed to death for years. This conference is to discuss frontiers.

Given that the presentations will be on unusual subjects, there will be a minimal requirement for having published previous work. Any publicly available source of writing, for instance a blog, would qualify an attendee. Applicants with no publicly available writing at all will not be considered. Attendees are encouraged to write with ambition and daring after the conference ends.

Attendance will be limited to about fifteen people. There will be a nominal fee of $50.00 per person to help cover room rental, and lunch will be provided. A discounted rate at a convenient bed and breakfast hotel is available.

Durango has a lot to offer visitors. Conference attendees may wish to arrive early or stay late to take advantage of the nearby attractions: beautiful mountains to the north and east, desert sandstone canyons to the west and south. The steam train from Durango to Silverton and back is a fantastic experience, as is soaking in the hot springs nearby. Within a short drive are Mesa Verde National Park, Canyons of the Ancients National Monument, Monument Valley and the Navajo Reservation, Great Sand Dunes National Park, etc. Local opportunities for hiking, biking, and boating abound.

There are direct flights to Durango from Denver and from Phoenix. Many attendees will likely come from the Denver area, so carpooling from there may be possible.

To submit a proposal, either send a message to me at christopher dot mcmahon on gmail, or join the writing-about-testing mail list at http://groups.google.com/group/writing-about-testing and submit your proposal there.

The deadline for submissions is Jan 1.
Invitations to the conference to be sent Feb 1.
The conference itself will be May 13/14.

Here are some possible frontier subjects for presentations:

System Administration

There is a surge of interest in recent times in a concept called "DevOps". DevOps proposes an alliance among software developers and system administrators in order to create the best possible production environment experiences. Testers need to be a part of that conversation.

Data Visualization

Not only our applications, but the whole world around us generates incomprehensible amounts of data, and the only way to make sense of it all is to render that data in a visual or tactile fashion. Testers need to understand these technologies in the service of their work.

Frameworks/Mashups

Good test automation today happens at every level. A single framework may exercise the user interface, call REST or SOAP APIs, and reach into a database, all in the course of a single test suite. Myriad tools for such testing exist, and knowing how to get such tools to talk to each other for a particular purpose is becoming a critical skill for testers.

User Experience

Great strides have happened in user experience work in the last few years, and there are exciting advances on the horizon. Testers have largely ignored the conversation happening among user experience experts.

Web Design

New javascript libraries like jquery are making things possible in browsers that were unimaginable just a few years ago. Flex, Flash, and Adobe Air are making huge inroads into application UI design. HTML5 is on the horizon and looming. Testers need to know what is happening in this arena, and largely do not.

Web Services (REST/SOAP)

Twitter, Facebook, and the bleeding edge of web applications are no longer about the UI. Today it is all about the APIs, and the third party applications that use those APIs to bring killer experiences to users. Testers need to know how web services and APIs work.

Environments/VMs/Cloud Computing

Managing test environments has always been challenging. New cloud computing services in some ways make such work more challenging, but the reward is a vastly simplified process for provisioning test environments. This work needs public exposure.

Agile Methods

Agile methods work, but even today, no one knows exactly why. The explanations we have are frequently facile and often abused. Testers could be the ones to provide the well-considered explanations for the effectiveness of agile methods.

Process Work/Quality Assurance

QA has a bad reputation in the testing community that it does not deserve. I have said before on stickyminds.com and in Beautiful Testing that QA is not evil, that it is work that still needs doing, and that often testers are in a good position to provide quality assurance. Bring back real discussion about Quality Assurance.

Aesthetics/Artistic Performance

There is a wealth of knowledge available from disciplines within the Liberal Arts that apply directly to software development. Testers can help bring that knowledge over to the world of software development.

One Year Writing for SearchSoftwareQuality.com

2010-10-03T18:53:00.001-06:00

I generally do not post links to pay-wall or registration-wall sites, but today I am sincerely happy and proud to publish a link to SearchSoftwareQuality.com.

For each of the last twelve months, I have written at least two 1000-word articles for SSQ. In the last year, SSQ has published nearly forty individual pieces of mine, a book-size body of work.

I am sure some of those articles are better than others, but I wrote every one to the best of my ability with all sincerity, and I truly believe that every one of those articles contains at least one interesting idea intended to help people working in software testing and software development.

I would particularly like to thank my editors at SSQ, at first Jan Stafford, later Yvette Francino. The SSQ editorial staff is professional and efficient. Both Jan and Yvette have given me an enormous amount of freedom and encouragement over the last year, and it has been a real pleasure working with them both. I especially appreciate their tolerance on the few occasions when I pushed that freedom to the limit.

I would also like to thank the Writing About Testing mail list. A great number of people on that list have been immensely helpful, freely giving comments and constructive criticism, providing new ideas, and just being generally smart and encouraging human beings. In the very near future we will be announcing the CFP for the second Writing About Testing conference in the spring, which should bring even more new ideas and new voices into the public discussion of software development and testing.

Finally, I want to thank Matt Heusser specifically. Matt introduced me to SSQ a year ago. I would never have had this opportunity if it were not for his generosity.

not about testing: a bit of writing

2010-08-31T09:12:00.004-06:00

I've been neglecting my blog, mostly because I have been doing a whole lot of professional freelance writing on the subject of software dev and test, and really enjoying it a lot.

A few months ago I also submitted a piece to the Mountain Gazette, one of my favorite magazines, available for free around the West. They always publish really good writing.

Mountain Gazette was soliciting pieces on the subject "My Favorite Mountain". To their surprise, they got more than 200 submissions, of which they could only publish 11. I submitted a piece, it was rejected, but I don't mind, I've been reading the issue, and there are some really great essays.

So since it isn't going to appear anywhere else, I figured I would publish it here:

----

I don't have very far to go to get to my favorite mountain. I go out
my front door and take a right, and I walk about a mile through my
neighborhood of mostly middle-class houses, some Victorian, some like
mine vintage 1930s-40s, a few more modern. I say "hello" to my
neighbors as I work my way a little uphill to the trailhead near the
electrical transformer station.

The way up the west side of the loop trail is kind of a slog, moving
from about 6500 feet of altitude to about 8200 feet through pine
forest and the occasional meadow, but every once in a while a mountain
biker comes barreling down from the top and catches some air on a nice
jump. That's fun to watch. I used to be pretty good on a mountain
bike, but the lure of adrenalin doesn't call so strong now. Now I get
a thrill holding one of the horned toads that seem so common on this
mountain but that I never find anywhere else.

Nearing the top I get peekaboo glimpses of the La Plata range to the
northwest, so I know I'm getting close. The forest opens up, and a
final climb takes me to the overlook where I can see the La Platas to
the northwest, the 14ers of the Weminuche Wilderness off in the
distance to the northeast, and all of the river valley laid out below.
There are always birds moving across the sheer cliff below my feet,
often crows, sometimes buzzards, once or twice eagles. Sometimes a
man-made glider works the thermals over my head.

My mountain isn't one of those tall craggy ones like on the Coors beer
label, it's an uplifted slab of shale and sandstone, and coming down
the broken edge of the slab is the best part of the walk. The view
gradually shifts from the north to the east, hiding the tall peaks but
revealing broken sandstone ridges marching off into the distance. The
college sits below one ridge, and my town is laid out along the banks
of the river moving downstream to the south, where the sandstone
canyons start.

When I finish the loop around my mountain, if I don't feel like going
straight home, just a couple blocks out of my way is the tap room for
one of the local breweries, a fine place to finish off a four-hour
walk on a nice afternoon on my favorite mountain.

Writing About Testing wrapup

2010-05-30T14:55:00.001-06:00

On May 20 and 21 some of the brightest people in the field of software testing met in Durango Colorado for the first ever Writing About Testing conference. We participated in a diverse set of activities: formal presentations, ad-hoc demonstrations, collaborative exercises, lightning talks, and informal discussions of topics of interest that ranged from the role of media, to finding the time to write.

I started my software testing career in the bad old days of the mid-1990s. Both Open Source software tools and agile methods were highly controversial at the time. And while many of us were doing amazing and innovative work, the entrenched culture of software development was highly skeptical that what we were accomplishing was valid, or even sane. I think there is a real danger of a return to those days, and I wanted to create a community where people working out on the edges of software creation could hone their ideas in a supportive community, and from what I saw at w-a-t, that community now exists.

Open Source and Agile both succeeded for three reasons: they fostered a laser focus on the technical aspects of software tools; created general support for communities of dedicated practitioners; and provided philosophical/theoretical frameworks within which to accomplish the work. And the information coming out of the Open Source and Agile communities was so valuable that the institutional trade media was forced first to pay attention, and then to participate actively in the promotion of those cultures.

At the Writing About Testing conference we discussed REST architectures and wiki-based test frameworks like Selenesse. (All three principals of the open source wiki-based test framework Selenesse were in the same room.) We discussed data visualization and the challenge of managing enormous datasets.

We discussed new ways of working being discovered and propagated from places like Agilistry and from within particular companies like Socialtext, 42lines, and others.

We discussed new ways to consider software users and consumers, and the implications of the increasingly common phenomenon of near-real-time interaction with those who enjoy and depend on our software.

We discussed what it means to actually do the work of software testing today, in the real world.

We discussed a lot of other stuff, too.

The most important thing I learned is that as software becomes more ubiquitous in the world, the work of software development is becoming radically diverse, as are software business models, as are the skills necessary to be successful in creating software. This has particular implications for software testing. Both the practice of software testing itself and the hiring of software testers are undergoing significant changes, with no end to the evolution in sight.

The software tester of the future will no longer do one thing in one way. The software tester of the future will be expert in some aspects of software creation. Testers will seek out teams that need someone with their particular set of skills and expertise, and teams will seek out people with particular sets of skills and experience to maximize the benefit to the users of their software. Some of the areas of expertise represented in the room at w-a-t:

deep database knowledge, framework programming, and exploratory testing
API and architecture expertise, user experience testing, and process knowledge
system administrator skill, scripting/development ability, and multibyte character processing knowledge
management experience, programming and architecture expertise
software security and software performance
data wrangling, visualization knowledge and deep experience in online communities
business expertise and business communication skills
Quality Assurance. As I've noted before in a number of places, QA is not evil.

Software testers of the future will invest in a range of skills and experience, and the teams that hire them will audition software testers based on their ability to use those skills and that experience to further the goals of those teams. Software testers who do only one thing in only one way will be relegated to the sidelines, doing an increasingly limited sort of work for a diminishing number of jobs.

It would not surprise me to see the term "software tester" itself gradually disappear over time. Instead, those of us who call ourselves "testers" will more and more say instead "I am an expert in X, Y, and Z, and I have a deep interest in A, B, and C. If that mix of skill and experience is what your team needs, then you need me to work on your team."

Those of us writing about testing face some interesting challenges. In the 90s the major communication channels were the trade publications and the research consultancies. Those organizations still swing a very big stick, but two trends seem very clear: for one thing, the cutting edge has moved away from the big institutional publications, out onto blogs, social networks, and loosely-organized communities of practice; at the same time, the major media have become more conservative, and are generally less likely to publish controversial or cutting-edge work. But that means that major media are caught in a bind, because as it becomes more attractive to publish highly original work outside the major media channels, the major media channels find themselves hungry for content. The entire situation is very fluid right now, and that provides remarkable opportunities for new voices in software to be recognized quickly.

It is an interesting question whether or not there will be a second Writing About Testing conference. Right now enthusiasm is high, but I wonder if a second conference would have as much impact as the first one did. For now I am postponing a decision on whether to pursue a second conference next year. I have not abandoned the project; over the next six months or so I will be talking with the original participants and with potential future participants to see if a second conference next year would be valuable to those of us working in software testing in the public arena. In hindsight, there are a few things I would do differently the second time around, and I suspect that I will get a lot of ideas from others as well.

On a personal note, I am immensely pleased and proud that the Writing About Testing conference and the community that sprang up around it have been so successful. I have invested a lot of energy in w-a-t over the last six months. After the conference ended, I went on a 5-day backpack trip in the remote canyon country of SE Utah to clear my head and reflect on it. I am fascinated by the ruins and artwork left by the Anasazi in the remote canyons of this part of the world, the most recent of which is about 800 years old, and the oldest of which I just saw is about 7000 years old. There is a mental phenomenon known to people who make such trips called sometimes "re-entry". After spending significant time in a very remote desert region contemplating the remains of a culture that thrived from 5000 BCE to 1300 CE, adjusting again to a world of streets and lights and computers can be jarring.

In the light of re-entry, Writing About Testing was a very good thing.

watch your language

2010-05-05T21:44:00.002-06:00

For a number of years I've been writing about treating great software development as a very specialized subspecies of the performing arts.

Some time ago I reviewed a piece of writing from a software person inspired by the concept of artistic software, but who had no background in the arts at all. It showed: the most egregious error was that instead of using the term "performing arts", this person used the term "performance art". The rest of the piece was earnest but the author's lack of expertise (in art, not in software) was painfully obvious.

The performing arts are music, theater, and dance. Performance art, on the other hand, can be dangerous stuff.

But artistic software development is only a minor representative of a number of new concepts in the field bubbling madly just behind the zeitgeist. For example, methods of harnessing immense amount of data in order to make them comprehensible to human beings are about to change all of our lives, both in software development and in the world at large. Professionals working in the field refer to this as "data visualization", or as "visualization" in the broader sense, which encompasses a wide variety of technical endeavor.

A diagram is not visualization, just as performance art is not the performing arts. To misuse such terms not only spreads ignorance and misconception, but is also a grave disservice to those experts actually working in such fields.

Consider a few terms that once pointed to specific concepts and practices, but which today are laughably devoid of meaning: "Enterprise"; "Web 2.0"; "Service Oriented Architecture"; and "Agile" is coming up fast.

If you plan to use a technical term, please be familiar with the concepts that underlie the term. If you (mis)use a technical term because you heard it somewhere and you think it sounds cool, you do a grave disservice to your colleagues actually working in those trenches.

bad agile estimation

2010-03-27T11:20:00.005-06:00

Depending on how you define the term, I have been on at least five and as many as seven agile software teams. Two were brilliant; two were poisonous; the rest were just flaky. A big part of the poison stems from not understanding how to do agile estimation.

This is part of a message that showed up on a mail list I lurk on:

I'm working with a team that does great work. They are skilled and
work well together.

They also average about 50% or less in meeting their sprint
commitments. And don't seem to mind.

"There's a lot to do we just didn't get to it all."
"We'll do that in the next sprint."
"Yeah, that's not working yet."

These are the kinds of statements during the sprint or in the retrospectives.

How do I help this team look at the problem to solve it, instead of
just living with it?

Since the list name has the word "Scrum" in it, I will assume this person is a Scrum Master.

The first misunderstanding here is to know that all estimations are wrong. But when we do estimation in an agile way, we find over time that as we come to share a consistent idea of what a "point" means, that our estimations will be consistently wrong in the same way. This allows us to plan with a high degree of accuracy.

So if you have a agile team that consistently estimates 20 points per sprint and achieves 10 points per sprint, then the capacity of the team is 10 points, and 10 points is the figure you need to use for planning purposes.

The term "meeting their sprint commitments" bothers me a lot. For one thing, insisting that a team complete more stories than the capacity of the team can support is a well-known recipe for technical debt and poor quality. For another thing, it's not "their" sprint commitments, it is our sprint commitments. Finally, I object to describing this situation as "a problem" for the team to look at.

Remember what the job of the Scrum Master is? The Scrum Master has only one job on the team: to remove impediments. If there is a business reason to achieve a consistent velocity of 15 points instead of 10 points, then the Scrum Master should examine the situation for impediments to remove, not try to force the team to meet some sort of imaginary capacity in order to satisfy the requirements of what is essentially an old-style Gantt chart.

Artful Making

2010-03-25T17:13:00.003-06:00

I never read business books, I mean I NEVER read business books. But after Marlena Compton read my chapter in Beautiful Testing, she recommended that I read Artful Making by Rob Austin and Lee Devin, subtitled "What Managers Need to Know About How Artists Work".

I've been writing about creating artistic software for some time now, but with a copyright of 2003, this book pre-dates my endeavors and I was surprised not to have heard of it.

Austin and Devin are professors at Harvard, Austin of business and software, Devin of theater. Early in the book they recount how they began the conversation that led to writing the book:

We were surprised to discover common patterns and structures in our separate domains... Some recent ideas and methods in software development, especially in the so-called "agile" community, seemed almost identical to theater methods. As this became more obvious, an idea dawned on business professor Rob: These artists are much better at this than we are. (emphasis NOT mine)

They cite four qualities of artful making: Release, Collaboration, Ensemble, and Play. They address all of these qualities well except for Ensemble, of which more in a moment...

They devote a significant portion of the book to distinguishing knowledge work from industrial work, which is very much worth reading. Then they assuage managers' concerns about security, uncertainty, and fiscal responsibility.

I have three criticisms of the book. One is based purely on my own personal biases, another is a gaping hole in their argument, one that does not get the attention it deserves. The third is more or less wishful thinking on my part.

First, this is not a book for practitioners. This is a book for executives who wander by the agile team room and wonder why it's so noisy and the floor is covered with index cards and there is a box of Lucky Charms on the table and the scrum master is wearing a viking hat. It does not tell you how to do artful making, it only tells you what artful making looks like.

Second, the book glosses over two very important points related to actual practice. The first one is explicit, in the conclusion, discussing the quality of Ensemble, they say "An ensemble at work on a project is a group that exhibits the quality of Ensemble". They admit the tautology, but they have very little to say about how to foster or even recognize the quality of Ensemble in a group. The second point that gets even less attention than "ensemble" is talent. It is impossible to succeed at artistic performance without a talented group of performers. I assume it is difficult for managers to acknowledge that their workers lack talent; and it is very difficult to define what makes one person talented and another not; but I found the lack of discussion of the talent of those doing the work somewhat disturbing.

Finally, and ultimately, the book does not go far enough. On page 40 there is a graphic with the title "Characteristics of Artful Making in Agile Software Development and Play Making" with a lot of common practices. What I would much rather see is a graph entitled "Artistic Performance" listing common practices of Theater, Dance, Music, and Software Development. I'm going to keep working on that graph.

take responsibility for UX

2010-02-04T07:44:00.003-07:00

I am really starting to dislike the term "User Experience", but I'll get back to that.

In the mid-90s I was a bass player in the acoustic music scene in the South, living in Atlanta. If you happen to know Atlanta, to give you some perspective, my band opened New Years Eve at Eddie's Attic in 1994, and headlined New Years Eve in 1995 and 1996.

Eddie's Attic was and still is one of the most important and influential clubs on the acoustic music circuit in the South. Also on that circuit is a club in Nashville called The Bluebird Cafe.

The Bluebird is interesting because it enforces a strict no-talking policy. If you talk to your companions at all during a performance at the Bluebird, you are asked to leave the room.

At one time back in the 90s there was an intense discussion among people on the acoustic music scene as to whether Eddie's should implement a no-talking policy like the Bluebird's. As far as I could tell, the musicians who advocated the most for such a policy were the mediocre performers (you really couldn't get onstage at Eddie's if you were outright bad). The more that the performers had inattentive audiences, the less able they were to command a stage and hold the attention of the audience, the more likely they were to support a no-talking policy.

At the time there was a low-circulation newspaper devoted to the Atlanta acoustic music scene that interviewed me on the subject. In that interview I said three things: first, that if you intend to get on stage in the first place, it is your job to command that stage and compel the audience to listen solely by means of your own talent; second, that if you consistently have talkative audiences that don't pay attention, then you should either work on improving your performance, or else stop performing at all; third, that a no-talking policy robs performers of valuable feedback during the course of the performance.

I dislike the term "User". I think the word "user" has bad connotations and associations. I think it is too easy to turn "users" into "lusers" in our own minds. I far prefer the term "audience" to describe those who consume our software. It is only inanimate objects that have users. Performers have an audience.

Given that preference, I think the use of personas as proxies for types of users may not be a very good practice. It seems far too easy to exclude valuable segments of the potential audience and to miss valuable feedback by blinding oneself by the limits of the particular personas being considered as users.

I think a far more interesting and valid approach to UX work is to do things like instrument servers so that we can track in real time the activity of the largest groups of people using our software that we can muster, and to have that activity influence our development and delivery work as nearly instantaneously as possible. And of course, doing that also lets us know when the audience is not paying attention. This is exactly the feedback loop that audience applause provides performers on stage.

Writing About Testing list/conf update

2010-02-01T10:45:00.000-07:00

The writing-about-testing mail list began in September 2009, and has already played a part in a remarkable number of achievements:

The following writers appeared in print for the first time or contracted to be published in print for the first time:

Abby Fichtner
Dawn Cannan
Catherine Powell
Lanette Creamer
Parimala Shankaraiah

Lanette Creamer and Matt Heusser collaborated on an article about test automation in waterfall and agile projects.

Alan Page, Matt Heusser, and Marlena Compton published the "Code Coverage Cage Match" collaboration. http://www.stpcollaborative.com/knowledge/538-heusser-v-page-code-coverage-cage-match

Matt Heusser started a book project "Testers at Work".

Adam Goucher signed a contract for a book on Selenium.

Fiona Charles edited the Women of Influence issue of Software Test and Performance, with contributions from many list members. http://www.stpcollaborative.com/magazine/year/2010

Yvette Francino landed an editorial position at SearchSoftwareQuality.com

Dawn Cannan and Lisa Crispin collaborated on a piece for Agile Record #1. http://www.agilerecord.com Dawn's first conference presentation ever was in Second Life. http://www.passionatetester.com/2010/01/welcome-to-virtual-agile-world.html

If you would like to join the list, contact me or any other member for information. If we don't know you or your writing, please point out links to some public examples like blog posts, public documentation, conference papers, or similar work when you do so.

The deadline to apply for the Writing About Testing conference is now past. The conference is completely full with 15 attendees. Any future applications will go onto a waiting list. The attendees will be:

Elisabeth Hendrickson (http://testobsessed.com/)
Lisa Crispin (http://lisacrispin.com/wordpress/)
Geordie Keitt (http://tester.geordiekeitt.com/)
Marlena Compton (http://marlenacompton.com/)
Rich Hand (http://www.stpcollaborative.com/)
Joey McAllister (http://www.stickyminds.com)
Marisa Seal (http://thetestingblog.com/)
Jon Hagar (http://www.swtesting.com/)
Ben Simo (http://www.questioningsoftware.com/)
Marisa Burt (http://www.burtconsultinginc.com)
Lanette Creamer (http://blog.testyredhead.com/)
Yvette Francino (http://www.yvettefrancino.com)
Rick Scott (http://rickscott.posterous.com/)
Dawn Cannan (http://www.passionatetester.com/)
Chris McMahon (http://chrismcmahonsblog.blogspot.com/)

looking back, looking forward

2010-01-01T10:18:00.004-07:00

Looking Back

The world of software testing today is radically different than it was on this day a decade ago. On New Year's Day 2000 I had been a dedicated software tester for about three years. I was a leader on a team testing an application that provided location information to the dispatchers who handle 911 calls. I was intensely interested in the most progressive thinking about software testing available, because when we released a bug to production, someone could die because of it.

I remained interested in the most progressive thinking about software testing throughout the decade. Looking closely, we owe a vast debt to three people: James Bach, Bret Pettichord, and Brian Marick. If they didn't supply every breakthrough idea in software testing in the last decade, one of them was nearby when it happened.

There was a shot across the bow in 1996 when Bach published "Test Automation Snake Oil". This would be the first of a relentless assault on proprietary test tool vendors and the intellectual bankruptcy of the approach to testing that such vendors were selling.

In 2001 Pettichord published the incendiary "Hey Vendors, Give Us Real Scripting Languages". These articles and others that followed were the beginnings of the big push not only for open source test tools, but also for encouraging testers to increase their skills to make the best use of such tools. The tools themselves were still some years away, but the conversation started here.

Meanwhile, not long before that in 2000, Bach and his brother Jonathan had published "Session Based Test Management" in a magazine called Software Testing and Quality Engineering, known today as Better Software. The ideas fueling Exploratory Testing had been around for some time at this point, but skepticism was widespread that ET was a worthwhile test approach. SBTM was the first treatment of a large-scale measurable ET process. To my mind, this single article turned the tide in making ET a valid and acceptable practice.

Automation and ET are the touchpoints for how we work today. But at this time we had no guidance on when and how to use them. Then 2002 saw the publication of Lessons Learned in Software Testing by Cem Kaner, Bach, and Pettichord. A lot of us had read Dr. Kaner's earlier work, and Beizer, and quite a number of other books on software testing, but we found that those practices mapped poorly to our own experience. While Lessons Learned has gotten a lot of valid criticism over the years, that hodgepodge of techniques and approaches and theoretical frameworks was the first accurate description of what it was like to be a tester on the ground early in the 21st century.

One of the tenets of Lessons Learned is the idea of "no best practices". Shortly after the publication of Lessons Learned, Pettichord began defining the Four Schools of Software Testing, of which the Context-Driven school is fueled by ET and reinforced by Lessons Learned. Context-Driven testing is alive and well around the world today.

At this point we understood ET fairly well and the Context-Driven school of software testing was far and away the best match for our actual day-to-day working experience. But the lack of open-source test tools was galling. Dedicated testers were falling farther and farther behind developers in terms of the power of their tools.

In 2003 Pettichord and Brian Marick began teaching their 'Scripting for Testers' course. Marick had written a small time-clock application, and they used a library called "iec" created by Chris Morris to drive Internet Explorer and automate some tests of the application.

I downloaded SfT a number of times but could never get it to work properly. I begged several managers to let me attend SfT but I never got to go in those early days. Later I would teach SfT myself on three different occasions, at the STARWest conference, the STAREast conference, and at Agile2006.

In 2004 a programmer named Paul Rogers took the SfT code base and rewrote it from scratch. Pettichord and Rogers released the rewritten SfT as Web Application Testing in Ruby aka Watir. I was the first user of Watir. Watir was the first open source web-testing tool robust enough for actual production work. I was responsible for many of the early enhancement requests, particularly for frames and iframes. To my knowledge, Watir was the first tool in history with robust support for frames.

In the meantime, Marick had been one of the authors of the Agile Manifesto, a document that would eventually change our entire approach to software development.

Also, for all of the decade so far, Marick had been the technical editor at STQE/Better Software magazine. We would devour every issue because Marick was publishing more of the most innovative, inspiring, and mind-bending stuff about software development than any other source. Marick would often solicit material for the magazine via his blog. He published my first article in Better Software in 2004 when I answered one.

Pettichord had been hosting a small peer conference called the Austin Workshop on Test Automation for some time, and 2005 was the first to feature Watir prominently. I attended that AWTA, where I met a number of people with whom I would later work on various other projects. I would also attend AWTA in 2007, one of the formative experiences of my last several years.

Pettichord went to work for Thoughtworks at about this time, where I would follow him around 2006. There we met Jason Huggins, who was developing an in-house time-and-attendance application that had an innovative test harness. Pettichord would be instrumental in getting Huggins' test harness released by Thoughtworks as open source. The project was called Selenium.

Marick left the editorship of Better Software, Pettichord now works in private industry, and Bach has turned a lot of his effort to his memoir in recent times. But without the work of these three in this last decade, the world of software testing would be a very different place.

Looking Forward

I can't imagine what the next decade will bring for software testing. But I'll make a prediction: women will be the biggest innovators in the field.

Just as Bach's "Snake Oil" article was an indicator of things to come, I think Lisa Crispin's and Janet Gregory's
Agile Testing book points the way to our future. Now in multiple printings and multiple languages, the book is a comprehensive statement about the state of the practice today, and has some hints about where we'll be going.

In recent months I've been part of a community of software testers who write publicly on the subject of software testing. A remarkable number of them are women, and a remarkable number are doing really original, ground-breaking work. Here are some women that will shape the next decade of software testing:

Marlena Compton is working with the visualization of large sets of data and with social networks.

Catherine Powell should publish her first piece, on software architecture, very soon.

Lanette Creamer has created a number of innovative approaches to software testing. She'll be making the rounds of conferences this year.

Elisabeth Hendrickson is an international superstar. She has just opened her own 'agile practice space' called Agilistry.

Marisa Seal and I have taken over Gojko Adzic's WebTest project that provides a portable, platform-independent bridge between Selenium and Fitnesse. We intend to rename the project Selenesse. Marisa recently became a committer on the Fitnesse project.

Agilistry, practice space for Agile software development

2009-12-28T15:18:00.001-07:00

Elisabeth Hendrickson and her colleagues with Quality Tree Software have announced the opening of Agilistry, their 'immersive training space' in Pleasanton California.

I've been following Elisabeth's announcements along the way to making this happen, and from what I can tell, the Agilistry facility is set up as real agile working space... except conceived and designed from the ground up by some of the most intelligent, experienced agile practitioners in the world today.

When I think of "training space" for software, I think of a trainer behind a podium with a whiteboard and a projector and some handouts, with trainees sitting at tables facing the podium.

This isn't that. Agilistry isn't training space at all; Agilistry is *practice* space. It even says so on the web site: "Agilistry Studio; a practice space for Agile software development"

And I'll bet that not many people know how to use a practice space.

For beginners, practice space is expensive but very possibly worth it. Say for instance that management informs your development organization that next month you will be "going agile". They're going to dismantle your cubicles, bring in some tables and whiteboards and index cards and sticky notes, and they expect your productivity to soar.

As a beginner, your first responsibility is do your research, improve your skills, set some reasonable goals, that sort of thing. But if you are going to be thrown into the soup anyway, it might be a good investment to get as good a start as you can. A beginner team at Agilistry won't be able to take advantage of (or even recognize) the more sophisticated aspects of what Agilistry is offering, but getting off on the right foot could very well mean the difference between success and failure. Having the Quality Tree staff available in such a well-designed space would be a huge boost to a beginner team. If nothing else, Agilistry can prevent beginners from having to rearrange the furniture every day until they get a good configuration.

Practice space is critical for intermediate groups. Intermediate groups have enough experience to have gotten some taste of success, but still need to adjust, and practice, and analyze, and adjust, and practice... Because intermediate groups have had some success, they are often under intense scrutiny. A practice space is a place where the group can go by themselves to work slowly, do a lot of analysis, make a lot of mistakes, and correct them, or at least protect themselves from making the same mistakes again and again. This is the biggest stumbling block I see for intermediate groups: so many of them retreat to a practice space only to make the same mistakes in the same way, sometimes without even recognizing that they are making mistakes. I see having the guidance of the people from Quality Tree available to help intermediate groups is the best value Agilistry offers.

Advanced groups have the skill and experience to learn new concepts and implement new practices instantaneously in the course of their work. For advanced groups, the value of practice space comes when one project ends and another begins. As long as the team is stable and the goals are well-known, an advanced group can implement new ideas on the fly, without really needing practice space so much. But when the goals are met and the project ends, and it comes time to start a new project or to gather a new team, having a separate practice space to repair to for planning and practice is invaluable.

Finally, a practice space provides a community. Groups move in and out, groups meet each other, groups exchange information about what they work on while they practice. Over time, a practice space can generate an entire "scene" of groups that know and respect each others' work. Knowing Elisabeth and Quality Tree, I have a feeling that will happen.

We musicians know all about practice space. Beginners have to rearrange the furniture in the garage. Intermediate musicians scrimp to rent a smelly soundproof room in a warehouse somewhere. A touring band can rent or borrow an empty club stage. Sting rents a fully-supplied medieval castle in France for the whole group, but only for a few weeks until the band goes on tour. I've rented every kind of practice space you can imagine, from dingy rooms carved out of warehouses to fully-equipped stages. Having a "practice space for Agile software development" fits right into my notions about how proper software development should be supported: the more the agile development environment resembles the ecosystem that supports the performing arts, the better performers that environment will produce.

Congratulations Elisabeth and Agilistry!

Selenesse, nee WebTest

2009-12-12T09:35:00.003-07:00

Marisa Seal and I have officially taken over maintenance of Gojko Adzic's 'WebTest' project. If you want to know why, I wrote about it here.

Marisa is doing the coding (so far; I've never written any Java and would like to learn) and I'm being something like a Customer. I've used similar tools in the past, and I know what I want this tool to do.

Eventually we're going to need documentation for our first release (0.9?), so it seems like a good idea to write down what we're doing, and this seems like a fine place to do it.

The first thing is that we want to change the name of the project to avoid confusion over the other tool with the same name, Canoo WebTest. My favorite so far is 'Selenesse'. Not only does it give a sense of the bridging nature of the tool, it also is very close to the word for the high-level selenium language known as 'Selenese', and since Selenium borrowed a lot of structure from Fitnesse early on, it's kind of poetic to bring it back around like that.

Marisa wants to host the project on Github not Sourceforge. I've used them both, so that switch will probably happen. This has the added benefit of leaving the old project on Sourceforge if anyone ever has a use for it.

We're going to change the namespace not to use "neuri", Mr Adzic's own company name.

The project will support SLIM and scenario tables from the beginning. I'm really liking scenario tables.

The project will have better support for Se-IDE-reported convenience methods. For instance, we've implemented support for clickAndWait, with more such coming.

The project will have a new approach to methods with two arguments. Because of how SLIM is built, methods with 2 arguments such as type() or select() that are normally spelled

| type | | |
| select | | |

are implemented as

| type | | in | |
| select | | in | |

In the course of working on Selenesse, Marisa has also become a committer on the Fitnesse project, so eventually we can support both styles of 2-argument commands.

One big change we're making is to have all Se commands return boolean values to Fitnesse. (This is what makes Fitnesse turn a line red or green; otherwise, if the command fails, Fitnesse returns an exception not a test failure.) At the moment, WebTest returns boolean values only for explicit assertions such as verifyFoo or waitForFoo. I want to emulate SWAT and WikiTest and have every test step return a boolean so that Fitnesse records each test step as a pass or fail. I have a long justification for doing this I've written about elsewhere.

Except that we hit a snag with this. We found that we need the Selenium command check(). Unfortunately, "check" is a reserved word in Fitnesse/SLIM. My suggestion here is to emulate WikiTest and Se-RC in Perl, which provides two versions of each method; a method that does not return boolean, such as click() or check(), and a method with the same name with 'Ok' appended that *does* return a boolean, e.g. checkOk(), clickOk(). If I'm not mistaken, this scheme is available in Se-RC in Perl, so having both versions of the method available would add a certain amount of compatibility between Selenesse and (at least) Se-RC in Perl.

At the moment all our work has been in Java, but already we have interest from a .NET user who may join the project also. The Java work is going to take priority, because we need that for our real job, but .NET support should come along eventually.

Finally, I have to acknowledge my own deep respect and gratitude for the people at Socialtext and at Ultimate Software who build the WikiTest framework and the SWAT tool. I've been an expert user of both, and Selenesse will be borrowing the best features of both projects.

Selenesse, though, will be far more portable, with far fewer dependencies than either SWAT or WikiTest.

telecommuting policy

2009-12-01T10:29:00.002-07:00

Many of the members of the writing-about-testing group are also experienced telecommuters. It turns out that a number of us have had bad experiences when telecommuting for organizations that lack a sane and rational telecommuting policy.

So we wrote one. The following document has been through more than 80 revisions and contains text contributed by 5 separate authors, and also reflects the comments of even more participants. We hope you find a generic telecommuting policy useful.

Telecommuting Policy

Some organizations want to offer telecommuting as an amenity for work/life balance. Other organizations do not want to have to rely on local talent to build exceptional teams. Other organizations want to use telecommuting to reduce the cost of infrastructure such as office space.

Regardless of the reason, any organization wishing to succeed with telecommuting must have a policy for handling remote working. An institutional standard must be in place such that everyone telecommuting or interacting with remote workers has had a similar experience using similar tools to accomplish similar work. This is such a policy.

Any team member who wishes will be allowed to telecommute if: the whole team agrees to monitor and be responsive on all of the communication channels required by the organization for telecommuting teams. Team members doing solo or pairing work must keep the rest of the team apprised of their status, and should not be incommunicado for long periods of time.

Telecommuters who are not responsive on all of the required communication channels will be questioned and may be subject to having their telecommute privileges revoked or other disciplinary action.

Here we describe how to implement this simple policy.

For Telecommuting Beginners

As stated above, an effective telecommuting culture makes constant use of a set of communication channels. As a first step to implement this mode of working, an organization with little experience with telecommuting can have an expert supply compelling, collaborative training sessions, presented using teleworking technologies. These high-value training sessions provide a focus as the organization implements these policies and procedures for the first time, or consolidates such procedures already in place, and supplies incentive for everyone attending to participate and contribute. Although everyone will attend the training via the communication channels, interacting with the trainer, the material, and one another electronically as though they were telecommuting, training may take place locally and some or all attendees may be co-located. This enables smooth technical support and a controlled experience, affording natural opportunities for feedback on the policy.

Prepare the Team and Prepare the Environment

We strongly recommend that the organization put an IRC (Internet Relay Chat) server in place and have everyone in the organization know how to use it. The authors do not know of another messaging service that provides even a fraction of the features available in IRC. We recommend an institutional channel like "#social" for casual messages like "how was your weekend", as well as institutional channels for groups that collaborate closely, for instance "#dev" or "#my_team".

Before the remote training session begins, the following equipment must be in place for every member of the team:

A working headset
A VNC server and client for desktop sharing (1)
A working instance of Skype, or other VOIP service that provides conference-calling ability (2)

Agree to Participate

Every member of the team must agree to participate in all of the communication channels required by remote workers. During business hours, every member of the team must be monitoring at least the following channels and be prepared to answer on them quickly when appropriate:

email
instant messaging such as Yahoo IM, AIM, or similar service
VOIP/Skype
yahoo IM video (3)
wiki (4)
IRC in both #social and #

We consider these channels to be a minimum standard for telecommuting. Telecommuters require high bandwidth communication channels to be effective. If they are disconnected from the ongoing work of others on the team, the lack of critical information will make them unable to contribute; the risk of wasted work is far higher than for those on co-located teams. Having a required set of standard institutional communication channels mitigates the risk of wasting the work of telecommuters.

Other communication channels like a microblogging service, IRC interfaces to Twitter or Facebook, or other such communication channels may also be appropriate. The authors are aware of at least one organization experimenting with virtual worlds like Second Life for distributed teams.

Ongoing Remote Working

Telecommuting Policy

The organization must have a telecommuting policy to refer to once the training sessions are finished. Since we have built an institutional standard, where all of the telecommuters who participated in the training have had a similar experience, using similar tools to accomplish similar work, we can now put such a policy in place.

Upon finishing any required training, any team member who wishes will be allowed to telecommute at will under the same requirements that were agreed to when working in the training session: the whole team must agree to monitor and be responsive on all of the communication channels required by the organization. Team members doing solo or pairing work must keep the rest of the team apprised of their status, and should not be incommunicado for long periods of time.

It should be understood that telecommuters who are not responsive on all of the channels mentioned above will be questioned and may be subject to having their telecommute privileges revoked or other disciplinary action.

During the training experience, we assume that some workers were on-site, interacting with the trainer (and with each other) using the channels described above. When such workers are remote themselves, there may be a need to have on-site workers available on their office phone extensions. The open source PBX project "asterisk" supplies VOIP connections to office phone extensions, as do a number of commercial phone applications such as Avaya.

Meetings

Meetings should be structured such that remote employees can participate on an equal basis with those on-site. On-site meeting rooms must be equipped with adequate speakerphones. A multi-user IRC or IM 'back channel' where both on-site and remote participants may comment on the ongoing discussion in the meeting increases meeting bandwidth and makes it much easier for telecommuters to raise interruptions.

Travel

Travel requires a careful balancing act. Face-to-face time allows coworkers to connect on a human level and can be incredibly productive and beneficial. However, full-time telecommuters are telecommuting for a reason, and mandatory travel that inflicts hardship upon them harms the company in equal measure. Have a clear travel policy for telecommuting workers.

Appendix: Tools

(1) Desktop sharing is required, both for pair-programming and for demonstration of issues with software. Employees should be able to screen-share spontaneously, without the need to schedule a slot in advance. Tools such as Microsoft LiveMeeting or Webex or YuuGuu may substitute for the open source VNC. When working in a *nix environment, screen(1) is more productive than VNC for collaborating in a text-only environment. Examples of when to use screen(1) are, for instance, when using Vim or when doing system administration from the command line.

(2) The open source 'asterisk' project provides a professional-level PBX with multiple conference-call and office numbers available. Proprietary tools such as Avaya or other systems may provide the same sorts of features. What is required is the ability to have a group of people speaking together without echo, distortion, or other interference. Employees should be able to hold impromptu conference calls without the need to book a conference extension in advance.

(3) Some people find video screens of individuals unnecessary if those individuals are available on other channels. For remote workers interacting with an on-site team, having a camera in the team room is valuable so the remote workers can see who are at their desk, or engaged in conversation, or otherwise disposed.

(4) Teams need to manage significant amounts of project documentation and other text. Storing documents on network drives and sharing documents by email is far too low-bandwidth for remote workers to negotiate effectively. As well, these media do not allow for effective multi-user collaboration, nor do they provide an easily accessible revision history for documents, nor do they provide a single authoritative place to access a given document. A full-featured wiki like Confluence, Socialtext, Mediawiki, or Twiki is required for the access to constantly changing information that remote workers need. In our experience, Microsoft SharePoint has shown to be subpar for this purpose.

ui test tool search

2009-11-18T18:50:00.002-07:00

I now work for 42lines. I was hired along with Marisa Seal to start the QA/testing practice for a very small totally distributed/remote software shop implementing agile processes as they make sense. I like this place.

One of the things that 42lines wants to do is to begin UI-level test automation. I have a lot of experience doing this, but I've never done it from a standing start, so this was a great opportunity to get a good look at the state of the practice for UI-level test automation.

For the last 3 years or so I've been using keyword-driven test frameworks that use a wiki to manage test data. I like these wiki-based table-based keyword-driven frameworks a lot. I'm a little suspicious of the BDD-style frameworks like Cucumber and others based on rspec-like text interpretation. Anecdotal evidence suggests that analyzing the causes of failing tests within BDD-style frameworks is an onerous task; also, I suspect that since BDD-style frameworks map closely to story coverage, there ends up being a lot of duplication of test steps. With wiki/table/keyword frameworks, I know from experience that it is possible to design tests with great domain coverage, little duplication, and scale to numbers of test steps in the suite in the five figures.

I was really hoping to find something great in Ruby. I did find that Fit has been ported to a number of languages, including Ruby, but it's still a pretty primitive ecosystem out there. I didn't find any significant bridges between Fit and Watir or Fit and Selenium-RC in Ruby.

(For the record, it would have been great to be a contributing member of the Watir community again. As far as anyone on the Watir team can tell, I was the very first Watir user, and I still think it is a brilliant project.)

So the best-of-breed wiki-based test management system seems without a doubt to be FitNesse. I've used it and I like it, and Marisa is something of an expert in FitNesse and its accoutrement. FitNesse comes in two flavors: Java and C#. 42lines is a Java shop. Selenium Core is in Java. I got no time to fuss with JRuby or whatever the latest coolest implementation of Ruby in the JVM is. It makes every kind of sense to use Java/Fitnesse and Java/Selenium for this project.

So FitNesse and Selenium in Java are beyond dispute the best-of-breed in their respective fields. I thought they went together like peanut butter and jelly, pizza and beer, and SURELY there would be a canonical reference implementation of an integration between these two awesome tools.

Sadly, I was mistaken.

There are to my knowledge three reasonable FitNesse+Selenium integration projects. The first one, fitnium, I dismissed mostly on the basis of rumors of bugs and rumors of lack of support. And in the official documentation, I found that the project was taking on features (Flex and Flash support) for which I had no need. Also, I object to abstracting/renaming Selenium methods for purely arbitrary reasons, as it makes the (very nice) Selenium documentation unusable in the tool.

That left us with a choice of StoryTestIQ and WebTest. We spiked them both.

StoryTestIQ has a beautiful UI, but it has too much magic. The killer flaw for us was that STIQ must run on the same host as the SUT. Even though it is possible to invoke *chrome/FF and *iehta/IE modes from the framework, the framework itself will not support cross-site operations. That's a non-starter for us.

And that left us with WebTest. The brilliant Gojko Adzic began the project but abandoned it about a year and a half ago. It has the feel of something partially-done. And yet in our spike with WebTest we found that we could not only invoke *chrome and *iehta modes, but we could also swap out the very latest FitNesse version and the very latest Selenium version for the antique versions shipped with WebTest, and the WebTest integration code still worked. To my mind, this speaks of good design.

Marisa and Gojko gave a joint presentation on DbFit at Agile2008. Gojko has been known to answer my emails. WebTest is a really obvious choice under the circumstances.

So as of today Marisa and I are administrators for the WebTest project on Sourceforge. I hesitate to say "owners", because neither of us knows much about Java (although Marisa certainly knows more than me) but the code for WebTest is so simple and readable that I am completely comfortable describing what it does and does not do.

Today I made some really nice demonstration tests in FitNesse driving Selenium via WebTest. We're going to make this work.

backchannel

2009-11-05T10:52:00.002-07:00

Kent Beck mentioned recently that he can't think of a situation for which Google Wave is indispensable. Jason Huggins left this amazing comment on Beck's blog post. Wave could very well be indispensable for implementing a "backchannel".

A backchannel is a multi-user space for commenting on some action being shared by everyone on the channel, whether it be listening to a speaker at a conference or attending a meeting during which people take turns speaking. A backchannel is perhaps most commonly implemented on IRC, but any reasonably robust multi-user messaging tool will serve.

But it's not only conference attendees that need a backchannel. Coming up I have a couple of articles recommending required communications channels for distributed teams and very large teams, wherein I strongly recommend having a backchannel during team meetings. It really is a critical core practice for distributed teams.

But even given an IRC backchannel, a lot of information gets lost during the interaction. The chat goes by very quickly, and it is impossible to type all the ideas one has into the backchannel (along with everyone else) fast enough or efficiently enough to capture everything that everyone thinks. Running a backchannel on a wiki is far too inefficient to be effective.

But Wave changes that. Not only can multiple users all contribute to the backchannel at once in real time; but after the meeting or the presentation, Wave provides a persistent, shareable, editable artifact that captured at least the basic premises of everyone's thoughts at the time. Any valuable aspects of this artifact may be edited and enhanced and otherwise manipulated after the fact.

I really want to try this out soon.

when to use analogy

2009-10-15T15:10:00.002-06:00

Analogy is a useful device when used to describe the things that we work on; but it is actively dangerous when used to describe our work itself.

When we're building software, it is useful to be able to say for example "it's like a library, where someone can make use of a feature and then stop using it so that someone else can use that feature" or "it's like a train, where there are just a few places where anyone can get on or where anyone can get off"

But when we use analogy to talk about our work, we invite misconception and misinterpretation. To say "writing software is like making a cake" invites misperception. Does writing software involve flour and sugar? Is there frosting involved?

Of course someone who says such a thing *probably* means that to write software one must do certain steps in a certain order-- but it is far better to describe the actual steps ("red, green, refactor" is not an analogy) than to invoke analogy, because analogy always misleads to some extent, and we have known for many years that the cost of failed communication on software projects is extremely high.

CFP: Peer conference "Writing About Testing" May 21/22 Durango CO

2009-10-01T15:15:00.002-06:00

CFP: Peer conference "Writing About Testing" May 21/22 Durango CO

I am organizing a a peer conference for people working in software testing and software development who write about their work in public. The conference will be organized LAWST-style, much like the recurring Austin Workshop on Test Automation or the Tester-Developer/Developer-Tester conference I helped organize in 2007.

The original proposal is on my blog here: http://chrismcmahonsblog.blogspot.com/2009/08/proposed-peer-conference-writing-about.html

There is significant demand for public information about software testing and software development and software process. This conference is for people who want to influence the public discourse on these subjects through public writing.

If you are interested in the subject, but not necessarily interested in the peer conference, there is a private mail list for writing about testing whose members include both writers and publishers. I (or any other member of the list) will add you to the mail list if you send us your email address. I am at christopher.mcmahon on gmail. Your email will be used for no other purpose than to join the list.

New voices are particularly encouraged to apply for the conference and to join the mail list. If we don't know you or your writing, please point out links to some public examples like blog posts, public documentation, conference papers, or similar work when you do so.

A number of prominent writers and also new voices in the software testing community have have expressed interest in attending the conference. In an effort to break down some of the wall between writers and publishers, at least two publishers of software testing material also have expressed interest in attending. The conference will have a definite agile slant, but I don't want to exclude great writers working in other situations.

This peer conference is run like many others: send a position statement saying why you want to attend, and what relevant material you would be prepared to contribute to the conference. Examples of relevant material might be a presentation, interesting experience, or even a really good set of questions. People will be invited based on those position statements. As of now, I will be the one evaluating these, but I will be sharing them freely with others on the mail list whose judgment I trust. Even better, join the mail list and publish your position statement there.

I may or may not cap attendance at 15. If we get a significant amount of great position statements, I might increase that number.

There will be no fee to attend, but you will be responsible for your own transportation and lodging. A discounted rate at a convenient bed and breakfast hotel within walking distance of the conference will be available for a limited number of rooms.

Here are the relevant dates:

Dec 1 2009
Deadline for position statements to be emailed to me or to be posted on the writing-about-testing mail list.

Feb 1 2010
Invitations to the conference sent. Waiting list established if necessary.

May 21 and 22 2010
Conference.

Durango has a lot to offer visitors. Attendees may want to arrive early and stay late. The Taste of Durango street festival is Sunday May 23, and we also plan to have an outdoor excursion that day. There will likely be a number of attendees driving from the Denver area Thursday May 20, so carpooling from Denver or Colorado Springs could be possible for the 6-8 hour drive. Direct flights to Durango are available from Denver, Phoenix, and Salt Lake City, or the Albuquerque airport is a 4 hour drive in a rental car. Let me know if you need more detailed information about getting to Durango or about the conference itself.

UPDATE: I have just been informed that there is no longer a direct flight to Durango from Salt Lake City. United, Frontier, and USAirways fly to Durango from Denver and Phoenix.

against kanban part 3

2009-09-22T15:09:00.002-06:00

At the agile2009 conference, David Carlton and Brian Marick presented something they called "Idea Factory", an overview of three sociological systems by which the scientific community comes to regard a certain thing as fact.

Carlton's presentation was particularly intriguing. He cited work pointing out that disparate communities come together in what are called "trading spaces" in order to pool aspects of their expertise to create new work.

One of the signs of a "trading space" is a creole or pidgin language created by the participants in the trading space and used by participants in the trading space to negotiate among themselves. Such language may seem weird or impenetrable by outsiders, but participants wield it in order to accomplish things among themselves.

In contrast, Six Sigma, ISO9000, and now Lean/kanban, are imposed upon the agile development trading space. They are concepts and processes and ideas forged from managing factories, fitted (roughly and badly) into a culture that, when healthy, has no need of them. Just a quick glance at the hundred or so two-column translation tables in the book "Lean-Agile Pocket Guide for Scrum Teams" gives a clear indication that it takes serious effort to map Lean processes onto an indigenous Agile framework.

To my mind, the agile/XP/Scrum community conceived three brilliant "trading space" type of ideas around which our work revolves:

First, the idea that at a certain time, we will release a predictable and well-known set of running, tested features to production, over and over again, reliably, puts the emphasis on actually shipping things for people
to use. This emphasis on succeeding within a time box consistently (and the subsequent celebration by both the agile team and by the customers!) is utterly critical to the success of agile teams, and Lean/kanban throws it out the window.

I had a number of Lean advocates tell me that "you can still have iterations with kanban", but the radical de-emphasis of the time box and of the ceremony of releasing features to the customer at the end of every scheduled development cycle is a huge step backward. In other writing I have suggested that software development without a regular, public, scheduled, celebrated release of features to production cannot be agile, regardless of what other processes are involved in such software development.

Second, the fact that the language to describe this work (iterations; big visible charts; information radiators; test-driven development; continuous integration; pair programming; automated testing; etc. etc. etc.) comes
directly from those who practice the work and is indigenous to that work. These terms have no meaning outside the realm of agile software development. This insistence on our own creole or pidgin language gives the agile software development community a high degree of self-reliance, and a degree of resistance to corruption.

The language used by Lean/kanban software folk is strewn with corrupted Japanese words and phrases ripped from a generation of Japanese manufacturing. In my experience such words (kaizen; pokyoke) are used more to intimidate and confuse the uninitiated rather than to actually further the work at hand. There are perfectly good words in the agile "trading space" to describe such concepts without resorting to arbitrarily adding terms from an entirely different culture, the culture of Japanese automobile manufacturing.

Third is that every Scrum process for every individual team contains the seeds of its own destruction. Every agile retrospective contains an opportunity to break the rules for a good cause. Want to scale your team up to three times the recommended size? Go for it. Want to replace pair programming with institutional code review, replace CI and TDD with a hair-trigger build system plus dedicated testers expert in the debugger? Go for it.
There is no room for creativity on the factory floor. Any significant change to a manufacturing process involves significant retooling expense. In software we have no such expense. We can change our development processes right now, and then change them again an hour from now, and again an hour after that. The agile frameworks and tools we use give us fast feedback that tells us what works and what does not.

The Lean/kanban term that probably concerns me the most is "waste". On a factory floor, this term has a reasonably clear meaning. In a software development team this word has dangerous implications.

I am a dedicated telecommuter, and the best telecommuter situations I have worked in have provided proxies for the "water cooler": wikis where we can post photos of our gardens and of our vacations, messaging systems where we can talk about our music or even our religion. On the face of it, this is waste; in practice, it binds a team tightly together and increases productivity.

This healthy social interaction in the course of the work is a symptom of a healthy "trading space". Besides the pidgin/creole language created universally by devs + BAs + testers, there is a creole created within individual teams to describe their work and how it affects their lives.

The foreign concepts Lean/kanban offer present a clear risk of corruption of the natural discourse of agile software development, and its other concepts like "waste" and "value stream mapping" pose a threat to the social interactions that make working in healthy agile teams such a productive, easy, and even joyous experience.

against kanban part 2

2009-09-14T15:44:00.001-06:00

After conversations with various well-meaning folk at the Agile 2009
conference about some details of Lean and kanban, I remain more
opposed to a general implementation of them than ever.

But I'm willing to grant that a single kanban call might have value.
Liz Keogh had a good example: a group of agile coaching consultants, each with multiple clients, each client moving at different rates, with more clients
waiting for attention. The consultants track their work with cards on a board. (Note: tracking work with cards on a board is NOT kanban...)

When one consultant finishes with a client, that consultant adds a
card to the board saying GIVE ME WORK.

As I understand it, this is the essence of kanban: as Eric Willike told me, the essential image of kanban is a colored card in an empty basket, sent up the line to have the basket filled again.

But what a frighteningly low-bandwidth transaction! This means of communication is for infants: GIMME followed by either YES or NO. Put a bunch of infants in a room and you will certainly observe social behavior and even happy cooperation; but those social transactions cannot be very complex. A social system based only on GIMME: OK/NO is a poor substitute for real adult communication.

But it is a perfectly adequate system for interacting with inanimate objects, such as on a factory floor.

I was asked to supply my favorite metaphor so we could apply lean/kanban to it, and of course I chose a musical performance: a group of experts negotiate a set of things to perform at a certain time at a certain place for a paying audience who will be delighted by the performance. At the arranged time and place, the group of performers demonstrate their expertise for the delighted audience. Then they do it again two weeks later.

My interlocutor then asked me to imagine how to deal with the transportation and the instruments and the catering and the sound and lights and the hotel, etc. etc. etc., implying that lean/kanban is an appropriate system with which to negotiate the larger ecosystem in which a software development team works.

Thinking it over later, though, I still think that not only is kanban a poor fit, it's a dangerous approach when applied to any transaction more complex than filling a basket with parts.

The essential problem is that arranging a performing arts tour or releasing successful iterations to production requires complex negotiations among people at every stage. Kanban defines only a single, infantile transaction: GIMME: OK/NO. If this single
transaction is the only model you have for dealing with people, your project, whatever its nature, is doomed to fail.

On the other hand, if your communication model is complex, for example as they say in the Scrum world, if your story is a placeholder for a conversation rather than an empty basket to be filled, you are doing something-- but whatever it is, it is not kanban. If your cards on a board represent more than just Work In Progress but instead represent complex agreements among groups of people, you have stepped outside
the lines of Lean and kanban and you are doing something altogether different.

The risk I perceive here is that organizations will not only attempt to reduce all of their software development practices to infantile GIMME:OK/NO transactions, but that they will pursue such oversimplifications to great expense in the name of "value stream
mapping". This is an approach that blends the brain-dead process descriptions of ISO9000 with the outrageous process design expense of Six Sigma.

As I've noted many times, processes designed to manipulate physical objects map very poorly to software development.

proposed: peer conference "Writing About Testing"

2009-08-16T12:38:00.006-06:00

It seems like there is more demand than ever among the technical publications for information about software testing. Experience reports, theoretical pieces, tool documentation, all seem to be in great demand right now.

At the same time, I think the overall quality of what I've been reading about testing is declining, as people rush to meet that demand without adequate preparation or knowledge.

I'm interested enough to do the legwork to host such a peer conference, assuming the potential participants were willing to come to Durango. If not, I'd be willing to travel to attend such a conference, and to help facilitate, promote, or whatever is needed to bring it off.

I envision soliciting participation from two groups: established writers with a history of publication with outlets like SQE, STP, InformIT, O'Reilly, etc. etc.; and also new voices looking to start writing for these sorts of publishers (I very much hope that there really are new voices writing about testing very soon). New voices should have some sort of verifiable writing ability, like a public weblog or conference papers, or some other reasonable means to evaluate their work.

As for content, I'd like an Open Space style event, but I envision themes like:

Ethics of public discourse: the difference between reporting and opinion; proper citation; avoiding false pretences and misunderstandings; criticising ethical lapses on the part of others.

Voice and style; tone and timbre. Academic/scientific writing versus colloquial and general writing. Grammar, structure, how to engage the reader.

Working the business: queries and pitching, invoicing, negotiating with publishers, working with editors and art departments.

Professional development: improving technical skills, and being able to write about it.

Impact of blogging, Twitter, social media that shows extensive writing, like Facebook and LinkedIn and such.

As for a date, I envision some time within the next year, but not before January 2010. Assuming it's in Durango, there would be a lot of amenities available. In winter, there is good skiing available at a couple of places nearby. Spring is a shoulder season, but it's a great time to go to the canyon country. Summer has river sports, biking, and mountain trips, all in town.

Durango has a new library with meeting facilities that should be adequate for a fairly small group. For a larger group, the local college makes its facilities available for minimal cost. This conference, like most such, would be a break-even proposition.

There are a number of alternative locations to Durango. SQE is based in FL, as is the company I work for. STP is mostly in Nashville, not far away. The Bay Area or Chicago or Portland or Austin all might be possible locations, as I believe there are facilities for such gatherings easily available. I'd be particularly interested in Denver, Phoenix, SLC or Albuquerque also, as they are only a short plane ride or a day's drive away.

If this idea interests you, please leave a comment here, send me email, or drop me a line on Twitter.

a year of STPedia

2009-07-18T17:22:00.003-06:00

This month marks one year of the column Matt Heusser and I co-write every month for Software Test and Performance magazine, "STPedia". It was great to mark our anniversary in a double-length column on "Agile Testing in Practice" in July's newly redesigned print version of ST&P and to help out with the launch of ST&P's redesigned portal, stpcollaborative. ST&P are changing things around for the better.

The format of the column is that we have a paragraph or two introducing the topic for the month, then we define dictionary-style a set of terms related to the topic, then usually we draw some sort of conclusion from how the terms are related. ST&P has a 6-month editorial calendar, so at this point we've treated many subjects twice. And really, how much is there to say about source control? From here on out we're going to range a little more widely in our choice of topic.

A little Inside Baseball, for those who are interested in how the writing gets done: we have no set process. Some of the time one of us will write the whole draft for the other to edit. For instance, September's column is mostly Matt, and October (at this point at least) is mostly me. Other times one of us will start the column, run out of steam, and the other will finish it. I remember one column in particular where I just could not come up with a conclusion that made sense. Matt rewrote the last half so that it all made sense. Sometimes one of us will write some of the definitions, and the other will handle what's left over. Matt has a wider range of expertise than me, so he likely wrote the really technical stuff. If you read the column month to month, you probably get about 50% of each of us over time.

The fact that we've been at this a year with no end in sight speaks to our mutual respect. Matt is a bright, honest, motivated, dedicated guy. And fun to work with. I think we still have a lot to say, I hope you'll consider reading our stuff.

against kanban

2009-07-11T12:54:00.002-06:00

I've been somewhat alarmed by the enthusiasm with which the agile community has embraced Lean manufacturing principles, especially kanban. While I'm sure that kanban is effective in some circumstances, I am also convinced that it exposes software teams to real risk of impaired function.

Like many of my qa/tester colleagues who were working in the field in the 90s, I used to be a real process-head. I was part of projects that enthusiastically implemented Six Sigma, CMM, ISO9000, as well as other approaches like RUP, spiral development, etc. etc. We thought then that if only we could get the process right, we could crank out great software just like factories crank out cars and engineers crank out bridges.

Six Sigma and ISO9000 were lifted straight from manufacturing. CMM was the product of a lot of analysts who looked at some properties of a selected set of software projects.

Then the Agile movement came along, and with it new processes, the most significant of which turned out to be XP and Scrum together. These were revolutionary, because they were software development processes actually created from scratch by people who were very very good at creating software. Obviously, because these processes were created by smart people creating software, these processes map very well to how we do excellent software development.

We can argue about the details: is it better to have stable velocity or increasing velocity? should we estimate using points or using ideal hours? Regardless of these details, if your team measures velocity and uses velocity to plan iterations, and if your team always does pair programming, test-driven development, and continuous integration, then you will either release excellent software every iteration; or you will quickly expose and face the reasons why not.

Kanban doesn't make it necessary to face your inadequacies. I have two examples:

At one point my dev team had a chance to interview a self-appointed "kanban expert" on the subject of kanban and Lean. At this point the team had successfully shipped something like 25 of the last 27 iterations, so releasing iterations to production was routine, we had the whole process pretty well dialed in. And we had some killer questions for the kanban expert.

Under this questioning, it turned out that the kanban expert's team had been using a Scrum/XP process, but they blew iteration after iteration after iteration. From his answers to our questions, it seemed that his team lacked the skill and discipline to be consistently successful iteration after iteration. So one iteration, when they knew they'd blown it one more time, they simply stopped doing iterations or planning to release features. This is apparently when this person attached "kanban expert" to his resume.

Before we started releasing every iteration, my team had been using a kanban-like process. We would identify a set of features, then branch our trunk code for each feature under development. When a feature was finished, we would merge the feature branch back into the trunk code. This way we always had features being developed, and we could release trunk safely at any time.

We had a lot of very bright developers on the team. Some of them were maybe a little too bright. With no timebox pressure, we had people doing a lot of gold-plating, making architecture changes that had no relation to the feature being developed, making (cool!) experiments in the underlying code, and generally making merging not very fun.

Furthermore, it was an enormous headache for the marketing people. Without timeboxes or estimates, it was difficult to know when a particular feature would be available, which made it almost impossible to do effective marketing for the product. When we switched to planning iterations based on our velocity and our backlog, we made the marketing people very, very happy.

The fundamental difference between software development and manufacturing or engineering is that in software development we do not manipulate physical objects. We manipulate concepts, and algorithms, and process. Processes designed to optimize the handling of physical objects simply do not map to the work we do when we create software. I'm sympathetic to those who see estimation and iteration planning as waste, but I think the risk inherent in not having a mechanism to expose our inadequacies is too much risk to take, even for the best of teams. We have no warehouses to fill up or assembly lines to halt, we have only the contents of our heads and of our code repository. For software development, the only truly useful metric is: can you deliver running tested features on a consistent schedule? Iteration timeboxing exposes that metric in the most obvious way possible.

What we find in both literature and anecdote about the application of processes taken from manufacturing and engineering, puzzlingly, is that for every team for which the process is successful and helpful, there are other teams who implement the same process only to meet with failure.

What I suspect is that ANY process or methodology (that is not actively destructive), applied to a skilled, disciplined, high-functioning, motivated team, will succeed, regardless of the process. Likewise, any process applied to a low-functioning team will likely fail.

With an XP/Scrum sort of iterative process, we will know very soon exactly how and why we are failing. This seems to be not true of kanban or of any other software development process taken from the manufacturing arena.

validate order with regular expressions

2009-07-07T14:42:00.003-06:00

Today I wrote some UI tests to validate that certain text appears on pages in a particular order. Since I've done this now in two different jobs using two different tools, I thought it might be of interest to others.

There are a couple of reasons you might want to do this. For example, a page might have a list of records sorted by certain aspects of the record. Here's a crude example:


|name |date|description|
|chris|july|tester     |
|tracy|june|gardener   |

One set of critical tests validates that name/date/description appear in the right order; that chris/july/tester appears in the right order; and that tracy/june/gardener appears in the right order.

Another critical test is that chris appears above tracy in the display.

The first challenge for the tester is to identify and be able to address the smallest part of the page that contains the text under consideration. Every tool does this in a slightly different way, but Firebug is your friend here.

The next challenge is to make sure that your particular framework knows how to interpret regular expressions. I have seen this in place in Selenium-RC and SWAT, and I have rolled my own in Watir. (But for all I know Watir might have this kind of thing automatically by now. It's been some time since I've used Watir on a real project.)

Now, to check the horizontal order, you construct a test that looks something like


|assert_text|id=text_container|name.+date.+description|
|assert_text|id=text_container|chris.+july.+tester    |
|assert_text|id=text_container|tracy.+june.+gardener  |

To test that chris appears before tracy on the page:


|assert_text|id=text_container|chris[\W\w]+tracy|

Regular expressions are for matching text. The first set of tests is fairly straightforward. The "." operator says "match any character". The "+" operator says "match 1 or more instances of whatever". This means that text like "namedate" would cause the test to fail (which is exactly what we want) but text like "name foo date" would pass (which might not be what we want but it's still a decent test, we could write a fancier regex if that happened to be critical).

The second test is a little trickier. The trouble is that by default "." does not match any newline characters. And since chris has to appear above tracy in the page, there will always be a newline involved. Most if not all languages have a way to tell the "." operator to match a newline, but some of them are really awkward. (I'm looking at you, .NET.)

So instead we make a little workaround: the \W means "match anything that's not a word" and "\w" means "match anything that is a word". "Word" is defined as any letters plus any numbers plus "_", but it doesn't matter. [\d\D] (for digits) would have worked as well, since the point is that one of those expressions will always match anything we encounter between one interesting string ("chris") and the other interesting string ("tracy").

UI test framework design using Watir

2009-04-24T13:45:00.009-06:00

I wrote a little toy test automation framework recently.

It's a keyword-driven framework, and the test data is kept in a CSV file. The test data look like this:

goto,http://www.google.com,,
text_field,name,q,pickaxe
button,name,btnG,click
match,Programming Ruby,text

This is the classic Google Search example from the Watir documentation.

I used CSV because it's the simplest implementation. This is, after all, a toy. In the real world this might be HTML tables or wiki tables or a spreadsheet. What we want is for anyone to be able to write a test quickly and easily after a little bit of training on FireBug or the IE Developer Toolbar. (Test design is a different issue. Let's not talk about that now.)

The framework grabs the test data and interprets each row in order. A simple dispatcher invokes the correct execution method depending on what the first element in the row of test data is. This has two advantages.

For one thing, there are a finite number of types of page elements to deal with, so our methods to manipulate them will be limited in number.

For another thing, we can write custom test fixtures using our own special keywords, and make our own little DSL, like I did with the final row above that starts with "match".

Reporting pass/fail status is left as an exercise for the reader.

This simple framework could easily be the start of a real test automation project. It's robust, it scales well, and it can easily be customized, improved, and refactored over time.

[UPDATE: the comment below about leaking memory is interesting. I didn't say it explicitly in the original post, but I think test data (CSV files) should represent fewer than about 200 individual steps. My vision is that this script is invoked for a series of CSV files all testing different paths through the application, and each CSV file is handled by a new process. Paths through the application with more than about 200 steps make it much harder to analyze failures. The fewer the number of test steps, the better.]



require 'test/unit'
require 'watir'


class ExampleTest < Test::Unit::TestCase


   def goto(args)
      @ie.goto("#{args[1]}")
   end

   def text_field(args)
      @ie.text_field(:"#{args[1]}",'q').set("#{args[3]}")
   end

   def button(args)
      @ie.button(:"#{args[1]}","#{args[2]}").click
   end

   def match(args)
      assert_match("#{args[1]}",@ie.text,message="FAIL!")
   end

   def test_google

      @ie = Watir::IE.new

      @command_array = []

      File.open('watir_keywords.csv') do |file|
         while line = file.gets
            @command_array << line
         end #while
      end #do

      @command_array.each do |comm|
         args = comm.split(',')


#arguably 'case' would be nicer here than 'if', but this gets the job done.
         if args[0] == "goto"
            goto(args)
         elsif args[0] == "text_field"
            text_field(args)
         elsif args[0] == "button"
            button(args)
         else args[0] == "match"
            match(args)
         end #if
      end #do

   end #def

end #class

a really good bug report

2009-04-19T20:39:00.002-06:00

I had occasion to use FireWatir recently and found a strange bit of behavior trying to manipulate a select_list element. I posted an executable bug report to the Watir-general mail list. (Please read the whole (short) discussion.)

I have not been working closely with Watir for some time, so I missed some obvious diagnostic paths. Even so, though I got a reasonable explanation for the observed behavior, I pushed it farther, wondering why I did not receive a NoMethodError trying to do what I did.

Please note: at no time did I ever editorialize; nor criticize; nor invent solutions. I merely discussed observed behavior with interested parties in disinterested tones. And upon examination, I had turned up not one, but two different bugs.

I am pleased once again to have contributed to the Watir project. Furthermore, I think the discussion linked above is pretty close to a Platonic ideal of bug reports.

at liberty

2009-04-05T21:23:00.002-06:00

blogging is so old-school. Thanks again Jeff!

I have been amazed by and touched by and thankful for and proud of all the people who have sent nice messages, said nice things, and sent me leads since I was laid off from the best job I've had in a decade.

I am seeking a telecommuting software testing/QA job or related work. I have expert-level skills and experience, a shockingly good resume, and awesome references from some of the best software people in the world.

agile COBOL is no oxymoron

2009-04-05T09:36:00.007-06:00

Dr. Brian Hanks of Fort Lewis College said on Twitter just now:

"Agile Cobol - the new oxymoron"

I have to disagree. In the late 90s I worked testing a life-critical 911 data-validation system written in COBOL and C. I was the tester when we migrated from key-sequenced data files (basically VSAM, or very close) to an SQL database (albeit one with no relational integrity-- we had to write that into the code).

When I joined the company, system releases were chaos. By the time I left, system releases were orderly, done on a regular basis, with great confidence. We evolved what was essentially an agile process to regularly ship running, tested features that our customers wanted.

But we broke every rule in the book (of the time) to do it. We had customers talking to developers all the time, we had sysadmins reviewing features before release, we had testers reading code and running debuggers, we (quietly) ignored test plans in 5-inch binders.

The Agile Manifesto validated this way of working. It was the first public acknowledgment that we were not alone in thinking that it took people working together all the time to release good software. Suddenly I had somewhere to point to say "See? Someone else thinks that working this way is a good idea!" As the Agile Manifesto became more widely known, any number of old-school mainframe people came out of the woodwork to talk about how they had been doing similar work for a long time. The Manifesto did not come into being from nowhere.

I want to talk about a few test techniques I used in a couple of COBOL code bases that are creative analogs of modern techniques, but using old tools.

Human Unit Testing

My mainframe career was spent all on Tandem systems. Tandem had a GREAT debugger, and a GREAT proprietary scripting language. As a tester, the first thing I did whenever new code got checked in was to crank up the debugger, set a breakpoint at the new code, step back and forth modifying the values of variables trying to cause a failure. When I did cause a failure, I would step back to a system level and use the UI or the batch processes to attempt to generate the data to cause the failure. Most of the time this was possible. Then I filed a bug report.

It is worth noting, now that the statute of limitations has probably run out: I was completely confident that our system would survive Y2K. Not because of the system test plan in the 5-inch binder (a worthless and stupid exercise required by someone in another part of the building whom we never talked to and that I ignored outright) but because I had personally examined the use of every instance of every date in every place in the entire code base by hand, in the debugger. That took significant time, but it was worth it.

Debugger + macros + scripting = automated unit testing

In hindsight I should have done a lot more of this.

Tandem's scripting language allowed you to start a program, call the debugger, and load something called a 'macro'. The macro would drive the debugger, allowing you to set breakpoints, modify and capture values during the course of the run.

I used to use this technique a lot to accomplish what Jonathan Kohl calls "Computer Assisted Testing". Use automation to accomplish the tedious steps of bringing the system to a state where something interesting is just about to happen, then let human beings take over.

If I knew then what I know now, I could have built some awesome standalone regression test suites using this technique.

Golden Snapshot

I worked in another COBOL code base that was a disaster. Actually, the whole company was a mess. To give a sense of how bad it was: the ball of mud was far too big for the compiler. Whenever the size of the main procedure had to be cut, the devs didn't peel off a module or a functional area; they just cut lines 5000-6000 or whatever and called it a day. Utter chaos.

And improving the code base would have threatened the jobs of the most senior devs. The best developers in the company were given impossible assignments, sabotaged along the way, blamed for the failure, and then demoted. Not nice.

The code was so bad it literally could not be automated at a unit or integration level. So I did two things.

First, we designed a "smoke test" set of manual test cases. We had the best tester in the shop run each test very carefully by hand. At the end of each reference test run we took a snapshot of all the system files and stored them in directories named for each test case.

For subsequent manual runs, the tester would execute the steps, then (figuratively) press a button that caused a diff of the saved files vs. the current system files. Any discrepancies between the state of the files was cause to suspect a bug.

In step 2 of this system, I used a tool called VersaTest from a company called Ascert to begin automating the stuff that the manual testers did. This tool was basically a proxy for the UI and allowed me to drive the system guts by bypassing the UI. To this day, I have never had a better customer support experience than I did with Ascert. I was calling them constantly, and one of their customer support people became a real friend I still speak with now and then.

This is pretty much a last-resort automation option. For one thing, it is extremely expensive up front. For another thing, debugging failures is depressingly hard.

Stubs

I was teaching myself Perl at this time also, which came in really handy on several occasions.

At one point I was assigned to a small side project. One of the better devs had run afoul of his peers again and been sentenced to work on this not-sexy side project. This dev is responsible for one of my favorite quotes of all time. After we'd been working together for a while and the project was starting to flesh out a bit, I noticed that his COBOL was quite a lot better than the code in the main code base and the code I'd seen from other devs. He said "Just because it's COBOL doesn't mean it HAS to be ugly".

This project was a transaction system. Our code would send an transaction, and an outside party would send back one of 4 messages: Success; Failure; Queued for Later Processing; or Later Processing Complete. But their test interface was really flaky and was down most of the time. I wrote a little Perl script that would reply to our system with one of the 4 messages depending on whether the last digit in the transaction number in the record was 1, 2, 3, or 4. It was rather clever and simple, and that little script saw use for the whole life of the project. This little side project was also neat because the team was me, the dev, and a business guy, and we were iterating fast, all together, all using the code, all analyzing the work at the same time.

Meanwhile, in the big sexy part of the company, there was a highly visible project to do transactions with a very important partner, but our dev schedule was ahead of theirs, so interaction testing was some way out. When I was informed that the code was "done", I read the spec and implemented a little Perl network server that could read and write messages over TCP/IP. I pointed our code at it, which immediately fell over dead. I started reporting bugs on the code. The dev actually complained to my boss about this. He could not understand how someone could report bugs when the other side of the transaction was not complete. One day though, he turned all the way around and started being nice to me. I suspect someone had reviewed his "done" code and chewed him out.

No Oxymoron

So it is possible to have automated unit testing, integration testing, and system testing in a COBOL/mainframe environment. Furthermore, it is not only possible to develop COBOL according to the Agile Manifesto; I think the points made by the Manifesto came from the cultures of successful mainframe projects.

Agile software development has been around for a long, long time. It just didn't have a name until those guys got together at Snowbird and cranked up the Agile movement.

Ruby net/http+SSL+Basic Auth

2009-01-03T13:28:00.003-07:00

Since one post from this blog is (or at least used to be) a little chunk of official documentation for Basic Auth in Ruby, I decided I should write this down also.

Socialtext is switching some of the SaaS servers to be https all the time. So I had to rejigger my selenium test-runner script, because it reads (test data) from and writes (test results) to the wiki.

I found this very elegant example.

My take on it looks like

def put_test_results_to_wiki
         http = Net::HTTP.new(@test_host,443)
         req = Net::HTTP::Put.new(@put_loc, initheader = {'Content-Type' =>'text/x.socialtext-wiki'})
         http.use_ssl = true
         req.basic_auth @test_user, @test_pass         
         req.body = ".pre\n" + @content + "\n.pre"         
         response = http.request(req)
         puts response
     end

What is a little peculiar about this is that Basic Auth is very much optional for SSL connections. Normally the client would do a GET, the server would return a cookie, and the client would use that cookie in subsequent transactions. I found a nice example of that here, but as of today the server seems to be down.

artistic testing

2008-11-24T21:40:00.002-07:00

I like to point out that historical, well-vetted aesthetic frameworks are useful for evaluating software.

But what if similar aesthetic frameworks could not only validate software itself but also *predict* software test techniques?

For example, let's examine overarching schools of painting: first came realism, where painters attempted to represent as much as possible what appears to the human eye. We can test realistic scenarios, we do this all the time. I would suggest that most bugs are not discovered under what we understand to be realistic circumstances.

Then came expressionism, where painters attempted to express things not seen. Likewise, test techniques moved on to persona testing (imagining the use of software by peculiar users) and soap-opera testing. Such techniques expose significant bugs.

Then came abstract expressionism. Think of Jackson Pollack. Are there abstract expressionist test techniques? Of course there are.

I would really like to hear of any other explorations along these lines...

two agile antipatterns

2008-11-22T16:01:00.002-07:00

Both of these have been crossing my path with increasing frequency in recent months (not at work, but in conversations with others whom one would think would know better):

1) Stories are scheduled work for features. Stories should not include:

testing
refactoring
documentation
undefined customer support
calling your mom

The problem with writing stories for stuff other than scheduled work on features is that you completely undermine the concept of velocity.

Velocity is the number of story points accomplished per iteration. If you assign story points to activities other than scheduled work on features, you cheat the business of accurate information about release schedules. (Although it might make you feel better to be accomplishing so many points each iteration, if you are not delivering features, you fail.)

2) Doing it "by the book".

Part of the agile culture is to have retrospectives at the end of each iteration. Each retrospective is an opportunity to change your process.

So if you're doing 2-week iterations, at the end of a year, you've had 25 or so opportunities to change your process. At the end of two years, 50 process changes.

If after all of these retrospectives, you are doing your work the same way you were at the beginning, you fail.

Finally joined Twitter

2008-11-12T17:25:00.001-07:00

I'm chris_mcmahon there if you want to follow me.

testers as navigators

2008-10-08T09:47:00.002-06:00

Not long ago Audrey Tang started working for Socialtext. It has been a remarkable experience, as Audrey is mostly fixing bugs based on bug reports submitted by our testers.

Audrey is in Taiwan and another of our testers is working from Bangladesh (although he is usually in Vancouver BC). Today they needed to hash out some details of a bug report so they shared a VNC session on a machine hosted in Palo Alto CA. It took only a couple of minutes to arrive at an agreement that the bug was fixed.

Audrey's comment in IRC really struck us all: " having an excellent qa team is like driving with a first-grade GPS navigation system :)"

We have worked very hard to create consistent, readable, executable bug reports for real, visible bugs. Having a developer as good as Audrey validate that work means a lot.

a test technique

2008-09-06T22:09:00.002-06:00

Very recently I was part of a bug hunt: an effort to find as many defects as possible in a UI in a short period of time.

I am not an outstanding UI tester. I have worked with testers whose eye for detail, line, color, consistency, work flow, etc. etc. make them really outstanding testers to have examining the front end. But I've spent most of my career on green-screen apps, test automation, environment maintenance, stuff like that. I lack the really expert artist's eye that great UI testers have.

That said, I was on a roll on this particular bug hunt, finding a lot of nice, sophisticated bugs. It was classic ET: decide a test approach, determine your next test based on the results of your last test; and when you stop finding bugs, change your test approach.

I'd followed all the logical paths I could think of; I'd tried really big inputs and really small inputs; I'd tried permissions errors, navigation errors, and checking error messages themselves. I tried some crazy stuff. I started to run out of reasonable test approaches. Toward the end, when I was getting tired and finding fewer and fewer bugs, I hit on a neat test technique that found a really cool bug:

Here's what I did: I sort of unfocused my eyes, and addressed the UI spacially instead of logically. Moving from top left to bottom right, I manipulated everything I could find to manipulate. Then I went across the bottom; across the top; up and down in lanes; just clicking and typing and choosing whatever was located next to the last thing I manipulated.

I found this great bug this way: there was a very small link in the footer of a particular area on a page that had exactly the same label as a much larger link elsewhere on the page. Even though we had four expert testers closely examining the application, no one had discovered that the little link in the footer was wired up completely differently (and totally wrong) than the big link with exactly the same label. Until I unfocused my eyes and considered everything on the page as equal and logically unrelated, except by space. Because I approached the UI spacially instead of logically, I found the little piece that was wired up completely wrong.

technical investment

2008-06-23T22:34:00.003-06:00

My colleague Matt Heusser is doing this workshop on technical debt soon.

I posted a little meditation on techdebt as impedance mismatch, but that seems to me to be trivially true, facile.

Since I made The Software Artists public, though, I've been fielding a lot of questions on the software-test mail list that has me rethinking my position on technical debt. So here is what I *really* think:

Technical debt doesn't really exist. At least, it doesn't exist for high-performing software teams. It's a useful concept for those encumbered by poor programming and poor testing, but the idea ceases to be useful about the time the code base becomes manageable.

I have a graph and and I have a metaphor that both explain my position.

I'll just describe the graph because I'm too lazy to hunt down some free software to draw it. This graph describes a couple of different projects I've been involved in, for a couple of different employers.

The critical line on the graph is "regression bugs released to production". Or you could title it "predictable bugs released to production" if you wanted to be more inclusive, but I like the regression idea best.

Another line on the graph is "features released to production". This is where we make the product better.

A third line on the graph is "test coverage". I don't really care about the distinction between unit tests, integration tests, acceptance tests, or whatever. Just stipulate that it is possible to achieve some significant automated test coverage of the product.

Over time, the graph of a productive, high-performance team creating software will likely show:

Early on, the number of regression bugs released to production will be a small number, then will go to zero. At Socialtext, we have a suite of Selenium-based regression tests. About a year ago, we rapidly increased the number of test steps from around 1000 to around 4000. The number of regression bugs released to production went from 'a few' at 1000 test steps to zero at 4000 test steps. That line on the graph has remained at zero, and the test suite now has about 7000 test steps.

The number of features added to the software will rise significantly. Good developers working well, releasing to production every two weeks: you get a lot of features.

Test coverage (and by implication, refactoring): this is the interesting part. Test coverage rises at a much smaller rate than the rate at which features are added. Picture the difference: as the number of features rises dramatically, and the amount of test coverage rises less dramatically, the amount of technical debt increases.

Yet the number of regression bugs released to production remains zero.

This works because that area of the graph between test coverage and features released doesn't really represent debt. We know of course that the number of *possible* tests for any reasonable feature far exceed the number of *necessary* tests for that feature. The only line that matters to us the 'defects released' line. We choose our tests to minimize defects, not to maximize coverage. Rather than go for feature-for-feature coverage, we weave a web of tests that is likely to catch any regression errors.

We INVEST in automated tests (and refactoring, etc.) to MINIMIZE RISK. I have to emphasize again that this can really only be done with a skilled and disciplined team in a reasonably well-designed code base. In a less-expert environment, the idea of technical debt might be more useful.

So now to the metaphor:

I have a friend who is a successful artist and gallery owner here in our small town in the West. She started in sculpture more than 25 years ago, moved to oils, and recently has fallen for acrylics and is doing some of the best work of her career. She tends toward landscapes both real and imagined, although she sometimes takes for a subject flowers or saxophones.

She's a Western female painter of landscapes and flowers: why is she not as successful as Georgia O'Keefe?

If we had been talking about software, the answer would probably involve 'technical debt'. But we can see on the face of it that such an answer is silly. There is no such thing as technical debt in a practice or in a performance.

However, the concept of technical *investment* does clearly apply in both cases. Georgia O'Keefe was an expert and shaped her career in a certain way. And she had a lot of luck. My friend Karyn is also an expert and shaped her career in a certain way. And she had her own share of luck, also.

The careers of both of these successful women show a lot of investment, but different kinds of investment in different areas, different people, different times, with different results.

The important thing is not that Ms. Gabaldon is less famous than Ms. O'Keefe: the important thing is that both succeeded in their work.

So where would the idea of technical debt actually be useful? Consider the team encumbered by poor programming and poor testing. If the number of predictable defects released to production on your project is high, consider studying the literature of software development and of software testing in order to bring the skills of the team up to a level where technical investment makes more sense than technical debt.

The Software Artists: Citations for Part Three

2008-05-31T13:57:00.002-06:00

Citations For the section “Practicing, Rehearsing,
and Performing Software”

Programmer, poet and guitarist Richard Gabriel's quote is from
his proposal for a Master of Fine Arts in Software:
http://www.dreamsongs.com/MFASoftware.html

Apple CEO Steve Jobs' quote is widely noted, but is documented
in context by Andy Hertzfeld here:
http://www.folklore.org/StoryView.py?story=Pirate_Flag.txt

The practice/rehearse/perform rubric was first presented on the
author's blog in September 2007:
http://chrismcmahonsblog.blogspot.com/2007/09/become-better-software-artist.html

The title of this paper comes from the author's blog post of May
2007 in response to Jonathan Kohl's recommendation of theater
and musical experience for software testing:
http://chrismcmahonsblog.blogspot.com/2007/05/here-come-software-artists.html

A performance by Ella Fitzgerald and Count Basie available on
YouTube was a strong influence on this paper, cited in response
to another of Brian Marick's thoughtful blog posts:
http://chrismcmahonsblog.blogspot.com/2007/05/example-of-analogy-monks-vs-music.html

Luke Closs of Socialtext is a performing juggler and an excellent
agile developer. His talk “Juggling for Geeks” is a fine
performance: http://revver.com/video/317034/juggling-for-geeks-alpha/

The author performed the jazz standard “These Foolish Things”
on a $25.00 green ukulele at the Developer-Tester/Tester-
Developer Summit at Google in Mountain View in February
2007, to demonstrate (as good software testers know) that is is
possible to produce excellent results even with inferior tools.
http://googletesting.blogspot.com/2007/03/developer-testertester-developer-summit.html

The Software Artists: Practicing, Rehearsing, and Performing Software

2008-05-31T13:49:00.002-06:00

Software development is a performance exhibiting
skills developed by an individual—often in groups of teams...
-Richard P. Gabriel

Real artists ship.
-Steve Jobs

Since the language of art criticism provides useful tools for
evaluating software, and the practice of teaching music provides
useful approaches to teaching software, it is reasonable to think
that the practice of artistic performance should provide useful
ways to approach creating software.

Only one way to build software has ever been acknowledged:
software is designed, coded, and tested. Sometimes the
design/code/test cycle is done a single time, sometimes
design/code/test is done iteratively or incrementally, sometimes,
as in Test Driven Development, the steps are moved around, so
that the cycle goes test/design/code.

But this is not how human beings create a performance. Instead
of design/code/test, artists have a cycle of
practice/rehearse/perform.

Since human beings have been creating performances since
prehistoric times, it is worthwhile to see if creating software might
benefit from the practices of artists.

Practice

Before an artist is prepared to perform, the artist must have a
certain set of skills. Even the most naive or improvisational artist
will have practiced some aspects of their craft before performing.
Practice is where the artist acquires and improves the basic skills
of the craft. A musician will use practice to learn scales and
chords and rhythms, an actor will practice lines and passages, a
painter will practice brush strokes and coloring.

By practice the artist acquires not only the fundamental
knowledge of the craft, but also masters the necessary tools for
that craft. A musician will learn an instrument, a painter will
learn brushes and pigments and canvases, an actor will learn
speech and movement.

When practicing for software, the software creator will learn
applications and programming languages, design patterns and
editors. The software creator will also learn operating systems
and network protocols, the basic frameworks on which software
is created.

When practicing for a particular software performance, a software
creator might use a “spike” in the agile sense, or a simple
prototype of some feature or features to be developed.
Regression tests could be considered as a tool for practice,
because such tests reinforce good software behavior.

Rehearse

As Dr. Gabriel says, software is created by groups. But whether
solo or in a group, artists rehearse before they perform. Rehearsal
is where artists behave as if they were performing. In rehearsal,
the various aspects of the performance are worked out, and the
performers examine the boundaries of the performance to make
certain that what is expected of them is in fact not only possible,
but will also be attractive to an audience.

Integration testing and functional testing might be tools of
software rehearsal, also beta versions and “dogfooding”, using the
software internally or among a small group before releasing the
software. User experience testing might be part of rehearsal.
Pair-programming could be a tool of software rehearsal, as could
code inspection.

Investigation is a key part of rehearsal. After practicing, artists
still need to know where misunderstandings exist, where aspects
of the performance may break down, where skills may not be
adequate to the demands of the performance.

Iteration is a key part of rehearsal. Rehearsal often involves a
close examination of the performance looking for ambiguity and
wrong expression. Upon discovering such issues, the small part
of the work exhibiting the problem is reworked until the
performers agree on the proper way to conduct the performance.

Listening to a symphony orchestra rehearse is enlightening. A
major symphony orchestra is comprised of the most
accomplished musicians on Earth. These musicians have
dedicated their lives to practice and performance. A symphony
orchestra is also an extremely expensive operation.

In a symphony rehearsal, the conductor or music director will
have the orchestra play only a few bars from passages that the
conductor believes to be problematic in some way. Based on
only a few seconds or minutes of playing, the conductor will
adjust the performance of various players and sections of the
orchestra in a very short period of time. These adjustments are
understood to also apply to the rest of the performance.
Some things about artistic performance could greatly improve
software performance:

For an artistic performance, everyone on the project has a role on the stage. The best agile development projects emulate this behavior, since everyone on the team is always actively involved in creating the software.
If rehearsals for an artistic performance are poor, the performers may retire to practice more before continuing with rehearsal. The notion of a software “spike” is very much like this, when the software creators put their regular work aside in order to learn something new about a relevant aspect of creating the software.
Performers are rarely micro-managed. Successful artistic projects and successful software projects assume that the performers have a certain level of competence, and remove people who do not have that level of competence.
The value of practice and rehearsal is measured qualitatively not quantitatively. For both software and art, the creators worry first about the excellence of the performance, and only later about the number of tickets or copies of the performance to be sold.

Perform

Serious artists perform their art for an audience. Serious software
creators release software for their users.

Performance is where value is realized, where expertise is on
display, where practice and rehearsal pay off: for the performers,
the creators, the investors, and for the audience.

Maybe it is time for the software development practice to begin
referring not to “the users” but to “the audience”.

To the Studio

Artists' studios, rehearsal studios, movie studios, and recording
studios are busy places. They are full of the artists' tools, toys,
and works in progress, and they are arranged for the convenience
and inspiration of the artists.

A factory floor is arranged for the convenience of managing the
product.

This tension between the artist and the manufacturer is not new.
For instance, Andy Warhol's studio was called The Factory.
Many movies and photographs of The Factory exist. The irony is
clear.

It is notable that cubicle farms, departments, and isolated offices
are giving way to bullpens, open spaces, war rooms, co-working
space, home offices, even coffeehouses and brewpubs. The
correlation to artists' studios is clear.

It is notable that a small number of software creators are already
being treated like artists. Linus Torvalds, David Heinemeier
Hansson, and Richard Stallman all come to mind.

Perhaps in the future the software artists will practice and
rehearse in their software studios so that they can release software
performances to their audiences. The artists and the audiences
and everyone else involved in these software performances will
understand the value of these performances, and the feedback
between artist and audience will increase the value of these
performances. People have been doing exactly this in art for
thousands of years.

The Software Artists: Citations for Part Two

2008-05-31T13:45:00.001-06:00

Citations for the section “Pedagogy and Practice in
Software and in Guitar”

Some of Joel Spolsky's writing about working with young
software creators is at these links:
http://www.joelonsoftware.com/articles/CollegeAdvice.html
http://www.joelonsoftware.com/articles/ThePerilsofJavaSchools.html
http://www.joelonsoftware.com/items/2008/01/08.html

Thoughtworks University had been called “Thoughtworks Boot
Camp” until the author had to explain the term upon crossing an
international border in 2005. A description of Thoughtworks
University is here:
http://www.thoughtworks.com/work-for-us/TWU.html

Some representative citations from IEEE publications about poor
software education are:
http://ieeexplore.ieee.org/Xplore/login.jsp?url=/iel5/8994/28540/01276501.pdf
http://stevemcconnell.com/ieeesoftware/eic12.htm
http://csdl2.computer.org/persagen/DLAbsToc.jsp?resourcePath=/dl/mags/so/&toc=comp/mags/so/2002/05/s5toc.xml&DOI=10.1109/MS.2002.1032848

The citation from CrossTalk is here:
http://www.stsc.hill.af.mil/CrossTalk/2008/01/0801DewarSchonb erg.html

Samples of the discussion of the apprentice/journeyman/master
model in software are at these links:
http://butunclebob.com/ArticleS.UncleBob.TheProgrammingDoj o
http://www.mcbreen.ab.ca/SoftwareCraftsmanship/

Of particular interest is Dave Hoover's account of hearing Bob
Martin's presentation at Agile2007:
http://redsquirrel.com/cgi-bin/dave

The developers at Atomic Object also believe that software is a
craft:
http://www.atomicobject.com/pages/Software+as+a+Craft

Brian Marick's blog post about skill, discipline, ease, and joy is
here:
http://www.exampler.com/blog/2007/05/16/six-years-later-what-the-agile-manifesto-left-out/

This interview with Robert Fripp was very helpful:
http://emusician.com/em_spotlight/fripp_philosophy_guitar_craft/

Eric Tamm's biography of Robert Fripp has a wealth of
information. Chapters 10 an 11 concern Guitar Craft, and were
very helpful:
http://www.progressiveears.com/frippbook/contents.htm

The Guitar Craft website is
http://www.guitarcraft.com/
The site behaves oddly sometimes though, so a direct link to “A
Preface to Guitar Craft is here:
http://www.guitarcraft.com/?np=5&&id=4

A number of Robert Fripp quotes are here:
http://www.brainyquote.com/quotes/authors/r/robert_fripp.html

The description of Bob Martin's ideas of “professionalism” come
from the abstract of his presentation at Agile2007:
http://www.agile2007.com/agile2007/index.php?page=sub/&id=649

The Software Artists: Pedagogy and Practic in Software and in Guitar

2008-05-31T13:28:00.002-06:00

Very few people graduating with a college degree in Computer
Science or Information Technology are prepared to write
production code or to test production software.
Companies that hire recent college graduates often have special
training for such employees before they may work on actual
projects. Joel Spolsky has written extensively about how his
company Fog Creek trains young software workers. Publications
of the IEEE mention the issue frequently. The well-regarded
software consultancy Thoughtworks has “Thoughtworks
University”, a six-week “boot camp” style training event for new
hires hosted in Bangalore. Even the journal CrossTalk: The
Journal of Defense Software Engineering says:

It is our view that Computer Science (CS) education is
neglecting basic skills, in particular in the areas of
programming and formal methods.

The Craft of Software

Besides the mentorship approach Spolsky takes and the “boot
camp” approach Thoughtworks takes, Robert C. (Uncle Bob)
Martin, Pete McBreen, Dave Hoover, and others have for some
time advocated a master/journeyman/apprentice system for
teaching software, much like the ancient guild systems. Mr.
Hoover in particular reports notable success with this approach.
But using 15th century teaching techniques for a 21st century craft
seems unlikely to succeed on any reasonable scale without
considerable adjustment.

For one thing, both the in-house training systems and the guild
systems of pedagogy tend to keep effective practices secret.
Knowledge is passed from person to person directly, without
involvement from the community.

For another thing, craft alone is not sufficient to produce excellent
software or excellent software creators.

Brian Marick has a remarkable weblog post from May 2007
called “Six years later: What the Agile Manifesto left out”. In it
he cites four values in which software creators should invest:
skill; discipline; ease; and joy. Of this list, Marick seems to
equate “skill” with Martin's idea of “craft. “Ease” and “joy”
follow from the practice of skill and discipline. The most
remarkable aspect of this essay is that “discipline” is a new idea
in the public discourse of software practice.

Martin's idea of “craft” and Marick's idea of “discipline” exactly
echo the language of the teaching philosophy of the well-known
guitarist Robert Fripp.

Robert Fripp: Craft and Discipline

Those who do not know Robert Fripp's musical work may know
him as the creator of the new sounds for Windows Vista. He has
been at the forefront of technology in music since the late 1960s.
Since 1985 he has been teaching an approach to music and guitar
called Guitar Craft. Many of his students, for instance the
California Guitar Trio, are quite successful in their own right.

There are sanctioned Guitar Craft programs in Italy, Japan,
Mexico, North America, South America, Spain, and the UK.

The two axes of Fripp's teaching methods are Craft and
Discipline. Fripp uses these terms in a way very similar to the
way that Martin and Marick use them. But unlike Fripp's notable
success at teaching music over the last two decades, software has
no single successful long-running education project with a
consistent philosophy.

While a mature discussion of software craft exists, that discussion
has yet to demonstrate any repeatable successes. Martin
discusses software craft in conjunction with professionalism, but
the milieu in which “professionalism” is exercised is somewhat
limited: professionalism for Martin is “...about behavior, attitude,
and professional ethics”, and that “Writing clean code is a matter
of both skill and character. “

These are fine and noble sentiments, but ethics and character have
little to do with actually creating great software.

Brian Marick, in the same essay where he identifies the four
values, explicitly says that “...new Agile teams... are
floundering.” While his essay identifies the problem, it fails to
explain how to exercise skill, discipline, ease, and joy in the
service of creating software, except to pronounce that these
qualities are necessary.

By contrast, Guitar Craft has thousands of words from Robert
Fripp and others on how the interaction of Craft and Discipline
are necessary for performing this music on guitar:

The invisible side of craftsmanship is how we conduct
ourselves within the process of acquiring craft, and how we act
on behalf of the creative impulse expressing itself through music.
In time, this becomes a personal discipline. With discipline, we
are able to make choices and act upon them. That is, we become
effectual in music.

It is notable that when Fripp re-formed the band King Crimson in
1981, they chose to name their new album “Discipline”.

Like Martin, Fripp uses the apprentice/journeyman/master
metaphor, but adds an additional role of “genius”. Each role is
delineated in a number of ways:

The apprentice acquires skill.
The craftsman acquires sensitivity.
The master acquires vision.
The genius attains freedom.

The apprentice is noisy.
The craftsman shapes sound.
The master shapes silence.
The genius is silent.

The apprentice practices the craft of craft.
The craftsman practices the art of craft.
The master practices the craft of art.
The genius is artless.

Beyond the four roles, Guitar Craft also recognizes seven levels
of accomplishment. The details of the seven levels are not
relevant to this discussion, except to point out that the well-
delineated roles combined with levels of achievement that
correspond to the student's commitment to practicing Guitar Craft
provide a well-designed structure for communicating the
principles of Guitar Craft not only to the community itself, but
also to those outside the community who may be interested in
examining the practice of Guitar Craft. Software craft has no
such well-defined hierarchies of learning or commitment.

There are of course technical details of Guitar Craft, having to do
with posture, tuning, left- and right-hand techniques, etc. etc.
Such details are beyond the scope of this paper. Such details,
while important when playing the guitar, are not relevant to the
discussion of whether software craft can learn from Guitar Craft.

While it is not actually possible (or desirable) to separate the
philosophy of Guitar Craft from the actual practice of playing the
guitar, there is so much in the written history of Guitar Craft that
is provocative for “Software Craft” that mining the Guitar Craft
canon for software methodology could be a valuable exercise:

If an apprentice does not hear what a master hears, is
then that quality not present in the music? Yes and no.
In the world in which the apprentice lives: no.

...if I cannot manage to persuade the members (of the
group) of what I see to be the next course of action,
how do you expect the group to deal with the
expectations of thousands of people. It is not possible.

These are ten important principles for the practice of
craft:
Act from principle.
Begin where you are.
Define your aim simply, clearly, briefly.
Establish the possible and move gradually towards the
impossible.
Exercise commitment, and all the rules change.
Honor necessity. Honor sufficiency.
Offer no violence.
Suffer cheerfully.
Take our work seriously, but not solemnly.
You have to find a way of recognizing and describing
the quality inherent in the music. If you fail to do that,
then the piece itself is compromised.

The history of Guitar Craft demonstrates that the ideas of Craft
and Discipline together can create a successful international, self-
perpetuating teaching and performing regime. And the language
today's most skilled and prolific software creators that it seems
worthwhile to find out if ideas from Fripp's way of teaching
music might translate to the similarly creative process of teaching
software.

The Software Artists: Index of Links to All Sections

2008-05-18T11:12:00.002-06:00

Explanation
Introduction

The Value of Software
Citations for Part One

Pedagogy and Practice in Software and in Guitar
Citations for Part Two

Practicing, Rehearsing, and Performing Software
Citations for Part Three

The Software Artists: Citations for Part One

2008-05-18T11:04:00.002-06:00

Many of the ideas in this paper were first presented at the Austin
Workshop on Test Automation in early 2007. The substance of
the talks appeared on the author's blog shortly afterward:
http://chrismcmahonsblog.blogspot.com/2007/01/craft-and- discipline-larks-tongues-in.html

Most of the New Criticism and Structuralist citations are from
Wikipedia, except:
Child Jr., William C. 2000 Monroe Beardsley's Three Criteria for
Aesthetic Value: A Neglected Resource in the Evaluation of
Recent Music. Journal of Aesthetic Education, Vol 34, No. 2
(Summer 2000), pp 49-63 doi:10.2307/3333576
The author did not know that unity/variety/intensity had been first
presented in music criticism until reading this article.

The author also referred to
Adams, Hazard, Searle, Leroy (eds.) 1986 Critical Theory Since
1965 University Presses of Florida/Florida State University Press
for background information.

Bruce Schneier on “security theater”:
http://www.schneier.com/blog/archives/2006/08/terrorism_secur. html

The Software Artists: The Value of Software

2008-05-18T10:45:00.004-06:00

Previous: The Software Artists: Explanation
Previous: The Software Artists: Introduction

Philosophy of Art and the Value of Software

Manufactured goods generally have a clear relationship between
cost, price, and value. In software, as in art, the value of the work
is more often completely unrelated to cost and price. Operating
systems provide a great example: the Windows audience for the
most part must use Windows regardless of cost or price. The
Mac OSX audience generally chooses to use OSX regardless of
price, and often explicitly because of the aesthetic experience of
using OSX. Linux has no cost at all, and a price that varies
wildly, and it also has a dedicated audience.

The value of software, like the value of an artistic performance,
lies in the ability of the software to attract and keep an audience.
The software industry would benefit immensely by turning the
tools of artistic criticism on software.

The 20th century in particular saw a great proliferation of critical
theory of artistic work. Among the most important ideas were
The New Criticism and Structuralism. Both provide fine tools
for evaluating software.

The New Criticism: The Intentional Fallacy and Aesthetic Value

The New Criticism was the most important school of literary
criticism in the middle of the 20th century. An essay this length
can touch only lightly on some key ideas of New Criticism, but
those ideas turn out to be quite valuable.

The New Critics believe that once the authors finish their works,
they disappear from the milieu in which the work exists. New
Critics do not involve themselves in what the author intended to
create; they concern themselves only with the work as it exists.
This principle is called the “Intentional Fallacy”: what the author
intended has no part in the value of the author's work.

Creating software can be seen in a similar light. Once the
software is released, the team that created it cannot control how
the software is used, nor who uses it, nor what it is used for. The
value of the software must lie in the software itself and in how it
is used, not in how it was created or in what it was intended to do.

The most important tool of the New Critics is a technique called
“close reading”. From Wikipedia: “close reading describes the
careful, sustained interpretation of a brief passage of text. Such a
reading places great emphasis on the particular over the general,
paying close attention to individual words, syntax, and the order
in which sentences and ideas unfold as they are read.”

Creating software has its own kinds of close reading: code
review, unit tests, and feature tests are all ways in which software
creators emphasize the particular over the general. But one of the
most important results of close reading, and one of the most
important aspects of value to New Critics, is to identify and
examine the works for ambiguity. These ambiguities are
examined for their value: some kinds of ambiguity add to the
value of the work; some kinds of ambiguity detract from the
value of the work.

Software whose ambiguous features allow the user to do more
than the developers intended will be more valuable-- think of a
wiki. Software whose ambiguous features stop the user in his
tracks has much less value-- think of the Windows Vista security
regimen.

Aesthetic Value: Unity, Complexity, Intensity

Monroe Beardsley, who wrote “The Intentional Fallacy” along
with William K. Wimsatt, also proposed a way to measure the
value of a particular work. Beardsley states that the value of the
work resides in three criteria: unity, variety, and intensity.
Applying these criteria to software is an interesting exercise.
Software may be unified if all of its features support a central
activity or theme. Software may have variety if the features cover
a wide range of activity. Software may be intense if using the
software is a compelling experience.

Periodically examining a software project throughout the
development process using these criteria is a fascinating exercise.

Structuralism: Signs and Myths

Structuralism became popular later in the 20th century. Where
New Criticism seeks to examine particular works disconnected
from intent or affect, Structuralism is rooted in linguistics and
anthropology, and seeks to examine works in the context of their
language and of their culture.

Two aspects of Structuralism are of particular interest: the idea
that linguistic signs can be decomposed into the signifier and the
signified; and that works can have deep structures that reflect
cultural values that can be represented as myths.

Signs: Signifier and Signified

A linguistic sign is a speech act: a word, a sentence, a poem, a
book. It is not at all unreasonable to treat the use of a software
feature or a software user interface as a sign.

The signifier is the “sound pattern” of the sign. It is what
happens in the physical act of speaking, or in the personal act of
reciting to ourselves. Furthermore, signifiers are the tokens by
which people communicate. Signifiers are what people send each
other as they participate in culture.

The signified is the “concept or meaning” of the signifier. The
signified is the true act or true feeling or
physical/emotional/apprehended manifestation of the signifier.
With the ability to separate signifier and signified, the
Structuralists look for common deep elements and mythological
underpinnings of artistic and cultural works.
The value of any particular work, then, can be evaluated in
several ways. A work might have value to a Structuralist if:

Signs are rich and easily available
Signifiers in the work are rich and pleasant
Signified are deep and clearly delineated by their
The work reflects a rich understanding of the cultural mythology and milieu

Exercise: New Critical and Structuralist Evaluation of Browser Behavior

New Critics contribute the idea of close reading in order to
evaluate unity, variety, and complexity of a work, looking for
ambiguity in the service of a rich aesthetic experience.

Structuralists contribute the ability to separate signifier and
signified within a work, in order to evaluate the richness of both
the experience of the work itself and the extent to which the work
relates cultural mythology, either that of the work itself or that of
the consumer of the work.

As a thought experiment, use these tools to measure the value of
this behavior:

When Internet Explorer opens a web page that contains MPEG3
files (for instance
http://www.drummingstyles.com/Genres/Latin/Bossa-
Nova/index.html), it by default prevents the user from loading the
page, and pops up a warning message that reads

(IE6 and IE7) Do you want to allow software such as
ActiveX controls and plugins to run?

(IE6) A script is accessing some software (an ActiveX
control) on this page which has been marked safe for scripting.
Do you want to allow this?

(IE7) This website wants to run the following add-on:
'Windows Media Player' from 'Microsoft Corporation'. If you
trust the website and the add-on and want to allow it to run,
click here...

A New Critic can use close reading on this behavior to see if it
exposes any ambiguities. If those ambiguities contribute to the
unity, intensity, and variety of the experience, then the behavior is
valuable.

Since the only executable files on the page are mp3, the false
reference to ActiveX sends the reader down a false path. Since
there is no explanation for why an mp3 is considered to be
ActiveX, this falseness only detracts from both the unity and the
intensity of the experience.

Furthermore, since the software requires validation of behavior
that the reader requested anyway (the false “ActiveX control” is
in fact “safe for scripting”) this behavior can only detract from
the whole point of the work, which is to play the mp3 file while
reading the text of the page.

So a close reading of the the behavior in question in a New
Critical sense shows that the software reports false information.
Although the behavior could be construed as adding variety, it
does so for no good reason, and prevents the experience of the
work itself, which is to play the mp3 files while reading the page.

A Structuralist would have a different interpretation. The
references to ActiveX are a signifier, but it is unclear exactly what
is being signified, since the obvious signified (a real ActiveX
control) does not actually exist.

The Structuralist finds a clue in IE7, where under some
circumstances, when loading a page with mp3 files, warns “This
website wants to run the following add-on: 'Windows Media
Player' from 'Microsoft Corporation'. If you trust the website and
the add-on and want to allow it to run, click here...”

Microsoft itself is concerned about security, or at least about the
appearance of security. Perhaps the false warnings are intended
to warn about the possible insecurity of the Microsoft Windows
Media Player itself.

Looking at the wider culture, everywhere from airports to banks
to offices, there is a demonstrable trend toward what Bruce
Schneier calls “security theater”. Security theater is
“countermeasures that provide the feeling of security while doing
little or nothing to actually improve security”.

A Structuralist will see that the culture that produced this work
values “security theater” while providing only the appearance of
security, so the Structuralist will concede a certain value to this
Internet Explorer behavior that a New Critic would not.

Citations for part one

The Software Artists: Introduction

2008-05-11T16:06:00.001-06:00

The people who create software are not factory workers. Nor are they engineers, in the sense that engineering is the “practical application of the knowledge of pure sciences, as physics or chemistry”. But the software industry continues to treat software workers as if they were factory workers or construction workers. The software industry also attempts to value software as if it were a manufactured product.

But making software is a fundamentally creative process, more similar to performance than to manufacturing. Like art and music, software has an audience that is involved in a personal way with the software. And the people who create software are much more like performers than they are like construction workers.

If it is true that software is much more like art than it is like manufacturing, then the tools of artistic criticism should be useful for evaluating software.

It should also be possible to apply successful approaches to art or music pedagogy to software pedagogy.

Furthermore, it should be possible to manage software projects in the same way that artists manage performances, with better results than if the software projects were managed like manufacturing or engineering projects.

The Software Artists: Explanation

2008-05-11T15:33:00.003-06:00

I wrote a paper some time ago to submit to the Conference of the Association for Software Testing, but the paper was not accepted for the program. I'm on the waiting list if another presenter drops out, which seems unlikely at this point.

I think it is likely that my paper was not accepted at CAST was because it is somewhat similar to a presentation from Jonathan Kohl and Michael Bolton, two of the best testers in the universe. I intend to publish my paper here on my blog in the hopes that Jonathan and Michael and other Software Artists will have access to as much relevant information as possible to support their position.

I think that one enormous reason that few people take software-as-artistic-practice seriously is because of a perceived lack of practical application: manufacturing and engineering provide venerable examples of measurement tools, education curricula, and market strategy-- assuming that you believe that software is an engineered and manufactured product. Those of us who believe that software is an artistic process have failed to provide compelling alternatives to the tools provided by engineering and manufacturing.

I set out to start to remedy that situation with this paper.

It is a long piece, so I'll publish it in several sections in the coming days, as I have time to get it right and looking good. I have extensive references also. My plan is to publish references as separate blog posts after each section, so that people who don't care can skip them and people who do care can relate them to recent posts.

technical debt as impedance mismatch

2008-04-18T14:39:00.002-06:00

My colleague Matt Heusser will be hosting a peer conference on "technical debt" later in the summer. I've been thinking about the subject and realized that technical debt could be interpreted as a kind of impedance mismatch. Impedance mismatch happens in acoustics, electric current, and many other places. Here are a couple of examples that everyone will understand:

trying to fill a swimming pool with an eye dropper

trying to get a drop of water from a firehose

I can think of two software examples from my own career, one involving tools, the other involving skill.

An application I was testing the other day has two javascript prompts in a row. Selenium correctly recognized and dismissed the first prompt, but it is not able to see or manipulate the second one. As a result, I have an unfinished automated test, and technical debt in the form of selenium hacking, manual testing, and future test maintenance. This is the eyedropper-swimming-pool example: no matter how much effort I expend, there is no way I can overcome the nature of the problem at hand. I'm forced to improve my tools in order to overcome technical debt.

In my career I have been a tester on two different COBOL systems. One was very nicely designed and maintained, the other was a total nightmare. The developers of the nightmare system were very poor coders. COBOL makes it easy to do dumb things, and the codebase was full of dumb things from people who didn't know any better. This is the water-drop-from-a-firehose problem: gross efforts were accepted when fine manipulation was required. The developers have to improve their skills in order to overcome technical debt.

If this really is a valid comparison, it might even be measurable in the way that electric current and sound pressure levels are measurable. If not, it is still worth noting that having the appropriate tools and the appropriate level of skill in the project team will likely minimize technical debt.

advice on a selenium problem?

2008-03-18T12:09:00.002-06:00

I thought I'd mention this here. If anyone has a theory, please leave a comment or find me somewhere...

I'm using Selenium to drive a Javascript WYSIWYG editor. I type some text into a largish textarea; save the page; open the page again in the editor.

Upon doing this, IE7 can properly read and parse the contents of the textarea, but Firefox thinks the textarea is empty.

I can set up an environment to reproduce this for an interested party.

my lecture at Fort Lewis College

2007-12-06T22:44:00.001-07:00

Boulder Ruby Group: telecommute panel Nov 14

2007-11-13T21:51:00.000-07:00

I'll be part of a panel on distributed agile working/telecommuting/remote working tomorrow evening Nov 14.

If you can get to Boulder, stop in. If you're interested in following along, there'll be an IRC channel running. (probably freenode #socialtext, unless we change our minds.)

Also, I have a wiki dedicated to the topic. For tomorrow evening the wiki will be totally public. I intend for the wiki to stick around as a repository of information about working in distributed teams. Leave a comment or send an email if you'd like access to that wiki.

test design economy

2007-11-02T22:38:00.000-06:00

The other day Jeff Fry sent me an IM out of nowhere and in the course of the conversation asked me what I was doing these days.

I blithely answered "not so much coding, more test design". Which was really too short an answer, but there's IM for ya.

In a nutshell, Socialtext has a really nice Selenium-based automation framework, and recently I've been attacking the backlog of creating automation for fixed bugs. When I'm doing manual (Exploratory) testing on these code changes, I'm willing to spend time investigating odd combinations of things, multiple recurrences of things, all sorts of unusual experiments to find where exactly the boundaries of the functions lie.

But automated tests are expensive: they are expensive to create, and they are expensive to run. So when writing the automated regression tests, I'm trying to minimize the number of test steps while maximizing the coverage of the code in question.

Here's an example:
| type_ok | email_page_subject | Fw: Fwd: Re: Email subject %%start_time%% |

We needed to test that email forwarded or replied to, on the same subject, would appear on a single wiki page. Manually, I tested every combination of "Fwd:", "Fw:", "Re:", after investigating all the permutations that various email clients might employ.

But when I wrote the automated test, I chose the most pathological case that I knew would succeed, in order to minimize run time and maximize the chance that the test would catch a regression problem.

Automated GUI testing is too expensive to cover all the cases that a human being can conceive. There is a craft and a discipline involved in creating automated tests that are both effective and economical.

I am very much enjoying exercising that craft and discipline.

actual code: create vs. sort

2007-09-27T21:33:00.000-06:00

don't test for blocking conditions: an example

2007-09-26T22:39:00.000-06:00

Matt Heusser quoted me on this at the Google Test Automation Conference. I've had this idea for some time, that many tests both manual and automated waste time explicitly testing for conditions that don't need tests.

I ran into one of those today. Or rather, my boss did. (He's not really a boss so much as he is an experienced colleague who has more conversations with management than me. Socialtext has very little hierarchy. )

We have a lot of automated tests that we created very quickly. Many of them have TODOs left over. I've been cleaning up the tests and implementing the TODOs.

One of the tests had a whole list of TODOs that had already been done, so I deleted the TODO list. My boss saw this and asked me why. In particular, one of the TODOs had been to create a record of a particular type, and my boss didn't see a test for this.

I pointed out that I had written an explicit test for sorting records of exactly that type, and three records of that type were created in the setup for the sorting test.

Had the record-creation failed, the tests for sorting would have failed. It's a nice economy of effort.

sysadmin week in QA

2007-09-21T09:24:00.000-06:00

Go on, click the link, read the blog post, then read the source. Socialtext's REST API is pretty great.

Luke Closs/Distributed Agile/Agile Vancouver

2007-09-19T20:47:00.001-06:00

If you're near Vancouver, you should be there.

I'm interested in what Luke's going to say, and I work with him.

Joe Zawinul

2007-09-12T00:32:00.000-06:00

Joe Zawinul died today. He was 75. He wrote "In a Silent Way", and one of my favorites "Mercy Mercy Mercy" (hard to find an original version on teh internets), a hit for Cannonball Adderly.

He played in Miles' Bitches Brew band. He founded Weather Report and dealt with Jaco.

Bring on the artists.

Become a better software artist

2007-09-09T22:36:00.000-06:00

There's been an interesting discussion on the software-testing mail list about the failures of Six Sigma and other process-quality religions to map effectively to software development. I think you might get better software the same way that you get better music.

Practice.

Almost every musician starts by playing along with recordings. After that, playing along with the radio builds improvisation skills. After that, playing along with other people broadens and deepens the communication spectrum. Throughout all this, work on the basics like scales and chords and tempo don't stop. The conscientious musician practices as many styles of music as possible. The professional musician may not enjoy blues, or orchestral works, or jazz, but there are certain forms that all professionals master in order simply to be professional.

Rehearse.

Music is constructed for listeners. Rehearsal is where the details of the performance are designed and implemented. Rehearsal is not where musicians improve their skills; it is where musicians, together, plan to present their work to an important audience.

Perform.

Real artists perform. They create music for real listeners, who pay for the privilege of doing so.

Trying to apply Six Sigma or Lean or CMM in order to get better musical performance doesn't seem very smart. But humans have been performing music for our entire time on the planet. We know pretty well how to go about getting good performances. If it's software:

Practice.

Work the exercises in the books. Then make up your own exercises, or look for interesting questions. (My first serious program was to throw the I Ching in Perl, using the instructions from the back of the Wilhelm/Baynes translations as an algorithm. The script also calculated the changing lines, and printed both hexagrams to the console. I learned a lot.) Learn the basics, and keep expanding: conditionals, loops, objects, design patterns. Learn other languages, even if you don't like them much. (I've started Head First Java 3 times, I'm slack.)

Rehearse.

Share your programs, work with others to improve theirs. Publish your code as a demonstration or a work in progress. Get feedback, and read and comment on others' work. Join an open source project. Or a couple of them. Write documentation-- if you can't write it down clearly, then you don't understand it well enough.

Perform.

Real artists ship. -Steve Jobs
Make stuff for users. As often as you can. All kinds of users, all kinds of products.

apropos of not much: started a photo blog

2007-08-30T23:25:00.000-06:00

I've been thinking about doing this for some time, and now it's happened. I got a nifty little bullet-proof camera and started a photo blog of the things I see from day to day. Considering where I live, some of them are pretty amazing.

so where's the testing

2007-08-17T17:55:00.000-06:00

The Socialtext job is on craigslist now and I've been reviewing resumes. We have several great candidates, but most of the people who've submitted resumes don't seem to be able to help out much.

In particular, I'm seeing a lot of resumes that talk about how their owners do almost everything *except* actually test software. There seems to be lots of reporting and managing and reviewing and meetings and stuff, but actual testing experience seems to be pretty thin on the ground.

Or maybe that's just how people write resumes.

Socialtext is hiring a tester/QA person

2007-08-13T14:13:00.000-06:00

The job announcement should show up here any minute. In the meantime, you can read the job description.

This is a great job, and we're looking for one outstanding person. You can google for Socialtext yourself (I recommend doing some research before you apply), and read the official job description. Here I'm going to talk about why this is such a great job, and the stuff you need to have to get it that isn't in the job description.

Socialtext has about fifty employees. A few are based in Palo Alto, but the development, QA, and support staff are spread across the US and Canada, along with a few overseas employees. If hired, you would be expected to telecommute. If you happen to live in the Bay Area, you'd be welcome to visit the office.

We're a small-a agile shop. We have standups, retrospectives, and iterations. We release small features frequently, and we install them on our internal wiki before we release to production.

Besides the wiki, we use irc, email, VOIP, VNC, and more, to communicate and share information. We're in constant contact with each other through the day.

Since so much of our communication is with text, excellent writing skills are critical. Since there are so few employees, abilities beyond testing are very important. If you know about business, or development, or marketing, or Open Source, that's important to us. You can have a real influence on how the company works.

We have some interesting company policies. For instance, the official vacation policy consists of five words: "Be An Adult About It". We have three salaries. Everyone who joins gets paid one of the three salaries. Everyone in the company (usually) has a quarterly travel budget. When you join, you get an allowance to buy yourself hardware. Most people have Mac laptops, but we have a couple of Windows users and a couple of Linux users.

A lot of our development work is done in public. Surf http://www.socialtext.net/open to see what we're up to. The coolest thing I have over there is three small tutorials for how to use our REST API with a Ruby client.

We have a lot of very smart people who hold strong opinions. We use strong language, because we don't have the bandwidth to play politics. We criticize ideas, not people, we work on fixing code, not fixing blame. Discussions get heated, but aren't personal. If you're offended by four-letter words in the service of making a point, this is not the place for you to work.

Besides unit tests, we have an automation framework based on Selenium. We've quadrupled the number of test steps since April, from about 1000 to about 4000. I'm using Watir and FireWatir to do basic analysis of page load times. If you know test automation, Selenium, or performance testing, you'd be very welcome. Performance in particular is just coming under close scrutiny. In the next six months we are planning to make significant improvements to all of our test architecture. It promises to be a really cool project.

We have about 10 developers. When we add another QA person, we'll have a pretty good dev/test ratio.

This is a great job with great people and a very interesting market.

If you'd like to apply, send your resume as text in the body of an email addressed to jobs@socialtext.com. You can send it to me, too.

just enough design

2007-08-07T10:13:00.000-06:00

So in recent weeks the number of test steps in our Selenium-based framework has quadrupled, from about 1000 to about 4000. The time it takes to run these tests is starting to cause significant pain.

So I set out to spike a little harness that would run sets of tests concurrently. (Note: it is *so* nice to have fork() available, after years of working in Windows environments.)

Trying to achieve the Simplest Test Harness That Could Possibly Work, I started out running all of the tests in the same test environment. Unfortunately, some of the tests change the state of the test environment, which causes other tests running concurrently to fail.

I started separating the state-changing tests (which may not be run concurrently) from the other tests (which may be run safely).

My colleague objected to this. Strongly. And he's right. He framed his argument in terms of old-school "software engineering" principles, but the argument is sound in terms of agile mantras like "make it work; make it good; make it fast". Segregating magic tests from non-magic tests will eventually cause failure. Or madness. Or both.

My next spike, I'm going to clone the test environment and run sets of tests in their own test environments. I think that's the *next*-simplest thing that could work, and it will get us a long way down the road-- until we have to design some more.

new open source license: Common Public Attribution License

2007-07-26T14:45:00.000-06:00

The Register has probably the best coverage so far.

This license is particularly valuable where the market is saturated with competitors, and for companies offering "software as a service".

OSI did the right thing, I think.

data-driven test framework with Socialtext wiki as repository

2007-07-18T15:25:00.000-06:00

Socialtext has a tradition of having "Wiki Wednesday" once a month, where people hack little wiki-related projects.

I had a little Wiki Wednesday myself today. Socialtext wikis have a very nice (and not very well documented) REST API. I've been figuring out to use this API in Ruby. I got to the point today where I could read and write pages to the wiki, so I hacked up a little framework using Watir that reads test data from one page and updates another page with the results of the tests.

Wiki markup makes parsing the data really easy, and the REST API makes updating pages really easy.

@test_data_page looks like
| Google Test | Google | Google Search | I'm Feeling Lucky |
This displays as a table in the wiki.

It's in dire need of some refactoring. Reading pages with more than one line of test data is left as an exercise for the reader. :-)

If you want to muck around with it yourself, you could get a free account from socialtext, or try the open source version or the VMWare image.



require 'net/http'
require 'test/unit'
require 'watir'
include Watir

class TC_goog < Test::Unit::TestCase
def test_goog

@user = 'user@socialtext.com'
@pass = 'password'
@host = 'mineral.socialtext.net'
@port = '33333'
@test_data_page = "/data/workspaces/admin/pages/testdata"
@test_result_page = "/data/workspaces/admin/pages/testresults"
@test_results = ""

  def get_page(loc)
    http = Net::HTTP.new(@host, @port)
    http.start do |http|
    request =
      Net::HTTP::Get.new(loc,initheader = {'Accept' => 'text/x.socialtext-wiki'})
    request.basic_auth @user, @pass
    response = http.request(request)
    response.value
    @page = response.body
    end
    return @page
  end

  def put_page
    req = Net::HTTP::Put.new(@test_result_page, initheader = {'Content-Type' => 'text/x.socialtext-wiki'})
    req.basic_auth @user, @pass
    req.body = @test_results
    response = Net::HTTP.new(@host, @port).start {|http| http.request(req) }
      puts "Response #{response.code} #{response.message}:
    #{response.body}"
  end

get_page(@test_data_page)
puts @page

test_data = @page.split("|")
test_name = test_data[1]
page_title = test_data[2]
button_name1 = test_data[3]
button_name2 = test_data[4]

ie = IE.new
ie.goto("http://www.google.com")

@test_results << @page
@test_results << "|" + test_name
  begin
    assert_equal(page_title,ie.title)
    @test_results << "|PASS"
  rescue 
    @test_results << "|FAIL"
  end 
  
  begin
    assert_equal(button_name2, ie.button(:name, "btnG").value)
    @test_results << "|PASS"
    rescue 
    @test_results << "|FAIL"
    end
    
  begin
    assert_equal(button_name2, ie.button(:name, "btnI").value)
    @test_results << "|PASS"
    rescue 
    @test_results << "|FAIL"
  end
  
  @test_results << "|"
  puts @test_results

put_page

end 
end

jazz programmer

2007-05-31T11:07:00.001-06:00

Ron Evans gets it...

venture capital in a town of 15,000 people

2007-05-30T21:09:00.000-06:00

This is cool.

I've written letters about this in local (not available online) and international (October 2006) publications, it's great to see some serious action from people who might make something happen.

The quality of life here is superb. There is a small but enthusiastic pool of talent. If you have a spectacular business idea; don't need too much funding; and the VC environment in Silicon Valley, NYC, Austin, Atlanta, or Chicago is too much of a freak show: come here.

"let your employees decide when and where to work"

2007-05-24T17:25:00.000-06:00

Time magazine has a refeshingly hype-free short article about 37signals. Half of the company works remotely.

First, kill all your meetings; they waste employees' time. "Interruption is the biggest enemy of productivity," he says. "We stay away from each other as much as we can to get more stuff done." Use asynchronous communication and software instead to exchange information, ideas and solutions. Next, dump half your projects to focus on the core of your business. Too much time and effort are wasted on second-tier objectives. Third, let your employees decide when and where to work so they can be both efficient and happy. As long as their fingers are near a keyboard, they could as easily be in Caldwell, Idaho, as in Chicago.

At Socialtext where I work, most of the company works remotely. Socialtext sells wikis, and internally, the entire company lives, breathes, and evolves on wikis. The part about asynchronous (PUBLIC) communication is right on. Not only is it effective communication, it is also a living knowledge base and repository for company culture.

I have had a lot of jobs. It's ironic that the company culture at Socialtext is so rich when the people in the company see each other so rarely.

here come the software artists

2007-05-11T12:30:00.000-06:00

Apropos of my software-as-music analogy, Jonathan Kohl has another nice non-manufacturing metaphor, software development as theater. It makes a lot of sense, especially since Jonathon's point is actually about the value of improvisation in the course of developing and testing software.

In software, improvisation saves time and creates new opportunities. On the factory floor or in the supply chain, improvisation is a disruption and causes chaos.

starting a work blog

2007-05-08T22:48:00.000-06:00

I'm starting a blog for the work I'm beginning do at Socialtext. It'll mostly be concerned with wikis, Selenium, FireWatir, Perl, and maybe a little Ruby. I'll probably be posting to this one a little less often, but you're welcome to read the other one if you'd like. A lot of the Socialtext development staff are keeping open work blogs, so mine fits right in.

oh heck yeah

2007-05-06T22:24:00.000-06:00

http://imgs.xkcd.com/comics/conspiracy_theories.png

an example of an analogy: monks vs music

2007-05-05T12:03:00.000-06:00

Brian Marick gives a very well-reasoned and well-researched talk on the state of agile development. In it he compares agile teams to monks. I've never liked this analogy (see the comments) and on several occasions I've suggested that agile teams are much more like jazz bands. The most important differences for me are that unlike agile teams, monks a) do the same thing in the same way over and over and b) don't have audiences.

And now I've found the perfect musical agile counterexample.

This is Ella Fitzgerald and Count Basie doing the song One O'Clock Jump. The song is a twelve-bar blues, which is the jazz equivalent of a database app with a UI. By which I mean: just as every programmer has built a database app with a UI, every American musician has played twelve-bar blues. It is a framework on which many many many songs are hung, from Count Basie to Jimi Hendrix to the Ramones.

This particular video is a great example of agile practice. Listen to how the voice and piano influence each other. This is a lot like pair programming, and it's a lot like TDD: voice does something; piano responds; piano does something; voice responds. And notice the eye contact. These people are intensely aware of what's going on instant-to-instant. They have no sheet music (BDUF). They are involved in an activity that takes intense concentration and skill, just like good software development. They are also clearly aware that there is an audience, just as good software development should be aware of the needs of the people paying the bills.

I particularly love it when Ella sings "I don't know where I'm going from here". She recovers immediately, of course. It doesn't get any more agile than that. They are clearly inventing the music as they go along, using skill and experience.

I like Cleveland Eaton the bass player too. I think of him as a tester. He is subtly influencing the structure of the song, changing the dynamics, tying themes together, providing little pushes to the main players. Also, his instrument is much less sophisticated than a grand piano or a human voice, much as testers' tools are less sophisticated then developers' tools. For a bonus, here's a video of the full Basie orchestra featuring a bass solo from Mr. Eaton on another twelve-bar blues. Again, the interaction between bass and piano is wonderfully agile.

Apropos of not much, I had the privilege of speaking with Count Basie and Cleveland Eaton twice. The Basie band played at my high school on two occasions more than twenty years ago, and those conversations, especially with Mr. Eaton, were an enormous influence for me.

Charley Baker has a blog!

2007-05-04T17:49:00.000-06:00

Watir Project Manager Charley Baker is blogging. First post is an example of driving scriptaculous AJAX with Watir. I know that Charley is doing really big, nifty things with Watir and Ruby. I'm looking forward to reading about them.

but, but, but... that's black box testing!

2007-04-23T15:33:00.000-06:00

Brian Marick has been thinking about test frameworks in Ruby.

It's nicely done, but the killer idea is in the next-to-last paragraph that says:

In this particular case, the app communicates with the outside world via XML over the network, FTP files, and a database, so there’s no need to link the fixtures into the code. (emphasis mine) If it worked through a GUI, some remote procedure call interface would have to be added, but I persist in thinking that’s not too much to ask.

This has been my approach to test automation for some time, but it's hard to get across the idea until you've done it and succeeded once or twice.

I'll go even further, and stipulate that there is always a way to slice the application such that a test automator can write fixtures without reference to the underlying source code, and still provide effective, efficient system-level testing.

fine tools and beautiful machines

2007-04-19T14:59:00.000-06:00

My new laptop for work arrived yesterday, and I've been spending the day getting it set up:

I've used Windows machines at work for my whole career, but I've always had Macs at home. It's going to be a real pleasure to use a Mac for work.

I had a birthday recently, and my wife completely surprised me with one of these, an Ibanez AG86:

It was on the wall at the local music store, and I would pull it down and play it every time I went in. A really well-made guitar, a pleasure to play, and a good value. Guitar is not my main instrument, but I've been motivated to try to cop a few Wes Montgomery tricks.

This computer and this guitar are both fine tools and are both beautiful machines. There is a time in every student's career when they have mastered enough skills that an excellent set of tools is required in order to advance. This seems to be that time.

i've always wanted one of these

2007-04-12T15:53:00.000-06:00

and now I have one of my very own.

RadView WebLOAD now open source

2007-04-12T10:03:00.000-06:00

This is great news, and should really shake up the tools market. Not only is it open-source, it's released under GPL, which is a pretty radical move.

Open source load test tools have been kinda sorta OK for some time, but performance testing is the only case where commercial alternatives are pretty much universally acknowledged to be superior to open source tools. Building good performance test tools is hard. What RadView has done has instantly changed the whole scene.

I am no kind of performance test expert, but I've worked a little bit with the standard open source tools for doing that work. The next time I need one, I have no doubt that I'll reach for WebLOAD first to see how it works.

make a change to Ruby

2007-04-05T12:38:00.000-06:00

OK, I can't resist Pat Eyler's "Bloggin Contest"

I started programming in Ruby after having programmed in Perl for a while. There is not very much I miss about Perl, but there is one thing: in Perl, you can locate a subroutine anywhere in the script; in Ruby, any call to a method has to be located physically below the definition of the method. (Yes I know that there are reasons that Ruby's interpreter does it that way, but still...)

I write a lot of scripts that are run by non-technical non-programming users, like QA people and business analysts. The nice way to set up such scripts is to put all of the editable bits right up at the very top of the page, so that the users can ignore all of the programming guts below the important variables and such. This is easy in Perl. In Ruby, I am forced to either put all of the editable parts at the very bottom of the page, so the non-technical user has to scroll through a lot of intimidating code to get to the important stuff; or I have to distribute a separate library file and use 'require'.

Here's examples:

########################
#Perl
sub i_am_at_the_top {
print "I'm at the top\n";
}

i_am_at_the_top();
i_am_at_the_bottom();

sub i_am_at_the_bottom {
print "I'm at the bottom\n";
}
#########################
yields:
>perl -w subs.pl
I'm at the top
I'm at the bottom

#########################
#Ruby
def i_am_at_the_top
puts "I'm at the top"
end

i_am_at_the_top
i_am_at_the_bottom

def i_am_at_the_bottom
puts "I'm at the bottom"
end
#############################
yields:
>ruby defs.rb
I'm at the top
defs.rb:7: undefined local variable or method `i_am_at_the_bottom' for main:Object (NameError)
>Exit code: 1

If Ruby's interpreter would make all methods available for use at runtime, it would make communication with non-technical script users a lot more appealing.

it wouldn't hurt you to use the compiler, either

2007-04-03T10:28:00.000-06:00

I surf the O'Reilly Network blogs pretty often, and this one from Chris Josephes caught my eye, especially the part where he says of system administrators

"I don’t expect them to be a full fledged C programmer, but I think it’s important to know how to build software. The candidates that have demonstrated these skills have also been more proficient in debugging, tracing system calls, and identifying performance problems ahead of time."

I think the same is true of testers.

If you are a tester who has never used a compiler (or even if it's been a while), I have an exercise for you to do the next time you need inspiration for a new test.

Find your local build person, or a sympathetic developer, and have them send you the compiler log(s) from building your particular application. Do text string searches in the log files for the terms "warning" and "error". Unless you are working in an extremely mature organization, I'll bet you find some warnings and errors from the compiler.

Those errors and warnings point to places where bugs are very likely to live.

tests to prevent bugs: coral snakes and king snakes

2007-03-26T11:29:00.000-06:00

I was presented with a PNG screen capture of an application some time ago and asked to brainstorm test ideas based on the static representation of a limited section of the application UI.

I think I acquitted myself reasonably well at the time, but I've been thinking about the situation since then, because there is an interesting aspect to this particular application: if this software has defects, the repercussions of such defects could have a drastic effect on the physical health of users.

Assume that this application exists in order to tell the difference between snakes, so as to warn people if they are dealing with one that is poisonous or not. This is a very far-fetched analogy, but it serves to make my point. Right now, the development is in early stages, and the software is able to tell the difference between rattlesnakes, cobras, and coral snakes. Tests for those are easy: rattlesnakes have rattles, cobras have hoods, coral snakes are red, yellow, and black.

But someday we're going to want to add more snakes, like a scarlet king snake. Before we go on, take a look.

I don't have any requirement right now to add a scarlet king snake to the system, nor will I know any time soon if there ever will be such a requirement. However, the risk of confusing the two snakes is so incredibly high that I feel compelled to mitigate that risk with a test.

The appropriate test, of course, is to add a record for the scarlet king snake anyway, and to make sure regardless of the state of development of the app that the app never, ever confuses the coral snake record with the scarlet king snake record.

But the snakes are incredibly similar. How to design the test? We have to do enough research to find the 100% distinguishing feature:

Red next to Black - is safe for Jack;
Red next to Yellow - will kill any fellow.

I put a test into my regression suite saying that regardless of any other criteria (which could change anytime: my app is under development) if red-is-next-to-yellow: always choose "coral snake"; if red-is-next-to-black: always choose "scarlet king snake".

I'd probably want to add a couple of other bellwether tests as well, for instance, to distinguish between cottonmouth water moccasins and other water snakes.

If these tests ever break, then my users' health is at risk, and my company is at risk for serious litigation.

Transactionless

2007-03-23T11:22:00.000-06:00

Welcome back Mr. Caputo...

A number of people have been commenting on Martin Fowler's "Transactionless" essay. I didn't realize how unusual that approach seems.

I've said a number of times in public that if I had not begun my career by reading COBOL instead of, say, Java, I probably would not have gotten so far so fast.

Likewise, if I had not begun my career dealing with ISAM files instead of relational databases, I would probably have a much shallower understanding of what databases are and how they work.

In fact, of all the projects I've ever been a part of, one of the most impressive, most critical, and most educational was converting several enormous databases full of life-critical 911 location information from what was essentially a fancy ISAM arrangement to a true SQL database. Even at the time, though, our target SQL db did not have full support for referential integrity, so I became quite familiar with code that handles both referential-integrity checks and also that handles commits and rollbacks based on those checks.

I once wrote an article called "Old School Meets New Wave". Its funny how certain old techniques return to use over and over.

agile telecommuters?

2007-03-12T14:28:00.000-06:00

I would be very interested in hearing if anyone other than Kent Beck has stories about telecommuting for software development projects, particularly agile-ish software projects, with high-bandwidth communication requirements and short iterations and such.

My original assumption was that it was a widespread practice, but now I think it's much more rare than I had assumed. Which is strange, considering the savings to the business on office infrastructure, etc. and the availability of communication tools like IM, Skype, VPNs etc. Not to mention cheap airfare when you have to make that site visit.

I am also interested in promoting the practice-- if you are a telecommuting wannabe, you should say so as well.

I want to live in paradise and no one else seems to want to (although, if you all did, it wouldn't be paradise any more)
-Kent Beck (on the extremeprogramming mail list)

one good thing about the term "QA"

2007-03-08T17:20:00.000-07:00

Most serious software testers of my acquaintance object to the term "Quality Assurance". There are any number of specific objections to the term, but the general objection is that competent software testing goes far beyond the definition of Quality Assurance.

However, searching for jobs online is a heck of a lot easier using the string "QA".

a comprehensive guide to Google SketchUp Help

2007-03-04T14:48:00.000-07:00

In case you ever want to navigate your way through all of the help available in Google SketchUp, here's a quick guide:

There is the Help Center with really basic information about downloading and installing, links to major areas, high-level FAQs and such.

The User's Guide (which interestingly enough looks fine in Firefox but gives me an "Error on page" in IE) has the real nuts-and-bolts of what each control does and how to use it. This is deep information about all aspects of the UI.

The Help Groups which is a link to all of the Google Groups that have to do with SketchUp.

Video tutorials which are fun to watch but might be a little slow for some tastes. But there are 26 of them, so if something is perplexing, the answer might be here.

Examples (actually model structures) of both realistic and wireframe buildings.

Quick Reference Card a one-page PDF describing in shorthand how to use all of the UI widgets

SketchUp Community with a link to the public Google Groups, but also another link to the "Pro" groups.

Ruby Help The introduction to SketchUp's Ruby API

Self-Paced Tutorials Very nice, basic, simple tutorials that run within SketchUp itself. Only 6 of them, and they don't cover the super-complex topics, but they'll take you a long way.

Send a Suggestion Just in case this doesn't cover what you want, people are listening :)

at liberty

2007-02-26T20:39:00.000-07:00

For the first time in many years, I find myself without a job.

Ultimately, I would like to find a test automation partial-telecommute job based anywhere I can reach easily from Denver (with a preference for Denver, Salt Lake City, or Phoenix).

In the short term, if you or your organization anywhere in the world would like:

customized, personal training in Watir, based on Bret Pettichord's "Scripting for Testers" class that I taught at STAREAST, STARWEST, and Agile2006,

or training in fundamental principles of functional test automation in Ruby and/or Perl, based on a curriculum I have had in development for some time,

or a start on a quick, effective test automation suite for Web Services, either SOAP or REST,

or basic training on ethical hacking tools like WireShark, NTOP, Nessus, and ettercap,

or any other short term project involving test automation or basic training in Ruby

please send me an email at christopher dot mcmahon at gmail dot com, or leave a comment on this blog.

The Bay Area TD-DT Summit happened

2007-02-24T23:15:00.000-07:00

Our theme had to do with "Writing Better Functional Test Code". What struck me more than any other aspect at the gathering was how many different ways programmers, scripters, testers, and managers are attacking the issues involved in writing (and maintaining!) functional tests.

We had demonstrations of imperative approaches, deterministic approaches, random approaches, frameworks using mocks and stubs and Watir, FITNesse and Selenium, frameworks using math and using pixels, we had Ruby and sed and C++ and Java and C# and SOAP and databases and files, we had BDD and TDD and performance approaches.

And we had a ukelele. But only for two minutes. That was my fault.

Elisabeth Henderson wants a large-scale consolidated approach to doing functional testing. But given what we saw today, I start to think that even if someone were to give Elisabeth her pony, other functional testers would still find a dire need for aardvarks and platypuses.

instant HTML encoding

2007-02-08T14:26:00.000-07:00

this is cool

resources for figuring out ODBC index() function

2007-01-29T16:27:00.000-07:00

Microsoft:
(edited: man what a day)
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/odbc/htm/odbcsqlstatistics.asp
(bottom of the page, thanks to Paul Rogers, I'm not sure I ever would have found it on my own.)

For Oracle, you probably want to avoid the indexes() function, and do queries straight to the system tables:
http://www.ss64.com/orad/ALL_IND_COLUMNS.html
http://www.ss64.com/orad/DBA_INDEXES.html

Announcing the Bay Area Developer-Tester/Tester-Developer Summit

2007-01-23T17:18:00.000-07:00

If you're interested, leave a comment on the blog or send me an email or an IM or smoke signals or something. Here is the flyer Elisabeth Hendrickson and I have been sending:

What: A peer-driven gathering of developer-testers and tester-developers to share knowledge and code.

When: Saturday, February 24, 8:30AM – 5:00PM

This is a small, peer-driven, non-commercial, invitation-only conference in the tradition of LAWST, AWTA, and the like. The content comes from the participants, and we expect all participants to take an active role. There is no cost to participate.

We’re seeking participants who are testers who code, or developers who test.

Our emphasis will be on good coding practices for testing, and good testing practices for automation. That might include topics like: test code and patterns; refactoring test code; creating abstract layers; programmatically analyzing/verifying large amounts of data; achieving repeatability with random tests; OO model-based tests; creating domain specific languages; writing test fixtures or harnesses; and/or automatically generating large amounts of test data.

These are just possible topics we might explore. The actual topics will depend on who comes and what experience reports/code they’re willing to share.

For more information on the inspiration behind this meeting, see two of my recent blog entries:

http://www.testobsessed.com/2007/01/17/tester-developers-developer-testers/

http://www.testobsessed.com/2007/01/19/where-are-the-developer-testerstester-developers/

This is an open call for participation. Please feel free to forward it to other Developer-Testers and Tester-Developers who you think are interested enough to want to spend a whole Saturday discussing the topic and sharing code.

Proposed Agenda:

* Timeboxed group discussions: “Essential attributes of a tester-developer and developer-tester (differences and similarities)” and “What tester-developers want to learn from developers; what developer-testers want to learn from testers.”

* Code Examples/Experience Reports (we figure we have time for 3 of these)

* End of day discussion: Raising visibility for the role of a DT/TD, building community among practitioner

If you’re interested in participating, please send me an email answering these questions:

* Which are you: a tester who codes or a developer who tests?

* How did you come to have that role?

* What languages do you usually program tests in?

* What do you hope to contribute to the Bay Area DT/TD summit? Do you have any code or examples that you’d like to share? (Please note that you should not share anything covered by a non-disclosure agreement.)

* What do you hope to get out of the Bay Area DT/TD summit?

Thanks!

Your hosts:

Elisabeth Hendrickson
Chris McMahon

Mountain West Ruby Conf March 16-17 Salt Lake City

2007-01-22T21:43:00.000-07:00

This looks like a fine outing.

I've been lurking as these folks assembled the Mountain West Ruby Conference. It looks like a blast. Chad Fowler is giving the keynote, and at fifty simoleon bucks, the price is definitely right.

I'm not much for skiing, but it'd be a blast if I could make it to the shindig.

that's so crazy it's not even wrong

2007-01-19T10:11:00.000-07:00

I'd Like a Pony, Please

The Watir mail list gets on a regular basis questions like "How do I do X in Watir? The commercial tool that I use gives me X, but I can't figure out how to do it with Watir." (Watir, for those who don't know it, is a set of Ruby libraries for manipulating Internet Explorer. That's all it does. )

(OK, Watir has accumulated a few bells and whistles over time, but mostly it just reaches down IE's throat to move IE's bones around.)

X is logging/database connection/encryption/special string handling/test case management/distributed test environment/data-driven frameworks/etc., etc. etc.

The answer, of course, is that Watir doesn't do that. Nor will it ever do that. Watir is not intended to be a replacement for an entire test department, it is merely a set of libraries to manipulate Internet Explorer. Ruby itself has a lot of goodness to do this kind of thing, but Watir never will.

The people that ask these questions are often quite expert. The problem is that they do not think of themselves as programmers.

That's So Crazy It's Not Even Wrong PART ONE

I once worked with a man who was an expert with the Mercury tools. We were using Ruby to test an application that used a lot of XML. At one point he asked me "Does Ruby give me any way to ignore parts of strings?" I told him that I could think of a couple of ways to do that, and I asked him to show me what he was trying to do.

He wanted to extract the data elements from XML documents by using what boils down to Ruby's gsub("foo","") to strip the XML wrapper parts.

Understand this: he was setting out to build an XML parser. From scratch. Using gsub("foo","") as his only tool. He did not lack boldness.

This of course is madness. I introduced him to REXML and regular expressions. I presume he is still using them.

The reason he set out on such an insane project was that apparently in the world of Mercury, the only string manipulation tool available to you is something a lot like gsub("string",""). My colleague was simply unaware of the wider world of programming, and of what was possible.

He did not think of himself as either a developer-tester, or a tester-developer. He thought of himself as a tester who is a consumer of things provided by others.

That's So Crazy It's Not Even Wrong PART TWO

Mercury is not the only culprit in this kind of brainwashing. I have encountered testers (and others) whose entire world revolves around Excel.

For instance, I know of someone who started out his test project by putting test data into Excel, then writing VBA macros to generate Ruby code on the fly and launch it from Excel. He had a difficult time seeing Excel as simply a repository for data, instead of the center of the entire testing universe.

Where's My Pony?

Programmers' main tools are programming languages. Many testers' main tools are expensive whiz-bang-multi-aspect thingies that, if they have programming languages attached at all, are primitive or crippled.

I would like to see more testers learn about programming. Not just how to do programming, but how to think about testing problems in such a way that the problems can be solved with programming. Programming remains the best way to manipulate computers. If you work closely with computers, you owe it to yourself and your colleagues to learn some programming.

If you've never seen an pony, how will you know what to look for when you go to the barn?

P.S. many thanks to Jon Eaves for the title of his blog, which I've stolen and then mangled.

significance of developer-tester/tester-developer

2007-01-18T21:54:00.000-07:00

This idea of programmers-who-test and testers-who-program is starting to inform my daily work.

Reading articles about fantastic projects that sail to the end is all well and good, those great stories are something we strive for, but accomplish only occasionally. From day to day, our designs could be better, our test coverage could be better; our test plans could be better, our test cases could be more powerful.

I wrote the same bug in a couple of places this week. Luckily, my Customer was a tester. One of the developers I work with had a merge/commit problem a couple of times this week. Luckily, I was his tester, and I've struggled with merges and commits myself. We're working on figuring out how to prevent such errors in the future.

As a toolsmith, my Customers are testers. I go out of my way to write readable code so that they can inspect what I am creating. I encourage collaboration, and my Customers are coming to the realization that I need their help to recognize the bugs and get to the end.

As a tester, I am a Customer for the developers. I am working hard to understand how they go about writing and committing code. I am sympathetic when something goes haywire. I treat that as an opportunity to learn more about design.

As a tester-developer, I am working hard to understand the design and programming constraints of the system. I'm building automated tests and exploratory tests that expose the weakest parts.

As a developer-tester, I am working to include all the (programming and non-programming) consumers of my test scripts I can find, because I recognize that I work best when I collaborate. Disappearing into a closet until I personally resolve every issue simply takes too long and is just too boring to consider.

You! Luke! Don't write that down! *

2007-01-16T21:24:00.000-07:00

Robert Anton Wilson died last Thursday after a long struggle. If you haven't read The Illuminatus Trilogy, go do that now. It's a lot like Pynchon's The Crying of Lot 49, except 20 times longer and a hell of a lot funnier. Get back to me when you're done. I'll wait.

A couple of years ago I was privileged to see the movie about him made in his later years Maybe Logic at the old Durango Film Festival, which renewed my admiration.

Robert Anton Wilson refused to be made ridiculous, either by his critics or by his diseases. Both his sense of humor and his sense of the sublime were extraordinary, even in his last years. He refused to fade away, he raised a ruckus until the end.

*If you're intrigued by the title of this post, and you're too feeble to read Illuminatus, email me and I'll explain it.