Skip to main content

Posts

Showing posts from 2007

Boulder Ruby Group: telecommute panel Nov 14

I'll be part of a panel on distributed agile working/telecommuting/remote working tomorrow evening Nov 14.

If you can get to Boulder, stop in. If you're interested in following along, there'll be an IRC channel running. (probably freenode #socialtext, unless we change our minds.)

Also, I have a wiki dedicated to the topic. For tomorrow evening the wiki will be totally public. I intend for the wiki to stick around as a repository of information about working in distributed teams. Leave a comment or send an email if you'd like access to that wiki.

test design economy

The other day Jeff Fry sent me an IM out of nowhere and in the course of the conversation asked me what I was doing these days.

I blithely answered "not so much coding, more test design". Which was really too short an answer, but there's IM for ya.

In a nutshell, Socialtext has a really nice Selenium-based automation framework, and recently I've been attacking the backlog of creating automation for fixed bugs. When I'm doing manual (Exploratory) testing on these code changes, I'm willing to spend time investigating odd combinations of things, multiple recurrences of things, all sorts of unusual experiments to find where exactly the boundaries of the functions lie.

But automated tests are expensive: they are expensive to create, and they are expensive to run. So when writing the automated regression tests, I'm trying to minimize the number of test steps while maximizing the coverage of the code in question.

Here's an example:
| type_ok | email_page_su…

actual code: create vs. sort

Grig and Simon are both still suspicious, so here is an example from the test itself:

This is Selenium RC in a keyword-driven framework where test data is in the wiki. Take a look at the "code" below and I hope you agree that an explicit test for creating a record is not worthwhile.

Any of these could fail in the setup, even though there is no explicit assertion about success:

| clickAndWait | link=Accounts | |
| type | name | 0sort1account%%start_time%% |
| st-submit | | |
| clickAndWait | link=Accounts | |
| type | name | 0sort2account%%start_time%% |
| st-submit | | |
| clickAndWait | link=Accounts | |
| type | name | 0sort3account%%start_time%% |
| st-submit | | |

and even if those didn't trigger a failure, these other assertions about sorting would fail in a really obvious way if any of the three records were missing:

| open | /nlw/control/account | |
| text_like | qr/sort1account.+sort2account.+sort3account/ | |
| clickAndWait | link=Name | |
| text_like | qr/sort3…

don't test for blocking conditions: an example

Matt Heusser quoted me on this at the Google Test Automation Conference. I've had this idea for some time, that many tests both manual and automated waste time explicitly testing for conditions that don't need tests.

I ran into one of those today. Or rather, my boss did. (He's not really a boss so much as he is an experienced colleague who has more conversations with management than me. Socialtext has very little hierarchy. )

We have a lot of automated tests that we created very quickly. Many of them have TODOs left over. I've been cleaning up the tests and implementing the TODOs.

One of the tests had a whole list of TODOs that had already been done, so I deleted the TODO list. My boss saw this and asked me why. In particular, one of the TODOs had been to create a record of a particular type, and my boss didn't see a test for this.

I pointed out that I had written an explicit test for sorting records of exactly that type, and three records of that type were …

Become a better software artist

There's been an interesting discussion on the software-testing mail list about the failures of Six Sigma and other process-quality religions to map effectively to software development. I think you might get better software the same way that you get better music.

Practice.

Almost every musician starts by playing along with recordings. After that, playing along with the radio builds improvisation skills. After that, playing along with other people broadens and deepens the communication spectrum. Throughout all this, work on the basics like scales and chords and tempo don't stop. The conscientious musician practices as many styles of music as possible. The professional musician may not enjoy blues, or orchestral works, or jazz, but there are certain forms that all professionals master in order simply to be professional.

Rehearse.

Music is constructed for listeners. Rehearsal is where the details of the performance are designed and implemented. Rehearsal is not where musicia…

so where's the testing

The Socialtext job is on craigslist now and I've been reviewing resumes. We have several great candidates, but most of the people who've submitted resumes don't seem to be able to help out much.

In particular, I'm seeing a lot of resumes that talk about how their owners do almost everything *except* actually test software. There seems to be lots of reporting and managing and reviewing and meetings and stuff, but actual testing experience seems to be pretty thin on the ground.

Or maybe that's just how people write resumes.

Socialtext is hiring a tester/QA person

The job announcement should show up here any minute. In the meantime, you can read the job description.

This is a great job, and we're looking for one outstanding person. You can google for Socialtext yourself (I recommend doing some research before you apply), and read the official job description. Here I'm going to talk about why this is such a great job, and the stuff you need to have to get it that isn't in the job description.

Socialtext has about fifty employees. A few are based in Palo Alto, but the development, QA, and support staff are spread across the US and Canada, along with a few overseas employees. If hired, you would be expected to telecommute. If you happen to live in the Bay Area, you'd be welcome to visit the office.

We're a small-a agile shop. We have standups, retrospectives, and iterations. We release small features frequently, and we install them on our internal wiki before we release to production.

Besides the wiki, we use irc, ema…

just enough design

So in recent weeks the number of test steps in our Selenium-based framework has quadrupled, from about 1000 to about 4000. The time it takes to run these tests is starting to cause significant pain.

So I set out to spike a little harness that would run sets of tests concurrently. (Note: it is *so* nice to have fork() available, after years of working in Windows environments.)

Trying to achieve the Simplest Test Harness That Could Possibly Work, I started out running all of the tests in the same test environment. Unfortunately, some of the tests change the state of the test environment, which causes other tests running concurrently to fail.

I started separating the state-changing tests (which may not be run concurrently) from the other tests (which may be run safely).

My colleague objected to this. Strongly. And he's right. He framed his argument in terms of old-school "software engineering" principles, but the argument is sound in terms of agile mantras like "…

data-driven test framework with Socialtext wiki as repository

Socialtext has a tradition of having "Wiki Wednesday" once a month, where people hack little wiki-related projects.

I had a little Wiki Wednesday myself today. Socialtext wikis have a very nice (and not very well documented) REST API. I've been figuring out to use this API in Ruby. I got to the point today where I could read and write pages to the wiki, so I hacked up a little framework using Watir that reads test data from one page and updates another page with the results of the tests.

Wiki markup makes parsing the data really easy, and the REST API makes updating pages really easy.

@test_data_page looks like
| Google Test | Google | Google Search | I'm Feeling Lucky |
This displays as a table in the wiki.

It's in dire need of some refactoring. Reading pages with more than one line of test data is left as an exercise for the reader. :-)

If you want to muck around with it yourself, you could get a free account from socialtext, or try the open source version or th…

venture capital in a town of 15,000 people

This is cool.

I've written letters about this in local (not available online) and international (October 2006) publications, it's great to see some serious action from people who might make something happen.

The quality of life here is superb. There is a small but enthusiastic pool of talent. If you have a spectacular business idea; don't need too much funding; and the VC environment in Silicon Valley, NYC, Austin, Atlanta, or Chicago is too much of a freak show: come here.

"let your employees decide when and where to work"

Time magazine has a refeshingly hype-free short article about 37signals. Half of the company works remotely.
First, kill all your meetings; they waste employees' time. "Interruption is the biggest enemy of productivity," he says. "We stay away from each other as much as we can to get more stuff done." Use asynchronous communication and software instead to exchange information, ideas and solutions. Next, dump half your projects to focus on the core of your business. Too much time and effort are wasted on second-tier objectives. Third, let your employees decide when and where to work so they can be both efficient and happy. As long as their fingers are near a keyboard, they could as easily be in Caldwell, Idaho, as in Chicago. At Socialtext where I work, most of the company works remotely. Socialtext sells wikis, and internally, the entire company lives, breathes, and evolves on wikis. The part about asynchronous (PUBLIC) communication is right on. Not only…

here come the software artists

Apropos of my software-as-music analogy, Jonathan Kohl has another nice non-manufacturing metaphor, software development as theater. It makes a lot of sense, especially since Jonathon's point is actually about the value of improvisation in the course of developing and testing software.

In software, improvisation saves time and creates new opportunities. On the factory floor or in the supply chain, improvisation is a disruption and causes chaos.

oh heck yeah

http://imgs.xkcd.com/comics/conspiracy_theories.png

an example of an analogy: monks vs music

Brian Marick gives a very well-reasoned and well-researched talk on the state of agile development. In it he compares agile teams to monks. I've never liked this analogy (see the comments) and on several occasions I've suggested that agile teams are much more like jazz bands. The most important differences for me are that unlike agile teams, monks a) do the same thing in the same way over and over and b) don't have audiences.

And now I've found the perfect musical agile counterexample.

This is Ella Fitzgerald and Count Basie doing the song One O'Clock Jump. The song is a twelve-bar blues, which is the jazz equivalent of a database app with a UI. By which I mean: just as every programmer has built a database app with a UI, every American musician has played twelve-bar blues. It is a framework on which many many many songs are hung, from Count Basie to Jimi Hendrix to the Ramones.

This particular video is a great example of agile practice. Listen to how the …

Charley Baker has a blog!

Watir Project Manager Charley Baker is blogging. First post is an example of driving scriptaculous AJAX with Watir. I know that Charley is doing really big, nifty things with Watir and Ruby. I'm looking forward to reading about them.

but, but, but... that's black box testing!

Brian Marick has been thinking about test frameworks in Ruby.

It's nicely done, but the killer idea is in the next-to-last paragraph that says:

In this particular case, the app communicates with the outside world via XML over the network, FTP files, and a database, so there’s no need to link the fixtures into the code. (emphasis mine) If it worked through a GUI, some remote procedure call interface would have to be added, but I persist in thinking that’s not too much to ask.

This has been my approach to test automation for some time, but it's hard to get across the idea until you've done it and succeeded once or twice.

I'll go even further, and stipulate that there is always a way to slice the application such that a test automator can write fixtures without reference to the underlying source code, and still provide effective, efficient system-level testing.

fine tools and beautiful machines

My new laptop for work arrived yesterday, and I've been spending the day getting it set up:

I've used Windows machines at work for my whole career, but I've always had Macs at home. It's going to be a real pleasure to use a Mac for work.

I had a birthday recently, and my wife completely surprised me with one of these, an Ibanez AG86:

It was on the wall at the local music store, and I would pull it down and play it every time I went in. A really well-made guitar, a pleasure to play, and a good value. Guitar is not my main instrument, but I've been motivated to try to cop a few Wes Montgomery tricks.

This computer and this guitar are both fine tools and are both beautiful machines. There is a time in every student's career when they have mastered enough skills that an excellent set of tools is required in order to advance. This seems to be that time.

i've always wanted one of these

and now I have one of my very own.

RadView WebLOAD now open source

This is great news, and should really shake up the tools market. Not only is it open-source, it's released under GPL, which is a pretty radical move.

Open source load testtools have been kinda sorta OK for some time, but performance testing is the only case where commercial alternatives are pretty much universally acknowledged to be superior to open source tools. Building good performance test tools is hard. What RadView has done has instantly changed the whole scene.

I am no kind of performance test expert, but I've worked a little bit with the standard open source tools for doing that work. The next time I need one, I have no doubt that I'll reach for WebLOAD first to see how it works.

make a change to Ruby

OK, I can't resist Pat Eyler's "Bloggin Contest"

I started programming in Ruby after having programmed in Perl for a while. There is not very much I miss about Perl, but there is one thing: in Perl, you can locate a subroutine anywhere in the script; in Ruby, any call to a method has to be located physically below the definition of the method. (Yes I know that there are reasons that Ruby's interpreter does it that way, but still...)

I write a lot of scripts that are run by non-technical non-programming users, like QA people and business analysts. The nice way to set up such scripts is to put all of the editable bits right up at the very top of the page, so that the users can ignore all of the programming guts below the important variables and such. This is easy in Perl. In Ruby, I am forced to either put all of the editable parts at the very bottom of the page, so the non-technical user has to scroll through a lot of intimidating code to get to the important …

it wouldn't hurt you to use the compiler, either

I surf the O'Reilly Network blogs pretty often, and this one from Chris Josephes caught my eye, especially the part where he says of system administrators
"I don’t expect them to be a full fledged C programmer, but I think it’s important to know how to build software. The candidates that have demonstrated these skills have also been more proficient in debugging, tracing system calls, and identifying performance problems ahead of time."I think the same is true of testers.

If you are a tester who has never used a compiler (or even if it's been a while), I have an exercise for you to do the next time you need inspiration for a new test.

Find your local build person, or a sympathetic developer, and have them send you the compiler log(s) from building your particular application. Do text string searches in the log files for the terms "warning" and "error". Unless you are working in an extremely mature organization, I'll bet you find some warnings a…

tests to prevent bugs: coral snakes and king snakes

I was presented with a PNG screen capture of an application some time ago and asked to brainstorm test ideas based on the static representation of a limited section of the application UI.

I think I acquitted myself reasonably well at the time, but I've been thinking about the situation since then, because there is an interesting aspect to this particular application: if this software has defects, the repercussions of such defects could have a drastic effect on the physical health of users.

Assume that this application exists in order to tell the difference between snakes, so as to warn people if they are dealing with one that is poisonous or not. This is a very far-fetched analogy, but it serves to make my point. Right now, the development is in early stages, and the software is able to tell the difference between rattlesnakes, cobras, and coral snakes. Tests for those are easy: rattlesnakes have rattles, cobras have hoods, coral snakes are red, yellow, and black.

But someday…

Transactionless

Welcome back Mr. Caputo...

A number of people have been commenting on Martin Fowler's "Transactionless" essay. I didn't realize how unusual that approach seems.

I've said a number of times in public that if I had not begun my career by reading COBOL instead of, say, Java, I probably would not have gotten so far so fast.

Likewise, if I had not begun my career dealing with ISAM files instead of relational databases, I would probably have a much shallower understanding of what databases are and how they work.

In fact, of all the projects I've ever been a part of, one of the most impressive, most critical, and most educational was converting several enormous databases full of life-critical 911 location information from what was essentially a fancy ISAM arrangement to a true SQL database. Even at the time, though, our target SQL db did not have full support for referential integrity, so I became quite familiar with code that handles both referential-integrity ch…

agile telecommuters?

I would be very interested in hearing if anyone other thanKentBeck has stories about telecommuting for software development projects, particularly agile-ish software projects, with high-bandwidth communication requirements and short iterations and such.

My original assumption was that it was a widespread practice, but now I think it's much more rare than I had assumed. Which is strange, considering the savings to the business on office infrastructure, etc. and the availability of communication tools like IM, Skype, VPNs etc. Not to mention cheap airfare when you have to make that site visit.

I am also interested in promoting the practice-- if you are a telecommuting wannabe, you should say so as well.

I want to live in paradise and no one else seems to want to (although, if you all did, it wouldn't be paradise any more)
-Kent Beck (on the extremeprogramming mail list)

one good thing about the term "QA"

Most serious software testers of my acquaintance object to the term "Quality Assurance". There are any number of specific objections to the term, but the general objection is that competent software testing goes far beyond the definition of Quality Assurance.

However, searching for jobs online is a heck of a lot easier using the string "QA".

a comprehensive guide to Google SketchUp Help

In case you ever want to navigate your way through all of the help available in Google SketchUp, here's a quick guide:

There is the Help Center with really basic information about downloading and installing, links to major areas, high-level FAQs and such.

The User's Guide (which interestingly enough looks fine in Firefox but gives me an "Error on page" in IE) has the real nuts-and-bolts of what each control does and how to use it. This is deep information about all aspects of the UI.

The Help Groups which is a link to all of the Google Groups that have to do with SketchUp.

Video tutorials which are fun to watch but might be a little slow for some tastes. But there are 26 of them, so if something is perplexing, the answer might be here.

Examples (actually model structures) of both realistic and wireframe buildings.

Quick Reference Card a one-page PDF describing in shorthand how to use all of the UI widgets

SketchUp Community with a link to the public Google Groups, b…

at liberty

For the first time in many years, I find myself without a job.

Ultimately, I would like to find a test automation partial-telecommute job based anywhere I can reach easily from Denver (with a preference for Denver, Salt Lake City, or Phoenix).

In the short term, if you or your organization anywhere in the world would like:

customized, personal training in Watir, based on Bret Pettichord's "Scripting for Testers" class that I taught at STAREAST, STARWEST, and Agile2006,

or training in fundamental principles of functional test automation in Ruby and/or Perl, based on a curriculum I have had in development for some time,

or a start on a quick, effective test automation suite for Web Services, either SOAP or REST,

or basic training on ethical hacking tools like WireShark, NTOP, Nessus, and ettercap,

or any other short term project involving test automation or basic training in Ruby

please send me an email at christopher dot mcmahon at gmail dot com, or leave a comment on this blo…

The Bay Area TD-DT Summit happened

Our theme had to do with "Writing Better Functional Test Code". What struck me more than any other aspect at the gathering was how many different ways programmers, scripters, testers, and managers are attacking the issues involved in writing (and maintaining!) functional tests.

We had demonstrations of imperative approaches, deterministic approaches, random approaches, frameworks using mocks and stubs and Watir, FITNesse and Selenium, frameworks using math and using pixels, we had Ruby and sed and C++ and Java and C# and SOAP and databases and files, we had BDD and TDD and performance approaches.

And we had a ukelele. But only for two minutes. That was my fault.

Elisabeth Henderson wants a large-scale consolidated approach to doing functional testing. But given what we saw today, I start to think that even if someone were to give Elisabeth her pony, other functional testers would still find a dire need for aardvarks and platypuses.

resources for figuring out ODBC index() function

Microsoft:
(edited: man what a day)
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/odbc/htm/odbcsqlstatistics.asp
(bottom of the page, thanks to Paul Rogers, I'm not sure I ever would have found it on my own.)

For Oracle, you probably want to avoid the indexes() function, and do queries straight to the system tables:
http://www.ss64.com/orad/ALL_IND_COLUMNS.html
http://www.ss64.com/orad/DBA_INDEXES.html

Announcing the Bay Area Developer-Tester/Tester-Developer Summit

If you're interested, leave a comment on the blog or send me an email or an IM or smoke signals or something. Here is the flyer Elisabeth Hendrickson and I have been sending:



What: A peer-driven gathering of developer-testers and tester-developers to share knowledge and code.

When: Saturday, February 24, 8:30AM – 5:00PM

This is a small, peer-driven, non-commercial, invitation-only conference in the tradition of LAWST, AWTA, and the like. The content comes from the participants, and we expect all participants to take an active role. There is no cost to participate.

We’re seeking participants who are testers who code, or developers who test.

Our emphasis will be on good coding practices for testing, and good testing practices for automation. That might include topics like: test code and patterns; refactoring test code; creating abstract layers; programmatically analyzing/verifying large amounts of data; achieving repeatability with random tests; OO model-based tests; creating domain sp…

that's so crazy it's not even wrong

I'd Like a Pony, Please

The Watir mail list gets on a regular basis questions like "How do I do X in Watir? The commercial tool that I use gives me X, but I can't figure out how to do it with Watir." (Watir, for those who don't know it, is a set of Ruby libraries for manipulating Internet Explorer. That's all it does. )

(OK, Watir has accumulated a few bells and whistles over time, but mostly it just reaches down IE's throat to move IE's bones around.)

X is logging/database connection/encryption/special string handling/test case management/distributed test environment/data-driven frameworks/etc., etc. etc.

The answer, of course, is that Watir doesn't do that. Nor will it ever do that. Watir is not intended to be a replacement for an entire test department, it is merely a set of libraries to manipulate Internet Explorer. Ruby itself has a lot of goodness to do this kind of thing, but Watir never will.

The people that ask these questions are oft…

significance of developer-tester/tester-developer

This idea of programmers-who-test and testers-who-program is starting to inform my daily work.

Reading articles about fantastic projects that sail to the end is all well and good, those great stories are something we strive for, but accomplish only occasionally. From day to day, our designs could be better, our test coverage could be better; our test plans could be better, our test cases could be more powerful.

I wrote the same bug in a couple of places this week. Luckily, my Customer was a tester. One of the developers I work with had a merge/commit problem a couple of times this week. Luckily, I was his tester, and I've struggled with merges and commits myself. We're working on figuring out how to prevent such errors in the future.

As a toolsmith, my Customers are testers. I go out of my way to write readable code so that they can inspect what I am creating. I encourage collaboration, and my Customers are coming to the realization that I need their help to recogniz…

You! Luke! Don't write that down! *

Robert Anton Wilson died last Thursday after a long struggle. If you haven't read The Illuminatus Trilogy, go do that now. It's a lot like Pynchon's The Crying of Lot 49, except 20 times longer and a hell of a lot funnier. Get back to me when you're done. I'll wait.

A couple of years ago I was privileged to see the movie about him made in his later years Maybe Logic at the old Durango Film Festival, which renewed my admiration.

Robert Anton Wilson refused to be made ridiculous, either by his critics or by his diseases. Both his sense of humor and his sense of the sublime were extraordinary, even in his last years. He refused to fade away, he raised a ruckus until the end.

*If you're intrigued by the title of this post, and you're too feeble to read Illuminatus, email me and I'll explain it.

developer-testers and tester-developers

After hearing about a number of system-functional-testing adventures at AWTA, Carl Erickson proposed that there are "developer-testers" and that there are also "tester-developers". In context, the difference was clear, and it's worth talking about the two approaches. (Assuming that I did in fact understand what Carl meant.)

Developer-testers are those developers who have become test-infected. They discovered the power of TDD and BDD, and extended that enthusiasm into the wider arena of system or functional tests. There is a lot of this sort of work going on in the Ruby world and in the agile world. For examples, look at some of Carl's publications on his site at atomicobject.com, or talk to anyone in the Chicago Ruby Brigade about Marcel Molina's presentation last winter on functional tests in Rails and 37signals. (Follow the link, the number of examples is gigantic!) The unit-test mentality of rigor and repeatability and close control survives in…

craft and discipline, larks' tongues in aspic

One of my lightning talks at AWTA had to do with finding better languages than engineering and manufacturing with which to describe software.






With all due respect to theLeaners, if software work were like factory work, few of us would do it.












I suggest we look to art, literature, and especially music for language to talk about software.


I mentioned first a couple of academic examples taken from New Criticism and Structuralism, like Monroe Beardsley's idea that value is based on manifest criteria like unity, variety, and intensity, or the Structuralists' idea that value is based on the degree to which the work reflects the folklore and culture of the milieu where it was produced.

If they'd done software, Beardsley would have been analyzing coding standards and function points, while the Structuralists would have been all strict CMM or strict Scrum/XP. If you're a objective-measurements, CMM, or Agile person, I encourage you to find corollary principles in the language …

stand back! I know regular expressions!

I get a kick out of xkcd, but the hero programming Perl swinging at the end of the rope had me laughing out loud.

what is "automated testing"?

Apropos of a recent discussion on the software-testing mail list, I was reminded of reading James Bach's Agile Test Automation presentation for the first time. It contains this one great page that says that test automation is:

any use of tools to support testing.

until that time I had held the idea that test automation was closely related to manual tests. I was familiar with data-driven and keyword-driven test frameworks, and I was familiar with tests that relied on coded parts to run, but I still had this idea that there was a necessary connection between manual tests and automated tests. I had this idea that proper automated testing was simply where the machine took over and emulated human actions that were expensive or boring.

That way, of course, lies madness. And expense.

It was reading this particular presentation that really lit the light bulb and got me thinking about all the other things that testers do, and all kinds of other ways to approach test automation. Here are…

the minimum number of tests

Elisabeth Hendrickson wrote about a case of complacency and lack of imagination in a tester she interviewed.

I agree with the gist of her argument, that good testers have a "knack" for considering myriad ways to approach any particular situation.

But I would also like to point out that just as a "tester who can’t think beyond the most mundane tests" is dangerous, so a tester who assigns equal importance to every possible test is just as dangerous, if not more so.

The set of all possible tests is always immense, and the amount of time to do testing is always limited. I like to encourage a knack for choosing the most critical tests. Sometimes this knack looks like intuition or "error guessing", but in those cases it is almost always, as Brian Marick put it in his review of Kaner's "Testing Computer Software", "past experience and an understanding of the frailties of human developers" applied well.

Intuition and error guessing take you …