Sunday, December 04, 2011

Just Fix It

On the writing-about-testing mail list recently was a discussion of defect tracking.  Given a good enough code base and a mature dev team, I think defect tracking is mostly unnecessary, and it's worth talking about why that is. 

Some time ago there was a popular meme in the agile testing community that goes "Just Fix It", but I haven't heard it mentioned in some time, and I think it's worth reviving the discussion.  The idea behind Just Fix It is to bypass the overhead of creating a defect report, having those defect reports go through some sort of triage process, and only then addressing the problems themselves represented by the defect reports.  You save a lot of time and overhead if you Just Fix It. 

For some time now I have specialized in testing at the UI, a combination of functional testing and UX work.  In my experience, in a good code base, important defects found at the UI level are almost always what I think of as "last mile" issues, where the underlying code has changed in some way but the hook for that code into the UI has been mangled or overlooked.  These are cases where unit tests are almost certainly passing, but the app is broken anyway.  Some examples:

  •  Explanatory text has disappeared or no longer describes function accurately.
  •  A widget that used to function no longer does.  For example, a Submit button no longer makes anything happen.
  •  A call to some underlying function is no longer correct.  For example, a Search function that used to return results no longer does.

While a Just Fix It culture is not necessarily agile, examples of Just Fix It are easier to describe in a typical agile situation.

Small Team

Many agile teams share a single space, making communication easy and instantaneous. In such a situation, a conversation like this might happen:

Tester: "Hey, the froobnozzle stopped froobing, anyone know anything about that?"
Dev: "Wow, I didn't realize my last commit would break the froob function, I'll Just Fix It."

This is a canonical example of what Lisa Crispin calls the "whole team approach", where testers, devs, and everyone else is working on the same stories at the same time in the same place simultaneously.

And if it's appropriate, there's no real reason a tester couldn't Just Fix It themselves.

Large Team

But some teams are too large for a conversation like this to be practical.  Assume a collocated team with a really big story board with dozens of story cards all being moved around a lot.  Say a tester finds an issue with the froobnozzle. 

Tester: grabs a red sticky note and writes brief description of froobnozzle problem. Puts sticky note on froobnozzle story card
...minutes later...
Dev: whoa, a red card on my story, better Just Fix It. 

Distributed Team

Distributed teams tend to have really sophisticated issue-tracking systems in place, where stories are represented in software of some sort, where they can be assigned, have their status changed, etc. etc.  If a distributed team is small enough, a tester will know that Joe is working on the froobnozzle story, so:

Tester to Joe on IM: "hi Joe, I think you might have just broken the froobnozzle."
Joe the Dev:  "whoa, good catch, I'll Just Fix It."

Large Distributed Team

On a large distributed team, identifying who might be in a position to Just Fix It can be complicated.  One strategy is to read the commit logs upon identifying a defect to see who or what may have caused the problem.  Another strategy might be to review all the stories in play to discover who might be working on the froobnozzle this iteration. 

But sometimes these sorts of approaches are too complicated or take too much time.  One pattern I have seen on several occasions in large distributed teams is to designate a knowledgeable person on the dev staff, or possibly a Scrum Master type, to represent the whole dev team for questions about behavior or function.  I have seen this role called the Face, and the Ninja, and the Disturbed.  

Tester:  "hi Face, I just discovered the that the froobnozzle got broken within the last day or so."
Face: "whoa, let me check that for you"
Face: "good catch, Joe broke that two commits ago, he's Just Fixing It"

Defect Found in Production

A customer probably reported it.  The fix is deployed to production within minutes or maybe hours of the report.  (Again, a good code base allows this.)

I worry that too often "root cause analysis" is a synonym for "blame".  Defects in production are almost certainly a process problem, and the place to address process problems are in retrospectives or similar conversations. 

Besides, if your team is releasing so many defects to production that you have to track them, you have bigger problems.

Won't Fix

True story:  just this week I was refactoring some Selenium tests and discovered a bug.  This was in a part of the application that is not exposed to customers, it is only for internal users employed by my company.  The bug was that attempting to enter a duplicate record causes an unhandled exception and the user is presented with an ugly stack trace.  This was an old bug, and was not part of the work of the current iteration.

I work on a large distributed team.  As I noted above, we have a sophisticated issue-tracking system in place.  All of the work we do is documented and tracked in this system.  We have no designated defect-tracking system, just a single monolithic sophisticated issue-tracking software application.

Upon finding the bug, I had a conversation with the dev who knows about that part of the code.  We agreed that this was a no-harm-no-foul situation, no data corrupted, minimal inconvenience to the user, no customer exposure.  We agreed that Just Fixing It right this minute wasn't very important.

So I created a new issue in the issue tracking system and assigned it to the dev who knows about that part of the code.  This issue has the same visibility and status as every other issue in the system.  My bug report issue will be treated the same as every other issue in the system, included in the backlog, and prioritized to be worked on with every other issue in the backlog. 

I don't even really think of that particular issue as a defect.  It's just a description of the state of a part of the application, some work that we might choose to do at some point.  I'm sure we'll Just Fix It pretty soon. 

Friday, September 30, 2011

a selenesse trick

Selenesse is the mashup between Fitnesse and Selenium I helped work on some time ago.  I keep encountering this pattern in my selenesse tests, so I thought I'd share it...

Every once in a while a test needs some sort of persistent quasi-random bit of data.  In the example below I'm adding a new unique "location" and then checking that my new location appears selected in a selectbox.   This is also a neat trick for testing search, or anywhere else you need to add a bit of data to the system and then retrieve that exact bit of data from somewhere else in the system. 

| note | eval javascript inline FTW! |
| type; | location_name | javascript{RN='TestLocation' + Math.floor(Math.random()*9999999999);while (String(RN).length < 14) { RN= RN+'0';}} |
| note | instantiate a variable and assign the new value to it in a single line FTW! |
| $LOCATION= | getValue | location_name |
| note | click a thingie to add the new data to the system |
| click | location_submit_button ||
| waitForTextNotPresent | Adding ||
| note | check that the newly added data appears as the selected entry in a selectbox FTW! |
| check | getSelectedLabel | location_site_id | $LOCATION |

Thursday, September 08, 2011

more UI test design (once more from Alan Page)

Before it gets lost in history, I want to riff off Alan Page once again, who made some excellent points.  But as someone who has been designing and creating GUI (browser) tests for a long time, I'd like to address some of those and also point out some bits of ROI that Alan missed. 

Making UI tests not fragile is design work.  It requires an understanding of the architecture of the application.  In the case of a web app, the UI test designer is really required to understand things like how AJAX works, standards for identifying elements on pages, how the state of any particular page may or may not change, how user-visible messages are managed in the app, etc. etc.  Without this sort of deep understanding of software architecture, UI tests are bound to be fragile.

I've said before that UI tests should be a shallow layer that tests only presentation, and that rarely tests logic, math, or any underlying manipulation of data.  If tests are designed in this way, then they will be robust and maintainable over the life of the app.

UI test design is a skill.  Designing such tests is no harder or easier than any other activity that requires skill and understanding.  The tools with which I am familiar provide enough power to create reasonable, robust, maintainable tests.

Finally, I think Alan is missing one aspect of automated UI tests that I find the most valuable of all. 

From green-screen mainframe systems to bleeding-edge web applications, in my experience every software system suffers from one particular sort of error that is always extremely difficult to see when testing manually:  when something goes missing. 

A search that used to return results no longer does.  A widget that used to be on the page no longer is.  A bit of text critical to the user's work goes missing. 

Actions that cause errors are easy to find when testing manually, as are errors of presentation.  Elements and functions that used to exist but no longer do are difficult to find:  it is not easy for a human being to see the absence of a thing, but such errors stick out like sore thumbs in an automated UI test suite.

In my experience, this is one of the most valuable aspects of automated UI testing, and one of the best reasons to invest in UI test automation.  The absence of a thing in the UI is simply not detectable with unit tests or with integration tests.  That critical bit of function that doesn't manage to cross the last interface to the UI is only detectable at the UI itself, and automated UI tests are very, very good at detecting errors where something has gone missing. 

Tuesday, August 30, 2011

Automated Test Design (riffing/ripping off Alan Page)

Alan just posted this:

This is the nicest test example I've seen in a long time, and I think it bears a little more analysis.

If I were in charge of the development of this app, I would make automated testing happen on 3 levels.

First, there would have to be some sort of test for generating a single random number. Which seems easy enough, but at this level, you really have to understand exactly what a call to rand() (or whatever) on your particular OS is going to do. I once wrote a script in Perl (praise Allah just a toy, not production code) that returned perfectly random numbers on OSX but returned exactly the same values every time on Windows. You'd better test that your call to rand() really returns random numbers, regardless of the OS it is running on.

At a higher level, you'd want to do exactly what Alan talks about in looping 100,000 times (although 100K seems like overkill to me). This is what I think of as an "integration test". You're going to have some method or possibly even an API call or REST endpoint like "generate_five_random_numbers_and_add_the_results". You're going to want to exercise that wrapper enough to convince yourself that the numbers are right and that the math is right.

For some time now I've been making my living writing automated UI/browser tests along with doing ET. Here's my take on a UI test for Alan's app:

Open the page
Check that text "Total" exists.
Check that the 6 textboxes exist.
Click the Roll button.
Check that the state of the page changes (however that happens).
Check that there are values in each textbox (even this might be too fancy, depending on the app and the test framework).

Also getting fancy, you could check that the 6 textboxes exist in the correct order. (Selenium's "glob:" feature makes this pretty painless.)

Doing math in UI tests isn't very smart. Doing data comparison over multiple runs in UI tests isn't very smart. Do that stuff at lower levels, where you can take advantage of programming power and speed. UI tests are a shallow layer where you simply check that the user has all the stuff they need to get the job done.