Skip to main content

don't test for blocking conditions: an example

Matt Heusser quoted me on this at the Google Test Automation Conference. I've had this idea for some time, that many tests both manual and automated waste time explicitly testing for conditions that don't need tests.

I ran into one of those today. Or rather, my boss did. (He's not really a boss so much as he is an experienced colleague who has more conversations with management than me. Socialtext has very little hierarchy. )

We have a lot of automated tests that we created very quickly. Many of them have TODOs left over. I've been cleaning up the tests and implementing the TODOs.

One of the tests had a whole list of TODOs that had already been done, so I deleted the TODO list. My boss saw this and asked me why. In particular, one of the TODOs had been to create a record of a particular type, and my boss didn't see a test for this.

I pointed out that I had written an explicit test for sorting records of exactly that type, and three records of that type were created in the setup for the sorting test.

Had the record-creation failed, the tests for sorting would have failed. It's a nice economy of effort.

Comments

Grig Gheorghiu said…
Hi, Chris. One observation I have is that 'explicit is better than implicit' -- so even if you're creating that type of record somewhere else, the knowledge about that test is buried in another test. That makes it hard for example for a tool to parse all your test and generate the list of all your automated test cases. Especially when the creation of the records is done in your case in the setup method.

Grig
Simon Dugard said…
My first thought when I read this post was the same as Grig's. Sure, you might not need to have an explicit test, but it can make finding the problem much easier.

Simon
In this particular case I think this might be OK since it deals with CRUD-related operations. It hard to write a fully-controlled, isolated tests for R that doesn't C.

The one general concern is whether the setup incidentally uses a given record type or it actually needs that particular record type. In your example, it's sorting that record type, so this test clearly tests the create path.

I agree that it requires "tribal knowledge" to know that there is implicitly a test using the C of that recording type, so this might be a case where some duplication is better.

Also, what happens when a particular requirement goes away and the test for it, which was testing something else, goes away? In the case of CRUD, this is probably not a real problem, so it's not too risky.

So I'm not convinced of the general principle, but I think it applies to this context.

Popular posts from this blog

Reviewing "Context Driven Approach to Automation in Testing"

I recently had occasion to read the "Context Driven Approach to Automation in Testing". As a professional software tester with extensive experience in test automation at the user interface (both UI and API) for the last decade or more for organizations such as Thoughtworks, Wikipedia, Salesforce, and others, I found it a nostalgic mixture of FUD (Fear, Uncertainty, Doubt), propaganda, ignorance and obfuscation. 

It was weirdly nostalgic for me: take away the obfuscatory modern propaganda terminology and it could be an artifact directly out of the test automation landscape circa 1998 when vendors, in the absence of any competition, foisted broken tools like WinRunner and SilkTest on gullible customers, when Open Source was exotic, when the World Wide Web was novel. Times have changed since 1998, but the CDT approach to test automation has not changed with it. I'd like to point out the deficiencies in this document as a warning to people who might be tempted to take it se…

Watir is What You Use Instead When Local Conditions Make Automated Browser Testing Otherwise Difficult.

I spent last weekend in Toronto talking to Titus Fortner, Jeff "Cheezy" Morgan, Bret Pettichord, and a number of other experts involved with the Watir project. There are a few things you should know:

The primary audience and target user group for Watir is people who use programming languages other than Ruby, and also people who do little or no programming at all. Let's say that again:

The most important audience for Watir is not Ruby programmers 
Let's talk about "local conditions":

it may be that the language in which you work does not support Selenium
I have been involved with Watir since the very beginning, but I started using modern Watir with the Wikimedia Foundation to test Wikipedia software. The main language of Wikipedia is PHP, in which Selenium is not fully supported, and in which automated testing in general is difficult. Watir/Ruby was a great choice to do browser testing.  At the time we started the project, there were no selenium bindings for …

Open letter to the Association for Software Testing

To the Association for Software Testing:

Considering the discussion in the software testing community with regard to my blog post "Test is a Ghetto", I ask the Board of the AST  to release a statement regarding the relationship of the AST with Keith Klain and Per Scholas, particularly in regard to the lawsuit for fraud filed by Doran Jones (PDF download link) .

The AST has a Code of Ethics  and I also ask the AST Board to release a public statement on whether the AST would consider creating an Ethics Committee similar to, or as a part of the recently created Committee on Standards and Professional Practices.

The yearly election for the Board of the AST happens in just a few weeks, and I hope that the candidates for the Board and the voting members of the Association for Software Testing will consider these requests with the gravity they deserve.