Skip to main content

Reviewing "Context Driven Approach to Automation in Testing"

I recently had occasion to read the "Context Driven Approach to Automation in Testing". As a professional software tester with extensive experience in test automation at the user interface (both UI and API) for the last decade or more for organizations such as Thoughtworks, Wikipedia, Salesforce, and others, I found it a nostalgic mixture of FUD (Fear, Uncertainty, Doubt), propaganda, ignorance and obfuscation. 

It was weirdly nostalgic for me: take away the obfuscatory modern propaganda terminology and it could be an artifact directly out of the test automation landscape circa 1998 when vendors, in the absence of any competition, foisted broken tools like WinRunner and SilkTest on gullible customers, when Open Source was exotic, when the World Wide Web was novel. Times have changed since 1998, but the CDT approach to test automation has not changed with it. I'd like to point out the deficiencies in this document as a warning to people who might be tempted to take it seriously.

The opening paragraph is simply FUD. If we take out the opinionated language

poorly applied
terrible waste
shallow, narrow, and ritualistic
pandemic, rarely examined, and absolutely false

what's left is "Tool use in testing must therefore be mediated by people who understand the complexities of tools and of tests". This is of course trivially true, if not an outright tautology. The authors then proceed to demonstrate how little they know about such complexities.

The sections that follow down to the bits about "Invest in..." are mostly propaganda with some FUD and straw-man arguments about test automation strewn throughout. ("The only reason people consider it interesting to automate testing is that they honestly believe testing requires no skill or judgment" Please, spare me.) If you've worked in test automation for some time (and if you can parse the idiosyncratic language), there is nothing new to read here, this was all answered long ago. Again, much of these ten or so pages for me brought strong echoes of the state of test automation in the late 1990s. If you are new to test automation, consider thinking of this part of the document as an obsolete, historical look into the past. There are better sources for understanding the current state of test automation.

The sections entitled (as of June 2016) "Invest in tools that give you more freedom in more situations" and "Invest in testability" are actually all good basic advice, I can find no fault in any of this. Unfortunately the example shown in the sections that follow ignores every single piece of that advice.

Not only does the example that fills the final part of the paper ignore every bit of advice the authors give, it is as if the authors have chosen a project doomed to fail, from the odd nature of the system they've chosen to automate, to the wildly inappropriate tools they've chosen to automate it with.

Their application to be tested is a lightweight text editor they've gotten as a native Windows executable. Cursory research shows it is an open source project written in C++ and Qt, and the repo on github  has no test/ or spec/ directory, so it is likely to be some sort of cowboy code under there. I assume that is why they chose this instead of, say, Microsoft Word or some more well engineered application.

Case #1 and Case #2 describe some primitive mucking around with grep, regular expressions, and configuration. It would have been easier just to read the source on github. If this sort of thing is new to you, you probably haven't been doing this sort of work long, and I would suggest you look elsewhere for lessons.

Case #3 is where things get bizarre. First they try automating the editor with something called "AutoHotKey", which seems to be some sort of ad-hoc collection of Windows API client calls, which according to the AutoHotKey project history is wildly buggy as of late 2013 but has had some maintenance off and on since then. I would not depend on this tool in a production environment.

That fails, so then they try some Ruby libraries. Support for Windows on Ruby is notoriously bad, it's been a sticking point in the Ruby community for years, and any serious Ruby programmer would know that. Ruby is likely the worst possible language choice for a native Windows automation project. If all you have is a hammer...

Then they resort to some proprietary tool from HP. You can guess the result.

Again, assuming someone would want to automate a third-party Windows/Qt app at all, anyone serious about automating a native Windows app would use a native Windows language, C# or VisualBasic.NET, instead of some hack like AutoHotKey. C# and VisualBasic.NET are really the only reasonable choices for such a project.

It is as if this project has been deliberately or naively sabotaged. If this was done deliberately, then it is highly misleading; if naively, then it is simply sad.

Finally I have to point out (relevant to the article section "Invest in testability", and again strong shades of 1998) that this paper completely ignores the undeniable fact that the vast majority of modern software development takes place on the web, with the UI appearing in a web browser and APIs offered from servers over a network.  This article makes no mention that selenium/webdriver is a UI automation standard adopted by the World Wide Web Consortium (W3C), that the webdriver automation interface is fully supported by every major browser vendor:  Google Chrome, Mozilla Firefox, Microsoft Internet Explorer, Opera, and most recently Apple Safari, or that the Selenium API is fully supported in five programming languages: C#, Java, Ruby, Python, and Javascript, and partially supported in many more.

Ultimately, this article is mostly FUD, propaganda, and obfuscation. The parts that are not actually wrong or misleading are naive and trivial. Put it like this: if I were considering hiring someone for a testing position, and they submitted this exercise as part of their application, I would not hire them, even for a junior position. I would feel sorry for them.


Vernon said…
Greetings Chris,

Thanks for putting that together.

I agree that things like "The only reason people consider it interesting to automate testing is that they honestly believe testing requires no skill or judgment." are not a universal truth. People are a little more sophisticated these days (and I will continue looking for more lines like this). I also get what you mean about the language and tone. I've been speaking about that on Twitter myself as it happens because crap content or not, we can instantly piss people off using terms like that.

That said...

As I re-read it, I still think the audience for this thing isn't someone like you. It's true the mistakes they list ARE old! The thing is, people are still making them.

I confess I'm probably not even at basecamp when it comes to learning these automated testing/toolsmith-ing (delete as appropriate) skills, so I definitely don't have the same experience as you or people like Noah Sussman, Elisabeth Hendrickson, Alan Richardson, Alister Scott and others when it comes to doing this stuff.

I do have experience of being on projects where teams have made the mistakes cited in the article though (in 2016 to boot!). I think the difference between those teams and the teams you guys worked on, is you guys learned from your mistakes, made improvements and were able to navigate through/around objections/misconceptions of peers and people up the food chain.

Obviously this is a guess but I reckon people hire people like you specifically because of your track record i.e. they recognise your expertise and are prepared to listen. What I usually see though, are people hired with a proficiency using a particular tool (typically WebDriver or some variant thereof) given the instruction to "automate the regression pack" and off they go. Little to no thought of "is that the best thing to do right now? If so is there a more effective way to achieve the same goal?".

Now I look at it, I can see your point about the Windows app examples too. That does seem odd since most people are working on web/mobile apps of some description. Perhaps they were trying to demonstrate an example of inappropriate tool selection or something?! In which case - success!

On Twitter you asked me what I'm going to do about it. Well my plan is to learn to code. It's very hard to convince coders and others there are more effective ways to use automation/tooling when you can't speak articulately about it and can't "show people your battle scars" so to speak, so that's my goal. Until I get there though, I need articles like this I can point people too.

I agree from your perspective it's like showing Garry Kasparov tic-tac-toe strategy and expecting him to be impressed but there it is.

What do you think?


joni piter said…
I see this article as a clear explanation of facts that are still misundertood, and that I often have to explain repeatedly during my testing lifetime! They are not preventing test automation, but instead clarifying the terms automation, tools, etc, and I still see lots of stakeholders that blindly follow the "Automation" myth, like having X % of tests automated, with no idea what it means exactly to Automate and when it can be efficient. So I find the article useful as a resume and to point to other people instead of keep explaining things over and over.

I would say that when a generic automation framework is idealized and created it aims web applications or browser forms fill in, but real world applications go much further from that. the kind of automation those tools offer are frequently (but maybe not always) very basic verifications and cover very little part of a complex system behaviour. At least my experience.

I must say that we use a lot of test automation with in-house tools, but that is not an automation like "click a button and result is out", it involves tester constant analysys, reaction/adaptation to the ongoing effort.

Joao Rodrigues
Danny Faught said…
An aside - the article you linked to, "Is Windows a First Class Platform for Ruby?", was posted in 2008. Do you have any newer references on that topic? I've been diving into Ruby more seriously lately and am curious to learn more about its viability.
Chris McMahon said…
Danny, yeah, my point was that Windows has been a second-class citizen in the Ruby community for a very long time. I haven't used Windows seriously in many years, so I'm not current.
Dave McNulla said…
There are a lot of ways to do things incorrectly. That isn't proof that doing those things is a bad idea. I guess if you are not capable of doing them correctly, that might be a reason. Probably better to learn to improve.

I do not believe that automated checks replace sapient testing. Automated checks definitely replaces manual checks quite well. Some benefits are lost when the human is not used, but time is gained in optimal circumstances.

Automation is interesting because of time savings from manual checking. Checking is necessary to prevent regression problems. I can think about what I am testing, explore it in great detail. When the dev team adds more code, refactors classes, etc., the exploration I did yesterday is not helping me that much now.

Automation is also interesting as a vehicle for behavior driven development. If I know how I want a feature to work, I can create a script to verify that, with code to support the verification being automated. It's only one checkbox for done, but it's an important one.

I may not be a CDT, but I try to be pragmatic in my testing. I take context into consideration every day, every decision. That has to include finding better ways, not reasons to avoid better ways. Automated tests and A/B testing will be part of the progress, when the context allows.
Jason said…
If the two authors weren't who they were, there's almost no way pepole would be trying so hard to come up with a positive interpretation of what is a transparently stupid paper that has almost nothing to do with modern test automation practices. That strongly suggests that a cult of personality exists around the authors of the paper. And that is a nasty business that I want no part of.
Timothy Western said…
Danny I can relate to the RUby on windows comment and vouch for what Chris says there. I ran into an issue in 2013 (just 3 years ago) because I wanted to do some self balancing of multiple tests at the same time, but most of the gems and tools available then relied on the LInux Fork command. Windows doesn't fork.

I'm sure there are otherways in which Ruby is a second or third class citizen on windows also.
Jon Hagar said…
Being a follower of the Context school (though maybe excommunicated for some of my actions in the test community), I think you (Chris) are right. There are tools and context where high levels of automation are possible for some testing (beyond simple verification-checking even). However as always, context is important. I am thinking the reference article was pointing out, in part, that automation in test of everything all the time may not be an ideal to be worked towards, and gave examples of where this may be true. In short articles, it is not possible to address every aspect and context of a subject like automation. So you (Chris) point out contexts were automation can be viable. Good.

I wrote about the "schools" at in the last few days. These articles and blogs point out another example of where I think we are having some civil discussion about a topic that inspires some passion (test automation). As a profession we need the discussion and not "war".
Rosie Sherry said…
I would love if we could share more of your thoughts and experiences about testing and test automation. I personally think it's important that we communicate better about both/all sides of the story.
Chris McMahon said…
Rosie, there are no "sides" here. This essay implies that the authors of "The Context Driven Approach to Automation in Testing" are either incompetent or lying. Now we know from Twitter that it is one of each.

My thoughts and experiences about testing and test automation are widely available in multiple places. In 2009 alone I published 50,000 words in dozens of articles that are all publicly available. There are video interviews and youtube videos. If you have any questions about my views, feel free to ask.
Chris Kenst said…
To be fair it's not THE Context Driven Approach to Automation in Testing, just A Context Driven Approach to Automation in Testing. Which is to say it's not the only approach (although possibly the only published / formally coalesced one) and doesn't speak for everyone in the community.
ConnorR said…
(Part 1 of n)

NOTE: Chris, I would have preferred to send this to your privately, but searched your site for your email and did not find it. Also, you have disabled DM on Twitter, so I am assuming you'd rather have public feedback.



First, you should know that the amount of animosity at which you have attacked this position prevents some people from hearing you. Not sure if you are aware of that or not, but I thought you should be if your goal is to reach people with your knowledge.

Now, I can personally set your tone aside and look at your words objectively, but after also reading the Twitter threads with replies between you and the authors (Michael and James), it seems that you had many misunderstandings of what their intentions with various exercises were. In general though, it's an odd practice to read someone's work, then not ask clarifying questions. Did you reach out to them privately before writing this blog? I feel that could have saved you a lot of mental anguish and the subsequent failing about on Twitter. I was embarrassed for you on that, because you seem like a smart guy, but were jumping to rash assertions, each of which kept getting knocked down by the authors.

In general, if something strikes you strongly, then maybe there's some depth there that needs to be explored before an assertion is made, right?

When I was a child and someone said something that came across as very odd/off/weird/incompetent/etc then I would react right away, and tell them how wrong they were, subconsciously assuming all along that my stance must be the only proper way of interpreting their words. However, as I grew into an adult, I shed some of that narcissism and arrogance, realizing that I have my own biases through which other ideas and opinions are constantly being passed, like a filter. Since my filter will coat anything I hear or read based on my own experiences, then I must internally concede that the resulting interpretation in my brain is at least somewhat flawed. The key, is that I must come to this realization before I speak, or I will undoubtedly cause harm. So now, as an adult, I hear something that sounds abrasive and question myself, internally first (e.g. What three or four meanings could this have?), then I ask questions externally only after doing that. As adults, I'd think we'd all seek to be more reflective and less reflexive, but this article comes across very much the latter.

ConnorR said…

(Part 2 of 2)
Did you try the Socratic method with the authors before forming these assertions?

Did you get clarifications on their context and intent before writing this blog? Do you feel you did your due diligence of making sure you position as well-informed as possible before clicking "Post"? That's a duty we all have if we want to be contributing constructively to the community. This is why I sometimes have blog posts in draft mode for months or a year in a couple cases, before actually finalizing them publicly. That is my responsibility. Your observations of the article, as outlined in this post, really come across as knee-jerk/surface-level assertions. On top of that, your poor handling of it and odd behavior on Twitter didn't help. I'm have a very hard time taking you seriously as a professional because at this point, I cannot tell if you genuinely care about the craft or are simply trying to plant a flag in the ground so people associate your name with something popular. I think you might actually care, are perhaps are just unskilled at how to articulate that. Worthwhile communities expect more introspection and exploration of an issue, before jumping to assertions; especially coming from people with your apparent experience level.

Would it surprise you to hear that James and Michael were the ones who actually showed me the value of automation?

Both Michael and James continually use tools in their teaching to write small bits of automation that help make their points or to further an exercise. I have taken RST in-person from both of them separately, then also additionally the RSTA Online class from James. If you actually sat down with them in person, you'd see how much they actually honor the use of automation as a tool by coding various tasks to assist with testing. They refuse to use automation as a battering ram to storm all castles, which tells me they are thinking critically. Perhaps that is what you wish them to do?

In contrast, you should know that the way you write about automation in this article actually pushes me away from thinking it is something enjoyable and fun. Calling James and Michael ignorant on Twitter came across very short-sighted and muddied your reputation, whether you acknowledge it or not. There is a lot I could learn from you, but it's going to be much harder to garner that information now, since you seem to want to put barriers up. I am guessing there is some personal history there of which I do not know about, but it seems very misdirected and poorly handled through the posting of this article, and the subsequent Twitter drama. I genuinely want to see others come together on this, and am still convinced that you wrote an article that is mostly not about what they intended, as if you are trying to address some other problem that did not exist. In reality, we're not as divided as people want to think, but text-only mediums sure make it come across that way.
Jared said…

Published 2008 originally, seems to have had updates. Given James is mentioned as a reviewer it may have something to do with his choice of example (book automates a windows text editor with Ruby Win32 libraries).
Noah Sussman said…
Wow did ConnorR just write TWO comments about he just can't believe the author understands the issue at hand?

And did he really just imply that any disagreement the author has, must be due to misunderstanding because the views in question are just so obviously reasonable?

You just cannot make this shit up. ConnorR's post is a shining example of all that is wrong with discourse in the testing community today. So glad that after many years of trying, some in the testing scene have managed to come together and have constructive dialog about how to move past the bad image our community has in the eyes of the rest of the tech world.
ConnorR said…
Noah - Based on the Twitter discussions that followed, it was clear that James and Michael were having to clear up confusion that Chris had (or maybe still has) about the intention of their exercises in their automation paper/article. You can see that here:

It'd be like me telling you "We're going to fix up this old house, but we cannot use a hammer for every single job" then you ranting about how incompetent I am at using a hammer. That's a completely different point and not at all what I was talking about. That reaction of yours would be based on a misunderstanding of my intent, when in fact I actually just have a genuine care for you to be a well-rounded craftsman.

Chris's post here assumes that the authors were trying to teach HOW to do effective automation when in fact they weren't. That's the problem IMO. I think Chris is using their paper as a chance to address some other deep-seated issues. Chris makes good points about automation, and is obviously very competent, but he doesn't seem to ever address what James & Michael's actual intentions were in writing the original paper. The authors' goal was not to show how to do competent automation - Their intent was to show examples of scenarios of what people try to do and why that fails.
Noah Sussman said…
This comment has been removed by a blog administrator.
pooja khot said…
This comment has been removed by a blog administrator.
pooja khot said…
This comment has been removed by a blog administrator.
pooja khot said…
This comment has been removed by a blog administrator.

Popular posts from this blog

Watir is What You Use Instead When Local Conditions Make Automated Browser Testing Otherwise Difficult.

I spent last weekend in Toronto talking to Titus Fortner, Jeff "Cheezy" Morgan, Bret Pettichord, and a number of other experts involved with the Watir project. There are a few things you should know:

The primary audience and target user group for Watir is people who use programming languages other than Ruby, and also people who do little or no programming at all. Let's say that again:

The most important audience for Watir is not Ruby programmers 
Let's talk about "local conditions":

it may be that the language in which you work does not support Selenium
I have been involved with Watir since the very beginning, but I started using modern Watir with the Wikimedia Foundation to test Wikipedia software. The main language of Wikipedia is PHP, in which Selenium is not fully supported, and in which automated testing in general is difficult. Watir/Ruby was a great choice to do browser testing.  At the time we started the project, there were no selenium bindings for …

Open letter to the Association for Software Testing

To the Association for Software Testing:

Considering the discussion in the software testing community with regard to my blog post "Test is a Ghetto", I ask the Board of the AST  to release a statement regarding the relationship of the AST with Keith Klain and Per Scholas, particularly in regard to the lawsuit for fraud filed by Doran Jones (PDF download link) .

The AST has a Code of Ethics  and I also ask the AST Board to release a public statement on whether the AST would consider creating an Ethics Committee similar to, or as a part of the recently created Committee on Standards and Professional Practices.

The yearly election for the Board of the AST happens in just a few weeks, and I hope that the candidates for the Board and the voting members of the Association for Software Testing will consider these requests with the gravity they deserve.