Skip to main content

Open letter to "CDT Test Automation" reviewers


Tim Western
Alan Page
Keith Klain
Ben Simo
Paul Holland
Alan Richardson
Christin Wiedemann
Albert Gareev
Noah Sussman
Joseph Quaratella

Apropos of my criticism of "Context Driven Approach to Automation in Testing" (I reviewed version 1.04), I ask you to join me in condemning publicly both the tone and the substance of that paper.

If you do support the paper, I ask you to do so publicly.

And regardless of your view, I request that you ask the authors of the paper bearing your names to remove that paper from public view as well as to remove the copy that Keith Klain hosts here.  For the reasons I pointed out, this paper is an impediment to reasonable discussion and it has no place in the modern discourse about test automation.


Cem Kaner said…
I don't think the paper was any good, but I think the content and the tone provide honest reflections of the views and analysis of some people who are seen as leaders in the field.

I don't think the paper should be taken down. I think it is part of their record. And if they eventually see it as poor work, they should say so. But even then, I don't take down my mistakes, I own them. I expect that from anyone else who wants my respect in the field.
Joris said…
Hi Chris,

Since we are apparently now looking to rid the field of software testing of any material that impedes any reasonable discussion, I was wondering who will lead the purification committee and who will be in it? I have some items that I really really want to add to the black list. Where can I submit them?

Best regards,

Chris McMahon said…
Posted on behalf of Alan Richardson, transcribed directly from what I received before Blogger removed it for what seems like login problems:

Alan Richardson has left a new comment on your post "Open letter to "CDT Test Automation" reviewers":

# "I ask you to join me in condemning publicly both the tone and the substance of that paper."


"Condemn - to express complete disapproval of."

I don't think so.

James and Michael asked me to review the paper. I read it. I wrote comments.
I sent the comments back to James and Michael. Some of my comments they took on board
some of them they chose not to (or the paper does not exhibit all the type of changes I described).
That is their right. They wrote the paper. It is theirs.

I'll point out some features of the paper that I agree with. The simple fact that I agree with some of it, means I can't condemn it:

* "There are many wonderful ways tools can be used to help software testing."
* Tools applied incorrectly add waste
* "Let's break down testing further..." section
* "Let your context drive your tooling".
* Case 1 and Case 2 are better than I remember them

Edit: I wrote more words here, but had to edit them out to fit the max comment length of this blog. Mainly I'm editing out longer examples of things I thought were good, and some I thought were bad, but it doesn't actually matter what I thought. People can read it and make up their own mind.

All of the above was written to say:

* I can't condemn it, because there are parts I agree with
* The parts I disagree with, I can ignore, and I don't have to condemn
* I gave comments to James and Michael, they may have reworked some of the text because of it, some of it they may not have. Its their paper, that's their choice.

I don't seem to take as much offence with the paper that you do. I don't feel that strongly about it.

# "If you do support the paper, I ask you to do so publicly."

I think there are useful parts in this paper. I think some parts I disagree with.

As with everything you read. Read it and make up your own mind.

# And regardless of your view, I request that you ask the authors of the paper bearing your names to remove that paper from public view as well as to remove the copy that Keith Klain hosts here. For the reasons I pointed out, this paper is an impediment to reasonable discussion and it has no place in the modern discourse about test automation.

No. Absolutely not.

You are asking me to engage in, and condone, censorship.


The fact that you asked for this. Is the reason I responded at length.

"this paper is an impediment to reasonable discussion".


You pointed out what you thought was wrong with the paper. I see nothing in your comments that demonstrate "this paper is an impediment to reasonable discussion". On the contrary, having pointed out the flaws, you can then go on to say what you think people can do instead. That is a prompt to reasonable discussion, not an impediment.

"it has no place in the modern discourse about test automation". People are entitled to describe their experiences of automating software. Even if they describe situations that they failed in the process.

Clearly you feel strongly about the paper.

I hope you create, or link to material, that offers alternative choices and demonstrates alternative views, which might open up options for your readers. Material that you do consider to have a place in the modern discourse of test automation, because I like to keep learning, and keep expanding my options.
Jason said…

My thoughts weren't directly asked for, but for whatever they're worth:

- As I made clear in my comments on your review of the Bach/Bolton paper, I think it's a steaming pile that is likely only being taken seriously because of who the authors are. I don't think it reflects a competent view of test automation at all, and unless they're willing to reconsider their views, I do not take Bach and Bolton seriously on the topic of test automation as a result of that paper.

- I would very much like to see/hear the responses from all the folks who reviewed the paper. It's possible I'll learn something, but even if not, I would appreciate more publicity behind the review process. Thank you for being willing to call out those folks by name.

- I have no interest at all in seeing the paper removed, and I'm not sure you asking for that really helps your cause. You seem concerned about, among other things, incompetence being tolerated and/or embraced within CDT. Doesn't having that paper as an example help you with your case? Aside from strategic considerations, I'm in favor of people writing whatever they feel like writing, and distributing it as widely as they would like, no matter how stupid or harmful their words might be. As long as those harmed by the words enjoy the same freedom to add their feedback, of course. I would view removal of the Bach/Bolton paper quite similarly to Bach (apparently) deleting some of his abusive tweets - scrubbing the evidence. I don't think that really helps anybody.

Aside from my partial disagreement, thank you for adding your voice and stating your concerns boldly.
Chris McMahon said…

The paper is not only incompetent, it is a cartoon. The language is so over the top you can almost picture those dastardly test automation people twirling their evil mustaches.

When I announced the review on Twitter, Bach and Bolton could not even agree between themselves whether it was intended to be example or counterexample. This led to Bach deleting tweets demonstrating his failure to understand the basic issues involved.

It seems clear to me that this paper is merely propaganda. It has no reasonable information about test automation in it at all, it's just a hit piece that's dragged on in one form or another for twenty years.

And I submit that having those names listed prominently as reviewers of the paper lend this piece of propaganda a legitimacy that it otherwise would not command.

I don't really care to demonstrate incompetence from CDT. Rather, as a professional software tester who can reasonably claim to be an expert in test automation, this paper is simply an unnecessary blight on work I care about deeply, and about which I have strong opinions.

I wrote the open letter to the reviewers to see who if anyone objected to having their names associated with such a blatant and unhelpful bit of propaganda.

I disagree that having the reviewers request the paper be removed constitutes censorship. As I said, this cartoonish piece of propaganda has no credible use in the modern discourse on test automation, and overcoming the obvious objections to it would be easier if it were simply removed.
Jason said…

Appreciate the response. Nodding towards Dr. Kaner, I'd probably still rather see Bach/Bolton publicly own up to the folly and propaganda in their paper (an outcome that doesn't seem terribly likely, I'll admit) than see the paper removed, but I think I understand where you're coming from quite a bit better than I did before. I think it's perfectly valid for the reviewers to ask their names to be removed from the paper. I'm also inclined to think that concerned software testing professionals asking for the paper to be removed, even if I disagree with it, does not constitute censorship. To me, censorship is a person or body in authority removing speech regardless of whether or not they were asked to do so.

Keep on keeping on. Your voice and passion for software testing is needed and appreciated.

Popular posts from this blog

Reviewing "Context Driven Approach to Automation in Testing"

I recently had occasion to read the "Context Driven Approach to Automation in Testing". As a professional software tester with extensive experience in test automation at the user interface (both UI and API) for the last decade or more for organizations such as Thoughtworks, Wikipedia, Salesforce, and others, I found it a nostalgic mixture of FUD (Fear, Uncertainty, Doubt), propaganda, ignorance and obfuscation. 

It was weirdly nostalgic for me: take away the obfuscatory modern propaganda terminology and it could be an artifact directly out of the test automation landscape circa 1998 when vendors, in the absence of any competition, foisted broken tools like WinRunner and SilkTest on gullible customers, when Open Source was exotic, when the World Wide Web was novel. Times have changed since 1998, but the CDT approach to test automation has not changed with it. I'd like to point out the deficiencies in this document as a warning to people who might be tempted to take it se…

Watir is What You Use Instead When Local Conditions Make Automated Browser Testing Otherwise Difficult.

I spent last weekend in Toronto talking to Titus Fortner, Jeff "Cheezy" Morgan, Bret Pettichord, and a number of other experts involved with the Watir project. There are a few things you should know:

The primary audience and target user group for Watir is people who use programming languages other than Ruby, and also people who do little or no programming at all. Let's say that again:

The most important audience for Watir is not Ruby programmers 
Let's talk about "local conditions":

it may be that the language in which you work does not support Selenium
I have been involved with Watir since the very beginning, but I started using modern Watir with the Wikimedia Foundation to test Wikipedia software. The main language of Wikipedia is PHP, in which Selenium is not fully supported, and in which automated testing in general is difficult. Watir/Ruby was a great choice to do browser testing.  At the time we started the project, there were no selenium bindings for …

Open letter to the Association for Software Testing

To the Association for Software Testing:

Considering the discussion in the software testing community with regard to my blog post "Test is a Ghetto", I ask the Board of the AST  to release a statement regarding the relationship of the AST with Keith Klain and Per Scholas, particularly in regard to the lawsuit for fraud filed by Doran Jones (PDF download link) .

The AST has a Code of Ethics  and I also ask the AST Board to release a public statement on whether the AST would consider creating an Ethics Committee similar to, or as a part of the recently created Committee on Standards and Professional Practices.

The yearly election for the Board of the AST happens in just a few weeks, and I hope that the candidates for the Board and the voting members of the Association for Software Testing will consider these requests with the gravity they deserve.