Wednesday, August 03, 2016

Sanction Keith Klain

I have asked the Association for Software Testing to make a statement regarding the relationship of AST with their former Board officer Keith Klain in regard to the lawsuit for fraud filed against Klain by his former employer Doran Jones.

The lawsuit alleges that Klain did a number of reprehensible things. What should concern the AST and the software testing community in particular is that Klain is alleged to have been a party to sabotaging and undermining the training in software testing given to disadvantaged people by Per Scholas in New York City.

Klain filed a "LETTER addressed to Judge Analisa Torres from A. Goldenberg dated July 20, 2016 re: Request for a Pre-Motion Conference Regarding Anticipated Motion to Dismiss".  Of concern to the software testing community is that this letter in no way disputes or denies Klain's behavior alleged in the law suit; this letter merely attempts to make the case that Klain's abominable behavior should not meet the letter of the statute itself for actual fraud. (1)

Then Doran Jones answered that letter with one of its own "FIRST LETTER addressed to Judge Analisa Torres from JASON H. KISLIN dated August 1, 2016 re: Plaintiff's Response to Defendant Klain's Request for a Pre-Motion Conference."  (2), which concludes in part:

"In addition, should the Court dismiss Doran Jones' CFAA claim, its remaining state law claims -- for breach of contract, breach of the duty of good faith and fair dealing, breach of fiduciary duty, tortious interference with prospective economic advantage, tortious interference with contractual relations, misappropriation of trade secrets, fraudulent inducement, and declaratory judgment -- will be properly before this Court because they do not require the Court to rule on novel or unsettled issues of state law."

This is reprehensible at best, immoral at worst. Not only should the AST sanction Keith Klain, but TechWell, SoftwareTestPro, and the conference organizations at which Klain presents should seriously reconsider their current and past association with Klain, and anyone else with whom he has any ties should consider the value of that association as well.

(1) Anyone may get direct access to court documents for this case for free or for a nominal fee by creating an account at PACER  The case number is Case 1:16-cv-02843-AT . Notices of actions are available publicly for free.
(2) This PDF does not render in my browser, but seems OK after downloading. For an original copy see (1) above.

Thursday, July 14, 2016

Open letter to "CDT Test Automation" reviewers


Tim Western
Alan Page
Keith Klain
Ben Simo
Paul Holland
Alan Richardson
Christin Wiedemann
Albert Gareev
Noah Sussman
Joseph Quaratella

Apropos of my criticism of "Context Driven Approach to Automation in Testing" (I reviewed version 1.04), I ask you to join me in condemning publicly both the tone and the substance of that paper.

If you do support the paper, I ask you to do so publicly.

And regardless of your view, I request that you ask the authors of the paper bearing your names to remove that paper from public view as well as to remove the copy that Keith Klain hosts here.  For the reasons I pointed out, this paper is an impediment to reasonable discussion and it has no place in the modern discourse about test automation.

Wednesday, July 06, 2016

Open letter to the Association for Software Testing

To the Association for Software Testing:

Considering the discussion in the software testing community with regard to my blog post "Test is a Ghetto", I ask the Board of the AST  to release a statement regarding the relationship of the AST with Keith Klain and Per Scholas, particularly in regard to the lawsuit for fraud filed by Doran Jones (PDF download link) .

The AST has a Code of Ethics  and I also ask the AST Board to release a public statement on whether the AST would consider creating an Ethics Committee similar to, or as a part of the recently created Committee on Standards and Professional Practices.

The yearly election for the Board of the AST happens in just a few weeks, and I hope that the candidates for the Board and the voting members of the Association for Software Testing will consider these requests with the gravity they deserve.

Thursday, June 30, 2016

Test is a Ghetto

If you read software testing news aimed at the general public, you might be of the opinion that software testing is done by, and *properly* done by:

The key of course is "minimal training". There is a class of software testers who have minimal programming skills, or system administration skills, or database skills, or any technical computer skills at all. These testers do honorable work and can be valuable members of a software development team. They have been my colleagues; I have helped hire them; and I have trained them in test automation. And I still do that sort of work myself sometimes, although others are better at it than I am.

However, their lack of technical skills mean that they tend to have lower status, lower income, and are often considered fungible, easily swapped out as economy dictates, or replaceable by automation. At least at the start of their careers, these testers can be caught in a vicious circle of not having the technical skills or critical abilities to advance their careers, while also lacking enough understanding of modern software development to see a way to improve their skills. Many stay in this circle indefinitely, where a whole mythology of the value of codified ignorance evolves. This is the software testing ghetto.

Ghettos exclude their occupants from the general discourse, but it is also true that ghettos are exploited by agents of the greater culture. Software testers, particularly junior-level software testers, especially such testers that are because of their circumstances ignorant of modern software development discourse, are especially prone to exploitation. If they are represented by an agent, then someone is paid to recruit them and train them. Someone is paid to negotiate their contracts. Someone sells their employers the tools they have been trained to use.

I have not found out anything about the training that autistic people, Aboriginals, or Malaysians receive; the original draft of this essay was to have provided a detailed analysis of the training provided by Doran Jones, Per Scholas , and Keith Klain in New York City. There is a wealth of material available, and I urge you to search online yourself.

However, in the course of doing that research, I discovered that Doran Jones has sued Keith Klain and Per Scholas for fraud over that software testing training program. The entire document (PDF download link)  is a fascinating look at the inner workings of those agencies that sell software testing services. The part of the lawsuit relevant to the software testing training begins at paragraph 89, for those who wish to read it. Also of interest given the stated list of Per Scholas partners is paragraph 37.

Under the circumstances, and given the nature of the claims against Klain and Per Scholas, I think it is not appropriate for me to publish my comments, but I would like to point out one fact not contained in the lawsuit documentation: the Association for Software Testing committed their resources  to the Per Scholas training program in NYC while Keith Klain was a member of the AST Board of Directors. I expect that the current and recent officers of the AST are extremely interested in the outcome of this lawsuit.

Klain says this about software testers: "Software testing is a strange business. It’s commoditized (sic), devalued, misunderstood, and goes through cycles of being chopped, changed, and lives at the front lines of imminent takeover by our robot overlords. Why anyone would want to be a professional software tester is even harder to understand." Read the whole thing

Interestingly, Klain and the people involved in this Per Scholas project are also the most vocal opponents of software testing certification, sometimes with questionable approaches to gutting certification efforts.

It makes sense that these agents of minimally trained software testers would oppose certification. A global, generally-accepted, inexpensive certification in software testing would allow those entry-level software testers with limited knowledge of modern software development culture to more easily be their own agents in that culture. The market for this sort of exploitation might shrink considerably. In hindsight, I wish I had said this explicitly when I tackled the topic in 2010.  As your career matures, your CV becomes more important than your certifications, but getting certified early on is a perfectly reasonable career move.

As Marlena Compton said in her 2015 essay "A Tableflip Guide: Transitioning from Tester to Developer"  "If you go to a testing conference you’ll find people talking about how you can stay in testing forever and how it is a great career path. I’ve noticed that, often, the testers who shout the loudest about staying in testing forever have carved out their own place in the power structure of the software testing industry." I urge you to read the whole essay.

I'll suggest further that those testers shouting the loudest may also depend on the minimally skilled testing ghetto for their livelihood, and may not have your best interests in mind.

If you as a software tester

are happy with your career path and prospects for growth
are happy with the skills you have and the prospects to develop them further
are respected by everyone on your development team and are treated as a peer
represent your own interests to your employer with good faith on both sides
have technical training available to you
understand technical aspects of software development other than testing

then this essay probably does not describe you. If these things are not true for you, you may be in the software testing ghetto.

Thursday, June 23, 2016

Reviewing "Context Driven Approach to Automation in Testing"

I recently had occasion to read the "Context Driven Approach to Automation in Testing". As a professional software tester with extensive experience in test automation at the user interface (both UI and API) for the last decade or more for organizations such as Thoughtworks, Wikipedia, Salesforce, and others, I found it a nostalgic mixture of FUD (Fear, Uncertainty, Doubt), propaganda, ignorance and obfuscation. 

It was weirdly nostalgic for me: take away the obfuscatory modern propaganda terminology and it could be an artifact directly out of the test automation landscape circa 1998 when vendors, in the absence of any competition, foisted broken tools like WinRunner and SilkTest on gullible customers, when Open Source was exotic, when the World Wide Web was novel. Times have changed since 1998, but the CDT approach to test automation has not changed with it. I'd like to point out the deficiencies in this document as a warning to people who might be tempted to take it seriously.

The opening paragraph is simply FUD. If we take out the opinionated language

poorly applied
terrible waste
shallow, narrow, and ritualistic
pandemic, rarely examined, and absolutely false

what's left is "Tool use in testing must therefore be mediated by people who understand the complexities of tools and of tests". This is of course trivially true, if not an outright tautology. The authors then proceed to demonstrate how little they know about such complexities.

The sections that follow down to the bits about "Invest in..." are mostly propaganda with some FUD and straw-man arguments about test automation strewn throughout. ("The only reason people consider it interesting to automate testing is that they honestly believe testing requires no skill or judgment" Please, spare me.) If you've worked in test automation for some time (and if you can parse the idiosyncratic language), there is nothing new to read here, this was all answered long ago. Again, much of these ten or so pages for me brought strong echoes of the state of test automation in the late 1990s. If you are new to test automation, consider thinking of this part of the document as an obsolete, historical look into the past. There are better sources for understanding the current state of test automation.

The sections entitled (as of June 2016) "Invest in tools that give you more freedom in more situations" and "Invest in testability" are actually all good basic advice, I can find no fault in any of this. Unfortunately the example shown in the sections that follow ignores every single piece of that advice.

Not only does the example that fills the final part of the paper ignore every bit of advice the authors give, it is as if the authors have chosen a project doomed to fail, from the odd nature of the system they've chosen to automate, to the wildly inappropriate tools they've chosen to automate it with.

Their application to be tested is a lightweight text editor they've gotten as a native Windows executable. Cursory research shows it is an open source project written in C++ and Qt, and the repo on github  has no test/ or spec/ directory, so it is likely to be some sort of cowboy code under there. I assume that is why they chose this instead of, say, Microsoft Word or some more well engineered application.

Case #1 and Case #2 describe some primitive mucking around with grep, regular expressions, and configuration. It would have been easier just to read the source on github. If this sort of thing is new to you, you probably haven't been doing this sort of work long, and I would suggest you look elsewhere for lessons.

Case #3 is where things get bizarre. First they try automating the editor with something called "AutoHotKey", which seems to be some sort of ad-hoc collection of Windows API client calls, which according to the AutoHotKey project history is wildly buggy as of late 2013 but has had some maintenance off and on since then. I would not depend on this tool in a production environment.

That fails, so then they try some Ruby libraries. Support for Windows on Ruby is notoriously bad, it's been a sticking point in the Ruby community for years, and any serious Ruby programmer would know that. Ruby is likely the worst possible language choice for a native Windows automation project. If all you have is a hammer...

Then they resort to some proprietary tool from HP. You can guess the result.

Again, assuming someone would want to automate a third-party Windows/Qt app at all, anyone serious about automating a native Windows app would use a native Windows language, C# or VisualBasic.NET, instead of some hack like AutoHotKey. C# and VisualBasic.NET are really the only reasonable choices for such a project.

It is as if this project has been deliberately or naively sabotaged. If this was done deliberately, then it is highly misleading; if naively, then it is simply sad.

Finally I have to point out (relevant to the article section "Invest in testability", and again strong shades of 1998) that this paper completely ignores the undeniable fact that the vast majority of modern software development takes place on the web, with the UI appearing in a web browser and APIs offered from servers over a network.  This article makes no mention that selenium/webdriver is a UI automation standard adopted by the World Wide Web Consortium (W3C), that the webdriver automation interface is fully supported by every major browser vendor:  Google Chrome, Mozilla Firefox, Microsoft Internet Explorer, Opera, and most recently Apple Safari, or that the Selenium API is fully supported in five programming languages: C#, Java, Ruby, Python, and Javascript, and partially supported in many more.

Ultimately, this article is mostly FUD, propaganda, and obfuscation. The parts that are not actually wrong or misleading are naive and trivial. Put it like this: if I were considering hiring someone for a testing position, and they submitted this exercise as part of their application, I would not hire them, even for a junior position. I would feel sorry for them.

Tuesday, June 21, 2016

Who I am and where I am June 2016

From time to time I find it helpful to mention where I am and how I got here. I have been pretty quiet since 2010 but I used to say a lot of stuff in public.

For the past year I have worked for, formerly the Salesforce Foundation, the independent entity that administers the philanthropic programs of My team creates free open source software for the benefit of non-profit organizations.  I create and maintain automated browser tests in Ruby, using Jeff "Cheezy" Morgan's page_object gem.  I'm a big fan.

My job title is "Senior Member of the Technical Staff, Quality Assurance".  I have no objection to the term "Quality Assurance", that term accurately describes the work I do. I am known for having said "QA Is Not Evil".

Before I spent three years with the Wikimedia Foundation , working with Željko Filipin  mostly, on a similar browser test automation project , but much larger.

I worked for Socialtext, well known in some circles for excellent software testing. I worked for the well known agile consultancy Thoughtworks for a year, just when the first version of Selenium was being released. I started my career testing life-critical software in the US 911 telecom systems, both wired/landline and wireless/mobile.

I have been 100% remote/telecommuting since 2007. Currently I live in Arizona, USA.

I used to give talks at conferences, including talks at Agile2006, Agile2009, and Agile2013. I've been part of the agile movement since before the Manifesto existed.  I attended most of the Google Test Automation Conferences  held in the US. I have no plans to present at any open conferences in the future.

I wrote a lot about software test and dev mostly around 2006-2010. You can read most of it at stickyminds  and TechTarget , and a bit at PragProg

I hosted two peer conferences in 2009 and 2010 in Durango Colorado called "Writing About Testing". They had some influence on the practice of software testing at the time, and still resonate from time to time today.

I create UI test automation that finds bugs. Before Selenium existed I was user #1 for WATIR, Web Application Testing In Ruby. I am quoted in both volumes of Crispin/Gregory Agile Testing , and I am a character in Marick's Everyday Scripting.

Sunday, March 27, 2016

Remarks on Wikimedia Foundation recent events

If you pay attention to Wikipedia culture and the WMF, you may know that the Executive Director of the WMF, Lila Tretikov, has resigned amid some controversy.

It is an extraordinary story, especially since, given the nature of Wikipedia culture, so much information about events is publicly available. I'll point you to Molly White's "Wikimedia timeline of recent events" as an excellent synopsis of Ms. Tretikov's tenure as ED.  The thing that strikes me most about that timeline is the number of people who left, and the long tenure of each person who departed. On the same subject, Terry Chay's note published on Quora also addresses this.

My own tenure at WMF was just over three years, from 2012 to 2015. In that time Željko Filipin  and I built an exceptionally good browser test automation framework, which at the time I left WMF was in use in about twenty different WMF code repositories. My time at WMF was roughly evenly split between Ms. Tretikov as ED and under the previous ED, Sue Gardner.

There are two things about Wikipedia and WMF that I think are key to understanding the failures of communication and culture under Ms. Tretikov's leadership.

As background, understand that everyone in the Wikimedia movement, without exception, and sometimes to a degree approaching zealotry, is committed to the vision: "Imagine a world in which every single human being can freely share in the sum of all knowledge."  I still am committed to this myself. My time at WMF absolutely shaped how I see the world.

Given that, what is important to understand is that Wikipedia is essentially a conservative culture. The status quo is supremely important, and attempting to change the status quo is *always* met with resistance. Always. There is good reason for that: Wikipedians see themselves as protecting the world's knowledge, and changes to the current status are naturally perceived as a threat to the quality or even the existence of that knowledge.

The other thing important to understand is that many of the staff at WMF come from the Wikipedia movement or the FOSS movement. Many (not all) of the technical staff began working with Wikimedia/FOSS software in college or even in high school, and ended up employed by WMF without ever experiencing how software is made and managed elsewhere. Likewise many (not all) of the management staff were (and are) important figures in the Wikipedia movement, without much experience in other milieux.

In practice, when attempting to make a change to Wikimedia software or Wikipedia culture, the default answer is always "no". No, you can't use that programming language, that library, that design approach, that framework. No, you can't introduce that feature or that methodology. 

So a big part of the work for those working in this culture is persuasion. One is constantly justifying one's ideas and actions to both one's peers and to management, and to the community, in the face of constant skepticism.  Wikipedians talk about "consensus culture" but in practice consensus is actually more along the lines of "grudging acceptance".  Sue Gardner's most recent blog post explains this better than I ever could.

And because so many Wikipedians have such a dearth of experience of other tech culture, NIH (Not Invented Here) is rampant. It was difficult to introduce proven, reliable, well-known tools simply because they were *too* well-known; they aren't *Wikipedia* tools, they don't have *Wikipedia* support, there is limited knowledge of them within the culture.

The result of these forces is that significant feature releases tend to be fiascos, but each fiasco of a somewhat different character. When WMF released the Visual Editor, the software was not fit for use, everyone involved knew it was not fit for use (or should have, they were certainly told), and the community rejected it for good reason. On the other hand, the Media Viewer *was* fit for use when it was released, but it was such a new paradigm that the community rejected it even more decisively than they had the Visual Editor. We could even speculate that had Media Viewer been as unusable upon release as the Visual Editor was, it might have received a kinder reception from the Wikipedia community.

Some notable exceptions to the fiasco release pattern were the Mobile Web work; the Mobile Web team did a great job and demonstrably made Wikipedia better, even if often over the occasional objections of their peers on the technical staff.  And the rollout of HHVM went well, as did the introduction of ElasticSearch, but none of these projects faced the Wikipedia old guard directly.

It also is notable that it took Željko and me three years to get our work accepted widely across all of WMF. Today I am building essentially the same system for (the philanthropic entity attached to as Željko and I did for WMF. I expect to have my project in the same position as the WMF project in one year, because I don't face the constant hurdle of having to persuade and persuade and persuade. Again, this is not necessarily a Bad Thing: the institutional skepticism and constant jockeying for acceptance of ideas, tools, and practices at WMF is a mechanism that protects the core mission of Wikipedia, even if it often makes the culture psychologically trying if not outright poisonous. You could argue that having to justify beforehand and evangelize afterward every step we took made the system that Željko and I built better than it would otherwise have been. If I seem to have a low opinion of the WMF understand that in my time at WMF I did some of the best work I've ever done, and I consider my time there to be the pinnacle of my career so far.

So it is perfectly understandable that Ms. Tretikov as Executive Director would want to launch an ambitious skunkworks project in secret. This is something CEOs do. CEOs have discretion over the budget, and they are responsible to shareholders for profits. But the Executive Director of the WMF cannot expect to hide a quarter-million dollar project engaging entities beyond Wikipedia without dire consequences, which is exactly what happened. Or was at least the final act in a long series of poorly executed maneuvers that alienated staff and community to the point of near-paralysis, and that caused a monumental loss of faith from the community as well as a huge loss of institutional knowledge as so many experienced staff abandoned the Foundation, or were abandoned by the Foundation.

I imagine that WMF and the Wikipedia movement will toddle on much as they always have. The Wikipedia vision of free knowledge for every human being remains compelling. And I hope that this troublesome period in the history of WMF can serve as a lesson not only to the Wikipedia community, but to the rest of us concerned with how best to make software work for our world.