Friday, September 27, 2013

Magic: strings and regular expressions in ruby Cucumber tables


I stumbled on an interesting trick that is worth documenting

Say you have a Cucumber Scenario Outline like so:

 Scenario Outline: Foo and bar
    When I click < foo >
    Then < bar > should appear in the page
  Examples:
    | foo        | bar                |
    | X          | =String or regex   |
    | Y          | ===String or regex |
    | Z          |  String or regex   |

and we use it like so:

Then(/^(.+) should appear in the page$/) do |bar|
  on(FooPage).page_contents.should match Regexp.new(bar)
end 

but this won't work, because the case with "=" will pass by accident if page_contents erroneously contains "===" instead.  And case Z would always pass no matter how many '=' characters are in the page.  (That's clear, right?  Say something in the comments if that doesn't make sense to you.)

Also, I really need to check for a leading space for case Z, so what I think I need is not a string but a regular expression, so those anchor carets below should work, right?

  Examples:
    | foo        | bar                 |
    | X          | ^=String or regex   |
    | Y          | ^===String or regex |
    | Z          | ^ String or regex   |

That seems reasonable.  I've made sure that the values for 'bar' are interpreted as regexes, so this should Just Work...

What I discovered is that the values for 'bar' are actually magically turned into *escaped* regexes, so what ends up being compared to page_contents is actually:

/\^\=String or regex/
/\^\=\=\=String or regex/
/\^ String or regex/


and that is no help at all.  Even though they are regexes, the anchor characters are being escaped within the regex so the arguments continue to function as strings.  That is, what is being sent to RSpec as the argument to match is equivalent to the literal strings

"^=String or regex"
"^===String or regex"
"^ String or regex"

and of course that isn't what we want at all.

Here is what I finally figured out:

First we put the regexes that we want to use inside a pair of some character, in this case single quotes.  (And it works for leading spaces in strings, too!)

  Examples:
    | foo        | bar                   |
    | X          | '^=String or regex'   |
    | Y          | '^===String or regex' |
    | Z          | '^ String or regex'   |

Here's the tricky part.  Before we do the match in RSpec we strip the single quotes, and hey presto the result is an unescaped regex *not* a string:

Then(/^(.+) should appear in the page$/) do |bar|
  bar = bar.gsub(/'/, '')
  on(FooPage).page_contents.should match Regexp.new(bar)
end 

That is, what gets sent to RSpec as the argument to 'match' ends up being

/^=String or regex/
/^===String or regex/
/^ String or regex/

and is correctly interpreted by RSpec as a regex-- not as an escaped regex and not a string either.

Sometimes you can have a little too much magic.

Sunday, July 28, 2013

giving a talk at Agile2013



So I'm giving a talk next week at the Agile2013 Conference about "radically open software testing".  It's about my experience over the last eighteen months or so founding and maintaining the QA/testing practice for the Wikimedia Foundation, the good folks who keep the lights on at Wikipedia.

I've done some little peer conferences, but I haven't presented at or even attended a big conference like Agile2013 since I talked about browser test design at Agile2009 in Chicago.  That worked out pretty well, at least Dave Haeffner liked it.  I know a little more about the subject since I gave that presentation also.

I think my presentation might be unusual.  I have nothing to sell.  I have no particular agenda to advance, except to encourage people to contribute to Wikipedia.  I intend to talk about some notable failures too.  What I'll be discussing isn't even particularly "agile", for whatever value the word "agile" has today any more.

I don't even have any slides.  I started making some, but they worked a lot better as high points or an outline for a long conversation than actual nuggets of useful information.  What if I met Tufte some day?  I'd rather just have a conversation and do demos.  Examples still come in handy.

So if you're in Nashville and interested in QA, testing, open software, free knowledge, ukulele, whatever, stop in for the conversation.


Monday, June 03, 2013

open testing Wikipedia mail list (and miagi-do)


For anyone interested in helping to test the software the runs Wikipedia, we have a new mail list dedicated to exactly that topic. The mail list is not even a month old, so the archives are pretty readable.     If you're interested, you can sign up here.

Let me expand on what I mean by "open testing" and by "interested".  At any given time, there are dozens of software development projects (at least) in progress in support of Wikipedia.  Some of these projects are for particular niches in the Wikipedia universe, but others are expensive, high-profile, in some cases world-changing.

Every one of these projects needs more testing than it gets.

So if you want to spend an hour looking for bugs in a Wikipedia feature under development, that's great.  We at the Wikimedia Foundation have worked with Weekend Testing Americas and other organizations on several occasions to do exactly that.

But would you like to help design every aspect of the ongoing test effort for major software development projects for one of the biggest web properties ever, and the biggest encyclopedia in the history of the world?  That's also possible.

And so is everything in between.  From the code to the documentation to the licensing to the communication, every aspect of this software development is open to scrutiny by anyone at any time.  This environment allows thoughtful contributions by an interested community, and I hope would be particularly attractive to thoughtful software testers.

That's what we're talking about on the Wikimedia Foundation QA mail list.  Feel free to join.

So about miagi-do...

A disclaimer up front:  I have been aware of the existence of miagi-do for some time, and many of its members are friends and acquaintances.  But I am not a member, nor I have ever had a conversation with anyone about any aspect of miagi-do.

So I was intrigued when the miagi-do folks started a blog, and I was particularly intrigued by the first   two posts. Or maybe four.

It's quite possible to certify yourself as a software tester by simply working openly, as the miagi-do people have.  And the software that powers Wikipedia is radically open, for testing, for everything.  And while thoughtless contributions are eliminated ruthlessly, no approval is required to contribute.  I think the miagi-do folks would agree with me:  If you meet the Buddha on the road, kill him.



Tuesday, April 02, 2013

Weekend Testing - Wikipedia Test Event Apr 6

Wikipedia is improving the new user experience.

Wikipedia is developing a new way to create accounts on Wikipedia and a new experience for new users when logging in.

Weekend Testing Americas has an online test event on the first Saturday of every month 10AM-1PM Pacific time (17:00-20:00 UTC).

 On April 6 WTA will be testing account creation, login, and the new user experience for Wikipedia . Justin Rohrman will facilitate.

 If you'd like to join this Saturday:


  • Send a message on Skype to weekendtestersamericas 
  • Join #wikimedia-dev on freenode on IRC 
  • Read over the Test Plan 
  • Test! 


 (If you can't attend the Weekend Testing session but you are interested in QA for Wikipedia, we have public QA events ranging from bug triage to Cucumber development every week.)

Tuesday, March 26, 2013

Who I am and where I am, March 2013




I am the QA Lead for the Wikimedia Foundation  I test Wikipedia.  My one year anniversary was in January.  I do exploratory testing and I do browser test automation with my colleague Željko Filipin.  I have a strong interest right now in community testing of all kinds.

I'll be talking about testing Wikipedia at some conferences soon:

GTAC April 23-24, NYC
Telerik Test Summit May 3-4, Austin TX
Wikimedia Hackathon May 24-26, Amsterdam
Agile2013 (pending acceptance) Aug 4-10, Nashville TN

I used to write about software a lot: SearchSoftwareQuality.com  (warning: registration wall), StickyMinds.com  and a couple of articles for PragPub.   I wrote a chapter for Beautiful Testing.

For many years I have been telecommuting from Durango CO where I created and hosted the Writing About Testing peer conference.  WAT is unlikely to happen again.

I am on my way out of Durango, but I will continue to telecommute.  In the coming months you might also find me near:

Santa Barbara CA
Tucson AZ
San Francisco CA
Southern Utah
Ithaca NY
Boulder CO
Dulce NM

I'm @chris_mcmahon on Twitter, christopher.mcmahon on all the Google services, chrs_mcmhn on Skype, although I rarely use Skype.  I am on a lot of Wikimedia IRC channels every day.  I am not on LinkedIn or Facebook.

Saturday, January 26, 2013

A community test event: mojibake and meaning



UPDATE 28 January the official announcement on the WMF blog.

The Wikimedia Foundation is developing new editing software for Wikipedia.  For the first time in history we intend to have a native rich text WYSIWYG editor for the Mediawiki engine that does not require users to know wiki markup syntax in order to create and edit articles on Wikipedia. 

Right now the Foundation needs help testing this editor, and we invite you to participate.

We have a suite of automated tests for the editor that is capable of checking the contents of existing articles, loading them into VisualEditor, saving the article, and checking that no information has been lost in the round-trip saving operation.  We are confident that the editor is robust enough to load and save any given content on the English Wikipedia. 

What is not clear is whether VisualEditor is robust enough to preserve information after that information has been altered:  additions, changes, deletions, other manipulations of the content of Wikipedia pages may or may not cause unintentional alterations to the contents of pages upon saving.  We just don't know. 

Of particular interest is manipulation involving non-Latin characters.  Various language scripts and other aspects of the greater Unicode character set should be handled correctly by the VisualEditor, and we want to find out if that is true. 

This sort of testing of course is best done by real human beings.  Ultimately this is a test not only that VisualEditor preserves altered text correctly, but that VisualEditor preserves *meaning* correctly.

This testing should be of interest not only to those who will eventually want to edit Wikipedia in their native language, but also to those interested in accessibility, internationalization and localization, maintainability and scalability as well. 

For the week starting 28 January 2013, the Foundation is providing extraordinary support for people testing non-Latin character handling in VisualEditor.  We will be available on IRC, on mail lists, on social media, etc. fielding questions and facilitating discussion of the behavior of VisualEditor handling of non-Latin characters.  It would be a fine thing if you could help us test VisualEditor.

We have created a test plan for the VisualEditor test event.

We plan to have more community test events in the future every month for Wikimedia Foundation development projects.  Coming up are sessions for "Echo", our new notification system for editors of Wikipedia, improvements to our Article Feedback system for critiquing Wikipedia articles, and more.  If you are interested in doing exploratory testing for Wikipedia, we encourage you to join our new Features Testing Group, the central repository for information about these ongoing testing exercises.




Sunday, December 16, 2012

Open browser test automation at WMF


Almost a year ago I started working as QA Lead for the Wikimedia Foundation.  Among other things, I had a mandate to create an automated browser testing practice at WMF.

From the start I wanted this project to be a world class, completely open, reference implementation of such a project, using the best and most modern tools and practices I could find.  I wanted this to be a project that anyone could read, anyone could run, and to which anyone could contribute.  I wanted this to be an industry standard implementation of a well-designed, well-implemented, working browser test automation project.

Around 2006 my career had veered off into a test automation approach that, while valid and useful in certain circumstances, would be inappropriate for the WMF project.  And in the years since 2006, the tools and practices that were immature at the time had grown into mature, stable, powerful projects of their own. 

I set out to educate myself about the details of the cutting edge of browser test automation in every language.  I visited Austin TX twice and San Francisco several times in the past year to discuss approaches to the project in person with experts on the subject. 

WMF hired Željko Filipin in October 2012 specifically for the browser test automation project, and just this week we opened the curtain and turned on the lights.  The WMF browser test automation project is in Ruby, using

* page_object gem
* watir-webdriver
* selenium-webdriver
* Cucumber
* RSpec
* rake
* Jenkins integration

The initial announcement is here.
A first pass at technical documentation is here, including instructions to run the tests locally if you want to see them in action on your own machine.
Some community concerns are here.
The main code base is managed in gerrit but there is a more accessible read-only mirror on github.
Our Jenkins instance is running on the Cloudbees service  using Sauce Labs  hosts, and the current test results are visible here.
If you would like to follow the project or contribute to it yourself, we are starting a community group you can join.  Feel free to add yourself to the page here  to follow the project or contribute to it.

Many people helped me along the way to making this project what it is.  Hopefully I haven't left out anyone, feel free to remind me if I did!

Jeff Morgan for creating the page-object gem and for answering tons of my questions
Jari Bakken, especially for this presentation about webdriver in Ruby.
Brahma Ghosh
Charley Baker
Alister Scott and his Watirmelon blog
Marlena Compton  and Matt Brandt  for discussing their experience at Mozilla WebQA with me.
Bret Pettichord for hosting  the Test Automation Bazaar
Jim Holmes of Telerik for hosting the Telerik Test Summit

And especially Željko for actually building the actual project! (And for kicking my ass  along the way, I still have a lot to learn.)