Skip to main content

ui test tool search

I now work for 42lines. I was hired along with Marisa Seal to start the QA/testing practice for a very small totally distributed/remote software shop implementing agile processes as they make sense. I like this place.

One of the things that 42lines wants to do is to begin UI-level test automation. I have a lot of experience doing this, but I've never done it from a standing start, so this was a great opportunity to get a good look at the state of the practice for UI-level test automation.

For the last 3 years or so I've been using keyword-driven test frameworks that use a wiki to manage test data. I like these wiki-based table-based keyword-driven frameworks a lot. I'm a little suspicious of the BDD-style frameworks like Cucumber and others based on rspec-like text interpretation. Anecdotal evidence suggests that analyzing the causes of failing tests within BDD-style frameworks is an onerous task; also, I suspect that since BDD-style frameworks map closely to story coverage, there ends up being a lot of duplication of test steps. With wiki/table/keyword frameworks, I know from experience that it is possible to design tests with great domain coverage, little duplication, and scale to numbers of test steps in the suite in the five figures.

I was really hoping to find something great in Ruby. I did find that Fit has been ported to a number of languages, including Ruby, but it's still a pretty primitive ecosystem out there. I didn't find any significant bridges between Fit and Watir or Fit and Selenium-RC in Ruby.

(For the record, it would have been great to be a contributing member of the Watir community again. As far as anyone on the Watir team can tell, I was the very first Watir user, and I still think it is a brilliant project.)

So the best-of-breed wiki-based test management system seems without a doubt to be FitNesse. I've used it and I like it, and Marisa is something of an expert in FitNesse and its accoutrement. FitNesse comes in two flavors: Java and C#. 42lines is a Java shop. Selenium Core is in Java. I got no time to fuss with JRuby or whatever the latest coolest implementation of Ruby in the JVM is. It makes every kind of sense to use Java/Fitnesse and Java/Selenium for this project.

So FitNesse and Selenium in Java are beyond dispute the best-of-breed in their respective fields. I thought they went together like peanut butter and jelly, pizza and beer, and SURELY there would be a canonical reference implementation of an integration between these two awesome tools.

Sadly, I was mistaken.

There are to my knowledge three reasonable FitNesse+Selenium integration projects. The first one, fitnium, I dismissed mostly on the basis of rumors of bugs and rumors of lack of support. And in the official documentation, I found that the project was taking on features (Flex and Flash support) for which I had no need. Also, I object to abstracting/renaming Selenium methods for purely arbitrary reasons, as it makes the (very nice) Selenium documentation unusable in the tool.

That left us with a choice of StoryTestIQ and WebTest. We spiked them both.

StoryTestIQ has a beautiful UI, but it has too much magic. The killer flaw for us was that STIQ must run on the same host as the SUT. Even though it is possible to invoke *chrome/FF and *iehta/IE modes from the framework, the framework itself will not support cross-site operations. That's a non-starter for us.

And that left us with WebTest. The brilliant Gojko Adzic began the project but abandoned it about a year and a half ago. It has the feel of something partially-done. And yet in our spike with WebTest we found that we could not only invoke *chrome and *iehta modes, but we could also swap out the very latest FitNesse version and the very latest Selenium version for the antique versions shipped with WebTest, and the WebTest integration code still worked. To my mind, this speaks of good design.

Marisa and Gojko gave a joint presentation on DbFit at Agile2008. Gojko has been known to answer my emails. WebTest is a really obvious choice under the circumstances.

So as of today Marisa and I are administrators for the WebTest project on Sourceforge. I hesitate to say "owners", because neither of us knows much about Java (although Marisa certainly knows more than me) but the code for WebTest is so simple and readable that I am completely comfortable describing what it does and does not do.

Today I made some really nice demonstration tests in FitNesse driving Selenium via WebTest. We're going to make this work.


Grig Gheorghiu said…
Chris -- also take a look at the Robot Framework. Think Fit + Selenium RC.

+1 on the Robot framework. Also there is a Ruby port of Concordion but I haven't tried it.

On Watir - It will be wonderful if come back as contributing member. I found a lot of your contributions in the code swarm video I made of Watir codebase :)
Michael said…
I ended up with the same issues with Fitnium, it looked nice and was fairly easy to script into the wiki if you got Cory Foy's converter to work. At then end though I needed more IE support and my last question in the Fitnesse group went unanswered, so I switched out to FitLibrary which worked out well once I converted the scripts. It's got some nice features and I am looking forward to it, if you get WebTest working well I might consider it again, but the inactivity drove me away even though it did work well.
Lisa said…
Chris, I need a demo please!

Grig, are you saying Robot Framework somehow integrates w/ Fit or FitNesse?
Grig Gheorghiu said…
Lisa -- no, Robot doesn't integrate with Fit, but it is modeled after Fit. It also has a really strong integration with Selenium RC. I haven't personally used it yet, but it's on my TODO list ;-)
"Anecdotal evidence suggests that analyzing the causes of failing tests within BDD-style frameworks is an onerous task"

I'm biased of course, but I haven't had that experience with Cucumber, and haven't heard about anyone else having that problem.

Cucumber gives you very informative stacktraces all the way back to the plain text line. The output also gives you very detailed information about line numbers, even for passing steps.
Camelia said…
Do you know what happened with site? From one month is down :(. I worked with WebTest, it is magic and I want to continue with it at my new job but ... I cannot find it. Can you help me?
TestWithUs said…
This comment has been removed by a blog administrator.

Popular posts from this blog

Reviewing "Context Driven Approach to Automation in Testing"

I recently had occasion to read the "Context Driven Approach to Automation in Testing". As a professional software tester with extensive experience in test automation at the user interface (both UI and API) for the last decade or more for organizations such as Thoughtworks, Wikipedia, Salesforce, and others, I found it a nostalgic mixture of FUD (Fear, Uncertainty, Doubt), propaganda, ignorance and obfuscation. 

It was weirdly nostalgic for me: take away the obfuscatory modern propaganda terminology and it could be an artifact directly out of the test automation landscape circa 1998 when vendors, in the absence of any competition, foisted broken tools like WinRunner and SilkTest on gullible customers, when Open Source was exotic, when the World Wide Web was novel. Times have changed since 1998, but the CDT approach to test automation has not changed with it. I'd like to point out the deficiencies in this document as a warning to people who might be tempted to take it se…

Watir is What You Use Instead When Local Conditions Make Automated Browser Testing Otherwise Difficult.

I spent last weekend in Toronto talking to Titus Fortner, Jeff "Cheezy" Morgan, Bret Pettichord, and a number of other experts involved with the Watir project. There are a few things you should know:

The primary audience and target user group for Watir is people who use programming languages other than Ruby, and also people who do little or no programming at all. Let's say that again:

The most important audience for Watir is not Ruby programmers 
Let's talk about "local conditions":

it may be that the language in which you work does not support Selenium
I have been involved with Watir since the very beginning, but I started using modern Watir with the Wikimedia Foundation to test Wikipedia software. The main language of Wikipedia is PHP, in which Selenium is not fully supported, and in which automated testing in general is difficult. Watir/Ruby was a great choice to do browser testing.  At the time we started the project, there were no selenium bindings for …

Open letter to the Association for Software Testing

To the Association for Software Testing:

Considering the discussion in the software testing community with regard to my blog post "Test is a Ghetto", I ask the Board of the AST  to release a statement regarding the relationship of the AST with Keith Klain and Per Scholas, particularly in regard to the lawsuit for fraud filed by Doran Jones (PDF download link) .

The AST has a Code of Ethics  and I also ask the AST Board to release a public statement on whether the AST would consider creating an Ethics Committee similar to, or as a part of the recently created Committee on Standards and Professional Practices.

The yearly election for the Board of the AST happens in just a few weeks, and I hope that the candidates for the Board and the voting members of the Association for Software Testing will consider these requests with the gravity they deserve.