Chris McMahon's Blog

"ResT" in the context of a software Application Programming Interface (API) stands for "Representational State Transfer". Like many software design patterns, ReST is a pattern , not an implementation . Before we dig deep into ReST though, let us talk about APIs in general. When we in the QA/test world talk about APIs, we usually mean APIs exposed to users in order to manipulate software systems efficiently from the outside. These are public-facing APIs, as opposed to internal APIs set up at boundaries between disparate subsystems within the application itself. There are any number of public facing API schemes that are not ReST. SOAP is still widely in use. CORBA was a pioneer design in public APIs. AJAX (Asyncronous Javascript And XML) is an API that enables things like sliding maps in a browser. And applications may have their own proprietary public APIs: the Salesforce API is a notable example; while similar to SQL and some other things, the Salesforce API is unli...

There was a thread on Twitter recently from people who caused major problems in software systems, and I was reminded of the worst bug I've ever seen. Long ago I had published this story with a major tech media vendor, but that article seems to have succumbed to bit rot, so I am going to tell that story here. It happened more than twenty years ago when I had only been in QA for a couple of years, but it is seared into my mind... In the late 1990s I worked for a company that handled 911 land line location software for major telcos. So when you choke on that chicken bone and call 911, you can't talk but they send the ambulance to where you live anyway. We handled about 75% of all the telephone numbers in the USA. So from time to time in those days, there would be an area code split, where a new area code is added to a populous area and phone numbers start getting the new area code where they had a different area code before. In the 911 world, we would maintain records for both old...

It often happens that someone creating automated testing programs using Selenium or a similar framework that controls the UI (User Interface) eventually finds it necessary to address a non-Selenium API (Application Programming Interface) of some sort. You might need to create, update, or delete test data. You might need to modify a system setting. You might need to determine the current state of the system. You might need to have your test send or receive a signal from some other part of the deployment process. Often the only way to do these things is through use of an API. API Defined An API is a relationship between a client and a server. The server controls some aspect of the system, and the client wants to see or change some aspect of the system that the server controls. Selenium-webdriver itself operates as an API. The server for the Selenium API is called WebDriver, and WebDriver exists only on the browsers that Selenium is automating. Your script controls the Selenium client tha...

Real paella is made in one pan. This recipe is "fake" because the seasoned rice is prepared separately from the other ingredients and added at the end. The advantage is that we can include a bigger variety of fresh vegetables. Feel free to vary the amounts below for your own taste.   Invest in preparation. Once you start cooking, this process goes very quickly. Seasoned rice: 2 cups long grain white rice. (I like jasmine rice, but any will do) 1 spice bottle of saffron (about 1 gram, get the biggest amount for the least cost you can find, more is better) Dried oregano, a healthy shake Cumin to taste (I go easy on cumin, too much smells like armpits) Black pepper, just a bit  Meat and seafood: 1/2 pound or more savory sausage links, like spicy Italian, Andouille, or Spanish chorizo, cut in thin medallions (Note that sausage is required because all the other ingredients rely on a little sausage grease in the pot. If you omit sausage, add a little oil to coo...

These are three of my favorite large scale recipes. They all feature complete and concentrated nutrition. They all involve a significant commitment of labor and time, so I learned to make them in large batches. Also, they can all be made in a concentrated form, and reconstituted at serving time by adding water. The marinara and green chile are particularly amenable to freezing and refreezing over and over. Enjoy... Marinara: 5-6 pounds fresh red tomatoes (or more), diced 1 head garlic, skinned and crushed 1 small yellow onion, diced 4 oz fresh basil, diced 4+ oz grated fresh parmesan cheese (optional, see vegan note below) 1 or 2 small cans tomato paste olive oil dried oregano to taste black pepper to taste salt to taste Dice the onion and crush or dice the garlic into a large pot. Cover with olive oil and cook until onion is transparent. Dice the tomatoes and fresh basil and add to cooked onion/garlic. Stir. Add tomato paste, grated parmesan, oregano and black pepper.  Simmer at l...

Like any recipe, you can tweak this for your own situation. First, you need a shared test environment to work in. I recommend beginning with a persistent test environment intended to model your production environment as closely as possible. While you may eventually evolve the ability to share a test environment with commonly-configured VMs or containers, having a persistent shared test environment at the beginning gives everyone the experience of keeping something running that looks a lot like the production system. (TOAD! DevOps!) For that matter, proper use of feature flags and such could make production itself a perfectly fine shared test environment. Sharing the responsibility of keeping the shared test environment up and running and valuable is a big part of understanding the value of the recipe. Second, you need some end-to-end tests (Testing!) to run against the shared test environment. You need to set up test data, tear down test data, type the typies and click the clickies...

TOAD is Testing, Observability And Devops. We think these three things are related. What would happen if we took each of the three aspects of TOAD and put strong emphasis on each in turn? What happens if we take our testing effort as far as it can go, and "dial it to 11"? Observability to 11? Devops to 11? The meaning of "dial to 11" will be different for different organizations: it might mean hiring staff, or investing in tools, or even just emphasizing the mission more than before. If we dial testing to 11, I think two things will result. For one thing, the number of observable incidents in production will likely go down, because the emphasis on testing means that more problems will be found and fixed before deploying to prod. I also think that rate of deployments (Devops) could potentially increase, because the decrease in observable incidents will make deployments safer. So: increase the T in TOAD to decrease the O and increase the D. With testing at 11, no...

TOAD is Testing Observability And DevOps My last blog post   was about why TOAD ideas are important and how TOAD ideas interact with each other. But even the most excellent TOAD systems occasionally release bugs to production. This post describes how to understand these problems and how to address them when they happen. And there is an experience report at the end! Most of the time we learn about problems in the production system because the system is observable. We see a performance problem, or a data problem, or a space problem, and we can use observability tools to narrow down and ultimately fix the cause(s) of the observed problem. There is also an entire class of problems that cannot be discovered with typical observability tools, but only by testing the system. Given that we find such problems in production, what can we do about it? Rollback and Replay The most drastic response to a production problem is to "roll back" the system to the last known good state...

TOAD is Testing Observability AND DevOps The AND is important! Disclaimer: In my career I have done most of the things I describe below, but never tied them all into one project. The following should be possible: Testing a System Suppose we have a software system of reasonable complexity. Suppose our system is comprised of a front end with a user interface (UI) and a back end, thus a client and a server. The front end and the back end communicate via an application programming interface (API) of some sort. This is a common architecture of many software systems. Suppose further that our front end and our back end are well-designed. They have unit tests, and meet whatever definition of quality you would want to apply. Because our system is reasonably complex, we want to have a suite of end-to-end tests that exercises the entire software-and-data stack,  that demonstrates that the users can do the things they need to do, and that the front end and the back en...

Possibly the most important aspect of an agile process is the retrospective. A retrospective usually happens in a team meeting, generally at the end of every agile iteration or sprint. While there are any number of ways to run retrospectives, the object is to discuss three questions:      What is working well?      What is not working well?      What should we change? The first question tends to be the easiest to answer, and it is tempting to just skip it, but that is dangerous. It is every bit as important to celebrate the team's success as it is to grapple with the team's issues and problems. Positive reinforcement is more powerful than negative reinforcement. The second question also tends to be easy to answer in teams where the members feel safe to discuss work honestly. It is important to surface problems so that the whole team is aware of the current issues and why those issues affect the work. The third question te...

There is a sentiment in software development that if you do not have a working BDD practice then Cucumber is just unnecessary overhead. I understand this position, but I disagree, based on my own experience and my own practice. I find Cucumber is particularly valuable in automated browser tests. I've been using Cucumber for automated browser tests at work just about every day for the last six years or so. At the same time, I have never worked with a full-on BDD team. Beyond BDD, here are three aspects of Cucumber I find particularly valuable: Cucumber creates a low barrier to entry for anyone at any time to contribute and understand the project. Cucumber's Given/When/Then syntax provides a design guide particularly well suited for browser tests, especially using "When" in a particular way. When a test fails, Cucumber provides a plain-English description of what the test does that may not be immediately apparent from the code or the nature of the failure. I tend ...

Some of the Watir community has been discussing the history of the project. Here I try to set down some notable things that I remember about that time. We have to start the story of Watir with Ruby. The first English documentation for Ruby was published in 2000, and it garnered a lot of interest, particularly as it was so amenable to creating Domain Specific Languages (DSLs), which caught the attention of a number of people working in testing, and in the Agile world. Python 2.0 also came out in 2000, and the dominant scripting language at the time was Perl. Of all the browsers available at that time, only Internet Explorer exposed an API for automating browser actions, via a COM interface. In 2001, Chris Morris published a Ruby library that exploited IE's COM interface called WTR, for Web Testing in Ruby. At that time, Open Source software was often seen as inferior or even downright suspicious, and was poorly understood by most businesses. Test automation products were exclusively...

This is the story of a really great project I did while working for Salesforce.org. I have done a significant amount of remote pair programming over the last ten years, but this project was extraordinary in a number of ways. For one thing, it was a really complicated problem that demanded a technically advanced solution. For another thing, it took almost an entire year to finish-- one hour per week. The Problem I will try to give you the background in a way such that your eyes don't glaze over: in order to work with data in a Salesforce instance via the API, you address "Objects" and "Fields".  (These are actually tables in a database that may be addressed by a poor and crippled version of SQL.) For example, here is a description of the Account object whose first field is AccountNumber.   If you are a developer on the Salesforce platform, you can add your own Field to the Account object, but you have to append ' __c' to it, like " MyField__c ...

As of January 2018 I resigned my position as "Senior Member of the Technical Staff, Quality Assurance" at Salesforce.org . I have more than twenty years experience in testing user interfaces and APIs across a wide variety of platforms. If you would like to contact me, my DMs on Twitter are open or by email at christopher dot mcmahon at gmail. I do not use Facebook, LinkedIn, or Skype. I have been working remotely for more than ten years. I enjoy telecommuting, it suits me nicely. In the past decade I have lived all over the western United States, including some time in Hawaii. Here are some points from my career that help tell the story of how I came to be here today: In 1997 I started testing 911 telecom location services, life-critical software for police/fire/ambulance dispatching for most of the USA. I tested these systems through Y2K and beyond. We saved the world. Don't let anyone tell you otherwise. In 2004 I was, as far as anyone knows, the first person ...

Managing test data is one of the most difficult parts of good testing practice, but I think that managing test data receives surprisingly little attention from the testing community. Here are a few approaches that I think are useful to consider for both automated testing and for exploratory testing. Antipattern: Do Not Create Test Data In The UI But first I want to point out what I think is a mistake that many testers make. When you are ready to test a new feature or you are ready to do a regression test, or you are ready to demonstrate some aspect of the system, the data you need to work with should already be in place for you to use. Having to create test data from scratch before meaningful testing begins is an anti-pattern (and a waste of time!), whether for automated testing or for exploratory testing. Create Test Data With An API This is usually my favorite approach to managing test data. Considering this scenario in the context of an automated test, I usually will have ...

Wait In modern web applications, elements on the page may come into existence as a result of actions that the user may take, or they may disappear from the page as a result of actions that the user may take. In order to test modern web applications in the browser, it is rare that tests do not have to wait for some condition or another to be true or false before the test can proceed properly. In general, there are two kinds of waiting in automated browser tests. In the language of Watir, these are "wait_until" and "wait_while". If you are not using Watir, you have probably already implemented similar methods in your own framework. My experience is that wait_until type methods are more prevalent than wait_while type methods. Your test needs to wait until a field is available to fill in, then it needs to wait until the "Save" button is enabled after filling in the field. Your test needs to wait until a modal dialog appears in order to dismiss it, the...

A conditional statement is the use of if() (or its relatives) in code. Not using if() is a fairly well-known test design principle. I will explain why that is, but I am also going to point out some less well-known places where it is tempting to use if(), but where other flow constructs are better choices. My code looks like Ruby for convenience, but these principles are true in any language. No Conditionals in Test Code This is the most basic misuse of if() in test code: if (x is true) test x elsif (y is true) test y else do whatever comes next It is a perfectly legal thing to do, but consider: if this test passes, you have no way of knowing which branch of the statement your test followed. If "x" or "y" had some kind of problem, you would never know it, because your conditional statement allows your test to bypass those situations. Far better is to test x and y separately:  def test_x (bring about condition x) (do an x thing) (do another x t...

There is a principle in software development "DRY" for " Don't Repeat Yourself ", meaning that duplicate bits of code should be abstracted into methods so you only do one thing one way in one place. I have a similar guideline for automated UI tests, DRYP, for "Don't Repeat Your Paths". I discuss DRYP in the context of UI test automation, but it applies to UI exploratory testing as well.  Following this guideline in the course of exploratory testing helps avoid a particular instance of the "mine field problem". Antipattern: Repeating Paths to Test Business Logic I took this example from the Cucumber documentation : Scenario Outline: feeding a suckler cow Given the cow weighs "weight" kg When we calculate the feeding requirements Then the energy should be "energy" MJ And the protein should be "protein" kg Examples: | weight | energy | protein | | 450 | 26500 | 215 | | 50...

When designing automated UI tests, one thing I've learned to do over the years is to start by creating the most minimal valid records in the system. Doing this illuminates assumptions in various parts of the system that are likely not being caught in unit tests. As I have written elsewhere, I make every effort to set up test data for UI tests by way of the system API. (Even if the "API" is raw SQL, it is still good practice.) That way when your browser hits the system to run the test, all the data it needs are right in place. For example (and this is a mostly true example!) say that you have a record in your system for User, and the only required field on the User record is "Last Name". If you start designing your tests with a record having only "Last Name" on it, you will quickly uncover places in the system that may assume the existence of a field "First Name", or "Address", or "Email", or "Phone". For som...

Recently on Twitter Sarah Mei and Marlena Compton started a conversation about "...projects that still use selenium..."  to which Marlena replied "My job writing selenium tests convinced me to do whatever it took to avoid it but still write tests..."  A number of people in the Selenium community commented (favorably) on the thread, but I liked Simon Stewart's reply where he said " The traditional testing pyramid has a sliver of space for e2e tests – where #selenium shines. Most of your tests should use something else. "  I'd like to talk about the nature of that "sliver of space". In particular, I want to address two areas: where software projects evolve to a point where investing in browser tests makes sense in the life of the project; and also where the architecture of particular software projects make browser tests the most economical choice to achieve a desired level of test coverage. What Is A Browser Test Project?  Also rec...