There is a principle in software development "DRY" for "Don't Repeat Yourself", meaning that duplicate bits of code should be abstracted into methods so you only do one thing one way in one place. I have a similar guideline for automated UI tests, DRYP, for "Don't Repeat Your Paths".
I discuss DRYP in the context of UI test automation, but it applies to UI exploratory testing as well. Following this guideline in the course of exploratory testing helps avoid a particular instance of the "mine field problem".
Cucumber is great for UI tests, but Cucumber is great for lots of other kinds of tests also. This example is a really poor UI test, for two reasons. Good UI tests take a single path through the application; and good UI tests should not test business logic.
The point of a UI test is to navigate from one point in the application to a final point in the application in order to demonstrate that the application allows users to accomplish the things they need to accomplish. In the example above, more than one pass through through this path in the application is pointless; if the first pass fails, all the subsequent passes will also fail, yielding no new information. And if the underlying mathematical calculations are in fact wrong, a UI test is a really bad place to have to discover that.
In the first example, the point of the test was to check the underlying calculation engine. In this example, the point of the test is to test boundary conditions and equivalence class partitioning. These tests are done far more efficiently at the API level or the method level. Doing this sort of thing in the UI is terribly expensive.
The most convenient situation is when I am certain that all the error messages are handled by the same error processing code. In this case, I am confident that having demonstrated the proper appearance of one error, all the other errors will behave in a similar way, and I do not have to repeat my path through the application to generate more than one error.
The inconvenient case is when I know that different errors are handled by different routines in the system. Even here, instead of going around and around the same path in the application, I prefer to create individual paths that target individual errors:
Note that in this example, the Background steps will have been accomplished by way of some sort of API or data-load operation. The individual Scenarios will in fact represent distinct paths through the application, since each path shares the smallest number of steps, and each path ends in a different and unique state.
A good suite of automated UI tests will each navigate as unique a path as possible through your application in the service of your users.
A good suite of automated UI tests manifests what I call "feature coverage". By "feature coverage" I mean that the test suite executes a web of unique navigation paths in a way that makes it very difficult to be unaware of the behavior of any given feature in your application.
I discuss DRYP in the context of UI test automation, but it applies to UI exploratory testing as well. Following this guideline in the course of exploratory testing helps avoid a particular instance of the "mine field problem".
Antipattern: Repeating Paths to Test Business Logic
I took this example from the Cucumber documentation:Scenario Outline: feeding a suckler cow
Given the cow weighs "weight" kg
When we calculate the feeding requirements
Then the energy should be "energy" MJ
And the protein should be "protein" kg
Examples:
| weight | energy | protein |
| 450 | 26500 | 215 |
| 500 | 29500 | 245 |
| 575 | 31500 | 255 |
| 600 | 37000 | 305 |
Cucumber is great for UI tests, but Cucumber is great for lots of other kinds of tests also. This example is a really poor UI test, for two reasons. Good UI tests take a single path through the application; and good UI tests should not test business logic.
The point of a UI test is to navigate from one point in the application to a final point in the application in order to demonstrate that the application allows users to accomplish the things they need to accomplish. In the example above, more than one pass through through this path in the application is pointless; if the first pass fails, all the subsequent passes will also fail, yielding no new information. And if the underlying mathematical calculations are in fact wrong, a UI test is a really bad place to have to discover that.
Antipattern: Repeating Paths to Test Boundaries, Equivalence Classes, and Errors
Here is another related example that is a variation on the testing-business-logic antipattern.Scenario Outline: deposit amounts
Given a user has an account
When we deposit "amount"
Then we should see "result"
Examples:
| amount | result |
| -1 | error |
| 0 | error |
| 1 | success |
| 1000 | success |
| 1001 | error |
In the first example, the point of the test was to check the underlying calculation engine. In this example, the point of the test is to test boundary conditions and equivalence class partitioning. These tests are done far more efficiently at the API level or the method level. Doing this sort of thing in the UI is terribly expensive.
Use Unique Single Paths to Test Errors
However, there is a legitimate argument for testing the appearance of error messages in the UI. My design approach for testing the appearance of error messages in the UI falls into two categories:The most convenient situation is when I am certain that all the error messages are handled by the same error processing code. In this case, I am confident that having demonstrated the proper appearance of one error, all the other errors will behave in a similar way, and I do not have to repeat my path through the application to generate more than one error.
The inconvenient case is when I know that different errors are handled by different routines in the system. Even here, instead of going around and around the same path in the application, I prefer to create individual paths that target individual errors:
Background: Given I have an account And the account contains "$100" Scenario: negative balance When I withdraw "$101" Then I should see the negative balance error Scenario: exceed balance limit When I deposit "$901" Then I should see the balance limit exceeded error
Note that in this example, the Background steps will have been accomplished by way of some sort of API or data-load operation. The individual Scenarios will in fact represent distinct paths through the application, since each path shares the smallest number of steps, and each path ends in a different and unique state.
DRYP in practice
A good automated UI test navigates a path through the application in order to demonstrate that your users continue to be able to accomplish the things they need to accomplish in your application.A good suite of automated UI tests will each navigate as unique a path as possible through your application in the service of your users.
A good suite of automated UI tests manifests what I call "feature coverage". By "feature coverage" I mean that the test suite executes a web of unique navigation paths in a way that makes it very difficult to be unaware of the behavior of any given feature in your application.