Automated regression testing in Agile: why, what, and how?

There’s a paradox surrounding automated regression testing of the UI in Agile. On the one hand, everyone tells you how important it is. On the other, it doesn’t seem to work right in far too many Agile teams.

Web application teams are the most susceptible to this. UI regression testing automation is something that very few companies do, even though visual bugs are incredibly common. So why does this happen and what can you do about it?

Why is automated regression testing so challenging in Agile?

The three cornerstone challenges of software testing in Agile are ROI, ROI, and — you guessed it — ROI. In the brave new world of agile product development, everyone frowns on large upfront investment of time and effort. The problem is you typically have to spend a whole lot of both time and effort to automate regression testing. More so for UI regression testing.

In addition to logic and user interactions, UI regression testing has to cover CSS bugs. Issues of this sort are hard to trace, and they require visual testing across screen sizes and browsers. Coincidentally, most UI testing frameworks lack a good default mechanism for visual testing. This brings us to another question: what can you do to fit automated regression testing into the framework of Agile?

What can you do to make automated regression testing of the UI economically feasible?

There are two strategies that agile teams use to automate regression testing on the UI level while keeping things cost-efficient.

1. Automation of happy paths with hand-coded tests. This strategy boils down to test case prioritization in regression testing. It dictates that testers only automate those parts of the UI which stakeholders deem crucial for functionality and/or conversions.
2. End-to-end regression testing automation with record-playback IDEs. In case with this approach, teams use automation tools to cover a wider scope of potential issues on the UI level.

Let’s take a closer look at how companies use these two strategies to automate regression testing of the UI.

Happy paths / test case prioritization with handwritten tests

When you code tests manually, you often end up with insufficient coverage. Real-life UI testing poses too much of a scope of work, and i simply takes too long to automate and maintain hundreds of coded tests.

So if covering the complete UI isn’t an option, focusing on the most critical aspect of the UI is a strategy that you can use. This strategy is as good as your test engineers and project managers are. Still, even with the best developers on-board, the more tests you write, the more overhead you get.

With happy paths, solid prioritization skills are paramount: stakeholders will need to agree on what parts of the UI require automated regression testing the most. This is something that you must do in the early stages of project development. Besides, your team has to decide whether to include tests into the DONE criteria, when to write tests, and how often to revise them. Answers to all of these questions will differ across teams and projects.

If you are working in a TDD environment, your automated regression testing suites can reuse some of the tests written by your UI developers. Testers can tackle visual bugs with screenshot comparison tools like Wraith or PhantomCSS. This will provide you with basic visual testing functionality — alas, at the cost of an increased maintenance burden.

Codeless automation of UI regression testing

The caveat with test prioritization is that it leaves parts of the UI compromised. In theory, you can fill the gaps with manual regression testing, but few teams have enough time to do this on a regular basis. Besides, there’s a wholly different option. If hand-coded tests can’t cover the whole UI, why not use a low-code or a codeless framework?

The codeless approach to automated regression testing is, essentially, as good as your record-playback IDE is. Automating regression testing with a record-playback IDE is something a manual tester without technical skills can do, and it excludes programming-related expenses from the equation. Besides, the creation of codeless tests takes much less time, making project management simpler.

Finding the right record-playback IDE isn’t always easy, though. In the worst-case scenario, your tool will only support editing tests via auto-generated code, which is a one-way ticket to maintenance hell. Luckily, more modern solutions allow for 100% codeless test editing.

There’s a whole lot of other features — aside from codeless test editing — that record-playback tools offer to streamline automated regression testing. Let’s take Screenster as an example of a modern solution for automated regression testing of the UI. We’ll look at the features that we think you should expect from every solution for UI regression testing automation.

Automatic verification of every UI element

Verifying everything on the web page has obvious advantages over handpicking the important elements. Full verification involves a lot of intricate manipulations with the WebDriver, but in case with Screenster, they happen under the hood. Instead of hundreds of lines of code, you get intuitive visual baselines and dashboards.

The idea of verifying everything is important because of the scope of UI testing and the unpredictability of CSS bugs. Where Selenium tests would leave most of the UI untested, leaking visual bugs, Screenster detects every single issue in the content that matters.

Self-healing locators for simpler maintenance

Broken locators constitute one of the top-3 reasons why regression tests break as your project grows. To make automated regression testing a simpler matter, Screenster employs a smart failover mechanism when dealing with locators.

When recording a UI test, the platform captures and stores all possible locators for each element. Whenever a locator gets broken by a UI change, Screenster automatically substitutes broken locators with the ones that remain intact.

Automatic timeouts

Selenium Waits are infamously hard to deal with en masse. Given that UI regression testing typically constitutes of hundreds of test suits, automatic handling of waits seems like a good idea. This is exactly what Screenster offers. When automation regression testing for web pages, the platform calculates optimal waiting time on a per-element basis.

Intelligent handling of dynamic UI regions

Some code-based screenshot comparison tools have a basic functionality for dealing with dynamic region like timestamps, userpics, or banner ads. When working with Wraith or Gemini, you can manually “blackout” particular regions by specifying its coordinates. Screenster can do the same, but with less input.

Instead of making you tinker with on-screen coordinates, the platform automatically detects dynamic UI regions during the optimization of a recorded test. During the further stages of automated regression testing, you can also create new ignored regions by clicking on an element in a UI screenshot.

Collaboration functionality

Being a team effort, automated regression testing requires collaboration-friendly tools. In case with a UI regression testing platform, this would typically imply a shared portal where teams can access and edit tests. This is exactly what Screenster offers, complete with color coded project and test health reports.

Multiple testing modes

Given how different QA automation teams can be in terms of size, skillset, and process coordination, flexibility is always great. As far as flexibility is concerned, Screenster can run the same tests in three modes allowing you to prioritize thoroughness or speed of test execution.

• The most thorough testing algorithm is called visual verification. In this mode, Screenster executes pixel-perfect visual comparison for every screenshot. Being the most precise out of the three options, visual verification is the recommended mode for pre-release runs of automated regression testing.
• In the content verification mode, the platform places a greater emphasis on content updates while decreasing the precision of visual testing. Content verification runs faster than visual verification, which makes it more suitable for everyday test runs.
• Minimal verification is a mode where Screenster focuses on user interactions with the UI. While doing so, the platform drops visual comparison for all test steps except for the first and the last one. The fastest out of the three modes, minimal verification is optimal for quick smoke testing sessions.

When it comes to Agile, simplicity often translates into greater ROI. With Screenster, UI testing becomes simple and cost efficient enough for teams to actually perform it. But does this mean that this solution is going to work for your project? There’s a great way to find out! Try our demo and automate a UI test for your website or web app.