What regression testing challenges exist on the UI level?

What regression testing challenges do you typically face when automating UI tests? And more importantly, how do you overcome these challenges? This post sums up our experience of dealing with UI regression testing and its pitfalls. But there’s one more question to ask before we can proceed: do you automate UI test in the first place?

I’m sure you know that quality control is inseparable from product development process. Some good amount of time and resources should be invested into it. New UI functionality must be covered with tests as soon as it is ready. And it is not enough to verify it once: each time an important change is introduced, the UI regression testing phase must follow.

Whereas unit- and API- level testing helps you catch the most critical bugs, UI regression testing mostly deals issues of a different nature. True, there’s plenty of logic underlying modern-day UIs. Still, there’s also a plethora of potential presentational (visual) and functional (UX-related) issues that people typically associate with GUI regression testing. And the problem with these issues? They are the first ones that users see.

Now, when it comes to catching UI bugs, a surprisingly large number of companies never move past manual regression testing. While not bad in itself, this approach has a major flaw. Manual UI regression testing tends to take more and more of the project time. An obvious solution for a big project is to automate this process. However, the UI regression testing automation, with all the benefits it brings, comes at a price.

Firstly, you need to invest into the automation in order to reap benefits later. And it could take a long time at the initial stage. The fact is, you need to spend resources to set up a framework such as Selenium, build tests (which should also be tested themselves). We’re talking about a lot of programmers’ time that could be spent on the development of features.

Secondly, the completed automated visual regression tests should be supported afterwards. Their coverage may become insufficient as the product grows and evolves. Sometimes tests should be amended or replaced with new ones. Some become outdated, so a cleanup is necessary from time to time.
Summing all that up, a proper UI regression testing automation requires a proper automation testing tool.

UI Regression Testing Automation Tools

As a rule, developer teams use multiple testing automation tools to cover the complete scope of potential issues posed by UI testing. These issues include visual HTML/CSS, UI logic in JavaScript, backend API and database state. Let’s take CSS regression testing as an example and explore the pitfalls of automating this specific segment.

One thing that makes CSS testing troublesome is the fact that the UI changes a lot. Changes can occur due to bugs or functionality updates. ‘Unexpected’ UI changes are typically considered bugs and need to be need to be dealt with. The problem is you have to be a very keen observer to ‘catch them all’. This is especially true for complex modern UIs powered by React, Angular, or Vue.

Naturally, each functional page built with HTML/CSS should be visually compared to the mockup, and functionally compared to the expected behavior. The task of manually comparing the UI shown on the screens to the expected state is not as simple as it may sound. Human eye might not catch some tiny details which, however, have much significance for the product. And every human is prone to testing fatigue which causes them not to notice even the most obvious problems.

Here is when the automation testing tool should jump in. However it brings some new pitfalls. Main challenges with UI regression testing automation are:

• Tests take too long to develop. Covering a single page often implies dozens, even hundreds of tests, which no one has time to write, ever. More often than not, automation engineers cover “happy paths” and conversion-related parts of the UI.
• Tests require development skills. This is really a pain point because developer’s time is expensive. While it’s a must for developers to write tests in TDD environments, very few startups have the budget to task developers with writing UI test.
• Existing record / playback tools produce bad code and are hard to use. Old-school tools like Ranorex and Sahi seemed like the silver bullet of UI testing automation. But in real life, the tests they produce are nothing more than autogenerated code that is really hard to maintain.
• Test maintenance is time consuming. Whether it’s hand-written or auto-generated, UI tests require maintenance because of how frequently the UI changes.

A simple and effective comparison of screenshots with easily renewable ‘basic’ design images would cover up the major part of your regression testing when it comes to UI. After that your QA team could concentrate on other important parts of the project scope: databases, load handling security etc.

UI Regression Testing on the cloud with Screenster

I have been automating my UI tests with Screenster for a while and I absolutely love it! Here is how it solves the typical CSS regression testing problems.

Problem. Tests take too long to develop.
Solution. Screenster is all web based with nothing to install locally. I can record and play tests back but since it’s all UI based I don’t have to deal with the code behind it. There is a simple interface for most common tasks and I can always add Selenium as part of my test if I really feel like it.

Problem. Existing record / playback tools produce bad code and are hard to use.
Solution. During recording, Screenster captures screenshots to record the baseline (‘expected’ images), then compares actual screenshots to it at the next run – no code writing involved!

Problem. Visual UI changes often and each time automated test must be amended.
Solution. Screenster allows to turn an actual screenshot into a ‘basic’ one in just two clicks thanks to its ‘Approve to Baseline’ feature.

Problem. Some elements are designed to change all the time. So an auto-test will always show a failure where everything is in fact OK.
Solution. All you need is to exclude those area from this test’s scope. With Screenster, you can create an ignore region for an area in the page you do not want to be analyzed (e.g. an ad). Moreover, Screenster automatically ignores dates if they constantly change after a right pattern (e.g. a current date & time element).

Problem. Buttons and other controls might change their id, properties or location in DOM causing an auto-test failing to find the right element to interact with.
Solution. Screenster has smart locator management mechanism that automatically finds renamed or moved elements. For ultimate control Custom CSS options lets you edit the CSS locator of an element, so that auto-test would perform fine after the element was moved.

You can download Screenster at our Portal. Feel free to leave a comment here about your experience with it!