Regression testing 101: everything you need to know plus a little moreMay 3, 2017
If you’ve been following this blog, you know that our team is pretty serious about regression testing. Visual regressions have been the arch-enemies of our product AjaxSwing, and we’ve even created our own automated regression testing tool to overcome them.
Today, we’d like to talk about regression testing, it’s key concepts and main challenges, and tools that help you overcome these challenges. Let’s start with addressing some of the basics.
What is regression testing?
Regression testing is a QA process that tells you if a previously written and tested program continues to work correctly after you’ve introduced some changes.
In other words, it helps you notice if you’ve unknowingly broken something in your code by an update or a fix of some sort. New bugs introduced with these updates and fixes are called regressions, hence the name of this type of software testing.
How to do regression testing?
Okay, so how exactly do you test for regressions? In a nutshell, you do it by continuously re-executing the same tests to ensure that system continues to function as expected. Once a test that passed during previous runs fires a failure, you know that you’re dealing with a regression.
As a rule, it’s a good practice to run regression testing before the release of every new version of your product. A typical regression testing routine incorporates the following stages:
- selecting which portions of the system to retest
- execution of test cases (manually or automated).
Another essential (and arguable) thing to note is that regression testing applies to any testing level, including unit, API/integration, system and acceptance testing. There is a widespread convention, however, that regression testing is mostly a part of system testing.
Extent of regression testing
The extent of a regression test can either cover the complete functionality of your software or focus on some of its key aspects. Aside from functionality, regression testing can deal with non-functional aspects of your project. Specifically, visual regression testing of a user interface often tackles presentation issues that many people would characterize as aesthetic.
Regression testing techniques
It is often impractical to strive to cover each and every aspect of your application with regression tests. Due to this fact, multiple regression testing techniques exist.
Full Regression Testing (Retest All)
When choosing this technique, testers strive to achieve a maximum coverage of all possible issues during every test run. This approach boils down to running all tests after every modification introduced to the project.
While it takes more time and effort, the retest-all technique is optimal when regressions are hard to predict and localize. A typical example is CSS regression testing where visual bugs may occur on a different web page or a in different UI state.
One could argue that since it is virtually impossible to predict which parts of the system are affected by the change. Consequently, the only way to guarantee that system functions as expected is to run full regression testing.
Regression Test Selection
The Retest All technique is reliable, but it often proves too time-consuming and costly. For this reason, it’s often wiser to organize your tests into suites that correspond to modules of your program. During each regression testing session, your QAs will focus on retesting only the modules that are known to be affected by the change.
In the case of regression test selection, each run focuses on the functionality that’s most crucial and/or relevant for a particular release. When following this approach, it’s vital to make sure that it provides you with a significant advantage over Retest All in terms of ROI.
Test Case Prioritization
Prioritizing the most essential tests is another way to optimize the testing process. When it comes to Test Case Prioritization, your QA team still runs all tests, but it decides on which ones to run first.
There are two widely-accepted approaches to Test Case Prioritization:
- General prioritization. The cases are selected for regression testing based on their relevance to all subsequent versions.
- Version-specific prioritization. Cases are chosen based on their importance for a specific version or update.
While Retest All and Test Case Prioritization involve roughly the same amount of resources, the latter enables you to ship important updates and fixes faster.
This term encompasses a wide spectrum of tactics that border on Retest All, Regression Test Selection and Test Case Prioritization. For instance, teams may opt for Regression Test Selection in case with minor releases yet choose Test Case Prioritization or Retest All for the major ones.
Regression testing best practices
Alright, we’ve found out what regression testing is and how it’s carried out. Let’s proceed to some of the regression testing best practices. Let’s look at a couple of recommendations that can make the life of a QA engineer simpler.
Or, to be more precise, automate everything you can. It’s next impossible to eliminate the need for manual testing altogether, but you can still keep it to a reasonable minimum. Basically, it’s paramount that you develop strategy for determining which tests to automate and which tests to execute manually.
The most common areas that require automation in 99.99% of cases include:
- Running the same regression test suite on different platforms (i.e. different operating systems, browsers, or devices).
- Running regression testing after a bug fix or a minor update.
- Data-driven testing (i.e. testing how your applications handles different input data).
Build customer personas and define your happy paths
Happy paths are clearly-defined test cases associated with widespread UX scenarios. The examples of these scenarios include signing in, adding items to a shopping card, using website navigation, etc.
In most projects that use code-based regression testing tools, the majority of automated tests revolve around happy paths (while edge cases become tasks for manual testers). Understanding of these happy paths stems from understanding of who your users are. This, in turn, involves creating well-defined customer personas.
Keep your regression test suites up to date
As your project grows, some of the added functionality will inevitably start lacking sufficient test coverage. Some of the tests may become obsolete or unnecessary. For these reasons, your QA team will need to occasionally revise your regression test suites.
Regression testing tools and software
We’ve covered some of the most widely used regression testing tools in our earlier posts on Selenium competitors and Selenium IDE alternatives. Still, it might prove handy to have a quick look at the most popular options:
- Code-based software testing frameworks. When it comes to web projects, Selenium seems like the undisputable leader in this department.
- BDD testing frameworks like Cucumber. Tools of this type use plain language parsers that enable QA engineers to write tests in a language that feels almost like human English.
- Enterprise record-playback IDEs like UFT, Progress Test Studio (Telerik), etc. Instead of (or in addition to) relying on hand-coded tests, IDEs record sequences of interactions with the software that’s being tested. They rerun these recorded interactions and use image comparison to detect regressions.
- Codeless cloud-based platforms like Screenster or Ghost Inspector offer a lightweight and easier-to-use alternative to enterprise IDEs. While using the same record-playback principle, cloud-based platforms typically offer a number of advanced features, like DOM comparison, automatic handling of timeouts and dynamic regions, collaboration functionality, etc.
Challenges of regression testing
Every process or methodology has its pitfalls, and regression testing is no exception. Namely, the advent of Agile has brought new challenges that testers have to overcome:
- Minimal documentation with changing project scope and frequent iterations. The pace at which Agile projects move leaves little time for both manual testing and the development of a comprehensive suite of hand coded tests. As a consequence, it is becoming increasingly more challenging to ensure efficient test coverage of a project.
- Choice of a regression testing tool and methodology. Manual regression testing is faster to set up in the short term, yet massively inefficient in the long run. Hand-coded tests, on the other hand, take more time to develop and are harder to maintain. With the right IDE, test automation takes less time and effort, yet IDEs are hardly compatible with TDD and the test-first approach.
- UI testing. The increasing focus on the UI/UX aspect of a software product makes UI testing more important than ever. Sadly, traditional code-based tools fail to provide a comprehensive test coverage of a UI.
Bottom line: the future of regression testing
The constant increase of speed at which software products are being built determines some of the key trends in regression testing. Today, more needs to be done in less time. Consequently, QA teams need to be on the constant lookout for more efficient regression testing methodologies — and more efficient regression testing tools.
As codeless IDEs and cloud-based platforms become better, their role in regression testing of web apps grows more prominent. When compared to code-based solutions, cloud-based IDEs offer significant advantages in terms of learning curve, test maintenance, collaboration functionality, and the ROI of regression testing.
But does this mean that an codeless IDE is optimal for every project — and, more specifically, the project you’re working on? The only way to find out is to try it for yourself. Check out the free online demo of Screenster below and tell us what you think.
Want to try Screenster on the cloud?Try Online