Why is manual regression testing still happening in 2017?

Where does manual regression testing fit in the software testing process? In fact, does it even make sense to test software manually, given the sheer number of automated regression testing tools available?

I bet you’ve seen people ask similar questions on StackOverflow — as well as a good dozen of articles exploring the topic. So why would you need another post on the same topic?

Well, here’s a reason to put my two cents into this same ole discussion.

The choice between automation and manual regression testing looks simple on paper. Most manual regression testing tutorials will tell you that the complexity of manual testing grows exponentially as your project scales. Automated tests, on the other hand, help you move faster than manual tests, they reduce the chance of human error, and they are more cost-efficient in the long run.

So, on paper, you should always strive to automate everything you can, right? After all, you can always have your developers write unit and API tests, and hire a QA team for everything else…

In theory, yes, that’s how things are supposed to work. In real life, however, things get more complicated. The more so if we are talking about small upstart projects built on a budget. Even more so if we’re talking about UI testing. Let’s look at a couple of situations where you’d find yourself adding manual work to regression testing.

Small teams that lack dedicated QA automation engineers

You probably know that Amazon, Apple and Google started as small garage projects, don’t you?

Sure, times were different back then, but even in the brave new world of today, tech companies often start lean, with 2–4 developers per tech stack. When it comes to small production teams of this kind, part of the regression testing often falls on the shoulders of less-skilled team members. Besides, quite a few companies will also task their non-technical staff (like project managers or business people) with doing regression tests on the UI level.

It’s difficult to exclude manual testing from this scenario. The problem is, each iteration of manual regression testing takes a ton of time, especially when it comes to doing UI testing for an emerging project. As a consequence, not having a dedicated tester in charge of doing these iterations implies wasting precious time, and it puts you at risk of making your regression tests superficial.

Exploratory and ad-hoc testing

Small teams with limited production budgets aren’t the only case that requires manual testing. Expanding automated regression testing with occasional manual tests is a practice that often proves beneficial for full-manned teams as well.

In particular, some teams claim they occasionally add sessions of manual exploratory UI testing to their regression testing cycles.

Whenever possible or economically feasible, these sessions involve non-QA staff, including programmers, designers, project managers, and even marketers. This approach might seem unorthodox, but it facilitates the discovery of fringe cases and usability problems.

It’s easy to see how occasional ad-hoc testing can benefit the process in a similar way. While different from regression testing in nature, exploratory and ad hoc testing help you detect critical bugs and expand your regression testing suite with valuable tests.

There is a problem with ad-hoc tests, though. By definition, these tests don’t follow a formal process, which makes them difficult to replicate. Besides, adding new cases to the regression suite mid-process means spending more time, which is why this practice isn’t popular.

Visual regression testing

Visual regression testing is a major pain point for teams that rely on traditional methods and code-based automation tools.

Let’s look at a case from Eric Ries’s book “The Lean Startup”. Say, one of your developers accidentally recolors of one of your buttons. This change will be obvious to anyone looking at the UI, but autotests targeting the buttons by their HTML tags, IDs, class, or position in the DOM will ‘overlook’ the bug.

Cases of this sort are frequent. An update to your code base can accidentally rewrite a LESS or Sass variable leading to a change of color, margins, or element positioning. An accidental change of z-index can render a pop-up invisible… To cut the long story short, UI will always remain one of the most unstable parts of your projects.

Sure, there is a huge variety of automation tools that handle bugs of this sort. However, far too many QA engineers will stick to the ones built for Selenium and PhantomJS because that’s what they use for low-level tests.
The bad news is WebdriverCSS and PhantomCSS are hardly an ideal choice — these two frameworks are just too complicated in terms of setup and test maintenance. Besides, they won’t offer much more than basic screenshot comparison, which is the exact reason many companies fall back on manual regression testing after experimenting with these two.

So what’s the right mix between manual testing and automation tests in 2017?

The three scenarios mentioned above have something in common. In case with each of them, manual testing isn’t really the best option — it just proves to be more practical than the traditional code-based approach to test automation. Yet, automation doesn’t have to imply hand-coding your tests.
If you have any experience managing regression testing projects, you can probably see that a record-playback solution would actually work better in each of the above cases.

A good tool of this kind could help you automate manual regression tests. With a solid set of test-editing features, a record-playback tool could also help you regularly expand your regression suites with cases discovered with exploratory and ad-hoc testing. Finally, with a good record-based testing tool, running and maintaining visual regression testing would be a breeze.
If only record-based solutions for visual regression testing weren’t so cumbersome…

As a matter of fact, this is the exact reason our team started working on Screenster, our very own platform for visual regression testing. This solution builds on record-playback principle used by tools like Ranorex or Sachi, but it adds a slew of advanced features into the mix. It allows you to run tests on the cloud, and it provides users with ample test editing features while handling things like dynamic regions, timeouts and locators.

When planning the initial roadmap of the project, we made a point of overcoming every shortcoming of old-school record playback solutions. And we’ve made sure our users won’t need to waste precious time reading manuals or tinkering with the installation package.

Bottom Line

A lot can be said about how we’ve implemented our vision of a record-playback solution that works, and a lot has already been said. However, the best way to see if Screenster can live up to your expectations is to try it yourself.

So is manual regression testing still a thing in 2017? As you can see, there still are quite a few cases that require actual interaction with the UI. Given this, my answer is ‘yes, but you can still automate it’.