Manual vs automation testing of the UI

Image credit: Freepik

Speaking of manual vs automation testing, which one do you prioritize on the GUI level? Most teams that I’ve seen want automation but start with manual testing in the early stages of product development. Later on, they add the automation of UI/UX happy paths with Selenium, Protractor, or other frameworks, gradually reducing the number of manual tests.

But is this approach optimal for modern-day UI testing? We’re basically talking about a tactic from five years ago, when “thin UI” was still a thing. When you do want to automate UI tests, you should pick a tool that’s quick and painless giving you low maintenance tests. And that’s not Selenium or Protractor.

Moreover, this approach doesn’t really account for the strengths and limitations of manual and automation testing. So what are these strengths and limitations?

Let’s take a closer look at the key advantages of automation testing over manual testing and vice versa. Check out the matrix below to see how the two compare on a high level or scroll down for a more detailed overview.

Automation testing vs manual testing comparison matrix

Why manual UI testing matters? Strengths of manual testing

Despite the industry-wide tendency to “automate everything”, reasons for why manual testing is important still exist. In certain cases, automation doesn’t produce a high enough ROI to justify it, so it’s okay to stick with manual. Let’s have a closer look at the advantages of manual testing over automation.

• Greater flexibility and adaptability. Manual tests are easier to adjust to changing requirements, which is a boon to UI development. Early-stage requirements for the GUI are exceptionally change-prone, more so in Lean and Agile. With this in mind, it generally makes little sense to write automated tests in the early stages of UI development.
• Humans look outside of the scope of test. Handwritten automation tests ignore everything outside of their narrow scope — they only deal with issues that are predictable and expected. When running a manual test, human testers will often notice whatever else might wrong with the UI.
• Testing of real-life user actions. There are layout bugs and design issues that you can only detect during real-life interaction with the UI. Manual testing comes down to reproducing real user interactions with an application, which makes it instrumental in catching these bugs.
• Lower initial investment. In the short term, manual UI testing is nowhere near as costly as automation. Manual testing doesn’t require professional programmers, there’s little setup involved, and you don’t need expensive tools to get things running.

Weak points of manual testing

The very idea of relying on human ability to perform tedious routine work has its limitations. Pushing manual testing beyond its limits will render it unscalable, inaccurate, and pointless for certain kinds of UI testing.

• Scalability issues. Running more manual tests is only possible via adding more human-hours, which implies hiring new people. Even though manual testers are less expensive than automation engineers, most software products reach a point when manual testing becomes economically unfeasible.
• Speed. The running time of manual tests is directly proportional to the size of your UI testing team and the skills of your testers. That said, it’s always slower than automation testing. Again, reaching a breaking point when you don’t have enough people to run your UI tests is inevitable with manual testing.
• Change blindness and human error. Humans are innately change-blind, which impairs our ability to notice minor visual differences. Considering that UI testing often comes down to detecting visual changes, automated image comparison might be more reliable then a human eyeballing the UI.
• Manual regression testing is pointless. Running the same tests manually for hundreds of times is wasteful. This is especially relevant for the GUI where regressions are ubiquitous. Once again, automation is a must for regression testing.

When to use manual UI testing?

Involvement of human testers has its benefits, but these benefits only matter in a limited set of scenarios. Manual UI testing is useful for those cases when automation is impossible or uncalled for, and when human involvement is necessary.

• Early-stage testing. It generally makes sense to start with manual testing once a part of the UI or feature is ready. At this early stage, there’s a high probability of defects that go beyond the immediate scope of the test. Detecting bugs of this sort is impossible without human observation.
• Unstable UI. Automation only makes sense when you’re certain you will be rerunning the test. If the UI or feature is unstable and you know the requirements will change a lot, it might be too early for automation testing.
• Ad-hoc testing. Quite obviously, there’s little use for automation in cases involving quick at-the-spot checkups.
• Exploratory testing. Human experience and creativity are indispensable to exploration, which makes a solid case for manual testing. Although very few teams do exploratory testing, it’s a good practice that I wish more teams followed.
• Usability testing. To evaluate and test usability, you need to zoom in on design matters, which hardly possible without a human tester. Still, automation of usability testing is possible with the right tool.

Benefits of automation testing

I’m sure you’ve noticed that the range of applications for manual testing is indeed quite narrow. In GUI testing, automation is welcome far more often, and there are several reasons for this state of things. Here are the most essential benefits characteristic of all types of automation testing.

• Long-term cost-efficiency. Even though UI tests are more fragile than unit and API tests, automation testing of the GUI still saves you money. The cost of running a suite of automation tests is negligible compared to the expenses associated with manual testing.
• Speed. It goes without saying that automation is a faster option, especially if you’re running UI tests on multiple machines or a dedicated cloud service. Besides, you can save additional time by using record-playback instead of hand-coded tests.
• Accuracy. When done right, automation testing excludes human error allowing for things like pixel-perfect visual testing.
• Reusability. Building reusable test code and test steps takes a skilled engineer. Nonetheless, the very concept of reusable UI tests makes automation superior to manual testing.
• Transparency. With automatic reporting, the results of running automated tests are immediately accessible to anyone in the team. If you’re dealing with manual tests, creating reports will also take additional time.

Disadvantages of automation testing

Automation offers indisputable advantages for GUI testing. Still, there’s also a number of essential problems that come with UI testing frameworks and old-school platforms like UFT, Sahi, or TestComplete.

• Large upfront expenses. Whether you’re hand-coding your UI tests or working with a desktop record-playback tool, automation requires considerable upfront investment. Moreover, automation testing has hidden costs associated with infrastructure setup and maintenance.
• Test maintenance can get tedious. Having to rewrite brittle UI tests to accommodate to every minor change is plain painful. It’s this factor that motivates testers to delay automation and only cover the most stable parts of the GUI with automation testing.
• Narrow scope. A typical hand-coded UI test only deals with UX touch points without actually verifying the UI. Unless there’s screenshot testing in place, hand-coded tests leak visual bugs which is never a good thing.
• Steep learning curve. The UI automation testing ecosystem has very few solutions that prioritize simplicity. Webdriver-based frameworks are only accessible to programmers, and most enterprise tools take weeks to learn.

When to use automation testing?

Automation powers some of the most essential processes in GUI testing. These include regression testing, screenshot comparison, and end-to-end testing.

• UI regression testing. By definition, automation testing is a go-to tactic for dealing with frequent changes introduced to a somewhat stable UI.
• UI screenshot comparison. Automated screenshot comparison increases the ROI and maximizes the precision of visual testing. With this in mind, it hardly makes any practical sense to task manual testers with catching visual bugs.
• End-to-end testing. The automation testing of complex user stories is possible with frawerks like Selenium and Protractor, yet dedicated tools are far more productive in this department. Modern record-playback features like object recognition with self-healing locators make tests less brittle, eliminating one of the major pain points of automated end-to-end testing.

Whether it’s manual testing or automation, UI testing is challenging

Knowing that both manual testing and automation solutions have major pitfalls, no wonder that UI testing is such a complicated matter. Returning to the UI testing scenario mentioned in the beginning of this article, manual testing quickly becomes too much of a liability. In fact, few teams can run enough manual tests to ensure software quality on the UI level. This is exactly why automation is an inevitable step.

On the other hand, the safety net offered by hand-coded tests has too small of a surface area. Most screenshot comparison modules and record-playback tools offer little value and are painful to deal with, which is why noone uses them. Problem is without visual testing, handwritten tests ignore most of the UI. Besides, the maintenance of automation testing suites tends to get burdensome.

Codeless automation with modern visual testing tools

The coverage of modern-day UI testing has a lot of blank spots. Overcoming this issue takes a better tool which would combine the advantages of automation testing and manual testing while leaving out the shortcomings. And it seems like tools of this sort are starting to appear.

The last few years saw the rise of several cloud-based platforms that seem to get record-playback right. These tools promise a simpler approach to UI automation testing with robust, manageable tests and a strong focus on intelligent visual testing. Here’s how these features are realised in our own solution Screenster:

• Low-code automation for higher ROI. Using record-playback as its core functionality, the platform allows for faster test creation with no need in programming. At the same time, Screenster supports WebDriver code for advanced functionality.
• Advanced functionality for simpler maintenance:
  • Intelligent record-playback. While most old-school record-playback solutions store tests as auto-generated code, Screenster offers fully codeless test editing and maintenance.
  • Self-healing locators. When automating a test, Screenster builds a DOM model of the UI in addition to capturing screenshots and user actions. The tool automatically captures multiple selectors for each UI element and uses them to restore broken locators.
  • Automatic handling of timeouts and dynamic UI regions. The platform automatically manages complex UI testing scenarios that involve asynchronous events and dynamic content.
• Smart visual testing. When automatically the screenshots of UI versions, Screenster reduces false positives by filtering out differences caused by rendering. At the same time, it uses pixel-perfect image comparison to detect layout bugs that a human tester is likely to overlook.
• Low entry threshold and a smooth learning curve. It takes a first time Screenster user less than 5 minutes to record a real-life UI test. Screenster is fully accessible to manual testers, and it can be used by non-testers as well.
• Multiple comparison modes. When rerunning a test, you can opt for quick content comparison or use thorough visual testing for increased precision.
• Collaboration functionality. Screenster pack powerful UI testing functionality in a form of a lightweight web application. Being browser based, Screenster allows for shared access to tests, and it facilitates collaboration with automatic reporting.

Here’s a quick demo of some of these features. In the video below, you can see how Screenster works for a real-life web application that millions of people use daily:

If intelligent low-code automation with 10x the ROI of Selenium tests sounds interesting, you can try Screenster online. Check out our demo and see how a more modern approach to Ui testing will work for your web application.