UI testing automation tools and frameworks
March 7, 2018
Table of Contents
When it comes to UI testing automation tools, there’s no shortage of options to choose from. Test automation frameworks, desktop applications, cloud-based platforms — the range of available solutions is broad, to say the least. But which one is right for your application?
The testers and developers in our team have collectively gone through dozens of automation testing tools and frameworks. We played around with Selenium before Selenium WebDriver, and we automated testing with more GUI frameworks or tools than most people can name. It’s this experience that has become the foundation of our own UI automation tool Screenster. The collective expertise of our team is also the main driving force behind this software testing tools list.
Long story short, here are the best UI testing automation tools of 2017–2018. You can click on the name of a testing solution in the list to view its detailed description or just scroll down and read further.
The best UI testing automation tools and frameworks of 2018
- Unit testing tools for the UI:
- End-to-end testing frameworks:
- Automation testing for BDD with ‘Living documentations’:
- Desktop applications:
- Cloud platforms for UI regression testing:
Mocha
Mocha is a simple yet feature-rich automation testing framework for behaviour-driven development with JavaScript. It automates the asynchronous testing of Node and browser code, has a clear syntax, and supports all major assertions libraries. Most teams use Mocha with Chai for assertions and Sinon.js for test doubles.
| Pros | Cons |
| More flexibility compared to similar testing tools | JavaScript fatigue due to the need to choose from external libraries |
| Fast, simple, developer-centric, and open-source | Mocha is in “dire need” of active contributors |
Jasmine (version 2.x)
Just like Mocha, Jasmine covers the automation testing of Node and browser applications. The two frameworks also have a similar syntax and are identical in terms of how they approach asynchronous tests.
There are differences too, though. Unlike Mocha, Jasmine comes with built-in expect-style assertions, spies for test doubles, and a command-line utility available out of the box.
| Pros | Cons |
| Almost all core functionality available out of the box. | No Sinon.js means no fake server for simple automated testing of AJAX requests. (You can still use Sinon.js with Jasmine, though.) |
Karma
Karma is a neat test runner that enables the browser to load and run test files. It generates reports, and it can automatically run regression testing whenever you change the code of your application. Karma works in tandem with Mocha or Jasmine, or other automation testing frameworks for unit and API-level testing.
| Pros | Cons |
| Leveraging the browser engine has obvious advantages for the automated testing of the DOM | Giving up the Node environment means no shell access |
Protractor
Authored by the team that built AngularJS, Protractor is the go-to end-to-end testing framework for Angular application testing. This automation framework ships with a test runner, which is always nice. In addition, Protractor’s accessors are almost awesome.
| Pros | Cons |
| Solid mechanisms for handling locators and waits | With no clever way of handling visual regressions, it’s not really end-to-end testing |
Selenium
Selenium is an automation testing framework familiar to every QA engineer. It’s been around since the last Ice Age, and Selenium WebDriver is the de-facto standard for browser automation. Even though Selenium has its shortcomings, it’s something that everyone knows how to use.
| Pros | Cons |
| Selenium is the Golden Standard of automated testing of the GUI. | Selenium locators are brittle, which makes Selenium tests brittle too. |
| Other automation testing tools and services use and/or support Selenium code. | No straightforward way to set up visual testing (all you get is simplistic screenshot comparison via third-party tools). |
Cucumber
Cucumber gains a competitive advantage over other testing tools and frameworks by merging tests and spects. The business-facing syntax of Cucumber’s “living documentations” offers a new take on automated testing for BDD. Thanks to this fact, Cucumber has gained quite a traction in the QA automation community, with a whole class of test engineers specializing in it.
| Pros | Cons |
| The framework is free and open-source. | It takes a specialized training to learn Cucumber. |
| There are implementations for JavaScript and Selenium. | It takes a lot of developers’ time and effort to set up automation testing with Cucumber. |
Katalon Studio
Katalon studio combines the feature sets of three different categories of automation testing tools. The platform offers record-playback with action and object recognition, as well as codeless manual test creation and scripting with Java and Groovy. Katalon Studio is free, with its revenue model built around paid customer support from a software testing vendor called KMS technology.
| Pros | Cons |
| The tool ships as freeware | For anything but really small and simple projects, test maintenance requires paid technical support |
| Decent documentation and active forum | Object reidentification could be more reliable |
| Codeless automated testing has a smooth learning curve |
UFT (with UFT pro)
The first commercial release of UFT (then known as QTP) was in 2001, which makes it older than Selenium, Firefox, and Facebook. Naturally, the testing tool has evolved a great deal since then. It also has a long history as a product of Hewlett-Packard, which adds to the positive reputation.
Feature-wise, we’re talking about a UI testing automation tool that caters to enterprises with all-in-one solutions. UFT covers automated testing on the API and GUI level, and it offers both scripted UI tests and record-playback.
| Pros | Cons |
| The tool has been around for 17 years | Record-playback only available via integration with Sprinter |
| Relatively small installation footprint compared to similar automation testing tools | Support for JavaScript and Python for test scripting is only available with LeanFT |
| Too expensive for a small team |
TestComplete
Another old-school solution on this list, TestComplete enjoys a positive reputation among enterprise teams. It boasts a better support for coded automation testing than UFT, with JavaScript and Python available along with other scripting languages.
In addition, TestComplete makes for a solid UI regression testing tool thanks to it’s GUI object recognition. Most of the locators generated by TestComplete for on-page objects continue to work after UI changes. Finally, TestComplete is a little less expensive in the long term if you compare it’s pricing to that of UFT.
| Pros | Cons |
| Supports JavaScript, Python, VBScript, JScript, DelphiScript, C++Script, and C#Script. | Larger installation/setup footprint compared to UFT |
| Flexible pricing model with granular control over what you’re paying for. | Still too expensive for smaller teams and startups. |
Applitools Eyes
Applitools Eyes is more programmer-centric than similar UI testing tools. It offers an API that adds cloud-based image comparison to your existing test automation infrastructure. A neat part of Applitools’ regression testing functionality is a visual dashboard that highlights the differences between old and new UI screenshots.
| Pros | Cons |
| Great support for automation testing tools and frameworks like Selenium, JUnit, etc. | Applitools is only accessible to test automation engineers with programming skills. |
| Applitools works for both web and mobile application testing. | Minor inconsistencies with Selenium (e.g. Viewport-based vs. Window-based screen size). |
| Great support for CI/CD and DevOps. | Applitools’ pricing policy could be more transparent. |
UseTrace
Like Applitools, UseTrace caters to professional testers, yet it doesn’t require programming skills. Instead of making testers write code, this platform offers UI-centric automation testing functionality in a form of a dashboard with selectable commands and test steps.
Working UseTrace still requires good understanding of software testing best practices and basic knowledge of Chrome developer tools. This said, UseTrace manages to keep things relatively straightforward and simple, as far as learning curve is concerned.
| Pros | Cons |
| Speedy, codeless test creation and maintenance. | The verification mechanism feels limiting (no verification for navigation). |
| Simple parameterization. | No automatic locators or automatic recognition of on-page elements. |
| Automatic waits. | No visual verification for the UI. |
| Notifications via JIRA, Slack, HipChat, Flowdock, and email. | No handling of UI areas with dynamic content. |
| CI support. | No support for Selenium. |
Ghost Inspector
Simplicity is what helps Ghost Inspector stand out from the competition. In a nutshell, we’re talking about a plugin for Chrome that enables basic regression testing for the GUI. The core functionality of Ghost Inspector revolves around record-playback with added functionality like test scheduling and screenshot comparison.
| Pros | Cons |
| Ghost Inspector is faster than most tools of the same type. | The automation testing functionality of Ghost Inspector relies on manual verifications. |
| The functionality of Ghost Inspector is fully accessible to anyone. | Maximum duration of a test session is 10 minutes. |
| Great reporting with videos, console logs, screenshots, and test steps. | No automatic selectors. |
| No automatic verification of navigation. |
Screenster
Screenster is a visual regression testing platform that my team is building, so consider this a disclaimer. The tool uses record-playback-verification to enable low-code automation of web application testing. The main focus with Screenster is the use of automation to increase the ROI of UI regression testing.
In particular, Screenster captures multiple selectors for each on-page element and uses them for self-healing locators that continue to work after the UI changes. The tool automatically verifies on-page content and uses a smart visual testing algorithm that minimizes false positives while keeping UI comparisons precise. Other features of the platform include automatic timeouts and detection of dynamic UI areas, as well as CI integration and collaboration features.
| Pros | Cons |
| It takes 15–20 minutes to master the core functionality of the tool. | Regression test runs are slower than in more simplistic solutions (e.g. Selenium IDE or Ghost Inspector). |
| No coding required (but you can use coded tests if necessary). | The design of Screenster is a work in progress. |
| Visual verification of the UI. | Limited reporting functionality (via email only). |
| Automatic self-healing locators for all UI elements. | |
| Automatic handling of waits and dynamic areas. | |
| Automatic verification of content. | |
| CI integration. |
Bottom line: to code or not to code?
The range of automation testing solutions for the GUI is huge, to say the least. In testing automation tools and frameworks only make for the top of the iceberg, with thing like testing clouds and crowdsourcing services also available.
This said, the primary question that many teams face is whether to choose a framework or a dedicated tool. While the latter offer a more developer-centric approach, dedicated tools promise better opportunities for increasing the efficiency of testing.
So which one do you choose? If smart, codeless record-playback with automatic verification and self-healing locators sounds interesting, try our demo and see what Screenster can offer for your web application.
I do really love how Screenster does its job!