9 automation testing tools to consider for web applications in 2018
June 25, 2018Table of Contents
The year 2017 saw an impressive number of important events as far as UI automation testing tools go.
In 2017, we witnessed the end of developer support for Selenium IDE and PhantomJS. We’ve also had a full year to evaluate how the 2016’s acquisition by Micro Focus has influenced UFT. Finally, the releases of Angular 5 and React Fiber have introduced new challenges for QAs, making GUI testing tools even more relevant.
Given all these news, what automation frameworks and tools will become the mainstream options for guarding the user interface against bugs in 2018? Read on for an overview of top web automation solutions across three categories:
UI automation testing frameworks | Desktop automation tools for enterprises | Cloud testing platforms |
---|---|---|
Selenium WebDriver | TestComplete | Screenster* |
Protractor | UFT Pro (LeanFT) | Ghost Inspector |
Cucumber | Eggplant Functional | Usetrace |
*Yep, thats us — and we’ve got awesome features to back up our claim for leadership.
UI automation frameworks. Selenium WebDriver, Protractor, and Cucumber
Code-based testing frameworks have their drawbacks, but they will persist driving UI automation in teams that have enough time to write UI tests. For these teams, Selenium, Protractor, and Cucumber will remain the preferred options 2018.
Selenium WebDriver: old but gold still got it
In 2017, people still consider it a must for every specialist involved in GUI testing to learn Selenium. And there’s a good reason to. Selenium WebDriver underlies the W3C WebDriver standard, it’s native to major browsers, and it’s used in all higher-level automated testing frameworks.
Being low-level, Selenium requires a solid grasp of good testing practices. The low-level nature of this tool also makes Selenium less productive than higher-level tools. Inspite of these two facts, Selenium has a reputation of the “golden standard” of automation testing of the GUI, which is why many teams still choose it over dedicated high-level frameworks. Specifically, many React teams prefer to test web applications with Selenium.
Protractor: test automation framework for Angular apps
While React doesn’t seem to have a UI testing champion, the situation is different in the Angular realm. When it comes to the automated testing of Angular-based web applications, Protractor enjoys the status of the go-to solution. Specifically, software testing engineers name several advantages of Protractor:
- Protractor is the brainchild of the people who wrote AngularJS.
- The framework has a clever way of handling Angular waits.
- Protractor’s accessors make for an effective Angular-specific locator mechanism.
- It’s always nice to work with a framework that ships with a test runner out of the box. In case with Protractor, the test runner is Jasmine, and there are many Jasmine fans out there.
Cucumber: BDD champion for UI
BDD frameworks stand apart from other test automation solutions. If you buy into the idea of “business-facing” Gherkin scripts, Cucumber is pretty much the only game in town. Capybara, the web-based automation framework for Cucumber, is also something you’d want to use for BDD-centric acceptance testing.
Desktop automation testing tools for enterprise: TestComplete, UFT Pro (LeanFT), and EggPlant
It’s not uncommon for large enterprises to gravitate towards all-in-one suites of automated testing tools that work for both web and mobile testing and have every feature one can imagine. The all-in-one nature of these products also manifests itself in how they automate tests.
IBM Rational Functional Tester, for instance, combines coded tests with screenshot comparison and “storyboard testing” which uses natural language for test editing. In the meantime, Telerik Test Studio emphasizes the comprehensive coverage of GUI, API, Load, and Performance testing.
While both of these two GUI testing tools offer a slew of neat features, this review will focus on other solutions. Specifically, we will zoom in on the tools featured in the Gartner’s Quadrant for automated software testing. The table below offers a comparison of TestComplete, UFT Pro (LeanFT), and EggPlant Functional.
Automation testing tool | TestComplete | UFT Pro (LeanFT) | Eggplant Functional |
Developer | SmartBear | Micro Focus (the app was previously owned by Hewlett-Packard) | Testplant Inc. |
Position in the Gartner’s Quadrant for 2017 | Challengers | Leaders | Visionaries |
Runs on | Windows (desktop app) | Windows, Mac, Linux | Windows, Mac OS X, or Linux |
Web testing | Chrome, Firefox, Internet Explorer/Edge, Safari | Chrome, Firefox, Internet Explorer/Edge, Safari | Chrome, Firefox, Internet Explorer/Edge, Safari |
Mobile testing | iOS, Android | iOS, Android | iOS, Android |
Visual testing | Image comparison | No (image comparison and record-playback are available with a UFT license) | Object recognition, image search, image recognition, and text recognition |
Test scripting | JavaScript, JScript, Python, VBScript, DelphiScript, C#Script, C++Script | Java, C# | SenseTalk (proprietary), or Java, C#, and Ruby |
Selenium support | Yes, via Java’s JUnit | Yes, via Object Identification Center, Objects locators & utilities | No |
Supported IDE | Microsoft Visual Studio, Eclipse | Microsoft Visual Studio, Eclipse, IntelliJ | Microsoft Visual Studio, Eclipse |
CI support | Jenkins | Jenkins | Jenkins, Bamboo |
Known issues | Slow to support new IE versions, weak with Java applets | Large memory consumption footprint, extensive infrastructure | The software is still buggy (even though the TestPlant team fixes them fast). |
Pricing model | Node-Locked License tied to a machine Floating User License tied to a user account | Seat-based license / 3 months Custom pricing for enterprises | Pricing information only available via contacting TestPlant |
Free version | Free trial | Free trial | Free trial |
Cloud testing solutions for codeles and low-code automation: Screenster, Ghost Inspector, UseTrace
In the last couple of years, a slew of web services and platforms emerged to enable efficient low-code automation of UI and functional testing. Like Selenium-based frameworks, these testing tools drive the browser to emulate user interactions with websites and web apps. Unlike Selenium and similar solutions, these tools test the whole UI instead of targeting separate user journeys.
Even though people sometimes compare cloud testing platforms to Selenium IDE, their functionality goes beyond simplistic test recording and screenshot comparison.
For instance, modern cloud-based UI testing tools are able to recognize on-page objects and UI areas with dynamic content. They can automatically ignore visual differences caused by rendering and handle timeouts with little to no human input. Being web-based, these tools also feature collaboration functionality. Finally, they support coded test scripts and continuous integration.
The table below will compare the three tools that offer advanced functionality for powerful test automation.
Screenster | Usetrace | Ghost Inspector | |
Functionality | Functional and smoke testing (UI) | ||
Visual verification | Yes | No (Screenshot comparison only) | No (Screenshot comparison only) |
Content verification | Yes (automatic) | Yes (manual) | No |
Navigation verification | Yes | No | No |
Automatic locators | Yes | No | No |
Self-healing locators | Yes | No | No |
Automatic waits | Yes | Yes | Yes |
Management of dynamic UI areas | Yes | No | No |
Built-in Selenium commands for in-browser coding | Yes | No | Yes |
Importing / exporting tests | Importing Selenium tests | Exporting tests as JSON | Importing and exporting Selenium tests |
Parameterizable tests / Variables | Yes | Yes | Yes |
Support of different screen sizes | Yes | Yes | Yes |
Creating a basic test takes < 3 minutes | Yes | Yes | Yes |
Fast test editing | Yes | Yes | Yes |
Reusable tests | Yes | Yes | Yes |
Supported browsers | Chrome, Firefox, IE | Chrome, Firefox, IE | Chrome, Firefox |
Collaboration functionality | Shared access to dashboards and test editing | Shared access to dashboards, tagging tests | Shared access to dashboards and test editing |
Reports | Screenshots and test steps | Screenshots and test steps | Video, screenshots, test steps console information |
Notifications | Email, Flowdock, HipChat, JIRA, Slack | Email, Slack, Webhooks, HipChat, PagerDuty | |
Installation/setup footprint | Fully web-based | Fully web-based | Requires a dedicated Chrome plugin |
Test server | Cloud-based or local | Cloud-based; NGrok tunnels to local machines | Cloud-based; NGrok tunnels to local machines |
Concurrent test runs | Yes | Yes | Yes |
CI support | Jenkins, TeamCity, Travis, Bamboo | Jenkins, Bamboo, CircleCI, Codeship, Visual Studio Online | Jenkins, CircleCI |
REST API | Yes | Yes | Yes |
Automated functional testing of the GUI with Screenster
Screenster, Ghost Inspector, and UseTrace narrow down on test automation on the UI level. Instead of offering you the broad functional scope of enterprise platforms, these tools do one thing, but they do it really well. Here’s what our very own product Screenster can offer for powerful website and web application testing:
1. Pain-free test maintenance
Screenster enables you to record and execute automated tests without hand-coding. While this is the same sort of functionality that you get with some enterprise IDEs, there’s one major difference. The tests generated by the record-playback tools of enterprise IDEs typicaly come as auto-generated code which is often a pain to handle.
Instead of making you fumble through autogenerated test code, Screenster allows for quick and fully codeless test editing. Given how frequent the UI changes can be, you’ll appreciate how much sense this feature actually makes.
2. Self-healing tests with smart locators
When creating the baseline of the UI, Screenster automatically builds complex locators that include all HTML attributes and structure-based selectors for every element. Should a UI change of any kind break one of the selectors, Screenster will automatically substitute it with another one. This way, you can perform test automation without having to worry about UI updates.
3. Intelligent visual testing with verification of visuals, DOM, and content
Screenster automates UI testing by capturing the DOM structure and the content of the page, in addition to taking screenshots of every UI state. All of this information is stored as baselines of each UX step, and Screenster checks new UI versions against these baselines during every test run.
When it comes to visual testing,Screenster does more than mere screenshot comparison. The platform distinguishes between style changes and content updates, providing you with valuable information on the nature of every UI bug. When looking at screenshots, Screenster sorts out the differences associated with anti-aliasing. It also detects UI elements with dynamic content and provides you with a simple functionality for managing these regions.
There’s a lot that can be said about the advantages of low-code cloud-based solutions, but giving them a try is worth a thousand words. Check out our demo, and you will see if cloud-based solutions have what it takes to rank among the top automation testing tools of 2018.
Looks like a great list, thanks! I liked that you’re talking about different web testing tools, no just the usual roster of frameworks. There’s a typo, though: in the last table, you have a full stop in one of the th tags.
Ghost Inspector is okay, but I kinda wish it could do more… Let’s see if Screenster has all that you’re writing about
I worked with TestComplete and UFT (back when it was owned by HewlettPackard). The two are pretty much similar. Do the same things slightly differently, but you’re basically gate the same out of the them. I also have limited experience with Katalon. The observation I have with tools like these is that teams that move to them rarely go back to Selenium
That’s the whole point. actually! There’s nothing inherently bad about coding your tests — as long as you have the people and the budget to do this. But most small teams don’t really have that — and they either write very few tests or run very few manual tests. Codeless IDEs could’ve done a lot in those cases where people use Protractor nowadays. But codeless IDEs still have a long way to go
looking at this list of automation testing tools, TestComplete seemed really awesome back in the day. now its foothold is basically undermined by apps like yours
Comparing Screenster vs Ghost Inspector, you’re obviously doing more in the visual comparison department. Your Content Comparison algorithm is something that really sets you apart. Hope you’ll keep on improving!
Thank you!
Would be great to see more tools compared in the last section. There’s, like, a dozen modern automated testing tools for web applications in addition to Screenster, Ghost Inspector, and Usetrace. For instance, I heard about LeapTest and iMacros on quite a few occasions
Would be nice to see open-source tools like Sahi Pen Source and Katalon Studio covered as parts of this list.
While I like the general idea behind open source automation testing tools (as opposed to just frameworks), there’s always a catch. Katalon isn’t really free: if you want decent technical support (and believe me you’ll need because it’s a smallish product built by a smallish company), that’ll be 2.5K/year charged as “business support” package. I’ve never tried Sahi but I heard that people end up using the Pro package (not sayin’ it’s a bad thing thought). Basically, there’s really no such thing as free lunch, and I’d rather know the price upfront…
Speaking of testing tools for web applications, this post should definitely mention Sahi
I wonder is these website automation tools work for wordpress, and I I really need them for a wordpress site.
Jest seems like a decent test automation framework. The thing works with TypeScript which is getting popular in 2018, and there’s built-in focus on React which is even more popular. There are other goodies as well, like code coverage reports tool and a mocking library. It’s also used by Facebook (naturally), as well as Twitter, Spotify, and Airbnb. I’d say Jest is worth considering is you’re working with JavaScript in 2018.
Screenster looks promising. I’m glad I discovered you
Jest is okay for sure
For some basic visual testing (i.e. screenshot-taking), you can use Modules like Protractor-snapshot. It’s more dev-centric, though. But than again, few companies that I’ve worked for have huge teams of real dedicated QAs for software testing (I’m guessing, that’s more typical of larger companies, startups don’t hire testers now-a-days).