9 regression testing tools for web automation

What regression testing tools will you actually want to use in your future projects? I bet Selenium will remain a go-to solution for many, even though the framework will celebrate its 14th birthday this year. But does this make it the right choice for test automation in 2018? In my opinion, it’s not necessarily the case.

Call me biased because I’m a part of a team building a Selenium alternative, but this doesn’t mean I’m wrong. There’s a whole bunch of powerful regression testing tools and frameworks out there, and it’s always good to know your options. As far as these options go, here are my personal favourites from 2017. I think it’s these 9 automation testing solutions that you might want to consider in 2018.

Top UI regression testing tools of 2017–2018

 
Note that we’re not just talking about frameworks here. The regression testing tools in the table above include codeless and low-code solutions that ship as either desktop or web apps. These two types of testing tools target non-programmers with features like image-comparison-based visual testing and record-playback.

In addition, the three types of regression testing tools cater to the needs of different teams:

• Code-based frameworks work for teams where the same people write product features and automate tests.
• Desktop tools target dedicated QA teams in large enterprises with all-in-one solutions for automated testing, bug tracking, and reporting.
• Web-based platforms focus on codeless or low-code automation for Lean teams of manual testers and non-QAs.

So which one would you choose? Read on for a quick overview of the 9 tools and those features that make them my top picks for 2018.

Selenium

I mentioned that Selenium isn’t the only game in town, but it’s still an option that you can’t ignore. Selenium is a must-learn testing tool which is familiar to the overwhelming majority of QA automation engineers. Besides, Selenium WebDriver is the W3C standard for browser automation, and other regression testing tools use it as well.

Advantages of Selenium

• Selenium is free and open-source, and it boasts a huge global community behind it.
• All major browsers and OSs support Selenium.
• You can write regression tests in Java, Scala, Python, Ruby, C#, PHP, Groovy, and Perl.
• Selenium support is must-have for third-party test clouds (like Sauce Labs and BrowserStack), crowdsourced testing platforms, and other third-party tools and services.

Cons of Selenium

• Development and maintenance of Selenium-based regression test suites are as labor-intensive as feature development.
• Selenium lacks built-in image comparison and record-playback, which means it doesn’t really support visual testing without third-party plugins.
• Configuration pains: setting up Selenium servers with Grids, CI, and collaboration is anything but a straightforward process.
• All Selenium locators except for ID-based ones are inherently fragile.

Protractor

An end-to-end testing framework for Angular, Protractor enables UI-centric regression testing for non-Angular applications as well. As far as practical value goes, this software testing framework has quite a few perks to offer.

Advantages of Protractor

• Being a wrapper around WebDriverJS (the official W3C WebDriver API), Protractor enjoys universal support across major browsers, OSs, and third-party platforms.
• Regression test cases for the GUI written in JavaScript.
• Protractor syntax is more concise compared to Selenium.
• Less configuration pains compared to Selenium.
• Accessors make for a precise and relatively robust locator mechanism.
• Decent compatibility with other frameworks and test runners like Jasmine, Mocha, or Karma.

Cons of Protractor

• Code-based end-to-end testing ignores visual bugs, which means its coverage of happy paths and UX scenarios is partial at best.
• Protractor shares the major shortcomings of Selenium: no built-in visual testing, low efficiency for test development, tedious maintenance.

Cucumber

Cucumber advertises itself as an “open source tool for executable specifications”. This platform uses a plain language parser called Gherkin for business-facing test suites written in a BDD style. Here are the key pros and cons of this approach.

Advantages of Cucumber:

• Cucumber merges tests and specs into always-up-to-date “living documentations” that act as a single source of truth for testing teams.
• The “Given-When-Then” syntax is accessible to non-programmers.
• BDD enforces a user-centric approach to test development. Cucumber tests are acceptance tests above all.
• Cucumber and Selenium make for a legit automated regression testing combo.
• A JavaScript implementation of Cucumber is also available for both Node and the browser.
• Cucumber is an open source tool with a thriving community.

Cons of Cucumber:

• Automated testing with Cucumber takes special training and a team of dedicated engineers.
• In the case of regression testing, the tests suites you build with Cucumber are as fragile as the ones built with Selenium.

TestComplete

TestComplete from SmartBear ships with a huge suite of features that streamline automation testing. Historically, its record-playback functionality has been more of an add-on feature, but it has improved noticeably in the past years.

Advantages of TestComplete:

• Supports both record-playback and handwritten tests (JavaScript, Python, VBScript, JScript, DelphiScript, C#Script, and C++Script).
• TestComplete automatically locates UI objects during test recording and continues to recognize most of them after minor UI changes.
• Reusable test steps.
• Out-of-the-box support for Selenium, Angular, Backbone, and other frameworks.
• Broad integration with IDEs and solutions for CI, version control, issue tracking, and application lifecycle management (including Visual Studio, Jenkins, Git, Jira, HP ALM, etc).

Cons of TestComplete:

• TestComplete is costly, to say the least.
• The functionality of TestComplete is extensive and complex enough to warrant a need for specialized training.
• No Linux or Mac support.
• Limited functionality for test editing and error handling.

Unified Functional Testing (UFT)

UFT, formerly known as HPE Unified Functional Testing (UFT) and formerly owned by Hewlett Packard, has quite a history behind it. This tool for automation still has a strong foothold in the enterprise community. Besides, its current owner/developer MicroFocus enjoys the status of a Leader in the Gartner Quadrant for Test Automation.

Advantages of Unified Functional Testing (UFT)

• UFT “unifies” GUI and API testing in one comprehensive tool.
• UFT Insight (the feature for image-based object capture) works for deprecated UI technologies (including Flash).
• Record-playback available via integration with HP Sprinter.
• Drag-and-drop test management with Canvas appeals to manual testers.
• The installation package for UFT is growing smaller from year to year, while other tools are gaining weight.
• Strong focus on mobile.
• Good collaboration functionality.

Cons of UFT

• UFT only supports VBScript out of the box; advanced code-based testing requires LeanFT.
• There are regression testing tools with more advanced visual testing functionality, available for a cheaper cost.
• Even the general functionality of UFT is quite complex to master.

Eggplant Functional

One makes Eggplant Functional stand out is its UI-centric nature. TestPlant views the UI as a “single source of truth for the user experience”, which makes a lot of sense. What’s more, Eggplant is one of the few desktop regression testing tools to work across Windows, Mac, and Linux.

Advantages of Eggplant Functional

• Supports Java, C#, Ruby, and SenseTalk for handwritten test scripts, as well as record-playback for codeless automation.
• The platform promises device-agnostic testing across OSs and software versions (for scripted tests).
• Decent support for Safari and Opera in addition to other major browsers.

Cons of Eggplant Functional

• Even though it’s simpler than other desktop tools, Eggplant still requires training.
• TestPlant’s pricing policy could use more transparency.
• Eggplant’s functionality for codeless automation needs a stronger focus on visual testing.

UseTrace

The premise behind Usetrace is that coded tests are too slow-paced for the brave new world of Agile software development. To speed things up, this regression testing tool substitutes scripting with modular, editable test steps that mimic user interactions with the UI. Chaining these steps into series of simulated interactions with the UI allows for codeless automation of regression testing.

Advantages of Usetrace

• Modular architecture reduces maintenance pains. Should the UI change, you can edit an existing test instead of automating a new one from the ground up.
• You can run recorded tests across multiple browsers. For instance, a test recorded in Firefox will run in Chrome.
• Automatic handling of waits.
• Support for parameterization  and random data.
• Support for different screen sizes
• Fast test creation and editing.
• Notifications via Email, JIRA, Slack, Flowdock, HipChat.
• Support for continuous integration tools (Jenkins, Bamboo, Visual Studio Online, CircleCI, Codeship).
• You can export tests in JSON.

Cons of UseTrace

• A typical UseTrace test doesn’t go beyond happy paths.
• No verification of navigation: you need to manually verify UI elements (e.g. click on on-page content to check if it’s the right page). This makes tests less robust.
• The tool doesn’t offer much in terms of visual testing or guarantee the integrity of the UI.
• No automatic locators.
• No automatic management of dynamic areas.
• No support for Selenium commands.

Ghost Inspector

Ghost Inspector was one of the first regression testing tools to prove that coding isn’t the most productive way to test the GUI. The platform offers smart record-playback in the form of a fast and lightweight browser application.

Advantages of Ghost Inspector

• The tool is simple enough for a non-tester to learn it in a day.
• Reusable tests and test steps.
• Import and export of Selenium and support for custom JavaScript code for advanced regression testing functionality.
• Support for pseudo-selectors (e.g. <code>:contains)</code>).
• Decent support for iframes and frames, Google Analytics, and embedded editors.
• Extensive reports with videos of testing session and console information in addition to screenshots and test steps.
• Support for drag and drop and file uploads.
• Parameterization with variables.

Cons of Ghost Inspector

• Limited verification of UI integrity (basic screenshots comparison).
• No verification of navigation: you need to manually click on on-page elements to check if the right page has loaded.
• No automatic content verification.
• No smart or selectors or self-healing selectors.
• A chrome plugin is necessary for the tool to work.
• 10-minute limit for testing sessions.

Screenster

Screenster is our own solution, so take this section for what it’s worth. This said, there are quite a few reasons to seriously consider this tool for the automation of regression testing in 2018. Screenster introduces a number of advancements into record-playback, verification, and selector management, which result in simpler, more robust automated tests.

Advantages of Screenster

• Smart record-playback-verification with automatic recognition of every on-page element.
• The platform uses a concept of UI baselines that store UI screenshots and the snapshots of its DOM structure.
• Self-healing locators: Screenster stores all locators for each element and uses this data to automatically restore locators in case of UI changes.
• Automatic handling of dynamic UI areas (the platform ignores UI regions with dynamic content during regression test runs).
• Automatic handling of timeouts: the platform automatically determines optimal wait times for complex page loading scenarios.
• Automatic verification of content.
• Editable test-steps are simple enough to manage for a non-programmer or non-tester.
• Recording a basic test takes under 5 minutes.
• Smart screenshot comparison algorithm ignores differences related to rendering.
• Support for Selenium commands and custom JavaScript code for extended functionality.
• Shared portal for collaborative management of test suites.
• Screenster tests can run on a local server or on the cloud.
• Fully web-based interface with user-friendly dashboards.
• Reusable tests and test steps.

Cons of Screenster

• Notifications via email only.
• The UI of the platform is in active development: the way some features look will change.
• The platform runs on Windows servers only, with Linux support still in development.

Wrap up: framework versus desktop tools versus web platforms

We’ve covered 9 tools in this post, yet we’re basically speaking about three types of solutions that enable regression testing automation.

Frameworks like Selenium, Protractor, and Cucumber boast a serious following. On the other hand, a growing number of teams move from code-based frameworks to dedicated testing tools. Moving to a right codeless solution helps you increase your ROI by saving you time and enabling manual QAs to automate tests.

As far as codeless automation of regression testing goes, desktop and web-based tools demonstrate different approaches to automation. Desktop solutions keep the legacy of old-school tools like Sahi, Ranorex, or Rational Functional Tester by shipping features in volumes. In the meantime, web-based platforms focus on doing one thing but doing it well.

So which one do you choose? If codeless automation with Screenster sounds interesting, there’s a simple way to find out if our product can meet your needs. Try our demo for free, record several tests for your application, and tell us what you think.