Let’s talk about Selenium competitors for UI regression testing

With so many UI testing tools posing themselves as direct competitors to Selenium, which one do you choose? And why opt for a Selenium competitor in the first place?

Don’t get me wrong, Selenium is awesome. Well, sort of…

Even today, 13 years since its launch, Selenium remains the number-one choice for testing in the browser. It supports the widest range of programming languages, it works with every major browser, and it runs on every major OS. No other testing automation tool can rival the flexibility and the functional scope of Selenium. Besides, none of the competitors is as popular as Selenium.

The above doesn’t mean, however, that Selenium is the best solution for every use case that its functionality covers. Namely, you wouldn’t want Selenium for end-to-end regression testing of complex UIs — or, at least, that’s the conclusion we’ve arrived at when testing own product, AjaxSwing.

In fact, our scepticism about Selenium grew to a point where we abandoned the tried-and-true framework and started looking at Selenium competitors in search of an alternative. Eventually, this led us to building our own tool for visual regression testing, but that’s a different story.

When working with Selenium, we’ve faced three major deal breakers.

1. With Selenium, you’re chasing moving targets

Having an API to automate real-world browsers is handy. Quite ironically, the same thing works to your disadvantage whenever a browser update rolls out.

Updating your Selenium library is the typical course of action that you’d take to address whatever potential issues might occur. The problem is, you’ll often discover that something small got broken because of the library update.

In this respect, testing with Selenium is akin to playing the Smack-n-bash game, and at some point, this starts to get tiresome.

2. UI testing with Selenium isn’t cost-effective

The way Selenium enables you to tinker with the browser API at low level is neat — specifically because of the flexibility of this approach. However, this approach also implies targeting hundreds of small parts of the UI with hundreds of hand-coded tests.

This, in its turn, implies that your team will need several experienced QA automation engineers who will spend weeks writing and maintaining hundreds of UI test.

Let’s admit it, very few companies actually have the budget to hire large QA automation teams. Your best specialists will always be busy testing logic-related stuff on the unit and API level where bugs are critical. As a result, smaller upstart companies will often fall back on manual UI regression testing or automate just a small part of UI tests.

3. Maintaining Selenium-based UI tests is tedious

If you’ve ever built a UI from the ground up or updated a really old UI, you know how many unplanned changes tend to get introduced mid-process.

Due to things of this sort, the number of UI tests tends to double by the time the product is ready to ship. At some point, your suite gets so large it takes days to run. Using Selenium Grid is one way to solve this, but it’s not exactly easy to pull off, plus it often requires dedicated specialists. Eventually, re-writing an ever-growing suite of Selenium tests will start consuming way more time than you can afford.

What do Selenium competitors have to offer?

Okay, criticizing things without offering much in the way of solutions is neither constructive nor helpful. Besides, I did mention that our team has gone through several Selenium alternatives, so sharing this experience might actually prove worthwhile.

As a matter of fact, we’ve considered several Selenium competitors, and here’s what we think about them.

Going old-school with record-playback IDEs

Selenium’s low-level nature is both its strength and its Achilles’ heel. So won’t it be wiser to automate actual human interactions with a record-playback solution of some sort? At least on paper, this functionality has been available for decades thanks to test automation IDEs like HP UFT and Sahi.

The great thing about old-school record-playback tools is that they enable non-coders to automate manual UI tests. This saves you time and effort when creating your UI test suite.

The not-so-great thing is that you still need a good programmer to actually use the tests recorded with UFT or Sahi. As for the not-at-all-great thing, the tests produced by these IDEs are a huge pain to work with. In fact, many QAs will argue that hand-coding UI tests is less troublesome than maintaining tests recorded with UFT or Sahi.

Aside from that, most old-school record-playback tools are heavyweight enterprise solutions that require local installation and cost you an arm and a leg.

Headless frameworks

Okay, record-replay IDEs don’t do the job, so maybe it makes sense to pick another code-based framework? Namely, you can work with higher-level frameworks like PhantomJS that substitute the real-browser with a headless WebKit-based alternative.

In addition to giving you control over updates, this approach helps you speed things up.

In theory, a headless browser eats up less resources providing you with a lightweight setup for UI automation testing. In practice, however, PhantomJS is almost as resource-hungry as Selenium. Besides, some of the bugs can only be caught in a real browser, so you basically trade performance for reliability.

A similar solution HttpUnit is more lightweight than PhantomJS. However, it’s simple built that uses a JS engine is even more different from a real browser, hence even more bugs leak into production.

Jasmine for simple unit tests, Protractor for end-to-end UI testing

So you can’t get performance and reliability in one package. But what stops you from using two different tools?

For instance, you could stick to something lightweight like Mocha or Karma for simple unit tests. Once you start UI testing on a system level, you’ll be looking for slower-but-reliable solution like Protractor or Jasmine. Best of both worlds, right?

This setup offers you way more that Selenium in terms of TDD and BDD support, in comes pre-integrated with other platforms, and it’s the same old WebDriver under the hood. Besides, coding in JavaScript seems like a sane thing to do for a web project. So can we stop here and announce our winner?

Turns out there’s a whole bunch of other tools that allow you to do just that while enabling your manual testers automate their UI tests, increasing the ROI of your project. If that’s what you’re looking for, bear with me for just a little while:)

Cloud-based Selenium competitors for UI testing

Let’s take another look at record-playback solutions. The idea of letting manual testers record real interactions with the UI and use them as a baseline for regression testing is awesome. The problem is everything else pretty much sucks. So why not take the good parts and get rid of the bad ones?

That’s exactly what tools like Ghost Inspector are trying to offer. Besides, that’s the functionality that we think we’ve achieved with Screenster, our very own solution for visual regression testing.

Instead of making testers tinker with code, tools of this type provide you with a wide array of editing features. The exact functionality differs depending on a particular platform, but here’s what you can expect from the decent ones:

• Advanced image comparison with automatic recognition of DOM structure.
• Detection of layout bugs, unwanted visual changes and CSS-changes, including cropped tests, changed colors, overlapping borders, etc.
• The ability to add, remove, skip, or alter tests in the suite.
• User-friendly means of handling timeouts, element locators, and UI areas with dynamic content.
• Automatic handling of Ajax updates,
• Cloud-based platform for running tests, in addition to the option of running tests on premise.
• A solid collaboration functionality seems like a way to provide any Selenium alternative with a competitive edge. This holds true because Selenium lacks decent collaboration functionality.

Last but not the least — ideally, tools of this kind need to make UI testing accessible to not just manual testers. If a testing platform is good enough, it will enable non-technical users to record, edit, and run UI test. While this might make tech-savvy folks skeptical, it’s easy to see how this approach can benefit production teams.

What QA automation software do you use?

Let’s take another brief look at the selenium competitors mentioned in this post. Below, you will find a table comparing different UI testing platforms. For this comparison, we’re using features that we thing matter most for UI testing automation.

 
Sure thing, the IDEs, frameworks, and platforms mentioned in this post are just a small portion of the tools available. Besides, this comparison is largely based on the experience of one production team, so I’d really appreciate if I you share your opinion.

So what’s your experience with UI regression testing? Do you stick with Selenium, or use a Selenium competitor of some sort? Also, what do you think about the tools belonging to the fourth group?