Selenium alternatives for testing automationMarch 13, 2016
Love and Hate with Selenium
If you spend enough time with automating web applications, you will most likely develop a love/hate relationship with Selenium. The truth is that it’s pretty much the only game in town to drive the browser through the API. That’s where the “love” comes from – you can start writing cross-browser code in your favorite programming language in a matter of hours. Sweet!
But spend enough time doing that for a real world application, and you will discover the ugly side of this coin that at times make you pull your hair out. In my experience the following 6 pain points are near deal-breakers for automation with Selenium.
1. Selenium tests are unstable.
WebDriver libraries version trail the auto-updating browser and there’s always something small that doesn’t quite work. Often when you get the new version of the Selenium libraries that is supposed to fix the issue, you discover that now something that used to work doesn’t work the same way anymore. Add to that multiple versions of browsers, and you are constantly chasing a moving target.
2. Good tests require good coders.
Yeah, I’m sure there are plenty of good developers automating web testing through Selenium. But for everyone’s benefit it’s optimal to have your best developers write product features and leave automation testing to lesser skilled people. For better or worse, many companies leave UI test development to junior developers or non-CS graduates, which results in at best mediocre code quality that is a liability over time. Writing a good Selenium test requires understanding of the browser, CSS, AJAX and application page structure.
3. Test maintenance is a drag.
This of course is true for any automated tests. The more of them you build, the more time you have to spend updating them for new features and dealing with false positives. The reason I hold Selenium culprit here is that it is a very low-level API that doesn’t offer much help to the developer in test maintenance like dealing with element locators, page screenshots and error capturing.
4. Selenium test suites take too long to run.
Write enough Selenium tests and your suite will take days to finish. No framework out there can make a browser run faster and Selenium has pretty cool support for grid that can run parallel tests on multiple computers. But very few people have actually set it up and again, it requires quite a bit of skill to keep that beast running.
5. Selenium IDE lacks crucial functionality
The concept behind the lightweight Selenium IDE is neat, but very few companies can actually use it for production purposes. The IDE doesn’t offer much in the way of functionality or browser support, and the tests it produces are, essentially, poorly implemented WebDriver code.
6. Low ROI
Selenium is de-facto the only browser automation API that at a low level gives you everything you need. But it requires a very capable technical test automation engineer, and even then the productivity is low.
I know what you are thinking. Sure, it’s easy to complain about the imperfect world of test automation. But unless there’s something better, this is just whining. Well, I’m here to share some of the promising alternatives that people have used to get to a better state.
IMHO the main problem with Selenium is that it’s just too low level. It’s definitely good enough for what it does, but if you want to increase the ROI, then I think you should consider picking up a higher-level framework or testing tool. In an ideal world, this tool should combine the simplicity of Selenium IDE with the advanced features offered by Selenium.
Old School: desktop record/playback IDEs
Back in the 90s, desktop-based record/playback tools were all the craze. Tools like Ranorex, LeanFT, TestComplete, Telerik and Sahi still fall into this category. Just like Selenium IDE, they allow a less technical tester automate the testing, this approach is less and less popular. While they can certainly do this job better than Selenium IDE, very few people consider them to be a real alternative to Selenium.
These tools tend to be a giant beast that every tester has to install, they take weeks to learn and in the end the record/playback approach produces brittle tests written in some language that tester has to continue to maintain manually. Over the long term these kinds of tests tend to cost more than well-coded tests written by hand on top of something like Selenium.
Most of the teams I talked with have abandoned old school IDEs in favor of hand-coded tests with a framework like Selenium, so that’s an alternative that I wouldn’t recommend.
First, let’s make sure we compare apples to apples. Selenium is an API for browser automation. It is primarily used to drive a real browser like Firefox and Chrome giving exactly the same environment as the end user.
An emerging alternative to a real browser is a “headless” browser like PhantomJS, which is a WebKit-based browser without any UI. It’s not what end users will use with a website, but it’s pretty darn close. The downsides are that PhantomJS is almost as heavy as a real browser like Chrome, but passing tests in PhantomJS do not guarantee that they will pass in Chrome.
Evolutionary: coding to a higher-level framework
So we want to stick with a real browser but get something less quirky and more productive than plain old WebDriver. In 2017 we have a lot of good choices for hand-coded tests, so let’s quickly look at the most popular ones.
- Pre-integrated with other frameworks
- BDD syntax for more readable tests
- Easier for MVC frameworks like Angular, React and Ember
- Improved asynchronous and timeout support
However, most of the fundamental issues that you’d run in Selenium are still the same. A lot of setup, testers have to be capable programmers, every action emulation and verification has to be coded manually, fragile locators break whenever the page changes, and there is no CSS or visual verification of any sort. All this results in brittle tests that take a long time to write and a long time to run.
Revolutionary: cloud based integrated platforms
Wouldn’t it be nice if there was a system that could just capture how humans normally do testing? This system would produce tests that could be used to regression test the application and detect changes.
The Holy Grail of automation testing has always been record/playback but the tools just didn’t quite get there. The main issue is that they never got good enough were no coding was needed, and the code they produced was so painful that a good developer would prefer to just write it from scratch.
Another major problem is that the majority of automated tests do very little to assert that the UI is not broken. None of them can fail a test if certain text is cropped or overlapped or colors are off. Missing images, inadvertent text changes and broken layouts just go undetected.
During the last couple of years, there’s been an emergence of a new generation of platforms that promise dramatic increase in productivity and the ease of use due to a fundamentally different model. Rather than coding to a framework, you get a cloud-based automation testing platform that works out of the box. Lets use Screenster, a codeless platform to see what the future of UI test automation may look like.
Screenster: codeless visual tests on the cloud
Introduced in 2016, Screenster is a web-based automation testing platform that is unlike anything else. Built for less technical testers and business users, it’s promises 10x productivity improvement compared to Selenium and other frameworks. How does Screenster address the typical problems of record/playback IDEs?
There is nothing for testers to install locally. All testers and developers use a shared server running on premise or on the cloud. And there aren’t any manuals to read either.
As the tester interacts with the application, Screenster creates a visual baseline of the application and stores it on the server along with the user input. It detects the dynamic portions of the UI and offers to ignore them from future comparison. Regression test runs detect the changes and show them to the tester for approval to the baseline or bug reporting.
Tests are stored in the database along with captured DOM and screenshots. Testers can browse page elements offline and update tests through innovative mouse-friendly UX. For skeptics: would you deal with code if you didn’t have to?
Differences can be approved to the baseline with one click. Smart selectors are used to ensure that tests don’t break when elements are moved on the page or page structure changes.
Screenster has intelligent logic that detects page transitions and automatically handles AJAX updates. No need to code timeouts and deal with complexities of asynchronous programming.
The prerequisite for using Screenster is that UI state is repeatable, meaning that repeat executions of the same tests produce the same result. The approach with codeless visual tests may not cover all use cases but it should produce a significantly better ROI for 95% of the commonly covered scenarios.
Want to learn more about Screenster? Go to http://screenster.io
Want to try Screenster on the cloud?Try Online