The Rise of Automated Testing Platforms

It’s 2017, and software testing is as important as ever — so is choosing the right automated testing platform. So how do you make the right choice given the number of available automated software testing tools?

In a sense, testing software features is like shipping goods. Your clients don’t care about what means of transportation you used to ship the product — as long as it gets there in time. You, however, will definitely want to make sure you’re using a vehicle that delivers goods in time, in pristine condition, and at the right cost.

Just for fun, let’s stick with the shipment analogy and examine both paid and open source automation testing tools for web applications.
apps..

Horse (Manual Testing)

If you’re shipping something small and time isn’t a priority, you can hand your package to a guy on a horse. This sort of logistics is as reliable as the delivery guy, but it’s simple to implement. Need to ship more? Hire a couple more horsemen.

Just like horse transportation, manual testing works fine as long as your product is small. But once things get bigger, testing manually becomes costly and time-consuming, so it’s not sustainable in the long run.

You realize you need a car (and an automation testing framework), so you go shopping.

Muscle Car (Test Automation Frameworks of the 2010s)

Cars cost money upfront, but once you own it, one car is better than a hundred horses.

Test automation frameworks like Selenium and Appium are indisputable standard for browser and mobile testing. They’ve been around for quite a while, and they have proven their worth.

But in 2017, manually coding tests and setting up automated testing infrastructure for CI is like shipping with muscle cars of the 60s. They are powerful, macho, and you can swap out the parts yourself. Without a doubt, a car beats any human on a horse (aka manual testers).

On the flip side, these metal beasts eat a ton of gas, they are clunky, and they break often. If you are mechanically inclined, you can tinker with them but you’d better know what you are doing.

Working with a low-level test automation framework like Selenium WebDriver, CasperJS, Capybara, Cucumber and Karma in a custom environment leads to multiple problems:

• Getting started is way too long. You argue and pick your test automation frameworks, configure them, integrate them into your process — all of this can last for days. Hint: you could use that time to actually automate tests.
• Auto-updating browsers get out of sync with WebDriver version causing tests to break.
• Testing with IE deserves a separate line. It’s like doing a flip into a pool – looks pretty easy, but try it and you’ll most likely do a belly flop. (Or in the worst case, land head first onto the edge – ouch!)
• Using a headless browser like PhantomJS is different than testing in real browsers, which leaks bugs into production.
• Browser control: need to kill hanging browsers, stage parallel executions, monitor host environment
• Framework gotchas: use setAttribute(‘value’, ‘myText’) for INPUT with WebDriver and it won’t work for React apps. Hint: should have used sendKeys(‘myText’) instead.
• Selectors maintenance: many elements won’t have IDs, so when page structure changes, your tests won’t be able to find their targets causing them to fail.
• The more tests you write, the longer they run. Sure, setting up your own grids is a solution, but it’s not for the faint-hearted.
• Screenshot/CSS testing: not supported in frameworks and simple solutions one can build quickly are pretty much useless. So all your formatting and CSS bugs have to be caught by manual testing or they are shipped to prod.

There are plenty of good test automation frameworks for writing JavaScript unit tests for the front end code, and it is certainly a good idea to push tests down the pyramid. But even with 100% pass rate of unit tests, you are not truly testing what the user will see. The only way to actually test this is to run end-to-end tests in a real browser.

Whether you like fixing your custom-tuned stick-shift muscle car, or you stick with it because that’s what you’ve always driven, you know for your growing business you need something more reliable and easier to operate.

You discover the Sports Sedan.

Sports Sedan (Cloud Platforms)

Fast, powerful, comfortable, easy-on-the-gas and with plenty of tech, sports sedans were the “it” of 2015 and 2016. Like muscle cars, performance sedans still offer a lot in terms of raw power, yet you don’t have to be your own mechanic to own and drive them.

There’s a tradeoff, however, that becomes evident in those cases when you actually want to be your own mechanic. With a sports sedans, you get a limited set of predefined factory features, and upgrades are either too expensive or impossible.

Now, there’s actually a whole lot of cloud-based automated testing platforms that provide you with the same level of comfort. Yet, they also have a more limited functional scope.

Running Tests on the Cloud

The following automated testing platforms don’t offer much for test creation or maintenance. On the other hand, they provide a readymade infrastructure for running your developed tests on the cloud.

• SauceLabs. One of the earliest cloud companies and still the leading platform for running Selenium tests on the cloud. It’s great for parallel execution, it captures screenshots and video, but it doesn’t provide much value beyond that.
• Browserstack.com. It’s been great for manual testing on remote VMs, and lately they’ve added automation testing support where you can upload Selenium tests, making it a direct competitor to SauceLabs.
• Xamarin Test Cloud. Provides cloud infrastructure for running tests written in Calabash, a proprietary framework for mobile apps.

Functional testing platforms allow them to scale test execution, but they do very little to improve the productivity of writing and maintaining the tests. To write good tests, your QA engineers need to be almost as good as developers. They need to understand the basics of coding, the structure of HTML page, CSS or XPath selectors, and need to know how to tie it all together.

In other words, sports sedans are definitely a great alternative for a muscle car. But if you want to focus on feature development and beat your competitors with shipping, it’s time to look at something revolutionary.

Like a self-driving car.

Self-Driving Cars (Codeless Automated Testing Platforms)

While it is debatable, many argue that when AI is built out enough, it will surpass human driving capabilities. Cars, they say, will be better because they are never tired or distracted, they have more sensors and they don’t make mistakes.

In an ideal world, drones will deliver products and cars will drive themselves, letting humans focus on more important things (like teaching AI to play ukulele). Obviously, this vision of an ideal future includes AI-driven testing platforms. These automated testing platforms will learn everything by themselves and eventually do most the work with little human input.

Like Tesla, these platforms are not quite there yet, but like Tesla, they show tremendous promise and already give you cool features that leave you excited and waiting for more. Let’s look at a few revolutionary platforms that are pushing the innovation forward.

Applitools – Visual Baseline for Your Coded Tests

Applitools is a web service that allows you to add visual testing to your existing tests. It has sophisticated algorithms for image comparison, but since it doesn’t recognize objects in the image (like text or buttons) it has inherent shortcomings. As of today, this automated testing platform has quite a few strengths, but also a couple of weaknesses:

• Good: powerful image comparison, SDKs and examples for all languages, a plugin allowing you to create visual baselines, visual comparison across resolutions. And the best part is that you can add it to your existing Selenium tests.

• Bad: doesn’t help in any way with ROI because it still requires learning and coding, no help for non-technical user, no collaboration, no dashboards.

Usetrace – Simple Codeless Testing on the Cloud

The great thing about this automated testing platform is that it targets codeless tests making things super-easy for non-technical users. In a nutshell, ease of use and productivity look like the key priorities for Usertrace.

• Good: works right after you sign up, super-easy, plus you get a remote browser on the cloud. There’s also a basic record/playback editor with support of multiple browsers and versions, simple verification for presence of or text inside an element, collaboration, scheduling and nice dashboards. Great tutorials and documentation.

• Bad: no baseline concept, and users have to actually understand and manage locators. Also, no CSS testing, no visual testing, no screenshots for steps, and the user has to manually create all verifications.

Screenster – Visual Codeless Test Automation

Screenster is a new all-in-one automated testing platform that combines powerful visual testing with simplicity of codeless tests. While not as mature as Usetrace and Applitools, Screenster combines the benefits of simplicity with the power of full automation.

• Good: works right after you sign up, easy to learn, has record/playback functionality. Also, powerful visual baseline, screenshots of each step, smart visual comparison, visual diff, automatic selector maintenance, automatic timeout maintenance, collaboration, and very high ROI as a result.

• Bad: doesn’t work for every site, not all browser versions supported, not mature yet.

Conclusion

Selenium WebDriver is awesome in many ways, but it doesn’t offer much in terms of ROI because it’s simply an API for automating the browser. This tool has a steep learning curve, and it relies heavily on your ability to write code.

Higher level frameworks like Protractor, Karma and Jasmine offer more structure and functionality out of the box, but they still require deep technical knowledge of the browser, selectors and coding in general.

Cloud platforms are improving the productivity of testers. Execution platforms like SauceLabs and BrowserStack provide a scalable environment for running Selenium based-tests, and solve the execution pains. But they do not solve test development pains.

Finally, there is a new generation of forward-looking automated testing platforms like Applitools, Usetrace and Screenster. These simplify test development and maintenance by offering you a UI-based web environment for test editing. What’s more, they enable collaboration, visual testing and lots of smart features that go far beyond the capabilities of frameworks, resulting in vastly improved ROI for technical and non-technical testers.

So basically, it seems that the nearest future of QA automation is about combining the right tools that are best suited for their own jobs. In some cases, nothing beats the reliability of a good ole muscle car, while sometimes a more progressive solution is what you really need. But is the software testing ready for its Tesla? What do you think?