Al in Software Testing: Artificial Intelligence Testing Methods and Tools

The advent of AI in software testing is imminent. When asked what part of the software delivery cycle needs artificial intelligence the most, most people name software testing. Mainstream media like Forbes herald the future of AI-driven testing. What’s more, associations like AISTA emerge to advance AI in testing automation.

So is it time to panic?:) Are testers going to lose their jobs soon? My guess is there’s still plenty of time before Skynet takes over…

In the past few weeks, our team took a close look at how different companies implement AI in software testing tools. We researched several UI testing platforms that claim to leverage artificial intelligence. Besides, we explored what business and tech experts have to say about the so-called “third-wave of testing automation”. So what is it all about?

The Third Wave of Test Automation: AI All the Things

Know someone who’s into third-wave cafes? If you think the third-wave of automation involves the same amount of hype, you’re not wrong. For those not yet familiar with the term (or rather, the buzzword), “waves” represent turnpoint solutions in UI testing automation:

• The first wave: old-school record/playback tools like QTP (now UFT), TestComplete, and the likes.
• The second wave: introduction of Selenium, the WebDriver standard, and countless Selenium alternatives that use WebDriver.
• Third wave: AI in software testing automation.

Buzzwords aside, it’s easy to see how artificial intelligence can change everything in testing automation — and we really are talking about everything.

Artificial Intelligence Testing Methods: How Will AI Change Software Testing?

If you’re a technical person, you probably know that artificial intelligence is more of an umbrella term. So when we’re talking about AI in software testing, what do we actually mean? Is it machine learning, image recognition, natural language processing, or maybe, expert systems? Each of these can have an impact on software testing automation:

• In automated UI testing, artificial intelligence can push visual screenshot comparison beyond pixel matching. Done right, this would mean better precision with fewer false positives. AI-driven comparison will support moved content, responsive layouts, and ephemeral values like dates.
• Neural networks and expert systems can analyze test runs, prioritize tests, and even automatically generate test scenarios.
• With genetic algorithms and deep learning, artificial intelligence will be able to produce maintainable test code. Some research in this direction is already available.
• With natural language recognition, artificial intelligence will validate voice input. This will prove useful when virtual assistants and voice search will become even more popular.

The points above are just the tip of the iceberg.

Part of the grand promise behind AI in testing automation is that it will test other AIs. But what is it going to look like? Imagine an artificial intelligence engine testing Cortana, Siri, and Google Assistant. I bet it’s going to look more complex than this:

AI in Software Testing Tools

AI-testing AIs aren’t quite there yet, but it’s already possible to find AIs in software testing tools. In fact, there are at least 5 UI testing tools whose that incorporate AI into automation.

1. Applitools Eyes: Visual Testing Driven by Artificial Intelligence

While not fully AI-based, Applitools claim to leverage artificial intelligence to some capacity:

Specifically, Applitools offer something called “AI-powered cognitive vision”. The feature is, essentially, a visual comparison algorithm that ignores differences a human eye can’t possibly notice. So if it’s a 0.5-pixel layout shift or anti-aliasing, the platform won’t fire a false positive. The grand promise behind Applitools is that everything works out of the box, with no setup or manual thresholding required.

That said, image comparison is all there is to it, so you are left to do 90% of the testing work. Applitools requires you to write your own tests, capture your own screenshots, at the right times, and then call their service. Still, it’s easy to see what potential artificial intelligence holds for visual testing (provided that it moves beyond automatic thresholding).

2. Testim: No More Brittle Locators

One peculiar fact about Testim is that its co-founder is also the ex-director of R&D from Applitools. Functionality-wise, Testing promises to unleash the power of machine learning to tackle test authoring and maintenance. What this means in a more practical sense is “auto-healing locators”. If you’re thinking this feature was available before the AI and ML hype, you’re 100% right. But hey, everything’s better with machine learning, right?

ML or not, Testim uses a sophisticated algorithm to automatically update element locators. For instance, should the ID change for some reason, the platform will automatically swap it for a new one. As a result, you waste less time fixing broken tests, which is a huge advantage for UI testers.

3. Mabl: a Virtual Assistant for Testers

The Mabl team puts a strong emphasis on machine learning in software testing. Or at least, that’s a huge emphasis their marketing. The team behind this project refers to Mabl as a “she/her”, and they call their recorder extension “Mabl trainer”. The overall UX of the platform revolves around personification and “AI-fication”, too.

Speaking of the actual functionality, Mabl automates tests using manual assertions and runs tests on schedule. It compares test runtimes, page load speeds, and UI screenshots to catch regressions. The data from test runs also fuels a machine learning engine which drives Mabl‘s notifications.

Despite the slew of useful features, Mabl’s hardly looks like full-fledged artificial intelligence or machine learning. Still, we’re talking about a pretty sophisticated platform for codeless tests and scheduled test runs. To give credit where it’s due, Mabl almost feels like an AI. It runs in the background and notifies you only if it needs your input, offering an unusual user experience.

4. Eggplant AI: (Semi-)Automatic Generation of Test Cases

Eggplant AI sees its claim to fame in the automatic creation of test cases, and it promises to reduce test scripting by 90%. Neat, huh?

There’s is a catch, though. To make Eggplant AI work, you need to map all states of every screen of your application using a dedicated canvas-like tool. This process is 100% manual, and requires special training, too. Once you’re through with this, though, the Eggplant AI will proceed to generate test cases.

You’ll also need Eggplant Functional, another tool from TestPlant, to carry out the automation part. In addition, test maintenance for Eggplant AI requires the knowledge of SenseTalk, a proprietary language integral to the Eggplant ecosystem.

5. Appvance: Record-Playback with Machine Learning

Appvance boasts a rather unique take on automation. It combs through apps with machine learning, and it applies cognitive generation to produce models known as “app blueprints”. Based on these “blueprints”, Appvance promises to generate thousands of test cases in minutes.

In addition to artificial intelligence in its software testing mechanism, Appvance relies on record/playback-based feature called Test Designer. The feature combines record/playback with screenshot comparison, data-driven tests, and automatic capturing of DOM and AJAX. The company also promises to produce cleaner auto-generated code for recorded tests.

AI or Not, You Need to Get the Job Done

Naturally, there was a reason behind our exploration of AI applications in software testing tools. We’ve been introducing the elements of artificial intelligence in Screenster during the last 3 years separating the hype from true game changers.

Here are five features for intelligent test automation that will have a significant impact on your ROI.

Intelligent Visual Testing

If you’re automating with Screenster, you’re working with one of the most advanced visual testing solutions to date. It’s so advanced you could almost call it an AI.

If you think about it, what could you expect from artificial intelligence in visual testing tools? You’d certainly want it to recognize individual UI elements, their DOM, visual representation, and content. Guess what, this is exactly how Screenster works.

Screenster looks into the DOM of the UI to single out individual elements. It runs pixel-perfect comparison on an element level to keep things super-precise while minimizing false positives. It recognizes text and dates, and it can detect and ignore dynamic UI areas. Just like certain “AI-powered Visual UI testing” solutions, Screenster is also able to sort out anti-aliasing and similar noise factors.

Self-Healing Tests

Do you really need machine learning for self-healing tests? When scanning the DOM of a UI, Screenster captures all DOM attributes, CSS and XPath selectors, and content for each element. Using this data, Screenster can restore broken locators (like, say, renamed IDs, moved elements, or changed content).

Automatic Verification of All UI Elements

Automatic verification makes for another reason to programmatically scan the DOM. While AI-based software testing tools require manual assertions and mapping, Screenster verifies all elements and content automatically. When running verification, it automatically determines optimal wait time for each element. This way, you don’t have to worry or even think about timeouts.

Simple, Maintainable Tests

Part of the role of AIs in software testing is making tests more maintainable and simple. Tools like Appvance and Eggplant strive to achieve this goal with machine learning. The problem is they still depend on auto-generated code and/or proprietary programming languages. Let’s face the truth, this is neither simple nor maintainable.

In the meantime, Screenster allows for fully codeless test maintenance. To make thing more fun, we also support custom Selenium and JavaScript code. This way, you can code if you really feel that in your case, coding would work better.

So does this mean artificial intelligence and machine learning are an overkill for UI testing? Not at all. In fact, we’re considering AI as a long-term development strategy. But while we’re not yet there, you’re still getting everything an AI-based software testing tool can offer.

Here’s an easy way to see if Screenster can really deliver in terms of functionality and performance. Sign up for our free only demo and give Screenster a try.