12 features that every UI testing automation tool must have
March 22, 2017
What software do you use for UI testing automation? For that matter, how did you come to choose the solution that you’re currently using?
Making lists is always handy, so let’s do exactly that. Below, you will find a list of requirements that we deem must-have for every modern tool used for automated user interface testing. When working on it, we’ve used the feedback from our customers — and from our competitors as well.
Besides, the checklist below pretty much mirrors our roadmap for Screenster, our very own platform for UI testing automation. Without further ado, read on.
1. Speed
Higher speed equals higher ROI. This is why it’s paramount that the solution you’re using for UI testing automation allows you to quickly create, edit and debug your suites. So if creating a basic test suite takes more than 30 minutes, you should probably start looking for another tool.
2. Codeless UI tests
Don’t get me wrong, coding is fun — but we think the real fun is to code features, not tests. And let’s face it, there’s really no such thing as a robust suite of hand-coded UI tests that covers the whole user interface, not just its separate elements. Unlike scripted tests, codeless IDEs offer a complete coverage of UI, UX, and customer journeys — and isn’t that the actual purpose of UI testing automation?
3. CSS testing
CSS is the single best reason why the automation of UI testing can be such a pain in the neck. Rewriting CSS rules often leads to visual bugs that aren’t covered by typical tests. So either you have a tool that captures CSS-related regressions automatically, or you end up making your end-users do the job of your QAs.
4. Beginner-friendliness and short learning curve.
In case with 99% of startups that launch on a budget, UI testing falls on the shoulders of manual QAs or even non-QA staff. So wouldn’t it be great to have a tool that’s simple enough for non-techies to pick up?
5. Test reuse
I’m sure you know that no problem should ever have to be solved twice. In real-life projects, however, you’ll sometimes find yourself having to create the same tests for different web pages or UI states. Given this, it’s a must for modern UI testing automation tools to provide you with a user-friendly way of reusing tests or test steps.
6. User-friendly parameterization
This one is actually related to the previous point. Duplicate test steps or code often contribute to the maintenance burden, which raises the need of smart parameterization.
7. Smart handling of dynamic areas.
The UI testing automation of websites and web apps often involves working with UI areas that have dynamic content. Dates, banner ads fetched from a third-party server, sections with related items… you name it. If the tool you’re using doesn’t handle these, you’ll be working with dozens of false positives during every test run.
8. Smart handling of timeouts
No matter how great you are at coding your tests, no sane person can possibly like to hand-code workarounds for test timeouts. So it’s pretty much a must for any tool used for UI automation testing to handle timeouts.
9. Collaboration
For many QAs, lack of decent collaboration features is a major pain-point of old-school UI testing automation solutions. By default, any cloud-based UI testing platform is bound to have a shared portal that allows team members to access and collaborate on test suites.
10. Support of coding whenever it matters
The ability to import and export scripted test seems like a neat functionality that can make any IDE-like solution more flexible (and hence more useful).
11. Minimal installation and setup pain
Setting up huge enterprise-grade QA automation solutions often takes days — and you often need a dedicated professional to do just that. No one likes to waste this much time, which makes it crucial for a UI testing automation to minimize installation pains.
12. CI support
Continuous integration is inherent to the modern software development process. In this context, it’s a bit surprising that only few modern-day automated user interface testing tools prioritize CI support.
Bottom line: let’s stop testing 5% of UI and move on to a more comprehensive solution
At this point, I guess it’s pretty obvious that the list above reflects our own vision of a modern tool for UI automation testing. Sure, you could argue with some of the points — specifically the one that states UI testing should be codeless. Yet, I’m sure we’re not the only ones who think that code-based solutions aren’t exactly suitable for UI testing automation.
If you have any experience working with Selenium, in particular, you know that it doesn’t verify anything but itself. Basically, all verifications are hand-coded, and usually, only the key elements are verified.
Having to hand-code you UI tests also implies targeting separate UI elements instead of covering UI as a whole. And we all need to admit it, writing tests for just a part of your UI was only okay when ‘lean UIs’ were still a thing. Today, in the era of rich logic-driven interfaces written in React and Angular, you need to test the whole UI, not just its parts.
Is this UI testing checklist for automated user interface testing tools only or are you talking about frameworks too?
I guess you could consider some of the points relevant to frameworks, but definitely no all of them. For instance, it’s kinda obvious that point 2 (Codeless UI testing) isn’t compatible with frameworks.
I’d say this list mostly focuses on those features that have the most direct impact on the ROI of UI testing automation. We’ve worked with multiple frameworks in the past, and we came to the conclusion that on the UI level, code-based automation isn’t efficient or effective. This is the reason why we decided to build Screenster:)
I’d add “Lightweight” as Point#13. Tools like UFT and Testcomplete offer advanced image-testing (also, both have a solution for locators handling, just like your tool). What they don’t have though, is an option of not having to install a ~3GB application suite of your computer when all you need is non-coded automated image comparison (with all those little extras like locators handling).
Here’s a #14: supports Selenium
You can criticize Selenium all you want to but there’s no point denying that having this sort of flexibility can be helpful. Ideally, I’d love to see a solution that combines Selenium with screenshot comparison in a manageable way
We actually have that too! Screenster can execute Selenium and JavaScript tests. There’s also minimal verification for executing Selenium-like series of actions and events (with verification for the first and the last UI state in a test).
We’ve been using similar guidelines when working on our own automated user interface testing tool! I like the overall idea of DOM scanning and how it works in Screenster, btw
Looks like a decent list of checkpoints for a good UI automation tool. I wonder is there’s a real chance to persuade people to move from handcoded Selenium. If your platform is as good as advertised, there’s a good chance manual testing firms will adopt it
№13 — Screenshot “history”. An archive of older screenshots may prove useful for the support of really old projects.
I came here looking for user interface testing checklist, but the overall idea underlying this looks nice, too. It would be great to see Katalon Studio, Ghodt Inspector, Screener, Screenster (popular name, I guess), and Usetrace compared based on these criteria (as well as the ones mentioned in comments)
My 2¢ is that you need the support of several programming languages. I’ve seen a bunch of tools that allow for import/export of Selenium, but what I would really find useful is the support of JavaScript for custom actions
I second that. Coding support is a must because no record-playback is too much of a “carpet bombing” approach. Sometimes, you really don’t need to look at the whole UI and would rather run a really small test that just looks whether something small works or not. Also, good support for Selenium/Java and JavaScript (mb with Protractor) would make for a really versatile solution, and that’s what most automated user interface testing tools miss, imho
I guess, the challenge with tools of this sort is to automate visual testing without making the process painfully slow. For a UI tester, slow execution is a dealbreaker. I guess this is the reason why nobody runs automated screenshot testing en-masse