A better way to do visual regression testingDecember 21, 2016
The concept of visual regression testing isn’t new, but it’s been in the spotlight lately. There are two factors causing this growth of attention.
First, the advent of rich UIs and responsive design has made it next to impossible to efficiently test web applications and websites without focusing on CSS and visual layout. Second, the ever-increasing competition among internet businesses forces companies to search for faster ways to create and run automated software tests.
Can manual testing do the job?
Given these two trends, manual regression testing doesn’t seem like a viable option — it’s just too slow and inefficient.
In fact, very few software development teams can afford to run visual regression tests manually after each UI revamp. Besides, humans aren’t that good at spotting visual differences in the first place due a thing called change blindness. Basically, if we don’t expect to see a minor change, we’re subconsciously bound to overlook it.
Now, the problem here is that tweaking CSS is all about unexpected changes. Once you add a new class or CSS rule on one age, it will almost certainly override something on a completely different page. As a result, your users are more likely to spot the bug than your testers.
Visual regression testing with PhantomCSS or WebdriverCSS
Now, as far as automation goes, popular code-based software testing solutions don’t seem to offer much in terms of ROI. Even though they spare you from having to eyeball each page in search of layout regressions, they also tend to over-complicate things that are supposed to be simple. Let’s look at a couple of examples to see what I’m talking about.
About a year ago, 10up’s lead frontend developer John Bellah posted this neat guide to visual regression testing with PhantomCSS. The article itself is great, but what really caught my eye was the top comment:
I bet anyone with experience of testing web UIs can relate to this. Sure, PhantomCSS enables you to automate visual UI tests, but the solution it offers is far too complicated to be something you can try out easily. This solution makes you install multiple modules and setup complex testing environments, it requires coding, and in the end, it still produces brittle tests. Even though visual regression testing is important, there’s just too much overhead in trying to make PhantomCSS work the way it’s supposed to.
Of course, PhantomCSS isn’t the only code-based tool out there that allows you to automate visual regression tests. The problem is other popular platforms aren’t making things easier. Take a look at WebdriverCSS, and you’ll face the same exact issues. In fact, solid programming background is even more critical with this one.
Aside from that, the functionality offered by both PhantomCSS and WebdriverCSS pretty much comes down to basic screenshot comparison that doesn’t really work for dynamic UI areas. In other words, your test will fail whenever it stumbles across dynamic content (like AdSense ads, dates, or sections with related posts). Sure, there are workarounds, but they require more hand-coding and/or additional modules.
Besides, pixel-to-pixel comparison means that your tests will be catching all layout changes, and it’s like AI revenge on you because they will be catching even the smallest change. As a consequence, the efficiency of your regression tests will depend on your ability to manually review these comparisons and check if the change is acceptable. It’s like going from a dark cave into a blinding light, when what you want is a pleasant Mediterranean balmy climate😌.
So what does this all bring us to?
PhantomCSS and WebdriverCSS reflect the problems inherent in any code-based solution for visual regression testing. Whenever you work with a low-level tool, you end up teaching your computer to browse your website for you — instead of reproducing actual user actions. So won’t it be better if your testing solution enables you to do the latter?
Now, reproducing actual user actions is what record/playbacks tools do — but they’ve never managed to do it right because they fail to deliver anything but basic screen-capturing functionality.
Visual regression tests with Screenster
So is there a solution that actually works for CSS regression testing? We’ve asked ourselves the same exact question when automating UI tests for our product AjaxSwing. We’ve researched a dozen of existing tools, and we’ve arrived as the age-old conclusion: if you want to do it right, do it yourself.
Two years later, we’ve launched Screenster, our very own cloud-based platform. This platform builds on the user-friendliness of the record/playback testing tools, but expands their functionality with a slew of advanced features.
Here are our solutions to the widespread problems of visual regression testing that QA engineers and companies face on a daily basis.
Code-free tests = higher ROI
We get it, coding is fun. What’s not fun, though, is having to spend days scripting hundreds of UI tests that don’t produce any business value. What’s even less fun is rewriting half of these tests every time someone adds a new menu item or removes a field from the subscription form. Besides, having to code your tests means that you’ll need QAs with solid programming skills.
So what if you use a codeless tool for visual regression tests? The immediate benefit is that your tests can be created by manual testers, or even product managers — and they’ll record these tests faster. We put Screenster to the test against Selenium and regardless of the differences in programming abilities, this approach provided 10 times the ROI based on the development time alone. This is actually the main reason why Screenster is code-free.
Screenster provides a fully functional integrated platform for visual regression testing, complete with with embedded browsers and a web-based portal. Your team is instantly productive and there are simple-to-follow reports for seeing the status of test suites and changes management.
Visual Baseline with Smart Comparison and Smart Locators
When creating Screenster tests, you record a series of actual user actions. Every time a user interacts with the UI, Screenster captures a screenshot of the page along with its DOM.
The screenshots from the first test run become the Visual Baseline. Once you change your UI and start running visual regression testing, Screenster will compare new screenshots to the baseline. When comparing the two, the platform is smart enough to disable anti-aliasing to filter out insignificant differences caused by rendering. At the same time, it’ll detect layout issues, broken fonts, and unexpected color changes even if the difference comes down to a single pixel. Basically, we’re talking about here are visual/CSS regression testing suites with more precise tests and fewer false positives.
Thanks to a feature called Smart Locators, Screenster will also recognize individual elements and store their DOM parent trees, CSS classes and IDs for future access. This way, even if the DOM structure of a page changes, Screenster will still be able to target these elements.
Add to baseline and Ignore rules
If a particular change is intended, you can add it to the baseline with one mouse click. In case there are dynamic areas on the page, you can exclude them from future comparison by clicking on the Ignore button in the dashboard. That’s right, there is no need to search for individual tests or hand-code workarounds — the feature is part of the native functionality of the platform.
When working on Screenster, we’ve made a point of streamlining visual regression testing. To do this, we addressed everything that we considered problematic in other tools. We’ve prioritized simplicity and efficiency, and we’ve put a particular focus on test maintenance. Finally, we’ve made sure Screenster tests provide an exact representation of what users see and how they interact with the UI.
There’s a lot that can be said about the possible benefits of this solution. However, the only way to tell of Screenster suits your project is to try it yourself. Test our platform by running your visual regression tests and tell us what you think about.
Want to try Screenster on the cloud?Try Online