Visual CSS regression testing toolJune 27, 2016
The case for CSS regression testing is very simple. When you change a class attribute, how do you know which elements and pages have you actually affected?
I’m here to tell that there IS a better way to run visual regression testing, and it’s totally pain-free! Companies like Google, HomeAway and Kaspersky have already signed up so let’s see what the buzz is all about.
Automating CSS testing with Screenster
What I’m here to tell you about is a tool that we’ve developed internally for regression testing. It’s called Screenster, and you can trust me when I say it was born out of pain. It was also born t support the UI testing for our company’s flagship product AjaxSwing.
AjaxSwing generates web UI for desktop apps. Making CSS and rendering changes was like a sumo wrestler walking on egg shells – you pretty much know you’ve broken something, but you don’t get to know until you shipped the product, customers upgraded and someone reported an error. Visual changes were some of the hardest to test and we’d agonize about making even a smallest change in fear of side effects. Many days and cups of coffee were wasted until we decided that there has to be a better life, even if it meant building something ourselves. Now, 2 years and a new product later, we have a fully integrated CI with visual baselines, and there fear of change is gone.
Screenster is a web-based automation testing platform that combines visual screenshot testing with DOM and CSS verifications. Screenster creates a visual baseline of the application as the user interacts with the web application. When recorded tests are played back Screenster uses a smart visual comparison algorithm that detects differences caused by changes to content or CSS styles of the page.
The idea is not new with lots of tools and frameworks taking a shot at it and not getting much traction. How is Screenster different?
Screenster adds a lot of “magic” to the record/playback and visual comparison algorithm which actually makes the platform truly usable for UI regression testing and even functional testing. Let’s take a look at some of that secret sauce.
Visual and DOM Baseline
Same page will look different in each browser and each resolution so Screenster supports baseline per browser per resolution. It also stores page DOM model that allows it to recognize page elements on the screenshot and apply magic to it.
Dynamic page elements such as ads or 3rd party content can be marked and ignored from future comparison.
Screenshots made by Screenster are used to detect even the smallest of changes. But failing a test because of 1-pixel difference is a recipe for maintenance hell. Screenster comparison is tolerant to insignificant differences due to anti-aliasing, minor content shifts and constantly changing text like date and time.
Differences can be approved to the baseline with one click or marked as a defect to be fixed.
Baseline and smart visual comparison are by far the most important concepts behind Screenster that make it pretty much the only out-of-the-box platform for CSS unit testing.
Scriptless Recorder that doesn’t suck
Do you really enjoy writing test scripts by hand? I love coding but I don’t know anyone who is excited about writing UI tests. Old-school record/playback tools have failed mostly because they spit out ugly code that is a major pain to maintain, and testers who rely on these tools are not coders in the first place.
Screenster doesn’t expect you to learn CSS or XPath selectors and then tinker with them to make things work. In a perfect world every element we interact with should have it’s own unique id or class, but in reality we often have to refer to elements through it’s parent tree. Screenster has sophisticated algorithm that automatically handles renaming of element ids and classes and structural changes in DOM tree. When all else fails and element can not be found, you can simply click on the new element in the page screenshot to update the selector.
OK, so what about all those AJAX issues we normally deal with through timeouts and nasty element lookup loops?
Screenster has intelligent logic that detects page transitions and automatically recognizes AJAX updates. Through the recorder it knows that user was looking for and it will wait just enough to get to the baseline state. If the baseline match doesn’t happen before the recorded per-step timeout, error is raised.
Screenster vs PhantomCSS, Selenium and other tools
So we all know CSS testing it’s a big pain. How come nobody built a real tool so far? Actually, there is a lot of tools out there so better question is, why aren’t people using them much?
The issue with CSS testing is that you can’t do it with a scripting framework like Selenium or Cucumber because it’s next to impossible to programmatically test things like overlapping text, cropped words, bad margins and wrong colors.
You also can not just grab a screenshot and do simplistic pixel-to-pixel matching. Screenshots will differ between browsers, operating systems, resolutions and development environments. There are often parts of the application like ad panels and date/time that is expected to change.
There are interesting ideas like PhantomCSS but they require complex setup, learning, coding, and the image comparison is too primitive for a real web UI.
The truth is that CSS unit test should be visual. No way around that. It should also be able to have some sort of a “visual baseline” concept that would allow to detect the differences introduced by a CSS property change.
But it should also be smart enough to know which changes to flag and which ones to silently ignore so that the human doesn’t have to review every little difference.
CSS and UI testing is super important but is underserved by tools. The prerequisite for using Screenster is that UI state is repeatable, meaning that repeat executions of the same actions with the same parameters produce the same visual result. Run your application against staged data, record the visual tests and your CSS testing is a breeze! Really 🙂