CI for CSS regression testing with Screenster

Now that the CI integration plugin has been successfully rolled out, it is time to recall how it all began… Ready for a tale about defeating a horde of CSS regression testing challenges with the help of a Jenkins server and a visual QA automation tool?

Why did we need CSS regression tests on our CI

Our team has realized the necessity of automating the visual regression testing process last winter, when our other product AjaxSwing gained a significant number of new users, who were actively requesting improvements. A new version was released each week, sometimes even more often. AjaxSwing generates web UI for desktop applications so you can imagine how much visual UI verification we had to accomplish each time a build was ready. It just seemed never-ending. We had unit tests in our continuous integration environment running against every new build, but they didn’t help with verifying the visual aspects of the UI such as CSS, formatting and layouts. That’s when our QA team decided it is time to try our new automation testing tool Screenster in real action, despite it being in its early Beta at the time.

We upgraded our CI environment to the following state:

1. A test server with AjaxSwing repository on GitLab
2. Jenkins server running on the same test server
3. A separate Screenster server with the last stable version of Screenster installed there — typically it is the latest released build
4. Automated CSS regression tests recorded on this stable server which perform visual verification of our AjaxSwing apps belonging to the regression scope.

We are running visual regression testing jobs for AjaxSwing either manually or by pre-set time — normally it is 3 — 5 times a day. A job includes the following operations:

1. Pulls all changes from a selected AjaxSwing branch to the repository on the test server
2. Prepares a build based on this branch
3. Launches CSS regression tests on the stable Screenster server
4. Sends a message with the test status to a person assigned to visual regression testing task after every finished test run.

The example above demonstrates how Screenster reports visual bugs in web UI. A screenshot captured during the latest run around 16:31 (see the time on the left page) contains differences from the baseline screenshot captured during the first run, around 16:23 (on the right page). These differences are marked with red color in order to alert a tester, who monitors the test results, about detected bugs.

Screenster saves the day!

Thanks to Screenster, today we have 98% fewer cases of our AjaxSwing users reporting visual issues caused by our changes. Moreover, we have recently started testing Screenster itself with Screenster! Visual regression testing process is the same here: a special job on Jenkins server creates a Screenster build with latest changes. After that, a test located at the stable server emulates user actions against this fresh Screenster including:

• Recording a visual regression test
• Running a test
• Editing a test (adding & deleting steps, adding ignore regions an approving steps)
• Overriding a test with new parameters (with the help of Screenster’s Extend feature)

This is a good example how Screenster, a CSS regression testing tool, well compatible with CI, enhances the web development team’s productivity! And what is your experience with automated visual tests on CI? Leave your comments and questions below!