Using Jenkins CI for UI testing. Will Selenium do?

If you’ve ever used Jenkins CI for UI testing automation, you probably know it’s a terrific scheduler. Sure, Jenkins isn’t the only tool of its kind, but continuous integration with Jenkins has several advantages over other CI platforms.

Namely:

• Jenkins is open-source, award-winning, and supported by an independent board of programming veterans from Yahoo!, Microsoft and RedHat.
• Being a mainstream CI tool, Jenkins has great documentation and a vibrant, booming community.
• Even though Jenkins CI is not as easy to configure as TeamCity, it’s still pleasantly manageable.
• Jenkins has hundreds of plugins available. Moreover, developers often give Jenkins higher priority over its Oracle-controlled counterpart Hudson when rolling out updates and new plugins.

Okay, Jenkins is uber-awesome… unless you use it to schedule Selenium UI tests

Jenkins does everything you’d expect from a robust CI solution. It’s great when used with most types of automated testing tools, too — as long as we’re talking unit and API tests. Things tend to get trickier, however, once you try using Jenkins with an old-school UI testing automation tool like Selenium.
Okay, that last statement might irk some Selenium fans, so let me explain myself while they’re getting their pitchforks and torches ready:). Or rather, let me ask you three questions:

1. What happens when your code update breaks style cascading, making your UI look all broken. Will Selenium or Jenkins “notice” the visual bug?
2. What happens when a Jenkins job running your Selenium tests stumbles on a hanging browser?
3. What happens when an Ajax-triggered event takes a little too long to occur?

The short answer to all of these questions is, “things will get nasty”.

Sure, Jenkins CI isn’t the one to blame — most of these problems are with Selenium. Actually, it’s not even that Selenium mostly falls flat on its face when it has to deal with hanging browsers or unexpected timeouts… My main grudge is that Selenium is not even capable of such a thing as end-to-end visual UI testing. That is, passing Selenium tests don’t mean the users won’t see broken UI.

Once Jenkins CI and Selenium finish running tests, your testers still might want to proceed to the manual UI checkup. You see UI testing has to be manual at this stage because Selenium doesn’t offer much in the way of visual testing. Sure, you won’t do these UI checkups every time, but wouldn’t it make sense to automate them anyway?

In theory, you can automate the UI testing step with a Jenkins CI + WebdriverCSS combo. The problem is both options add a lot of overhead without offering much value. With WebdriverCSS, you’ll deal with even more complex setup, yet what you get at the end of the day is simplistic image comparison. No on-page object recognition, no ignored regions. For most of the production UI, it’s not going to be enough.

Of course, I wouldn’t be bringing this all up if couldn’t offer a better solution:).

Jenkins CI with Screenster — continuous visual testing the way it’s supposed to be

The reason WebdriverCSS isn’t that great for an environment running continuous integration with Jenkins is that it does nothing apart from primitive comparing of screenshots. That might’ve seemed enough five years ago, but now things are different. And that’s exactly why modern-day UI automation tools like Screenster deserve attention.

Screenster is a visual testing platform that revolves around the record-playback-verification paradygm. This cloud-based tool does a lot more in the way of recognizing on-page UI elements, eliminating little pesky headaches like timeouts or locators, and streamlining the overall testing experience. What it offers, in a nutshell, is more features, more simplicity, and less maintenance pain. What’s better, it offers these benefits in the form of a fast cloud-based solution.

Let’s see how Screenster addresses the challenges of UI testing automation.

Visual testing

Visual testing needs to cover a lot more than simplistic screenshot comparison. While analyzing a UI, Screenster automates the manual tester’s routine by creating visual baselines of each UI state and user action. When doing so, the platform captures both screenshots and DOM snapshots of the UI, effectively recognizing on-page elements. Screenster also breaks down the testing sessions into editable steps.

Another cool thing about visual testing with Screenster is that the platform is able to sort out insignificant differences caused by rendering. While ignoring these falsy layout shifts, Screenster will alway actual visual bugs caused by changes in HTML, CSS, and JavaScript.

Hanging browsers

Most cases of browsers hanging mid-test lead to all sorts of problems, you need to hand-code workarounds to deal with issues of this sort. That is, you have to do this if you’re using Selenium (or any other code-based solution). If you’re running your tests with Screenster, however, the platform kills hanging browsers automatically. What this means for your CI is more streamlined experience of running Jenkins jobs.

Automation of timeouts

Timeouts are another little-yet-complex thing that you don’t have to worry about when using Screenster. When processing a user session, the platform automatically identifies AJAX updates and/or page transitions. This allows Screenster to determine optimal waiting time for each action.

What this all means from a practical standpoint is that your testers don’t need to tinker with waits to optimize the duration of job runs for Jenkins CI. When it comes to timeouts, Screenster handles everything for you.

Automation of locators

When capturing the DOM of a UI state or page, Screenster builds a full list of DOM parents for every element. These lists are updated whenever a change to the UI occurs. Thanks to this feature, the platform eliminates the need for hand-coded locators.

Detection of dynamic regions

Yet another thing that gets automated is the management of regions with dynamic content like ads, dates, avatars, etc. When running screenshot comparison with Phantom CSS, testers usually need to hand-code their way around these regions in order to avoid false positives.

In the case with Screenster, the platform detects the regions automatically and suggests ignoring them during future runs of Jenkins CI. All that’s left to do for the tester is approve the suggested regions with a couple of mouse clicks.

Simplicity and smooth learning curve

To crown all, Screenster is a simple cloud-based tool that virtually anyone could use. It doesn’t require programming skills or experience in testing automation (although manual testing skills will definitely be a plus). Setting it up is a pain-free experience — you don’t need to fiddle with installation packages, plugins or extensions.

Making Jenkins CI work with Screenster implies manual setup, but the process is fairly simple. In fact, let’s briefly cover this process.

Setting up Jenkins CI for Screenster

The first thing you need to do to add continuous integration with Jenkins to Screenster is to download this .zip archive with a .jar file (yep, that’s our CI client). Just like in case with any other CI client, this file needs to end up in a directory on your CI server.

Once you’re through with this, create a Jenkins CI job, name it and run in. After that, add a build step named “Execute Windows batch command” and run this command in it:

java -Duser=<userEmail> -Dpassword=<userPassword> -jar <screensterCIFolderPath>/screenster-ci.jar <projectList>

At this point, pay attention to the following things:

• screensterCIFolderPath — this stands for the path to the Screenster CI client on your CI server (make sure you’ve unzipped the archive).
• projectList — list of your projects, separated with commas.
• userEmail and userPassword — obviously, these two stand for the email and the password that you’ve used when signing in to agileengine.com.

Aside from this, there are a couple of optional parameters that you can learn about from our CI documentation.

Bottom line

By adding Screenster to your Jenkins CI jobs, you can automate the manual work of your testers without hand-coding a single test. What’s more, the Jenkins UI testing suites created with Screenster are faster to create and easier to maintain that those created with traditional solutions like Webdriver or PhantomCSS.

One more thing:). Besides Jenkins, Screenster supports other major CI tools like Bamboo, TeamCity and Travis.
So why not give it a try?