Better Alternative to Using Selenium for Continuous Integration Testing

UI testing automation is all the fuss today. In a way, this is due to an advent of new solutions that promise to outperform the “tried-and-true” tools of the past years. But how many of these automated testing tools (both new and old) are CI-friendly? Besides, how many of them are good at handling visual testing of websites and web apps?

Let’s see what the market of UI automation testing has to offer in the way of CI support. But first, let’s answer the following question.

Does solid CI support really matter for UI testing automation solutions?

The short answer is yes. After all, continuous integration solutions are the first line of defense against bugs and errors.

Let’s recall what a typical CI process looks like: one job creates a build, and another one launches unit tests on it. This setup allows for early spotting of most easy-to-find problems. If the build passes this stage successfully, testers can step in with their usual routine. They’ll take a quick look at the UI and, if everything is fine, proceed with functional testing.

Now, this process seems to have room for improvement, doesn’t it? Wouldn’t it be super awesome to automate this quick UI sanity check, as well as the overall visual testing routine? Let another CI job run automated visual tests against a fresh build.

Sounds fantastic, but this task requires a reliable automation tool with CI support. So what are the CI-friendly UI testing automation tools that we get to choose from?

Existing UI testing automation solutions for running Selenium on CI

Let’s start with arguably the most well-known combo, Selenium + Jenkins.

You’ve probably noticed how many homebaked Selenium-for-Jenkins solutions are out there. Basically, each of these solutions embodies a specific vision of an optimal automation testing tool. Naturally, many of these solutions are optimal for a specific purpose. Still, most of these solutions enforce numerous complexities:

• You will need to set up another testing framework (like jUnit) and use it as a base.
  Selenium Grid is often necessary because running a bunch of Selenium tests on one computer is problematic.

• At least several of these instances will require fine-tuning of access rights. For example, you need to enable Jenkins to start browsers.

• You have to spend time making sure that all ports that Jenkins and Selenium are using by default are free on each node of your network.

• Finally, you’ll need to run this system in ‘test mode’ to detect possible issues. Once you’re through with this, throw in more code to solve them, create additional batch files etc.

All in all, there are quite a few issues to address before you can even begin working on UI testing automation. Doesn’t this all look like like a huge pile of work?

Luckily, the Jenkins team has come up with their own solution, which simplifies things a bit. They have updated it in the summer of 2016, and it seems that many users have come to download it. The situation is pretty much the same for TeamCity, Travis, and Bamboo. Still, a problem remains: Selenium itself does not provide a good solution for CI integration.

UI automation testing with TestComplete plus Jenkins

Another big automation testing tool TestComplete has its own integration solution for Jenkins. Still, a unified integration plugin remains a dream, for the most part. At the end of the day, testers still have to invent their own means of integrating their UI automation testing processes with CI platforms.

This may look tiresome enough unless a team of developers and network integrators is assigned to that task. And it is just the beginning.

Once you’re done setting up your CI environment, it’s time to start thinking about the maintenance of your tests. As your project progresses, some of your tests will need updates or replacement. Aside from this, even a small change to your project may bring upon the need to reorganize the whole network.

A less technical person (such as a middle QA engineer with mostly manual testing experience or a business person) will face a bunch of difficulties while doing this job. These difficulties can be so huge that, ultimately, they will delay UI testing automation. No wonder that today, so many teams still prefer hiring additional manual junior QAs to avoid investing too much at the early development stage.

Selenium alternatives and new emerging tools for testing automation. How do they address this problem?

It’s no secret that Selenium has myriads of fans all over the world. Many of them coordinate their efforts in developing some local improvements. At some point, these local improvements become good enough for the general public.

Eventually, developer communities come up with decent CI integration solutions for Selenium, Cucumber, TestComplete and other prominent automation testing tools. So, is there a problem? Yes, there is. The UI testing automation niche needs a simpler and more efficient solution. Ths solution should be accessible to everyone, too.

Visual regression testing automation is the most obvious example of how low-level frameworks fail. The fact is, traditional platforms such as Selenium and Cucumber cannot perform this kind of web UI verification. They can perform functional testing and check for certain fields to be present. However, they cannot detect a changed font, a wrong text or even a broken image. But these bugs are no less important!

So what are the alternatives to Selenium? Are there more efficient solutions for running automated tests in continuous integration environments?

PhantomCSS

One of the most popular options is PhantomCSS. Unlike Selenium, this automation testing tool is actually capable of CSS testing. What’s more, it can compare web pages in a screenshot-by-screenshot format. Whenever a test step fails, Phantom displays a screenshot to a user.

As for the downsides, PhantomCSS is a cumbersome solution. Its installation implies downloading and setting up several separate modules. You will also need to fine-tune the whole framework once you’re through with its components. Basically, there’s a lot to do before you can get down to creating your tests.

Besides, Phantom uses its own built-in browser emulator and does not run in real Chrome or Firefox. This means that it’ll fail to trace browser-specific CSS issues.

Yet another common problem with Phantom is the learning curve. Just like to Selenium and TestComplete, PhantomJS requires an experienced developer. This means UI testing automation becomes the job of programmers, while non-coders are left outside the game. Wouldn’t it be more efficient to enable testers to create and manage all tests?

Finally, what about CI integration? As a matter of fact, Phantom itself does not provide any special integration tools with any of the popular CI clients. You can install it separately, on a slave/build instance of Jenkins or TeamCity so that its tests can run there. Another client, Travis, has created its own plugin for Phantom. The problem is this plugin has no multi-vendor CI integration solution either.

Screenster

Screenster is a new CSS automation testing tool that came out of beta on June 1, 2016. As of today, Screenster is the only project on the market that can boast full CI support for all major CI tools including Jenkins, TeamCity, Travis, and Bamboo. Thanks to the simplicity of its functional set, its CI plugin does not rely on any additional frameworks. The only thing that’s required is a basic configuration of the settings a CI tool.

Once you’ve downloaded the CI integration plugin, add the path to it in a selected CI client instance. You’ll also need to specify a few parameters including projects list, user email, user password, browser list and Screenster host address: https://screenster.io/documentation/screenster-ci-client.

Once you’re through with this, you’re good to start running CSS tests on your CI server. Screenster brings the following benefits to your visual regression testing process:

• Scriptless recorder: no coding needed to start with your visual tests. Screenster captures each user action during recording and then repeats it during a playback.

• Visual and DOM baseline: DOM model and actual screenshots are stored for a selected web UI after the test is recorded. Screenster will check for differences to the baseline during the future runs.

• Smart CSS selector: Screenster detects web page elements by capturing their CSS IDs. Selectors can be manually updated in a click. Screenster easily processes AJAX and can automatically detect dynamic areas.

• Easy maintenance: you can edit tests, add or delete steps, bulk override URLs of multiple tests at a time, export and import them. You can even call a test within another test.

Screenster is a live proof that it’s possible to have a universal solution. With Screenster, you can make seamless integration with all major CI solutions and then run UI automation testing after each new build. So, why search further?