Avoid Object Recognition Frameworks in your Test Case Automation

Avoid Object Recognition Frameworks in your Test Case Automation

There are shiny tools that promise to automate functional cases using object recognition technology.

Sikuli and Eggplant are examples of tools that allow you to interact with UI elements using images. It’s easy to fall for the promises of using these tools to automate almost anything: from native apps to complex web sites.

There are several things to consider before you invest time using these kind of tools.

Test Case Creation

This task can quickly become overwhelming. These tools require a set of images to use as identifiers to click, fill or interact with any element in your app.

I dare you to checkin your source code to a control system and see the commit that added 100 new files to the repository just to automate 1 single test case.

Test stability

Although there are ways to customize image recognition sensitivity on all tools, you will soon discover that even small changes will break your tests. Forcing you to either capture images again (and again and again) or accept the fact that your test will be failing from time to time.

Test portability

If you are planning to collaborate and create or run Test Cases using computers with different OS (Win, Mac, Linux ) or even people that are using different resolution on their screens, this might easily become a source of stress and frustration. Image recognition is just a comparison between two images and as I already stated even small changes might cause a failure.

Imagine plugging these cases into a CI server. You will create the perfect job to fail every time somebody is pushing code to the environment.

Data driven capabilities

When doing data driven testing you sometimes want to randomize your data in order to interact with different elements on a website. For example, you might want to choose a randomly selected set of products on a search landing page; in this case you will need to create a new image for each of the products you might interact with. Maintaining all these images and the scripts that use them quickly becomes cumbersome and difficult to maintain.

Test Speed and Computing Resources

If you do not mind waiting 10–15 minutes to run (and see them fail) object recognition is your best choice. Also consider the computer power used to run these kind of tools. Image recognition is expensive when you talk about computer cycles.

In summary I would say that these kind of technologies might have an application ( Super stable UI, running only in 1 machine, etc.. ). But considering the complexity of the current apps you might want to take a look to other tools that offer you a maintainable and stable set of test cases.

Some people would say that they would use Sikuli to test how the site renders on a device, and I would agree if we did not have access to more suitable tools like PhantomCSS that allows you to create a set of images used as a baseline (using CSS locators ) in the same computer that you will run the comparisons (IOU a blog post about PhantomCSS).

Thanks for reading. Enjoy your testing!!!

About Tacit Knowledge

Tacit Knowledge builds, integrates, and supports enterprise software for global brands. Tacit’s primary focus is Digital Commerce and we’ve won multiple awards for our work in this area. Our international experience extends to implementations within mobile commerce, social commerce and high-scale. Our world-class track record derives from specialization in key technologies and fielding teams with deep domain expertise. We also offer consulting services around product selection, creative and UX design, Agile coaching, system stabilization and performance tuning. And lastly, we offer 24/7 “follow the sun” application support, world class monitoring and alerting, and incident management.
Risk vs Reward. How investing in new headless technology from SAP paid off big dividends

Risk vs Reward. How investing in new headless technology from SAP paid off big dividends

Learn how one brand reaped big rewards taking a calculated risk implementing new headless technology.
Read More
Enable a better customer experience with SAP Spartacus

Enable a better customer experience with SAP Spartacus

Learn the immediate benefits from a successful headless commerce implementation of SAP Spartacus realized by a Tacit Knowledge client.
Read More
1 2 3 9