My name is Eric Bautista. I have 12 years of experience in Software Testing with the last 5 years as a QA Consultant for Tacit Knowledge—a technology enabler for brands and retailers. As a software quality specialist, I am keenly aware of new testing tools and frameworks released into the market. Some might be worthy of being included in our toolset, while others don’t make the cut. To build a robust toolset, we evaluate and based on our analysis we may or may not integrate. In the following article, I summarize the pros and cons of using Carina as a Web Automation Framework.
Reasons for evaluating new tooling
One of the benefits of being a Software Testing Consultant at Tacit Knowledge is the ability to evaluate and suggest automation frameworks and tooling to improve and streamline the work done by the QA team. Any recommendations should be made by a team member who either has experience with the tool (either through past usage or because they have invested time evaluating it) and can articulate the pros and cons.
Although it’s not possible to evaluate all testing frameworks and tools in the market, we may take the opportunity to do a Proof of Concept (POC) to validate the framework through the automation of different scenarios. Below are details on the POC I completed on Carina.
The Carina framework supports not only web applications, but also allows the creation of automation scripts for databases like MySQL and Oracle, REST services and mobile (iOS and Android) solutions. This sounds very interesting as there are very few automation frameworks that cover all of these testing layers. I am particularly interested in its web support so I will focus on evaluating only this component of the tool.
When installing an automation framework, we frequently use package managers such as Brew, Pip or Npm, which save us time and make the installation process easier; users do not need to perform multiple manual tasks other than typing a command. However, installation of the Carina framework requires more manual steps.
First, you should address some of the dependencies such as installing JDK 1.8+ and Apache Maven. Then, you will need to clone the Carina repository (https://github.com/qaprosoft/carina-demo), run some Maven tasks and download and execute Selenium standalone server. This procedure is not complex, but compared with the installation process of other frameworks, it is much more manual than automated.
One thing that caught my attention was the use of Selenium standalone server. Other selenium based frameworks (such as Gauge) use Selenium WebDriver instead, where users do not need to worry about manually starting Selenium server in order to be able to execute their automation tests. Again, this is not a complicated task nor does it require a lot of time to complete, but it is something you should manually perform so you can run Carina scripts.
Carina documentation is very simple – it contains installation instructions, information about Carina’s structure, and general information about features such as data providers. If you are looking for Java API documentation, you will be disappointed as it does not exist. I had to clone the entire repository to gain access to the source code. This is where I found that some functions were extended from Selenium. Definitely, this is an area that should be improved.
Carina’s user manual can be found in the following link http://qaprosoft.github.io/carina/.
How Carina Works
Before I started implementing my test scripts, I had to understand how Carina works. Carina is Java-based, extends Selenium features and uses TestNG as the testing framework. Carina’s support goes beyond web application and one very useful component is the Demo project on Carina’s repository that contains some examples of the different testing layers it supports. This Demo project has a couple of web-related scripts that allowed me to understand Carina’s architecture.
Carina fully implements the Page Object Pattern; it abstracts any page information from the tests. To achieve this, Carina created its own classes to handle Pages, UI Components and Web Elements. It also uses Selenium Page Factory to initialize the defined Web Elements within the Page Objects. In Carina, every existing Page extends from the AbstractPage class. If a component is defined, it should extend the AbstractUIObject class. Finally, every Web Element extends from the ExtendedWebElement class.
Advantages of Carina
All of these new classes implement features that are not part of Selenium and you can implement on your own, however, Carina has already done this for you. One key function that was implemented in the AbstractPage class, that is not already implemented in Selenium is the equivalent to “Wait Until Page Loads”. On every Selenium project I have been a part of, I have had to implement this myself. This is one of the many methods implemented in Carina. There are many other functions (such as saving a Page in a PDF format) that might not be used frequently, however, they part of the classes ready to be used and would certainly save time if/when they may be needed.
For UI components, this class allows you to reuse objects between multiple Pages. A good example, considering an e-commerce site, is the search bar located in the header. This component appears in the Homepage along with nearly every other existing page on the website. Instead of handling that component on each defined page, it can be implemented as an Abstract UI Component so you can share it between the existing Page objects. This feature definitely earns a point in Carina’s favor.
The Extended Web Element class is the biggest advantage in terms of reducing lines of code. The most relevant functions I found on this class, apart from a set of Waiting methods, have to do with Action related methods. In Selenium, some operations such as hover, drag and drop or double click should be implemented through an Action object. In Carina all of these operations have already been implemented within the Extended Web Element class. You do not need to worry about creating an Action object anymore.
Debugging in Carina
Once you implement automation scripts in Carina, it’s likely that you will need to debug them. Debugging scripts in Carina was not easy or straightforward. Running your script in debug mode slows down test execution drastically; by half or even more of its expected speed. In addition, the code is executed multiple times, as if multiple threads were running. I spent some time trying to set the number of threads to be used in Carina and I found a property that I set to “1”, but the behaviour remained unchanged. I have been unable to determine why this was happening.
Reporting in Carina
When my scripts were running in Carina, I was able to execute them and retrieve a test report. Carina’s report is generated as an HTML file and looks pretty simple including the list of test cases that were covered during the execution, their status, what has passed, failed or skipped. Carina does not currently include charts or a more sophisticated reporting layout.
Conclusion & Recommendation
Carina framework may not be suitable for use on all production projects. Carina’s community is active on Github and I can see ongoing updates, improvements and changes. This is a credit to Carina’s support team. Perhaps in the near future, we may receive a release that includes improvements on some or all the areas I’ve reviewed above.
Thanks for reading and happy testing!