Choosing a test automation suite

Choosing Autotest Suite

The problem with a lot of automation is that it just isn’t done to the best of it’s capability.

Pre–words

This document is an excerpt from research we prepared for one of our clients when they needed to choose a tool for automating their tests, so some specific requirements and considerations will not be disclosed. Please consider it a sample, not a comprehensive reference: it was written several years ago, and the tool landscape has changed since. Today we mostly use Playwright, Selenium and Cypress for web testing. Note: if you need something like that or you have any questions please feel free to ask us.

Introduction

Purpose

The main purpose of this document is to present the results of the research established by HiTech Service to compare several Automated Testing tools and find out the most appropriate solution, which is eminently suitable to business and technical requirements of the company project.

Scope

The scope of this analysis is limited by the explicit list of products for evaluation, and includes the following:

• Ranorex testing tools;
• SmartBear / Automated QA testing tools;
• Selenium testing tool

HP Mercury testing tools (namely ‑ HP’s Quick Test Professional (QTP)) was excluded from consideration by the customer due to its not acceptable licensing policy. Solutions were evaluated basing on the list of requirement and criteria specified by the company and listed in the "Customer Needs" section.

Customer needs

Automated Testing — is automating the manual testing process currently in use. This requires that a formalized "manual testing process" currently exists in the project. Minimally, such a process includes:

• Detailed test cases, including predictable "expected results", which have been developed from Business Functional Specifications and Design documentation;
• A standalone Test Environment, including a Test Database that is restorable to a known constant, such that the test cases are able to be repeated each time there are modifications made to the application.

The real use and purpose of automated test tools is to automate regression testing. To perform it there should be a database of detailed test cases that are repeatable, and this suite of tests has to be run every time there is a change to the application/component to ensure that the change does not produce unintended consequences. An automated test script is a program. Automated script development, to be effective, must be subject to the same rules and standards that are applied to software development. According to the Customer request, the needed automated testing tool should provide ability to perform the following types of testing:

• Regression Testing for HTML-based Web application in the Windows OS environment;
• Regression testing of Java applications (Java applets) in the Windows OS environment;
• Integration Testing;
• Functional Testing;
• Data-driven testing;
• Distributed testing.

The following testing is currently out of scope:

• Load/Stress/Performance Testing;
• Testing in the Non-Windows environment (Linux, Android, iOS);

There are also some additional features of the automated testing tool that should be taken into consideration:

• Price
• Inter-operability – ability to work together with other tools (nUnit, MS Visual Studio, build software)
• Capacity - particularly on load testing, how many users at once, virtual clients, multiple physical client spanning, support of synchronized tests running simultaneously on several computers etc.
• Perform coverage testing;
• Support scripting language: C#, VBScript, Jscript;
• Access several data sources (DBMS, Files, etc);
• Run test against multi-language applications;
• Command Line support;
• Report of run tests;
• Technical / Documentation support

Additional tools may be considered in the future in case of changes to the role and scope of automated testing within the project.

Risks

Evaluation of the tools revealed that neither of them can recognize and process items added to the image in the SRS editor (notes, labels, etc.) as separate entities. More investigation in this area is required, but there is a grounded expectation that the problem will be solved. Also, neither of the tools support functionality like Load/Stress testing and code performance. This functionality is usually delivered by the separate tools like load testing software (the former) and code profilers (the latter). SmartBear provides such tools (at the cost of separate licenses) while neither Ranorex nor Selenium have similar suggestions.

Conclusions and Suggestions

After thorough review and comparison of the tools, using the customer defined criteria, SmartBear TestComplete is recommended as the best fit to company needs. TestComplete provides all the functionality required by company, namely:

• Superior Support for Java, Flash and Flex applications;
• Superior Support for .Net applications;
• Support of required scripting languages;
• Interoperability with various compilers;
• Support of 3rd party components;
• Support for distributed testing and testing in the cloud (Amazon AWS);
• Ability to test web services;
• Great integrated development environment for tests creation and management.

The features mentioned above distinguish Test Complete from the competitors and allow its easy integration into development environment and testing process of different companies. Another benefit is easy learning and test development process. The tool has intuitive interface which will significantly reduce learning curve for existing and new team members. Test development and support time is also minimal due to wide support of scripting languages, integration with other tools and combination of all testing in one package. Ranorex tool seem to have practically the same functionality as TestComplete (except distributed and web services testing) but slightly higher price for the required licenses. Selenium tool is good for automation of web browsers but provides no support for testing other application types. Because of the limited technical support and test development environment (which, in many cases, requires usage of separate extensions), the learning curve and the cost of tests development, support and execution is expected to be on the same level as for TestComplete and Ranorex, even despite the fact that Selenium itself is free.