Have any questions? +1 646.844.5712 (US)

Case Study: Choosing Autotest Suite

Choosing Autotest Suite

The problem with a lot of automation is that it just isn’t done to the best of it’s capability.

Rosie Sherry

Pre–words

This document is an excerpts of a research that was created for one of our clients when they needed to choose a tool for automating their tests, so some specific requirements and considerations will not be disclosed.

Please consider this as a sample; not as a comprehensive reference of modern autotests frameworks.

Note: if you need something like that or you have any questions please feel free to ask us.

Introduction

Purpose

The main purpose of this document is to present the results of the research established by HiTech Service to compare several Automated Testing tools and find out the most appropriate solution, which is eminently suitable to business and technical requirements of the company project.

Scope

The scope of this analysis is limited by the explicit list of products for evaluation, and includes the following:

  • Ranorex testing tools;
  • SmartBear / Automated QA testing tools;
  • Selenium testing tool

HP Mercury testing tools (namely ‑ HP’s Quick Test Professional (QTP)) was excluded from consideration by the customer due to its not acceptable licensing policy.

Solutions were evaluated basing on the list of requirement and criteria specified by the company and listed in the “Customer Needs” section.

Customer needs

Automated Testing — is automating the manual testing process currently in use. This requires that a formalized “manual testing process” currently exists in the project. Minimally, such a process includes:

  • Detailed test cases, including predictable “expected results”, which have been developed from Business Functional Specifications and Design documentation;
  • A standalone Test Environment, including a Test Database that is restorable to a known constant, such that the test cases are able to be repeated each time there are modifications made to the application.

The real use and purpose of automated test tools is to automate regression testing. To perform it there should be a database of detailed test cases that are repeatable, and this suite of tests has to be run every time there is a change to the application/component to ensure that the change does not produce unintended consequences.

An automated test script is a program. Automated script development, to be effective, must be subject to the same rules and standards that are applied to software development.

According to the Customer request, the needed automated testing tool should provide ability to perform the following types of testing:

  • Regression Testing for HTML-based Web application in the Windows OS environment;
  • Regression testing of Java applications (Java applets) in the Windows OS environment;
  • Integration Testing;
  • Functional Testing;
  • Data-driven testing;
  • Distributed testing.

The following testing is currently out of scope:

  • Load/Stress/Performance Testing;
  • Testing in the Non-Windows environment (Linux, Android, iOS);

There are also some additional features of the automated testing tool that should be taken into consideration:

  • Price
  • Inter-operability – ability to work together with other tools (nUnit, MS Visual Studio, build software)
  • Capacity – particularly on load testing, how many users at once, virtual clients, multiple physical client spanning, support of synchronized tests running simultaneously on several computers etc.
  • Perform coverage testing;
  • Support scripting language: C#, VBScript, Jscript;
  • Access several data sources (DBMS, Files, etc);
  • Run test against multi-language applications;
  • Command Line support;
  • Report of run tests;
  • Technical / Documentation support

Additional tools may be considered in the future in case of changes to the role and scope of automated testing within the project.

SmartBear TestComplete

SmartBear’s TestComplete delivers automated functional, unit, regression, manual, data-driven, object-driven, distributed and manual testing in one easy-to-use and totally integrated package at an affordable price.

TestComplete shows the result in the following view. The reflection of results can be developed or customized.

TestComplete is a full-featured environment for automated testing of native Windows, .NET, Java, Flash/Flex, Silverlight, WPF (XAML) applications, web pages and web services in the Windows OS environment. It has been designed to free developers and QA departments from the massive drain of time and energy required by manual testing.

TestComplete is systematic, automated, and structured testing tool, with superior support for .NET, Java, Flash/Flex, Silverlight, Visual C++, Visual Basic, WPF (XAML), Delphi, C++Builder and web applications and services. It is equally oriented for testing 32-bit and 64-bit applications under all modern versions of Windows OS. With TestComplete it is also possible to test PowerBuilder, FoxPro, Access and other applications that utilize Microsoft Active Accessibility (MSAA) technology.

TestComplete is the ultimate solution to perform nightly tests and to help to get reliable daily builds of the developed software.

TestComplete tools and features let every member of the team, developers and non-developers, no matter how technically savvy they are, to contribute to the testing projects.

Functionality Excellent
Ease of learning and test development Good
Documentation and support quality Excellent
Price Satisfactory
Total Evaluation

Features and Benefits

  • Intelligent recording and simulating user actions;
  • Language Independent scripting (built-in support for multiple scripting languages: VBScript, JScript, C++Script, C#Script and DelphiScript);
  • No changes and/or additional instrumentation is required for the tested application;
  • Support for Third-party controls, such as Microsoft, Borland, Developer Express, Infragastic, etc. ( http://support.smartbear.com/viewarticle/26894/);
  • Support for Unicode;
  • Asynchronous test code execution;
  • Multiple compiler support;
  • Superior support for .Net applications;
  • Superior support for WPF applications;
  • Support for Windows Vista Technologies;
  • Support for Java applications;
  • Support for 32-bit and 64-bit applications;
  • Access internal objects, methods and properties of the tested application;
  • Calling functions located in .NET assemblies;
  • Calling functions located in DLLs;
  • Functional testing of Web-based applications;
  • Testing of Web Services;
  • Distributed testing;
  • Testing under different User Accounts;
  • Manual Testing with possibility to mix manual and automated steps within one test.

Ranorex

Ranorex allows quick and easy implementation of GUI (Graphical User Interface) automation scripts based on the common programming languages C# and VB.NET.

The Ranorex Premium Edition provides all necessary features and tools to automate applications with the most common programming languages. Use the power of object-oriented programming techniques to develop professional test and automation systems. Create, manage and debug automation projects with existing development infrastructure like Visual Studio.

Ranorex tool supports the following features:

  • Black-box testing;
  • Extended support for applications based on:
    • Microsoft .NET;
    • Visual C++;
    • Visual Basic;
    • Borland Delphi;
  • Support for 3rd-party controls (Infragistics, DevExpress, …);
  • Support of ActiveX controls;
  • Support for WPF applications;
  • Validation support;
  • Logging support;
  • Exception handling;
  • Image validation;
  • Calling functions located in .NET assemblies;
  • Calling functions located in DLLs;
  • C# and VB.NET development interface;
  • RanorexSpy Pro tool;
  • Testing of mobile applications (iOS, Android);
  • Tested applications require additional instrumentation before they can be operated by Ranorex.
Functionality Good
Ease of learning and test development Good
Documentation and support quality Excellent
Price Satisfactory
Total Evaluation

Selenium

Selenium is a Web browser automation tool written in Java with the very limited support of testing other application types and/or communication with them. Tests can be created using Java, Ruby or Python languages.

Selenium tool is provided in a form of a runtime engine without any environment to develop, debug and manage tests. All this functionality is provided by the separate stand–alone tools like Eclipse, etc.

Selenium supports the following features:

  • Recording and playback;
  • Java-based tests;
  • No changes and/or additional instrumentation is required for the tested web application;
  • Support for Flash/Flex technologies via separate extensions;
  • Poor testing environment (tests organization and management, team work, etc.);
  • Asynchronous test code execution;
  • Support for Java applications;
  • Functional testing of Web-based applications;
  • Distributed testing (load balancing);
Functionality Good
Ease of learning and test development Moderate
Documentation and support quality Moderate
Price Free
Total Evaluation

Comparison Matrix

Product Selenium TestComplete Ranorex
Vendor Selenium SmartBear (Automated QA) Ranorex
Description Web browser automation Automated functional testing Automated functional testing
Supported operating systems Windows

OS X

Linux

Solaris

Windows 98

Windows 2000

Windows XP

Windows 7

Windows 8 (not Modern UI (Metro) applications)

Windows server 2003

Windows server 2008

Windows server 2008 R2

Windows NT

Windows 2000

Windows XP

Windows server 2003

Windows Vista/2008

Tested applications’ languages / platforms Web applications Native applications

.NET

Java/JavaFX

Rich Internet Applications

Command-line utilities

Native applications

.NET

Java

Rich Internet Applications

Support for various Windows applications components Flash (ext. extension)

Flex (ext. extension)

Silverlight (ext.extension)

AJAX

Win32/64 support

COM support

Flash, JavaFX, Flex, Silverlight, WPF, AJAX

Web pages

Broad support for a number of custom UI controls

Win32/64 support

COM support

Flash, Flex, WPF, AJAX

Web pages

Android

iOS

Testing type support Web testing

Functional Testing

Regression testing

Distributed testing (load balancing)

Functional Testing

Cloud Testing (Amazon AWS)

Regression testing

Distributed testing (concurrent testing)

Manual Testing

Web Testing

Unit testing

Web Services testing

Functional Testing

Regression testing

Web Testing

Scripting language C#

Java

Python

Ruby

VBScript

DelphiScript

JScript

C#Script

C++Script

C#

VB.NET

Tests recording Yes Yes Yes
Support for several data sources Yes (ext. extension) Yes Yes
Test synchronization No Yes No
Test execution balancing Yes No No
Logging results Yes (current log only) Yes (all logs are preserved) Yes
Export reports Yes Yes
Testing multilanguage applications Yes Yes Yes
Support for Team Test work Yes Yes Yes
Command line Yes Yes
Integration with Microsoft Visual Studio No Yes Yes
Integration with the Build Software Yes Yes Yes
Technical Support No Yes (Excellent) Yes
Documentation Yes (Medium) Yes (Excellent) Yes
Pricing Free Node-Locked Enterprise: $1999

Floating Enterprise: $4499

TestExecute (runtime engine): $399

10% off the regular price promotion is valid until June 20, 2013

All licenses are perpetual and include one year of free upgrades/maintenance.

Maintenance renewal usually provided with the 20-30% discounts

Node-Locked Premium: €1480

Floating Premium: €3080

Node-Locked Ranorex Runtime: €389

Floating anorex Runtime: €780

All prices exclude VAT

All licenses are perpetual and include one year of free upgrades/maintenance.

Maintenance renewal costs from 20% to 50% of the license price

Risks

Evaluation of the tools revealed that neither of them can recognize and process items added to the image in the SRS editor (notes, labels, etc.) as separate entities. More investigation in this area is required, but there is a grounded expectation that the problem will be solved.

Also, neither of the tools support functionality like Load/Stress testing and code performance. This functionality is usually delivered by the separate tools like load testing software (the former) and code profilers (the latter). SmartBear provides such tools (at the cost of separate licenses) while neither Ranorex nor Selenium have similar suggestions.

Conclusions and Suggestions

After thorough review and comparison of the tools, using the customer defined criteria, SmartBear TestComplete is recommended as the best fit to company needs. TestComplete provides all the functionality required by company, namely:

  • Superior Support for Java, Flash and Flex applications;
  • Superior Support for .Net applications;
  • Support of required scripting languages;
  • Interoperability with various compilers;
  • Support of 3rd party components;
  • Support for distributed testing and testing in the cloud (Amazon AWS);
  • Ability to test web services;
  • Great integrated development environment for tests creation and management.

The features mentioned above distinguish Test Complete from the competitors and allow its easy integration into development environment and testing process of different companies. Another benefit is easy learning and test development process. The tool has intuitive interface which will significantly reduce learning curve for existing and new team members. Test development and support time is also minimal due to wide support of scripting languages, integration with other tools and combination of all testing in one package.

Ranorex tool seem to have practically the same functionality as TestComplete (except distributed and web services testing) but slightly higher price for the required licenses.

Selenium tool is good for automation of web browsers but provides no support for testing other application types. Because of the limited technical support and test development environment (which, in many cases, requires usage of separate extensions), the learning curve and the cost of tests development, support and execution is expected to be on the same level as for TestComplete and Ranorex, even despite the fact that Selenium itself is free.