The problem with a lot of automation is that it just isn’t done to the best of it’s capability.
Rosie Sherry
The problem with a lot of automation is that it just isn’t done to the best of it’s capability.
Rosie Sherry
This document is an excerpts of a research that was created for one of our clients when they needed to choose a tool for automating their tests, so some specific requirements and considerations will not be disclosed.
Please consider this as a sample; not as a comprehensive reference of modern autotests frameworks.
Note: if you need something like that or you have any questions please feel free to ask us.
The main purpose of this document is to present the results of the research established by HiTech Service to compare several Automated Testing tools and find out the most appropriate solution, which is eminently suitable to business and technical requirements of the company project.
The scope of this analysis is limited by the explicit list of products for evaluation, and includes the following:
HP Mercury testing tools (namely ‑ HP’s Quick Test Professional (QTP)) was excluded from consideration by the customer due to its not acceptable licensing policy.
Solutions were evaluated basing on the list of requirement and criteria specified by the company and listed in the “Customer Needs” section.
Automated Testing — is automating the manual testing process currently in use. This requires that a formalized “manual testing process” currently exists in the project. Minimally, such a process includes:
The real use and purpose of automated test tools is to automate regression testing. To perform it there should be a database of detailed test cases that are repeatable, and this suite of tests has to be run every time there is a change to the application/component to ensure that the change does not produce unintended consequences.
An automated test script is a program. Automated script development, to be effective, must be subject to the same rules and standards that are applied to software development.
According to the Customer request, the needed automated testing tool should provide ability to perform the following types of testing:
The following testing is currently out of scope:
There are also some additional features of the automated testing tool that should be taken into consideration:
Additional tools may be considered in the future in case of changes to the role and scope of automated testing within the project.
SmartBear’s TestComplete delivers automated functional, unit, regression, manual, data-driven, object-driven, distributed and manual testing in one easy-to-use and totally integrated package at an affordable price.
TestComplete shows the result in the following view. The reflection of results can be developed or customized.
TestComplete is a full-featured environment for automated testing of native Windows, .NET, Java, Flash/Flex, Silverlight, WPF (XAML) applications, web pages and web services in the Windows OS environment. It has been designed to free developers and QA departments from the massive drain of time and energy required by manual testing.
TestComplete is systematic, automated, and structured testing tool, with superior support for .NET, Java, Flash/Flex, Silverlight, Visual C++, Visual Basic, WPF (XAML), Delphi, C++Builder and web applications and services. It is equally oriented for testing 32-bit and 64-bit applications under all modern versions of Windows OS. With TestComplete it is also possible to test PowerBuilder, FoxPro, Access and other applications that utilize Microsoft Active Accessibility (MSAA) technology.
TestComplete is the ultimate solution to perform nightly tests and to help to get reliable daily builds of the developed software.
TestComplete tools and features let every member of the team, developers and non-developers, no matter how technically savvy they are, to contribute to the testing projects.
Functionality | Excellent |
Ease of learning and test development | Good |
Documentation and support quality | Excellent |
Price | Satisfactory |
Total Evaluation |
Ranorex allows quick and easy implementation of GUI (Graphical User Interface) automation scripts based on the common programming languages C# and VB.NET.
The Ranorex Premium Edition provides all necessary features and tools to automate applications with the most common programming languages. Use the power of object-oriented programming techniques to develop professional test and automation systems. Create, manage and debug automation projects with existing development infrastructure like Visual Studio.
Ranorex tool supports the following features:
Functionality | Good |
Ease of learning and test development | Good |
Documentation and support quality | Excellent |
Price | Satisfactory |
Total Evaluation |
Selenium is a Web browser automation tool written in Java with the very limited support of testing other application types and/or communication with them. Tests can be created using Java, Ruby or Python languages.
Selenium tool is provided in a form of a runtime engine without any environment to develop, debug and manage tests. All this functionality is provided by the separate stand–alone tools like Eclipse, etc.
Selenium supports the following features:
Functionality | Good |
Ease of learning and test development | Moderate |
Documentation and support quality | Moderate |
Price | Free |
Total Evaluation |
Product | Selenium | TestComplete | Ranorex |
---|---|---|---|
Vendor | Selenium | SmartBear (Automated QA) | Ranorex |
Description | Web browser automation | Automated functional testing | Automated functional testing |
Supported operating systems | Windows
OS X Linux Solaris |
Windows 98
Windows 2000 Windows XP Windows 7 Windows 8 (not Modern UI (Metro) applications) Windows server 2003 Windows server 2008 Windows server 2008 R2 |
Windows NT
Windows 2000 Windows XP Windows server 2003 Windows Vista/2008 |
Tested applications’ languages / platforms | Web applications | Native applications
.NET Java/JavaFX Rich Internet Applications Command-line utilities |
Native applications
.NET Java Rich Internet Applications |
Support for various Windows applications components | Flash (ext. extension)
Flex (ext. extension) Silverlight (ext.extension) AJAX |
Win32/64 support
COM support Flash, JavaFX, Flex, Silverlight, WPF, AJAX Web pages Broad support for a number of custom UI controls |
Win32/64 support
COM support Flash, Flex, WPF, AJAX Web pages Android iOS |
Testing type support | Web testing
Functional Testing Regression testing Distributed testing (load balancing) |
Functional Testing
Cloud Testing (Amazon AWS) Regression testing Distributed testing (concurrent testing) Manual Testing Web Testing Unit testing Web Services testing |
Functional Testing
Regression testing Web Testing |
Scripting language | C#
Java Python Ruby |
VBScript
DelphiScript JScript C#Script C++Script |
C#
VB.NET |
Tests recording | Yes | Yes | Yes |
Support for several data sources | Yes (ext. extension) | Yes | Yes |
Test synchronization | No | Yes | No |
Test execution balancing | Yes | No | No |
Logging results | Yes (current log only) | Yes (all logs are preserved) | Yes |
Export reports | Yes | Yes | |
Testing multilanguage applications | Yes | Yes | Yes |
Support for Team Test work | Yes | Yes | Yes |
Command line | Yes | Yes | |
Integration with Microsoft Visual Studio | No | Yes | Yes |
Integration with the Build Software | Yes | Yes | Yes |
Technical Support | No | Yes (Excellent) | Yes |
Documentation | Yes (Medium) | Yes (Excellent) | Yes |
Pricing | Free | Node-Locked Enterprise: $1999
Floating Enterprise: $4499 TestExecute (runtime engine): $399 10% off the regular price promotion is valid until June 20, 2013 All licenses are perpetual and include one year of free upgrades/maintenance. Maintenance renewal usually provided with the 20-30% discounts |
Node-Locked Premium: €1480
Floating Premium: €3080 Node-Locked Ranorex Runtime: €389 Floating anorex Runtime: €780 All prices exclude VAT All licenses are perpetual and include one year of free upgrades/maintenance. Maintenance renewal costs from 20% to 50% of the license price |
Evaluation of the tools revealed that neither of them can recognize and process items added to the image in the SRS editor (notes, labels, etc.) as separate entities. More investigation in this area is required, but there is a grounded expectation that the problem will be solved.
Also, neither of the tools support functionality like Load/Stress testing and code performance. This functionality is usually delivered by the separate tools like load testing software (the former) and code profilers (the latter). SmartBear provides such tools (at the cost of separate licenses) while neither Ranorex nor Selenium have similar suggestions.
After thorough review and comparison of the tools, using the customer defined criteria, SmartBear TestComplete is recommended as the best fit to company needs. TestComplete provides all the functionality required by company, namely:
The features mentioned above distinguish Test Complete from the competitors and allow its easy integration into development environment and testing process of different companies. Another benefit is easy learning and test development process. The tool has intuitive interface which will significantly reduce learning curve for existing and new team members. Test development and support time is also minimal due to wide support of scripting languages, integration with other tools and combination of all testing in one package.
Ranorex tool seem to have practically the same functionality as TestComplete (except distributed and web services testing) but slightly higher price for the required licenses.
Selenium tool is good for automation of web browsers but provides no support for testing other application types. Because of the limited technical support and test development environment (which, in many cases, requires usage of separate extensions), the learning curve and the cost of tests development, support and execution is expected to be on the same level as for TestComplete and Ranorex, even despite the fact that Selenium itself is free.