Selecting Automated Testing Tools

Author: SmartBear Software
Applies to: TestComplete 10


Testing is a critical part of the software development process. There are a lot of different automated software testing tools currently on the market. Some of these tools can perform only specific kinds of testing and work only with specific languages, like, for example, Java application unit testing. Other products support a wide range of applications and offer more features and functionality.

When you start your search for the right automated software testing tool, it’s important to create a list of requirements to review when choosing a tool for evaluation. If you don’t have a list of requirements, you may waste time downloading, installing and evaluating tools that only meet some of your requirements, or may not meet any of them.

Creating a list of requirements is not easy, since you have to take all of your testing needs into account. We have prepared a list of requirements, which are described below. Of course, the list does not include every situation, but it is a good start. You can modify and add to it to get a list that reflects the specifics of your software testing environment.

Support for Various Applications and Platforms

Some automated testing tools support only one type of application, for instance, only Java or .NET applications, while other tools support more application types. In general, the best option is to choose a tool that supports all development tools currently used in your organization, or all tools that you plan on using in the near future. Even if you build your application with one compiler, you might use other compilers in the future.

Selecting one "multi-compiler" software testing tool instead of several "compiler-specific" tools may save you money, but you must decide which is more important and whether it will save you time and energy when creating tests for future application versions. Purchasing and using several software testing tools requires additional time for training and creating tests. Do you have that much time? Does your company have enough budgeted to pay for this?

TestComplete is one of automated software testing tools that supports a wide range of Windows desktop, Web, RIA, and mobile applications:

  • TestComplete supports testing of any Windows program created with any compiler. It works with both 32-bit and 64-bit Windows, .NET, Java and Web applications and services:
    • Any.NET application created for.NET Framework version 2.0 - 4.5:

      Microsoft: Visual C#, Visual Basic .NET, Visual C++.NET, JScript.NET.

      Non-Microsoft: C#Builder, Delphi for .NET (VCL .NET), PowerBuilder .NET 12, Python.NET, Cobol.NET, and others.

    • WPF (XAML) and XBAP applications (.NET Framework 3 – 4.5)
    • Java applications (including Swing, SWT and AWT applications)
    • Visual C++ 6.0-12.0 applications (including ATL, MFC, WTL and wxWidgets applications)
    • Visual Basic 6.0
    • Delphi and C++Builder applications (both VCL and CLX applications)
    • Qt 5 applications
  • TestComplete provides access to methods and properties of web page elements displayed in Internet Explorer (both 32-bit and 64-bit versions), Mozilla Firefox, Google Chrome, Apple Safari, Opera Software Opera (both 32-bit and 64-bit versions) or any browser based on the Microsoft Web Browser control.
  • TestComplete can test web pages on the client computer and is not dependent on the server technology. You can use it to test web pages generated by HTML5, IIS, ASP, ASP.NET, PHP, CGI, AJAX, Atlas and other applications.
  • TestComplete supports functional testing of web services (including WCF).
  • TestComplete provides extended support for testing Rich Internet Applications based on Flex, Flash, AIR, Silverlight and JavaFX.
  • TestComplete provides support for Windows Store applications. You can work with them on Windows 8, Windows 8.1, Windows Server 2012 or 2012 R2.
  • TestComplete supports testing of applications for several popular mobile operating systems:
    • Android 4.0.1 – 4.4.x
    • iOS 6.x, 7.0 - 7.0.x

This extended support means that TestComplete can access internal objects, methods and properties of the application under test. QA engineers can use all of these elements in their automated tests. For instance, your tests can execute an object’s internal methods to initiate a unit test.

TestComplete also provides access to methods and properties of web page elements displayed in Internet Explorer, Firefox, Google Chrome or any browser based on the Microsoft Web Browser control. It can test web pages on the client computer and is not dependent on the server technology. You can use it to test web pages generated by IIS, ASP, ASP.NET, PHP, CGI, AJAX, Atlas and other applications.

Support for Various Operating Systems

Choose an automated testing tool that supports as many operating systems as possible. Make sure that the tool supports recent operating system versions. The fact that your customers do not run your application under Windows 8 does not mean that they will not use it in the future. If your application works under this operating system, your software testing tool should be able to work under it as well. In other words, the more Windows versions the software testing tool supports, the better.

TestComplete supports both old and new Windows operating systems: Windows XP, Windows Vista, Windows 7, Windows Server 2003, Windows Server 2008, Windows Embedded Standard 2009 and Windows Embedded Standard 7, Windows 8, Windows 8.1 and Windows Server 2012, including both 32 and 64-bit versions of these operating systems.

Support for Mobile Devices

If you need to test mobile applications, check if the automated testing tool provides support for them and if it supports the test operations you need. You may need to know the answer to the following questions: Can it simulate swipes and long touches? Can it simulate touching physical buttons? Does it support gestures (multi-touch events)? Can it install applications on a device? Can it access sensor data of a device? Do you need to prepare applications for testing or can the tool work with unprepared applications? Does the tool require root (superuser) permissions? Does it support emulators?

It is also important to check whether the tool can access an application’s internal objects and controls. If it can, you will be able to create more flexible and accurate tests. Using object-based testing, you can, for example, change the text of a text box by using one command - there is no need to input text letter by letter.

TestComplete provides extended support for the Android and iOS operating systems. You can test applications running on physical devices, on emulators and on Android x86 virtual machines. You can record and play back gestures on Android devices that support multi-touch.

You need to compile your tested application in a special way and, if you are testing an Android application, install TestComplete Android Agent to call the internal properties and methods of the application's objects and controls. TestComplete Android Agent also supports geolocation testing, working with devices’ built-in sensors and simulating local symbol input. If no preparations have been made, you can create image-based tests. If you use an iOS device, there is no need to prepare it in a special way.

To test Android applications with TestComplete, you do not need root permissions, and if you use an iOS device, there is no need to jailbreak it.

You can run the same test on multiple mobile devices, install, launch, test and delete applications on them, copy data from a device and store it safely on your PC.

Support for Various Software Testing Types

When choosing an automated testing tool, check whether it supports the test types that you are going to run. Can it simulate user actions over the tested application’s user interface, in other words, perform functional testing? Can it run unit tests built into the tested application? Can it run tests created with NUnit, JUnit, DUnit or MSTest? Does it support distributed testing? Does it have special features for synchronizing tests that run simultaneously on several network workstations?

Also, if you test your products using both manual and automated methods, you may want to choose a tool that provides specific support for manual tests. Also, don’t forget other important factors, such as, data-driven testing support and the ability to use the tool for regression testing.

There are many automated software testing tools that support only one or two testing types, for instance, only unit testing or functional testing. Some vendors require separate products for different testing types. For instance, support for distributed testing can be implemented with separate products or add-ons. If you wish to support these testing types, you will have to add the price of these products or add-ons to the price that you are going to pay for the product.

TestComplete supports unit, functional (or UI), regression, data-driven, manual and distributed testing of Windows applications. It provides special support for unit tests created in NUnit, JUnit, DUnit, MSTest, which allows you to run these tests directly from your TestComplete projects. This lets you use unit tests that were created with these tools earlier in your development cycle.

To perform coverage testing, check the application for resource and memory usage and find bottlenecks in it, you can use another SmartBear product called AQtime. AQtime is tightly integrated with TestComplete, so you can run coverage, performance and memory leak tests directly from your TestComplete projects.

Creating Automated Tests Without Programming

An automated testing tool must let you create tests quickly and effectively. The test creation techniques must be sufficiently powerful to allow performing various automated testing tasks. At the same time the automated testing tool must be easy-to-use and clear even for inexperienced QA personnel. A good automated testing tool may support several ways of creating automated tests. For instance, it may provide a simple, easy-to-use visual automated test editor and some other features, like automated test scripts, for performing advanced tasks.

Many automated testing tools provide only scripting capabilities for creating automated tests. Creating automated test scripts requires programming experience and may be difficult for inexperienced testers. TestComplete provides a simple but powerful alternative to automated test scripts: keyword tests (also called keyword-driven testing).

Keyword tests consist of operations that simulate user activities on the application under test (like clicking the mouse buttons or typing on the keyboard) or perform other automated testing actions: launching tested applications, posting messages to the test log, running other automated tests (for instance, low-level procedures or unit tests), use checkpoints to verify specific application functionality and more.

Keyword tests are very simple and easy to create. Testers do not need programming skills to create them. You can record these keyword-driven tests or modify them visually in a special editor. The editor’s intuitive UI, wizards and dialogs help even inexperienced users create powerful automated tests with ease. Keyword tests are an ideal feature for creating functional tests quickly.

But keyword tests are not the only automated tests that can be recorded or created visually. TestComplete can record automated test scripts that can be modified in the code editor.

Automated Test Scripting

Many automated software testing tools also provide scripting capabilities. Scripts are very important when performing automated testing. They let users create automated tests for specific application features, provide a variety of automated testing actions and perform operations that cannot be performed with standard features built into the automated testing tool. Automated test scripts are useful even if the automated testing tool provides easier ways to create automated tests, because script code typically provides more powerful functionality, it lets you perform actions that other automated testing types cannot.

If you prefer scripting for your automated testing, it is important to pay attention to the scripting language the automated testing tool uses. Using standard languages, such as VBScript or JScript rather than their dialects, will decrease the time needed to get acquainted with a product. Another benefit is that you can find information on these languages from third-party web sites as well as MSDN.

Two other important features to think about when choosing a tool that uses scripting is the code editor and script debugger. Check whether the editor supports features that simplify and speed up script writing: syntax highlighting, outlining, code completion, bookmarks, Ctrl-click navigation between script routines and many others. Check if the tool’s script debugger includes the needed debugging functionalities, such as breakpoints, step-over and step-into execution, Call Stack window and Evaluate dialog, and so on. Using all of these features you can easily find and correct bugs or find workarounds.

TestComplete supports several scripting languages: VBScript, JScript, DelphiScript, C++Script and C#Script, so, you can choose the language you know best and thus save more time.

TestComplete includes a powerful code editor that supports all modern code editor features: syntax highlighting, outlining, code completion, code templates, bookmarks and many others. TestComplete’s built-in script debugger supports ordinary and conditional breakpoints, step-over, step-into and run-to-the-cursor features, the Watch and Call Stack windows and special features for evaluating variables and objects during the script run.

Recording Automated Software Tests

Functional (or user interface) testing implies simulating user actions on the application under test. Implementing these tests even for a small application is a tiresome process. When choosing an automated testing tool, make sure the tool can record and play back user actions on the tested application. This feature will significantly streamline test development.

TestComplete records user actions as keyword tests or automated test scripts, which you can then edit and play back an unlimited number of times. Fast test development is not the only benefit. Recording also helps you understand how to create a keyword test or how to write automated test script code, how to address application objects and how to perform testing actions on them.

Creating Cross-Browser Web Tests

One of the most important goals in testing web applications is to verify that the tested web application behaves correctly in various browsers. Creating a test that will work properly for different browsers is a challenging task, as the object hierarchy, element properties, user interface and browser behavior differ from one browser to another and even from one version to another version of the same browser. QA engineers have to keep in mind browser differences and use tons of conditional statements to make tests function properly. Sometimes QA teams even create separate tests for different browsers. All this makes tests complex and difficult to maintain.

TestComplete offers a number of specific features that make creating cross-browser tests much simpler and easier. TestComplete hides browser differences, so you work with web pages, web elements, browser dialogs and message boxes in a unified manner. You can record or create a test for one of supported browsers and then run the test with few or no changes in other supported browsers. Special test objects provide information on browsers installed on a computer, so you can easily create tests that iterate through and check your web application in all browsers available on the current test machine.

Creating Cross-Platform Mobile Web Tests

Another important goal in testing web applications is to verify that the tested web application works correctly on various mobile devices. These devices have different screen sizes and orientations. Some web sites may display different content, which depends on the device you use. It is impossible to have all the devices your application will run on. That is why, it is important to be able to test applications by using a mobile browser emulator.

TestComplete can test web applications and web sites for cross-platform compatibility without any mobile devices. It emulates mobile browsers on your computer using the Google Chrome web browser. Chrome uses the same browser engine as most mobile browsers do. You can define the screen size of the emulated browser and the user agent string (web sites use it to identify the browser). TestComplete provides several presets for emulating mobile browsers launched on popular devices, such as Apple iPhone, iPad, Samsung Galaxy S3 and other smartphones and tablets.

Creating Automated Tests That Are Resistant to Changes

Normally, you start testing your application while it is still in development. So, it is quite possible that after you make changes to the application, the current tests will become obsolete and you’ll have to update them to work with the changed application windows and controls. Modifying the tests may take the same amount of time as modifying the application. That’s why smart teams like to create flexible tests that will be resistant to changes made to the application under test.

One approach that that decreases dependence on changes made to the tested application is the object-oriented testing model. Each window or control of the application under test is represented by an appropriate object with the name, for instance, WinFormsObject("Form1"), so, to simulate user actions on this window or control, you call the object’s methods and properties. For instance, simulating a mouse click on a button can be represented as a call to the Click method of the test object that corresponds to this button. If you use this approach, there is no need to change your test after the tested application has changed. For instance, if developers change a button’s width, height or position, the Click method will still work. If you simulate a click by coordinates, changing the button’s size or position will cause changes in the automated test. Nevertheless, in some cases, you may need to work with coordinates as well, so, it is nice if the automated testing tool supports both object-oriented and coordinate-oriented methods.

Another approach that lets you create robust tests is object mapping. As a rule, automated software testing tools address applications’ objects by their names or captions. However, during application development, programmers may change controls’ captions, so you will have to modify your tests in order for them to match the changed controls. That is why a good automated testing tool should be able to map an object name to a custom name and use this custom name to address the object in tests. However, the tool must be able to map an object not only by its caption or identifier. In this case, if a developer changes the tested application, you will only have to modify the mapping conditions while the scripts and other tests remain the same.

TestComplete includes special features that let you map application windows, controls and objects by their properties. Testers can map a short name to the tested object and then use this mapped name in automated tests. This simplifies working with objects, makes the created tests shorter and easier to understand. There is another benefit of name mapping - it makes automated tests independent of object properties, which may change from one build to another. Also, TestComplete supports the object-oriented approach to simulating user actions. Simulating mouse events by coordinates is also supported.

Support for Controls and Objects

Windows applications use controls to display information and interact with users. The operating system provides dozens of various controls that can be used in tested applications: edit boxes, list views, check boxes, menus, buttons and much more. This number is also extended with custom components offered by third-party vendors. These can be non-standard components, like a tree list or grid, or third-party analogs to standard components: custom menus, toolbars and many others. When choosing an automated testing tool, make sure it includes special features that simplify working with controls used in your application. It is important to determine whether these features are both for standard and third-party controls.

Another important factor is access to methods and properties of the application’s internal objects and controls. It provides methods and properties for simulating user actions and checking the results of test actions. For example, you can use the internal methods and properties of a grid control to change the value in a grid’s cell and then check if the change was successful.

TestComplete provides specific methods and properties that greatly simplify your work with standard Win32 controls, web page controls (including those of popular web UI libraries, such as JQuery UI, YUI, GWT and MooTools), Java and JavaFX controls, Qt, WPF controls and third-party components provided by Microsoft, Borland, ComponentOne, Developer Express, Infragistics, Syncfusion, Telerik and other vendors. In addition, TestComplete’s automated test engine can access the internal methods and properties of application objects and use these methods and properties to perform the desired testing tasks.

Supports Various Test Data Sources

When performing functional testing, it is a common practice to simulate user input into the tested application’s data forms. Thorough testing implies that you feed varied data to the input fields and that the tested application processes it correctly. When choosing an automated testing tool, check which data formats it can use, such as text files, XML files, database tables, and others.

With TestComplete, you can obtain data from text and XML files, Excel worksheets and databases like SQL Server, Oracle and MySQL. TestComplete includes special program objects and test operations that allow you to iterate through data records in external files and easily create data loops in your automated tests.

You can create and populate these data storages manually, or you can use TestComplete’s Data Generator to generate the desired test data automatically and save it either to TestComplete’s Table variable or to the specified Excel file. This approach helps you decrease the time spent on preparing test data. For more information on using this feature, see the Using Data Generators section in TestComplete’s help.

TestComplete also allows you to use script arrays, variables and constants to keep data for automated tests.

Running Automated Software Tests and Synchronization

The state of the application under test and the execution environment may differ from the state and conditions you expected. For instance, some network resource may be unavailable, or the desired window is not created; the web page loads very slowly or an unexpected modal dialog box appears. These and other similar factors may cause the automated software testing tool to decide that the test has failed and it should be stopped. A good automated testing tool must be able to handle these situations in the appropriate manner. For instance, it must be able to "wait" for the needed windows, "close" unexpected dialogs automatically and perform other actions that a human would do to continue the test.

TestComplete can handle such situations properly. For example, methods that work with windows and web pages can wait until a window appears on screen or until a web page is loaded; methods that work with processes can wait until the desired process starts and so on. TestComplete also includes special features that let you specify how the tool should handle unexpected windows without interrupting the test run.

Automatically Compare Results of Automated Test Actions

An automated testing tool should include features for automatic comparing of test actions results. For instance, if during the test run the tested application saves a customer list to a file, the automated testing tool should be able to compare this file with a baseline copy and determine whether the save was successful and the saved data is correct. The automated testing tool should be able to compare objects’ and controls’ properties, images, files and other objects.

TestComplete includes special tools, objects and functionality that you can use to compare files, images and object properties. Comparison can be implemented manually, or generated by TestComplete when you are inserting a checkpoint in your test. TestComplete includes special dialogs and wizards that can be used to create checkpoints for properties, objects, files, images, databases, tabular controls, XML documents, web services and web pages. By using the powerful test implementation tools, you can create automated tests that will compare complex objects and can also use regular expressions for comparisons.

Logging Automated Test Results

An automated software testing tool should keep a log of all testing actions and provide a detailed report for them. The log of a high-end software testing tool must contain different types of messages: errors, warnings and informative messages. In this case, you can easily determine what went wrong and why the test failed.

TestComplete includes a powerful test log that contains warning, error or informative messages as well as files and images. The images are captured by the Test Visualizer subsystem and automatically posted to the log during the test run, or they can be posted using special operations or scripting methods. The captured images illustrate test commands and simplify analyzing the automated test’s results.

Exporting Reports

An automated testing tool should give users the opportunity to export test results to external files. For instance, it may be useful, when you need to view test results on other computers that do not have this testing tool installed. Also, you should check whether the testing tool can export the test log so that you can send the results to your boss or colleagues.

TestComplete stores test results in XML files and allows you to easily export the test log data to several formats (XML+HTML, Excel, MHT), pack the results with WinZip, WinRar or some other archive tool and send the results via e-mail. You can perform these operations from the TestComplete UI or from tests.

Also, TestComplete provides access to the stored results. Since the test log uses the XML format, you can parse the results and retrieve the desired information.

TestComplete also allows you to create a bug report for test log data and post the report to an issue-tracking system.

Additional Needed Automated Testing Tool Features

To create powerful software tests, the test recording and playback features may not be enough. Choose a tool that provides many additional features. For instance, you may need the following:

  • A library of useful objects and functions to perform various operations. For example, reading data from and writing it to files, working with databases, working with date and time values, regular expressions and many others.
  • The ability to call Win32 API functions.
  • The ability to call functions that reside in DLLs and .NET assemblies.
  • The ability to call routines that reside in Java classes.
  • The ability to work with COM servers.
  • Access to the shell (the WSH scripting object) and the ability to call the operating system commands.
  • OCR functionality to retrieve data from applications’ windows and controls.
  • The ability to create custom add-ons that will add specific functionality to your software test projects.

TestComplete includes all of these features and offers many others.

Testing Multi-Language Applications

If you have customers throughout the world and your company develops or is planning to develop localized versions of your applications, you have to check whether the software testing tool can work with localized applications and operating systems and whether it supports Unicode. If the tool does not support foreign languages, then, most likely, you will not be able to use it to test localized software.

TestComplete supports Unicode and, therefore, it is possible to test multi-language software with it.

Automated Testing Tool: Support for Teamwork

Testers are members of the development team. They work together and communicate with developers, managers and other various people. When choosing an automated testing tool, you may be interested in features that support teamwork such as concurrent work with project files, sharing script code among several projects and integration with source code control and issue-tracking systems.

TestComplete supports all of these features. It allows you to share your test projects with several testers. You can easily link script units from one project to another. You can check project files in and check them out from a source control system directly from TestComplete. Finally, you can add a bug report to an issue-tracking system like Microsoft Visual Studio Team Foundation, Bugzilla, OnTime or JIRA, directly from the test log viewer, and thus notify developers about the problem immediately.

Command Line and COM Automation Support

Choose an automated testing tool that you can launch automatically, for instance, at a certain time or after the application build is over. You will be able to do this only if your automated testing tool has the command line or COM interface where you can specify run parameters.

TestComplete supports command line arguments and can work with COM. Other applications, for instance, SmartBear’s Automated Build Studio, use these features to call TestComplete automatically.

Integration with Team System and Build Software

Automated software testing tools are used along with build programs. For instance, the application "builder" can run tests at the end of each build. When choosing an automated testing tool, check whether it can be used with build programs.

TestComplete can be integrated with build software, for example, with SmartBear’s Automated Build Studio and with Microsoft Visual Studio, including version 2013 of this popular Microsoft IDE. Also, TestComplete projects can be called from Visual Studio test projects. TestComplete also includes a special assembly that you can use to run TestComplete’s software tests from MSBuild projects.

Technical Support

A very important aspect of selecting an automated software testing tool is the quality of the technical support provided. Prompt responses to your messages prepared by a professional support team will assist in resolving problems and save you time and energy.

Pricing Policy

When choosing an automated software testing tool, you have to determine whether its price is reasonable. You should take into account not only the product price, but also the price of add-ons (if they are needed), the support fee as well as training and upgrade prices. To get this information, you may need to call the company’s office. Some vendors have special prices that you can only learn about when talking to a representative from their sales department over the phone.

SmartBear offers a clear pricing policy. You can find TestComplete prices on our website: TestComplete Pricing.

As you can see, SmartBear offers a feature-rich product at an affordable price. As a result, not only large companies use TestComplete in their everyday work, but also small teams can afford it.

Also, you can save money by purchasing TestExecute licenses rather than TestComplete licenses for all of the computers that you are going to run software tests on, but that are not for creating and debugging software tests.

Training is a paid option. For more information on this, see our TestComplete Training webpage.

Free Trial Version

An important factor is the availability of a trial version. Even if a product seems to meet your requirements, the best way to determine that is to download and use it before purchasing. Since you are new to the product, you may have questions and need some assistance with performing tasks. So, another important factor is whether the company provides support for trial-version users.

SmartBear provides support for trial-version users. The trial version is a full-featured version, but it has a 30-day limitation and there are limitations on the size of your scripts. Download the trial version of TestComplete now.