Automated Test Performance Tips

Author: SmartBear Software
Applies to: TestComplete 9 - 10

 

TestComplete is an automated software testing tool for Windows and web applications. Automated tests provide a number of benefits to QA engineers: automated tests are more consistent, they make the testing cheaper and faster than manual testing. QA teams can perform more tests when running automated tests, however, for some applications even automated tests may take a significant amount of time and run for more than a day. See other QA testing tools.

QA teams often wonder whether it is possible to increase the automated test execution speed and how they can do this. There is no simple answer to this question, since there can be multiple reasons for test slowdown. Usually, a lot of short delays occur during a test run, for example, while TestComplete is searching for certain application or window. You can shorten these delays by removing some TestComplete modules. For example, you can use only the Web module when working with web applications. However, there may be a lot of other delays, such as extra time spent on capturing and storing images, or pauses between simulated user actions. Keeping that in mind, we suggest the following approaches for improving the performance of your automated tests:

Recommendations Common for All Applications
    1. Disable Test Visualizer
    2. Filter Open Application Processes in TestComplete
    3. Disable Unused Plug-Ins in TestComplete
    4. Optimize the Wait Time in Automated Tests
    5. Adjust Delay Options for Automated Tests
    6. Use Alternate Methods in Automated Tests
Recommendations for Desktop and Web Applications
    7. Increase the Double-Click Speed in the OS
    8. Turn Off Windows UI Visual Effects
    9. Turn Off Performance Counters for Playback
  10. Optimize Recorded Low-Level Automated Tests
Recommendations for Desktop Applications
  11. Optimize Recorded Low-Level Automated Tests
Recommendations for Web Applications
  12. Optimize Testing of Silverlight Applications
Recommendations for Mobile Applications
  13. Increase Gesture Execution Speed

 

Recommendations Common for All Applications

1. Disable Test Visualizer

TestComplete includes Test Visualizer that captures images of your tested application for each UI action performed during test recording and playback. These screenshots help you better understand the test flow and analyze the test run sessions. However, it takes additional time to capture images, which decreases the overall test execution speed. Moreover, captured images may occupy a lot of disk space, especially if you run large tests that include lots of UI actions or if Test Visualizer is configured to capture the entire desktop. So, disabling Test Visualizer may increase the speed of your tests and allows you to save the hard disk space.

Since version 9.30, TestComplete provides in-depth control over Test Visualizer data. It contains specific options that allow you to choose whether Test Visualizer will collect both images and object data, images only or no data at all. These options can be set individually for recording and playback. By default, TestComplete 10’s Test Visualizer uses the "capture images only" mode for both recording and playback. This saves the hard disk space and speeds up testing.

You can disable Test Visualizer entirely by changing its options:

  • Open your test project in TestComplete.
  • Open the project editor and then switch to the Properties page.
  • Choose the Visualizer settings group.
  • Choose Off in the Collect Test Visualizer data during recording and Collect Test Visualiser data during test run groups.
  • Select File | Save from TestComplete’s main menu to save the changes.
     

In earlier versions of TestComplete, Test Visualizer options are not separated for recording and playback, so you need to disable it entirely. This is done from the same menu.

2. Filter Open Application Processes in TestComplete

You can limit the number of processes to be recognized as Open Applications by commanding TestComplete to only recognize the tested applications. In this case, TestComplete will not spend time trying to expose objects of other applications.

To specify the process filter:

  • Open your TestComplete project.
  • Open the project editor and then switch to the Properties page.
  • Choose the Open Applications | Process Filter settings group. This group contains a list of processes to be included in the process filter and the Process filter mode box.
  • Select Use only tested applications in this box to command TestComplete to recognize only those Open Applications that are included in your project’s list of tested applications.

    You can also specify the desired processes name in the Processes list and then choose the Use only selected processes mode. This will tell TestComplete that only the processes added to the list should be recognized as Open.
  • Select File | Save from TestComplete’s main menu to save the changes.
     

3. Disable Unused Plug-Ins in TestComplete

TestComplete can access internal objects, methods and properties of tested applications. We call these applications Open Applications. The support for some Open Applications and third-party controls is implemented by various plug-ins in TestComplete. TestComplete uses these plug-ins to associate the control (or window) with a scripting object. By disabling the plug-ins, you can narrow down TestComplete’s functionality and speed up the search for objects and methods. This typically concerns the Desktop module, because this module provides support for various Windows applications (C++, C#, Visual Basic, Java, Delphi and so on) that use controls of various vendors (Microsoft, Developer Express, Infragistics, Syncfusion, Component One and so on). Recognizing objects in all these applications can slow down your test execution. The problem may also affect your tests if you have several TestComplete modules installed (for instance, Web and Desktop) and run tests that need only one of the modules.

To disable a plug-in, select File | Install Extension from TestComplete’s main menu and unselect the plug-in’s check box in the subsequent dialog. We recommend that you disable the plug-ins that concern Open Applications and third-party controls. For instance, if you test a Visual Basic application that uses standard controls, then you can disable the .NET Classes and MSAA Support plug-ins as well as the plug-ins that provide support for third-party controls (like Infragistics or Component One). But use caution! Don’t disable the plug-ins that are used by your automated tests.

4. Optimize the Wait Time for Automated Tests

When simulating user actions on controls and windows, TestComplete’s automated tests may need to wait until the desired window, control or object becomes available. The test engine lets you specify the wait time for individual test commands as well as for the entire project:

  • Keyword tests. In keyword tests you specify the wait time for individual operations in the Auto-wait timeout column. By default, this column is hidden. To make it visible, right-click somewhere within the keyword test editor, select Field Chooser from the context menu and then drag the column from the ensuing window to the editor.

    The operations, for which the Auto-wait timeout value is specified, will use this wait time. The operations, for which the Auto-wait timeout value is 0, will use the wait time specified by the project settings (see below).
  • Scripts. In scripts, the wait time depends on the method that is called. If you pause the automated test’s execution by a call to a Wait method (for instance, WaitWindow or WaitProperty), the wait time is specified by the method’s Timeout parameter. If you call methods that simulate user actions on a window or control, then the wait time is specified by project settings.

By default, if TestComplete does not find the needed control or window, it will wait for it for some period of time specified by the Auto-wait timeout project setting. To modify this setting:

  • Open your test project in TestComplete.
  • Open the project editor and then switch to the Properties page.
  • Select the Playback settings page and reduce the value in the Auto-wait timeout box.

To speed up the automated test’s execution, you can reduce the wait time by decreasing timeouts specified by the Auto-wait timeout project setting and using operation-specific timeouts (or the Wait methods) at the appropriate places of your automated tests. If you do this and use the keyword test operations or Wait methods with timeouts shorter than those specified by the project settings, the speed of your automated tests will grow.

Note: Using too small timeouts may cause your tests to function improperly: the automated test may decide that the needed object does not exist while it should just wait for a little while longer.
 

Some methods that simulate user actions contain parameters that specify the time to wait. For instance, the Close method of the Window object has a parameter that specifies the number of milliseconds the method will wait until the specified window is closed. By default the delay is 2 seconds. To speed up this method, you can pass 0 as a parameter. For example, call wndObj.Close(0) and then call the WaitProperty method to check if the window is closed, by “waiting” until the window’s Exists property is False.

A possible scripting alternative to the Wait methods are loop statements that pause the automated test’s execution until the desired condition is met (for instance, until a property equals a value or until the desired window appears on the screen). The Wait methods are faster than the appropriate loop statements, so, to speed up your test, replace these loops with the appropriate Wait method (WaitProperty, WaitWindow, WaitChild, WaitProcess or any other). The example below works with a desktop application, but you can use the same approach to work with your web and mobile applications.

Click here to see the example  

If you use the WaitWindow or WaitProcess method just to check whether the desired window exists, then set the WaitTime parameter to 0. In this case, the method will not wait until the window (or process) becomes available and will return the result immediately. The same concerns the WaitProperty method.

5. Adjust Delay Options for Automated Tests

The Playback group of TestComplete’s project properties contains a number of settings that specify time values that are used during the automated test’s execution:

  • Delay between events — Specifies the number of milliseconds to wait after the automated test script simulates any user action and before it continues with the next action. The more this value is, the slower the automated test’s execution is.
  • Key pressing delay — Specifies the number of milliseconds to wait after each keystroke is simulated. The more this value is, the slower the keystroke simulation is.
  • Dragging delay — Specifies the number of milliseconds it takes the mouse cursor to pass 20 pixels while performing a dragging operation during automated test playback. The more this value is, the slower the dragging events simulation is.
  • Mouse movement delay — Specifies the number of milliseconds it takes the mouse cursor to pass 20 pixels during automated test playback. The more this value is, the slower the mouse moving event simulation is.

Note: The Key pressing delay, Dragging delay and Mouse movement delay options are not used for mobile tests. Use the Delay between events option to speed up mobile test execution.

Check the values of these options and if they are large for some reason, consider decreasing them. To achieve the fastest test playback speed, use the following values:

  • Delay between events: 0.
  • Key pressing delay: 0.
  • Dragging delay: 5 ms.
  • Mouse movement delay: 0.

Note however that using minimal values may cause your automated test to function improperly.

6. Use Alternate Methods to Simulate User Actions in Automated Tests

TestComplete has more than one way to simulate most user actions. Recorded automated tests normally use high level methods because they are more concise, easy to read and encapsulate more verifications. However, more basic commands are available that can make playback faster.

For instance, the Sys.Desktop.KeyDown and Sys.Desktop.KeyUp methods work on a lower level than the Keys method of the Window or onscreen objects. Similarly, the Sys.Desktop.MouseDown and Sys.Desktop.MouseUp methods work on a lower level than the Click routine, and the Click routine. The latter, in its turn, works on a lower level compared to the ClickItem, ClickItemXY and other similar routines.

To increase the speed of text input into edit controls, we recommend that you use the wText property instead of the Keys action. The wText property directly sets the control’s text using low-level system functions and works faster than the Keys action that actually types in the text.

Typically, lower-level methods make fewer calculations and verifications and therefore they work faster than higher-level methods. So, if you are going to execute a group of high-level methods, you can perform the needed verification operations before this group is executed and then use lower-level methods to simulate user actions in your automated tests.

The disadvantage of the described approach is that replacing higher-level methods with the lower-level ones can make the automated test complex, less stable and less portable.

Recommendations for Desktop and Web Applications

7. Increase the Double-Click Speed in the OS

To simulate the double-click events faster, increase the double-click speed in the operating system’s settings:

  • Open the Control Panel | Mouse window.
  • Switch to the Buttons tabbed page.
  • Drag the slider in the Double-click speed section to Fast.
  • Press OK to save the changes.
     

This will not only increase the speed of double-clicks, but also speed up single clicks that are simulated at two points that are close to each other. When simulating these single clicks, TestComplete pauses the execution after the first click and waits until the double-click timeout elapses. Otherwise, the tested application may confuse two consecutive single clicks with a double click.

Note however that fast double-clicks may be inconvenient for the tester working on the computer.

8. Turn Off Windows UI Visual Effects

The Windows operating systems include a number of settings that let users apply various visual effects to menus, tooltips and windows. To increase the test speed, use the settings configuration that will provide the best performance.

The way you modify the settings depends on the operating system you use:

  • In Windows 8 and 8.1, Windows 7, Windows Vista or Windows Server 2008, 2012 and 2012 R2:
    • Open the Control Panel | System window.
    • Select Advanced system settings from the Tasks panel. This will invoke the System Properties dialog.
    • In the dialog, switch to the Advanced tabbed page.
    • Press Settings in the Performance section of the page. This will invoke the Performance Options dialog.
    • On the Visual Effect page of this dialog, select Adjust for best performance.
    • Press OK to save the changes and to close the dialog box.
  • In Windows XP or Windows Server 2003:
    • Open the Control Panel | System dialog box.
    • Switch to the Advanced tabbed page.
    • Press Settings in the Performance section of the page. This will invoke the Performance Options dialog.
    • On the Visual Effect page of this dialog, select Adjust for best performance.
    • Press OK to save the changes and to close the dialog box. You will return back to the System dialog box.
    • Press OK to close the Control Panel | System dialog box.

The disadvantage of the described approach is that the changes may affect the tested application and it will behave differently than what you expect.

9. Turn Off Performance Counters for Playback

You can use TestComplete to monitor the performance of your test computer via a special set of counters embedded in the Microsoft Windows operating system. All the counters that are available for Windows Performance Monitor are available to TestComplete, including the counters that allow you to monitor CPU load, memory and disk usage, disk errors and so on. While these counters help you optimize tests, it takes time to obtain this kind of information, which can slow down the test execution, especially on remote machines.

You can speed up the test execution by removing performance counters:

  • Select Tools | Current Project Properties from TestComplete’s main menu.
  • Select the Performance Counters category on the left.
  • In the subsequent window, choose the counter you want to remove and clickRemove.

This will remove the performance counter from the test project.

After removing performance counters, you will be unable to monitor the performance of your application, however, the test execution will speed up.

10. Optimize Recorded Low-Level Automated Tests

Normally, TestComplete records mouse and keyboard events without any delays to optimize the playback speed. The low-level procedures, however, record all user actions along with the delays between each action. If you are using the low-level automated test type then it may be possible to edit them to save time.

Recorded low-level procedures capture the exact actions and timing of the tester so they can often contain extra mouse movements that are not needed to playback the test. And they often capture delays between events that are longer than necessary.

You may be able to increase low-level automated test speed by deleting non-critical mouse movements and decreasing the delay time between events. If you try this tip, be sure to verify that the test playback still works as expected since your edits may significantly modify the low-level procedure behavior.

Recommendations for Desktop Applications

11. Turn Off the Debug Agent in TestComplete

Open Applications expose objects, methods and properties defined in the application code, so you can use these objects, methods and properties in your automated tests. Some applications are always Open to TestComplete while others are “closed” by default and you typically need to compile them with debug information to make them Open. Debug information allows TestComplete to detect the names of object methods and properties, names and types of method parameters and to obtain other critical information.

To parse debug information and get access to the methods and properties of the internal objects of applications created in Visual C++, Delphi and C++Builder and applications created with the Qt library, TestComplete uses a special subsystem called Debug Info Agent. Reading and parsing debug information can take some time, and this can significantly slow down the automated tests execution. So, if your automated tests do not use internal methods and properties of the application, you can disable the Agent and decrease the time TestComplete needs to obtain information about objects’ methods and properties.

To disable the Agent:

  • Open your test project in TestComplete.
  • Open the project editor and then switch to the Properties page.
  • Switch to the Open Applications | Debug Agent page.
  • Clear the Enable Debug Agent check box.
  • Select File | Save from TestComplete’s main menu to save the changes.
     

Recommendations for Web Applications

12. Optimize Testing of Silverlight Applications

TestComplete can obtain information about controls and objects of a tested Silverlight application in two different ways:

  • Via the Microsoft UI Automation technology.
  • Via TestComplete’s Silverlight Open Application Support plug-in.

To improve the performance of your automated tests, we recommend that you use the UI Automation engine to perform functional testing of Silverlight applications. With the Microsoft UI Automation technology, TestComplete partially exposes controls of Silverlight applications and does not provide access to internal objects and members, while the Silverlight Open Application Support plug-in allows TestComplete to fully expose Silverlight objects, including objects that are not really important for functional testing. This can slow down the search for objects and methods during test recording and playback, and thus, reduce TestComplete performance.

If you actually need to access Silverlight objects by using the Silverlight Open Application Support feature, you can increase the speed of automated test execution by reducing the number of exposed objects. To do this, enable the Extended Silverlight object tree project option:

  • Open your test project in TestComplete.
  • Open the project editor and then switch to the Properties page.
  • Choose the Open Applications | Silverlight settings group.
  • Clear the Extended Silverlight object tree check box.
  • Select File | Save from TestComplete’s main menu to save the changes.

Recommendations for Mobile Applications

13. Increase Gesture Execution Speed

You can use TestComplete to simulate gestures (multi-touch events) on Android devices that support a touch interface (simulating gestures on iOS is not supported at the moment). To record a gesture, you use the touch screen of the physical device, and you cannot change the gesture after recording it. Gestures may contain unnecessary delays, which slows down the test execution.

If you cannot change a gesture, you can increase the speed of the gesture playback. Note that if you increase the execution speed, the accuracy of the gesture will decrease.

To increase the gesture execution speed:

  • Open the Gestures Editor by double-clicking the gestures collection.
  • Select the Playback Acceleration (%) cell near the gesture you want to speed up.
  • Press F2 or click the cell twice.
  • Use the subsequent Playback Acceleration dialog to set the needed speed.

Are These Tips Helpful?

These optimization techniques are general, and we hope that they will be helpful to most users. However, you should remember that every automated test project is unique and can be improved in certain areas. So, some of the described techniques may have little effect on your project while others may help tremendously. If your optimization efforts succeed, feel free to share your experiences with other users on the TestComplete forums.