Distributed Testing Overview

Author: SmartBear Software
Applies to: TestComplete 9 – 10

About Distributed Testing

Distributed tests are used to test applications that work with multiple clients simultaneously: client/server solutions, web applications and so on. Unlike ordinary tests that typically run on a single computer and do not interact with other machines, distributes tests consist of multiple parts that work on different computers and interact with each other.

Testing a distributed application involves testing its client and server parts. You can always test them separately, but distributed testing gives you a way to test them all together. For example, the tests running on client machines can make simultaneous requests to the server, and at the same time, the server part can track and assess the server performance.

Client-server applications testing  scheme

The crucial points of distributed tests are synchronization, administration, and configuration. For better analysis of results, the tool that you use for running distributed tests should accumulate results of the entire test run and provide the ability to analyze individual results of the needed test part.

Performing Distributed Testing With TestComplete

TestComplete supports distributed testing and provides features for starting tests on remote computers, exchanging data between projects during the test run, and synchronizing test projects.

A distributed test you create in TestComplete consists of several test projects that are executed simultaneously on separate computers and test certain parts of the client/server application.

In TestComplete terms, the project that controls the distributed testing procedure and specifies what projects are to be executed on remote computers is called a master project. The projects that run on remote computers are called slave projects. Each slave project in its turn can be the master for the next level of projects.

Distributed test structure

Remote computers, where your test projects run, can be physical machines, virtual machines or cloud computers. They must be available in your network and have TestComplete or TestExecute installed on them.

How TestComplete Distributed Tests Work

When you run a distributed test in the master project, TestComplete performs the following actions:

  1. Connects to the remote computers.
  2. Creates user sessions on them, if necessary.
    Note: Creating an interactive user session on a remote computer is an important step. If it is not created, the remote TestComplete instance will not be able to simulate user actions. You can log in to a remote computer manually or use an existing user session for testing, but in most cases, it is more convenient when your TestComplete tests are configured to log in to a remote computer automatically.
  3. Copies your test projects to the remote machines.
  4. In order for TestComplete to be able to run tests on a remote computer, your slave test project must be on that computer. You can copy the project files manually, or configure your master project so that it copies the files automatically.
  5. Launches TestComplete or TestExecute instances on the remote machines.
  6. Commands the remote TestComplete to TestExecute instance to execute test projects.
  7. Accumulates test results.

Let's see what features TestComplete offers to automate these tasks.

About TestComplete Network Suites

In order to create distributed tests in TestComplete, you use a special Network Suite project item that provides communication between projects involved in distributed testing. Both master and slave projects must have that item.

Below is a sample view of a test project's Network Suite item in the Project Explorer panel:

Network Suite item

The child items of the network suite item allow managing tests involved in distributed testing and running them on remote computers.

The sections below provide detailed information on those items.

Hosts Item

The Hosts item stores a collection of hosts. Each host corresponds to a remote computer, virtual machine or cloud computer where you want to execute a slave project.

Hosts item

You can specify the host properties required for testing on that host in the Hosts editor:

Hosts editor

For each host, you must specify the following information:

  • The network name or IP address of the corresponding physical, virtual or cloud computer. This data is specified in the Address column of the editor.
  • Hosts editor address column

  • Information on the user account under which the test will be run. The Domain, User name and Password columns of the editor contain the user account information.
  • Hosts editor account information

  • How a user session will be opened on a remote computer. TestComplete can open a user session automatically after a test starts, or you can open a user session manually so that TestComplete can connect to the existing session. You specify how a user session is opened in the Login mode column of the editor.

    Hosts editor login mode

    For more information, see the Opening User Sessions on Remote Computers section.

  • The folder on the master computer where slave projects are located. You can create and configure slave projects on your master computer and then copy them to the desired remote computers where they are to be run. The Source path column specifies the folder on your master computer that stores the slave projects.

    If your slave projects belong to your master project suite, leave the default [Path to the current project suite] value in the column. If they belong to another project suite on your master computer, specify the path to the folder containing that project suite in the column. Then you can copy a slave project manually from TestComplete IDE, or you can configure your network suite to copy them automatically after testing starts. For more information, see the Copying Projects to Remote Computers section.

  • The path to the folder on a remote computer to which slave projects will be copied. If your slave projects are stored on the master computer and you want to copy them to remote computers, specify the path to the folder, where they will be copied, on the remote computers in the Base path column of the Hosts editor. For more information, see the Copying Projects to Remote Computers section. The path specified in the Base path column is also used as a prefix to the Project file name property values of tasks (see below).

Jobs Item

The Jobs item stores a collection of jobs. Each job in its turn stores a collection of testing tasks (see below) that will be executed on remote computers.

Jobs item

When you run a network suite, the jobs are executed in series one after another. You specify the job execution order in the Jobs editor:

Jobs editor

The jobs are executed in exactly the same order they are listed in the editor. However, the tasks within jobs are executed concurrently.

Task Item

Each Task item specifies what testing tasks are to be run on what remote hosts.

Tasks item

Each task can correspond to a test, project or project suite that you want to execute on a remote computer. You configure tasks in the Tasks editor where you specify what projects and what tests are to be executed on which remote computers, what application will execute the tests, how the logs will be stored and so on.

Tasks editor

For each task, you specify the following information:

  • Where the testing task will be run. In the Host column of the editor, you specify the name of the Hosts collection’s item that corresponds to a desired computer.
  • Tasks editor hosts collection

  • The project or project suite you want to execute on the remote computer. If your slave projects belong to your master project suite, leave the default [File name of the current project suite] value in the Project file name column. Otherwise, specify the path to the project or project suite you want to execute. If the Base path property of the corresponding host is not empty, specify the name of the project or project suite taking into account its value.
  • Tasks editor project file name

  • The test you want to execute. If you want to execute only a particular test, without executing the whole project or project suite, specify the test’s name in the Test column of the Tasks editor.
  • Tasks editor test column

  • What application will be used during the test run. Since you can use either TestComplete or TestExecute to run distributed tests on remote computers, in the Remote application column of the Tasks editor, you select the application you want to use (note that the application must be installed on the remote computer).
  • Whether the TestComplete instance running on the host computer will be used for testing. The value specified in the Use previous instance column of the Tasks editor defines whether the master project will close the TestComplete instance running on the remote computer and start a new one or whether it will use the existing instance for testing.
  • The shutdown behavior. The Action after run column of the Tasks editor contains a value that specifies what TestComplete should do after it finishes executing a task on the remote computer. You can configure your task to close the TestComplete or TestExecute instance after the task execution is finished, or to reboot or shut down the remote computer.
  • How the test results will be managed. Network suites allow accumulating test results on the master computer. In the Copy remote log column of the Tasks editor, you specify how the results should be managed.
  • Tasks editor copying remote log

  • The number of minutes the master project will wait for the remote computer to respond. This value is set in the Host timeout column of the Tasks editor.
  • A description for the task. In the Tag column of the editor, you can enter a short description for any task of your network suite.

SynchPoints Item

The SynchPoints item stores a collection of synchronization points that you can use to make your projects interact with each other.

SynchPoints

For more information on synchronization points, see the Synchronization section below and the Synchronizing Distributed Tests article.

The host, job and task items of the master project must have property values specified. The items of the slave projects (unless a slave project is the master for the next level of slave projects) may have no property values specified, or there may be no items at all. The only exception is the synchronization point items. All the projects you want to synchronize by using a specific synchronization point must have the SynchPoints collection containing the appropriate synchronization point items.

Copying Projects to Remote Computers

If your slave projects have been prepared and are stored on the master computer, you can configure your master project to copy them to remote computers automatically, or you can copy them manually using the Hosts editor.

The path to the folder where a slave project is stored on the master computer is specified by the Source path property of the host. The default [Path to the current project suite] value of the property corresponds to the folder that stores your current master project suite.

The path to the folder to which the project will be copied is specified by the host’s Base path property.

Both these properties must contain the correct values in order for the project to be copied successfully.

Hosts editor base path property

If you want to copy projects manually, select the desired hosts in the Hosts editor and click Copy Project to Slave.

Copying projects to slave

If you want the slave projects to be copied automatically, you need to set the network suite’s deploy mode to Automatic.

Deploy mode

When testing starts, TestComplete first copies the projects from the master computer to the target folders on the remote computers and then performs testing.

Opening User Sessions on Remote Computers

Running a testing task on a remote computer requires that a user session be opened on that computer so that TestComplete can perform the necessary test operations.

TestComplete allows you to automatically open a user session on remote computers before testing starts. That is, there is no need to log in to each remote computer where you want to run tests and open user sessions manually there.

You specify how user sessions are to be opened on a host by using the Login mode property of that host:

Hosts editor login mode

The following values are available:

  • Automatic (RDP Session) – When testing starts, TestComplete automatically opens an interactive user session on the remote computer.
  • Automatic (Console Session) – When testing starts, TestComplete automatically opens a user session on the remote computer and then retargets it to the remote computer’s console, so that the remote computer remains unlocked.
  • Manual – TestComplete does not open a user session, but when testing starts, it connects to the existing user session using the specified user account information.

To open user sessions (or to connect to existing sessions) TestComplete uses user account information specified for each host in the Hosts editor or on the Parameters page of the network suite editor.

Synchronization

Distributed testing implies that the projects executed on remote computers interact and exchange data with each other. This can be done by using the following:

  1. Synchronization points – Special points used in distributed testing. Upon reaching such a point, a project pauses and waits for the other projects to reach that point too. These points allow you to make sure that some specific actions in your distributed test are executed on several remote computers at the same time.
  2. Critical sections – Specific marked parts of tests that allow only one project to access some resources at a time. That is, while a project’s test is working with resources within a critical section, those resources are not accessible to other projects that have requested the same critical section.

For more information on how synchronization works during distributed testing, see the Synchronizing Distributed Tests article.

The States and Events of Distributed Tests

TestComplete allows you to track the state of your distributed tests and perform certain actions depending on that state:

  1. Distributed test states - each task, job or the entire network suite executed during distributed testing goes through a number of states, like Idle, Initializing, Running and so on. TestComplete provides special methods and properties for getting the state of a task, job or network suite. For example, the WaitForState method delays the test execution until the desired task, job or network suite is in the specified state. You can use these methods and properties to synchronize your tests.
  2. Distributed testing events - When the state of a task, job or network suites changes, or when the value of a network suite variable is modified, TestComplete raises special network suite events. You can create custom handlers for these events and use them, for example, to synchronize your tests.

Sharing Data Among Network Suite Projects

To share data among projects running on remote computers, you can use the following:

  1. NetworkSuite variables – Store data between test runs. The variables can take string, integer, boolean and double values. Network suite variables are specified in the master project and are shared among the master and all slave projects. If one of the projects modifies a variable, the other projects will use a new value.
  2. Shared Files – You can store the data required for distributed testing in shared files that are available in your network and access them from tests. TestComplete provides special program objects to work with files in tests. In order to avoid file access conflicts, you can use synchronization approaches listed above, for example, critical sections.

Verifying Distributed Tests

Before you run your distributed test, you can verify it, that is, you can check whether it will run successfully. You can verify the whole network suite or its separate hosts, jobs and tasks.

Verifying distributed test

During the verification, the master project checks whether the required remote computers are available on the network, whether the corresponding user session can be opened, whether TestComplete (or TestExecute) is installed and can be launched, whether the specified project exists and so on.

Verification process

If the verification passes successfully, you can run the distributed test.

Running Distributed Tests

You can start distributed testing by running a separate task, job or the entire network suite of the master project. This can be done either from the TestComplete IDE of from tests (TestComplete provides special scripting objects to work with network suite items in tests).

When a test is running, TestComplete connects to the remote computers where the test is being executed and displays the remote desktop windows of those computers on the Run State page. For each remote computer, the page shows the name of the current task and the task’s current state. This allows you to monitor the test progress without leaving the TestComplete IDE.

Run State page

If synchronization is used, the Run State page also displays the current synchronization event.

Run State page synchronization event

Viewing Distributed Test Results

After distributed testing is over, TestComplete automatically gathers test results. The logs of the tests executed on remote computers are copied to the master computer and are embedded in the master project’s log. That is, to view slave projects’ results, there is no need to log in to remote computers where those projects were executed, you just need to click the corresponding log item in the Log Items panel of the test log.

Slave project test log

The master log also contains logs for each executed task or job as well for the entire network suite. The logs contain messages posted by those tasks, jobs and by the network suite during the test run.

Master project test log

The job log also contains summary information on the tasks executed by the job.

Job log

If you do not want to store slave projects’ logs on your master computer or if you want to view these logs only if they contain errors, you can choose the appropriate option in the Copy remote log column of the Tasks editor.

Copying remote log

Do You Have a Step-by-Step Tutorial?

For detailed information on how to create distributed tests and on what information you need to specify in network suites, see the Distributed Testing Tutorial in TestComplete help.

Conclusion

In this article, we have provided general information about distributed testing, about its basic concepts and told you how to create distributed tests in TestComplete. We’ve also explained how you can prepare your distributed testing projects in TestComplete, how to run tests, monitor the test execution and analyze the results.

For step-by-step instructions on how to create a simple distributed test, please see our Distributed Testing Tutorial article or the Distributed Testing Tutorial in TestComplete help

TestComplete includes a sample distributed test project that demonstrates how you can create and run distributed tests. The project is shipped with TestComplete and by default it resides in the <TestComplete Samples>\Common\Distributed Testing folder.

If you need more information on synchronization in distributed tests, see our Synchronizing Distributed Tests article.

If you are interested in trying TestComplete for free, you can download a trial version from our web site (http://smartbear.com).