Synchronizing Distributed Tests

Author: SmartBear Software
Applies to: TestComplete 9-10

Distributed tests are tests that consist of several parts performed on different computers. Test parts interact with each other and strive for a single common result.

TestComplete supports distributed testing and provides features that allow you to create and manage distributed tests for any types of supported applications: desktop, web and mobile. For more information about this, see the Distributed Testing Tutorial article.

To ensure that all of the distributed test parts are performed in a certain order, you have to synchronize them.

TestComplete provides several features for distributed tests. This article provides detailed information about these features and demonstrates how you can use them in your distributed tests:

Synchronization Points in Distributed Tests

To synchronize the execution of different projects participating in distributed testing, you can use synchronization points (synchpoints). Synchronization points let you run the distributed test parts on several remote computers at the same time. In this case, the execution of participating projects is paused until all of these projects have reached the same synchpoint. After that, the projects simultaneously proceed. This allows you to make sure that different projects start performing specific actions at the same time.

In order to use synchronization points in your distributed tests, you need to add them to the SynchPoints collection in the NetworkSuite project item of each project you want to synchronize. Then, you need to specify places within the tests at which the projects should synchronize. You can do this in two ways:

  • By calling the NetworkSuite.Synchronize method in the desired places in your tests and passing the name of the desired synchpoint to it.
  • By calling the WaitFor method for the desired synchpoint in the desired places in your tests.

Synchronization points

When the project reaches the NetworkSuite.Synchronize or SynchPoint.WaitFor method, the distributed test will pause and wait until the other distributed test projects that use the same synchpoint call the Synchronize or WaitFor method. At this point, the state of these projects changes to the name of the used synchpoint.

Synchronizing

Using the Timeout parameter of the NetworkSuite.Synchronize or SynchPoint.WaitFor method, you can specify a time limit so that the given project waits for synchronization.

As soon as the last project calls the NetworkSuite.Synchronize or SynchPoint.WaitFor method, the execution of all distributed test projects will simultaneously continue, and their state will change back to Running.

Let us examine a simple example that demonstrates how you can use synchpoints to synchronize your tests.

Suppose, we have a distributed test that consists of a master project, Master.mds, which manages two slave projects, RemoteProject1.mds and RemoteProject2.mds.

First, we add synchpoints to each project we want to synchronize. To add a synchpoint, follow the steps below:

  • Open the project you want to synchronize in TestComplete.
  • Right-click NetworkSuite | SynchPoints in the Project Explorer panel and select Add | New Item from the context menu.
  • In the ensuing dialog, enter the needed synchpoint name, for example, NewSynchPoint, and press OK.

The same way, we add the synchpoint to the next project to be synchronized.

Adding synchpoints to a project

Then, we add the NetworkSuite.Synchronize("NewSynchPoint") method to each slave project before the code we want to synchronize.

RemoteProject1 Code (VBScript)

'Synchronize the projects using the NewSynchPoint synchronization point

NetworkSuite.Synchronize(NewSynchPoint)

'Actions to be synchronized

Log.Message("The slave projects are synchronized")

RemoteProject2 Code (VBScript)

'Synchronize the projects using the NewSynchPoint synchronization point

NetworkSuite.Synchronize(NewSynchPoint)

'Actions to be synchronized

Log.Message("The slave projects are synchronized")

To synchronize keyword tests, you can use the Run Code Snippet, Run Script Routine, or Run Test operations to call the NetworkSuite.Synchronize or SynchPoint.WaitFor scripting method. Add one of these operations to your keyword test before the operations you want to synchronize.

When one project reaches the NetworkSuite.Synchronize or SynchPoint.WaitFor method, its execution is paused, and it waits for the next project to reach the same synchpoint.

As soon as both projects reach NewSynchPoint, they continue being executed simultaneously.

Using Network Suite Variables in Distributed Tests

The NetworkSuite project item allows you to define a list of specific variables called network suite variables (or NetSuite variables).

These variables are common for the master and salve projects involved in distributed testing. If one project changes a network suite variable, the others use the new value. The network suite variables store data between test runs.

You can use these variables to exchange data among projects involved in distributed testing and to synchronize the execution of these projects.

Network suite variables are created and managed on the master project. When the test starts, slave projects obtain the variables from the master project and use them.

To create or modify network suite variables, follow the steps below:

  • Open the master project in TestComplete.
  • Open the NetworkSuite editor. To do this, double-click the NetworkSuite item in the Project Explorer panel.
  • In the NetworkSuite editor switch to the Variables panel.

Variables page

By using this page, you can add new variables to the project or modify the existing ones, specify their names, descriptions, default and local values. The default value is used when you open the master project on a computer for the first time, and it is common for all computers that use the project. The local value depends on the computer where the project is opened.

TestComplete includes several specific events that allow using network suite variables for synchronization.

For example, the OnNetVarChange event occurs when the value of a network suite variable changes. Thus, using this event in your distributed testing scripts or keyword tests, you can perform certain actions when changing network suite variables. To learn how to create an event handler for an event, see below.

TestComplete also provides the WaitForNetVarChange method that allows you to pause the distributed test execution until the network suite variable gets the needed value. The method returns True if the expected value is set to this variable and False if it is not. By using this method, you can stop the test execution if the variable does not get the needed value during a specified time period.

(VBScript)

' Halts the network suite’s execution

' If the VisualTestPassed variable does not get the True value

' Within the specified period

If NetworkSuite.WaitForNetVarChange("VisualTestPassed", True, 600000) = False Then

NetworkSuite.Stop

End If

To change suite variables during a test run, it is recommended to use critical sections. Otherwise, there may be various problems or invalid results. For more information about this, see Critical Sections below.

Using Network Suite Methods in Distributed Tests

You can synchronize your distributed tests using special network suite methods provided by TestComplete.

To suspend the test execution until the job, task, or network suite gets the expected state, use the Job.WaitForState, Task.WaitForState, and NetworkSuite.WaitForState methods.

(VBScript)

' If the network suite does not obtain the Running state

' Within the specified time period

' The test execution stops

If (not NetworkSuite.WaitForState(ns_Running, 1000 * 60)) Then

Runner.Stop

End if

To suspend the test execution until a particular network suite variable changes its value, use the NetowrkSuite.WaitForNetVarChange method in your project.

You can also use the Job.State, Task.State, Host.State, or NetworkSuite.State properties to check the state of the jobs, tasks, hosts, and network suite.

Using Network Suite Events in Distributed Tests

When a distributed test is executed, it passes through several testing stages, for example, initializing, verifying, running, synchronizing, stopping, and others. Some of them are executed consecutively, while others occur only when particular conditions are met.

In TestComplete, states of the network suite indicate stages of the distributed test. During the test run, you may need to perform actions when the state is changed. For that, TestComplete includes special events that occur when the state of the particular job, task, or host is changed: OnNetJobStateChange, OnNetTaskStateChange, and OnNetHostStateChange. You can create handlers for these events, and this way, perform custom actions when events occur.

There is one more event that you may use in distributed tests - OnNetVarChange. We spoke about it earlier when describing netsuite variables. The test engine raises this event when a network suite variable changes.

In order to process network suite events, a TestComplete project must include an Event control that contains the events. This control can be created only if your project contains the Events project item:

The Events project item

By default, it is added to the project. If for some reason you have deleted it, you should include it in your project. To do this:

  • Right-click the project node in the Project Explorer panel and select Add | New Item from the context menu. This will call the Create Project Item dialog.
  • In the dialog, select the Events project item, specify its name and location in the appropriate edit boxes and press OK to save the changes.

The Events project item will appear under the project node in the Project Explorer.

By default, after the Events project item has been created, it includes one Event control, GeneralEvents. This control contains a number of TestComplete events that you can use to create event handlers and perform certain actions. To see which events are included in this control, double-click the GeneralEvents node in the Project Explorer. TestComplete will show the Event Control editor in the Workspace panel:

General Events

The Event Control editor contains two lists: Available Events and Events to Handle. The Available Events list displays the events of the object that is specified in the Source object box. These events can be added to the Event control. The Events to Handle list displays the events that were added to the control.

The network suite events are not added to the Events to Handle list by default. To add these events to the list, right-click Network Suite events in the Available Events list and select Add selected from the context menu.

Then, you can create an event handler routine:

  • Select the needed network suite event from the Events to Handle list. Activate the Event Handler cell and click the New button displayed within this cell. TestComplete will show the New Event Handler dialog.

The New Event Handler dialog

  • In the dialog, you can specify the event handling routine or keyword test. To create a new event handler, follow any of the steps below:
    • To create an event handling routine, specify the unit that will hold it and the routine’s name (for example, GeneralEvents_OnNetVarChange). Click OK.
    • To create an event handling keyword test, select the KeywordTest item in the Test Containers tree, specify the test name, and click OK.

TestComplete will create an event handler and display it in the Code Editor or in theKeywordTest Editor. You can specify the actions to be performed when the specified event occurs.

Critical Sections in Distributed Tests

In distributed tests, it is quite possible that several projects use the same resources simultaneously. Network suite variables are shared for all projects engaged in distributed testing, and any project can change the value of any network suite variable.

Suppose, you have two projects that change the same network suite variable simultaneously or one right after the other, but you cannot say for sure what value the variable will have in the end. This may cause a loss of your test results or other unexpected and undesired behavior.

To solve the problem, use critical sections. A critical section is a specific marked section of code which can be performed by one project at a time. To avoid losing test results, we simply put the code that changes the value of the variable within a critical section. While one project works with the critical section, the resources it works with (in our case, the network suite variable) are accessible only to this project. Other projects that are going to use the same resources will have to wait until the first project leaves the critical section.

Critical sections can include not only code that modifies network suite variables, but also code that writes to any critical resource available from distributed test scripts and keyword tests (files, objects, programs, and so on).

Critical sections

Thus, critical sections let you synchronize projects that participate in distributed testing.

TestComplete provides special methods for the NetworkSuite program objects to enter and leave a critical section.

To enter the critical section, use the NetworkSuite.EnterCriticalSection method. To leave the critical section, use the NetworkSuite.LeaveCriticalSection method.

When calling these methods, you need to specify the unique name of the critical section you want to enter or leave.

(VBScript)

' Occupy a critical section

NetworkSuite.EnterCriticalSection("ChangeVar")

' Change the network suite variable within the critical section

NetworkSuite.Variables.FilesCount = NetworkSuite.Variables.FilesCount + 1

' Release the critical section

NetworkSuite.LeaveCriticalSection("ChangeVar")

When a project accesses the specified critical section, its state changes to Running in CS. The other projects that are requesting this critical section will wait for this project to leave it. They will have the Waiting for CS state.

Using the Timeout parameter of the NetworkSuite.EnterCriticalSection method, you can specify how long a project will wait for access to the needed critical section.

When the project leaves the critical section, the other projects can access it. They will enter the section in the order they requested it.

Conclusion

TestComplete allows you to create distributed tests and provides several features for synchronizing the execution of different projects participating in the distributed test. By synchronizing your distributed tests, you manage the way your tests react in certain situations when testing a client-server application. If you are interested in using distributed testing with TestComplete or would like to see what other features TestComplete has, download and try it for free.