Starting from TestComplete version 14.81, the Network Suite functionality is deprecated. Though it will not be removed from the product, we do not recommend that you use it to manage distributed test runs. We recommend that you use a CI/CD system instead.
This section explains how to migrate your distributed test runs from Network Suite to Azure DevOps.
In This Document
Prerequisite: TestComplete Test Adapter extension for Azure DevOps/Team Foundation Server
To run TestComplete tests from Azure DevOps pipelines easily, you can use the TestComplete test adapter. The adapter must be installed on all remote computers (agents in Azure DevOps terms), where your tests will run. To get the adapter, install the TestComplete Test Adapter extension to your Azure DevOps server. You can do it from the Microsoft Marketplace or, if you have a private Team Foundation Server, you can install the extension locally:
In a web browser, navigate to your team project web portal.
Click , then click Browse marketplace.
In the marketplace, locate the TestComplete Test Adapter extension.
Click Get if free:
Select the team project collection for which you want to make the extension available and click Install.
Copy the extension file (smartbearsoftware.install-testcomplete-adapter-task-for-tfs-15-40.NNN.vsix ) to a location available to your server.
By default, the extension resides in the TestComplete folder:
<TestComplete 15>\TFS Integration\smartbearsoftware.install-testcomplete-adapter-task-for-tfs-15-40.NNN.vsix
Connect to your team server. Click , and then click Manage extensions.
On the Extensions page, click Browse local extensions.
Click Manage Extensions. On the resulting Manage Extensions page, click Upload extension:
In the resulting dialog, select the TestComplete test adapter extension file and click Upload:
Click the extension in the list, and then click Get it free:
To make the extension available for your team project, select the collection to which your team project belongs, and then click Install:
How Network Suite elements translate to Azure DevOps
The image below shows an example of migrating TestComplete tests from Network Suite to Azure DevOps pipeline:
TestComplete: To manage test runs by using the Network Suite functionality, you use a special project (so-called “master”). It stores the list of computers where your tests will run (hosts), and specifies jobs and tasks to be run on hosts. It also defines synchronization points and Network Suite variables. You use this project to start and monitor the test run and to gather results from remote machines.
Azure DevOps: In Azure DevOps (and other CI/CD systems), a special TestComplete project is not required. You configure and manage test runs directly from Azure DevOps pipelines.
If you are new to Azure DevOps and pipelines, we recommend that you read the pipeline documentation on the Microsoft website:
TestComplete: In Network Suite, the remote computers, on which your tests are running, are called hosts.
Azure DevOps: Azure DevOps (and other CI/CD systems) support managing a pool of computers on which various tasks are run. Such computers are called agents. In Azure DevOps, agents can be Microsoft-hosted and self-hosted.
For TestComplete tests to be able to simulate user actions over the tested application user interface, the agent must be configured to run in the interactive mode.
For instructions on how to configure agents, see below.
TestComplete: In Network Suite, a job is a collection of tasks that run in parallel on specified hosts. Network Suite can have one or several jobs. Jobs are run in a series.
Azure DevOps: Azure DevOps pipelines also have jobs. A single job runs on a single agent and executes one or several steps. The job always executes the steps in a series, one after another. Jobs can be grouped into stages.
To run tasks simultaneously, you can configure parallel jobs by using various parallel strategies. The pipeline will create several instances of a job and run them concurrently.
To organize parallel job runs, your Azure DevOps organization must have parallel jobs purchased. To learn more, see Configure and pay for parallel jobs in the Azure DevOps documentation.
If you are not interested in parallel runs, you can run your jobs and tasks in a series.
TestComplete: a Network Suite task is a test to be run on a host as part of a job. Task properties specify the TestComplete project that contains the test you want to run, the test to run, the computer on which you want to run this test, and the test runner tool (TestComplete or TestExecute), and the way you would like to handle test results.
Tasks that belong to the same job run in parallel.
Azure DevOps: In Azure DevOps pipelines, a job consists of steps: tasks or scripts that run one after another. To run TestComplete tests as part of a pipeline, use the Visual Studio Test task.
Crucial differences from the Network Suite tasks:
You cannot assign the Visual Studio Test task to an agent directly. To do this, you use capabilities and demands. See below.
Unlike the Network Suite tasks that run in parallel, the Azure pipeline runs the tasks one after another in the order the pipeline specifies them. To have several tasks run in parallel, you can use parallel jobs. You can also configure the pipeline to distribute tests among several agents.
Distributed test in Azure DevOps – Example
To create and run a distributed test, you perform the following tasks:
Configure remote computers (establish the connection between the CI/CD server and the remote computer, install the tested application and test runner on the remote machine, and so on).
Copy test projects and data files to remote machines.
Run the test runner on each remote computer.
The following sections describe how to perform these tasks in Azure DevOps.
About the TestComplete project to run
Suppose we have TestComplete tests that we run on remote computers by using the Network Suite functionality. These projects are organized in the following way:
The project called
Maincontrols the test runs:
It has the Hosts collection that includes two hosts called
RemoteAgent2. For them, it specifies credentials used to open interactive user sessions.
Has one Job collection called
Job1. The collection includes two tasks that specify the tests to execute on each of the hosts.
RemoteAgent1. It executes
Test1that belongs to
RemoteAgent2. It executed
Test2that belongs to
Two test projects to be run on remote computers. These projects are called
Let’s examine how you can migrate this project set up to Azure DevOps.
1. Prepare and configure Azure DevOps
1.1. Install the TestComplete Test Adapter extension
To enable Azure DevOps pipelines to run TestComplete tests, install the TestComplete Test adapter extension to Azure DevOps. For detailed instructions, see above.
1.2. Create an Azure DevOps project
In Azure DevOps, if you still do not have a project to which you want to add your TestComplete automated tests, create one.
To learn how to do this, see About projects in the Azure DevOps documentation.
1.3. Configure pools and agents
An agent is a remote computer (physical or virtual), where Azure pipelines run jobs and tasks. To run automated tests, you will need agents that meet the following requirements:
Agents must run as an interactive process with the auto-logon enabled. See below to learn how to configure Azure DevOps agents.
Agents must have TestComplete or TestExecute installed on them. The TestComplete (TestExecute) instance must also be licensed.
Agents must have access to your tested application and all other files your tests use.
Register an agent pool
In Azure DevOps, agents are organized into pools. When you run a pipeline, it runs on an agent it selects from a pool that meets the required capabilities (see below).
If you still do not have a pool, create one, or ask your organization administrator to create a pool that you will use to run automated tests. To learn about creating and managing pools, please see Agent pools in the Azure DevOps documentation.
Download and configure agents
For Azure DevOps to be able to access computers where your tests will run, the agent must be running on them. To configure and run the agent:
Log in to the computer that you want to use to run your TestComplete tests.
There, launch a web browser, navigate to your Azure DevOps project, and then go to Project settings > Agent pools:
Click New agent.
In the resulting Get the agent dialog, switch to the Windows tab and click Download to get the agent installation files.
After downloading the files, unpack them and run the config.cmd file.
Follow the instructions to configure the agent. For detailed instructions, please see Download and configure the agent in the Azure DevOps documentation.
If your TestComplete tests interact with the GUI, make sure to configure the agent to run as an interactive process.
In our example, we will add RemoteAgent1 to the Test environments pool and configure the agent to run as an interactive process.
After the agent is configured, run it by running the run.cmd file. While the agent is running, it is available to Azure DevOps and can run jobs and tasks.
Repeat the steps to configure other agents.
Install TestComplete (TestExecute) on agents
On every agent you will use to run tests, install TestComplete or TestExecute. These TestComplete (TestExecute) instances must be licensed. The easiest way is to use a Floating user license.
Assign capabilities to agents
For the Azure DevOps pipeline to select proper agents to execute jobs and tasks, capabilities and demands are used. Capabilities are a collection of name-value pairs that specify what an agent can do or what software it has installed. A demand is a condition that specifies on which agent the task or a job can run.
To specify agents on which your TestComplete tests will run, you can create a capability named
TestRunner and set its value either to TestComplete or TestExecute (depending on the appropriate software installed):
In Azure DevOps, open your project and click Project settings.
Click Agent pools and then click the pool that contains agents where your tests will run.
Click an agent and switch to the Capabilities tab.
In the User-defined capabilities section, click the Add a new capability link.
Enter the capability name and value and click Add. In our example, we will create the
TestRunnercapability and set its value to TestExecute.
Repeat the steps to add the capability to other agents.
2. Place TestComplete projects to your Azure DevOps repository
For Azure DevOps agents to be able to access your TestComplete projects to run them, add the project files to your Azure project’s repository. How you can do it, depends on the repository you use. For detailed instructions on adding files, please see Supported source repositories in the Azure DevOps documentation.
3. Create an Azure pipeline to run the test
Note: In Azure DevOps, you can create and configure the pipeline either by using the visual editor or as a YAML file.
In your Azure DevOps project, go to Pipelines and then click New pipeline.
Specify the repository where your TestComplete project files reside.
Specify jobs that will run your TestComplete tests. In our example, we will have Job1:
Specify the agent pool and agents on which you want to run tests. In our example, we will run the job on agents that have the
TestRunnercapability set to TestExecute. It is the capability we have assigned to agents earlier:
If you do not have Visual Studio installed on your target agents, you can add the Install Visual Studio Test Platform task to the pipeline. The task will install the test platform that allows running tests on computers where Visual Studio is not installed:
Add the Enable support for TestComplete tests task to the job. This task installs the TestComplete test adapter to your agent and enables the TestComplete test support.
If needed, set the preferred test runner to use to run tests.
Specify how you want to handle test logs: upload them only when the tests fails, completed with warnings, or always.
Add the Visual Studio Test task to the pipeline and configure it to run TestComplete tests:
Configure the task to select tests by using assemblies.
Specify the project suite file (.pjs) that contains your tests and that the task will run.
Specify the folder on your agent where the task will search for TestComplete project files. By default, it will search the files in the agent’s working folder.
If needed, you can use a filter expression to specify test items of your TestComplete project that you want to run.
In our example, we will use the expression to run Test1 test item of the TestComplete project.
You may also specify the test run name to which test results will be assigned.
- job: Job1
- task: VSTest@2
testSelector : 'testAssemblies'
testRunTitle: 'Automated tests'
To run other test projects, you can add more Visual Studio Test task to the job. To run tests in parallel, you can enable the Parallelism for the created Job1.
Your organization must have parallel jobs. To learn more about purchasing parallel jobs, see Configure and pay for parallel jobs in the Azure DevOps documentation.
After enabling parallelism for Job1, configure each instance of Job1 to run a separate TestComplete test:
Create a variable (or variables) that will store values for each Job1 instances:
Use the created variable to specify the TestComplete test that the job should run:.
- job: Job1
- task: VSTest@2
testFiltercriteria : 'Name=$(test_name)'
testRunTitle: 'Automated tests'
Complete pipeline example (YAML)
The overall pipeline (with parallelism) will look as follows:
# A sample pipeline that shows how to run TestComplete tests
- stage: QA
- job: Job1
name: Test environments
demands: TestRunner -equals TestExecute
- checkout: self
- task: VisualStudioTestPlatformInstaller@1
- task: InstallTestCompleteAdapter@1
- task: VSTest@2
testRunTitle: 'Automated tests'
4. Run the test
Typically, you configure the pipeline to run on a specific event. For example, when new file versions are uploaded to the project repository:
You can also run the pipeline manually.
5. Get test results
This section describes how you can solve typical tasks that you may face when migrating your distributed tests to Azure DevOps.
Copying test projects to test agents
Before the testing starts, your test projects must be placed in a location where the test runner (TestComplete or TestExecute) can access them from the remote computers.
TestComplete: this task can be done automatically — you can configure your tasks to copy projects to hosts.
Azure DevOps: Typically, you place your TestComplete projects to your Azure project’s repository and then configure your pipeline to take the project files from that repository and place them to a target location on agents.
Opening user sessions
For your tests to be able to simulate user actions over the tested application GUI, agents, where the tests will run, must have an interactive user session opened.
TestComplete: you can store user credentials that will be used to open user sessions in Network Suite. For each host you add to your Network Suite, you can specify how user sessions should be opened on them.
Azure DevOps: configure test agents to run as an interactive process, and enable the auto-logon. When the agent starts, it will open the session automatically.
To learn more about configuring agents with the auto-logon enabled, see Self-hosted Windows agents in the Azure DevOps documentation.
In distributed testing, it may be crucial that tests that run on several remote computers interact with each other.
TestComplete: the Network Suite functionality offers the following means of interaction between projects:
Synchpoints - Can be used to pause the test execution on remote computers until a specific point is reached and then resume the test execution on all remote computers simultaneously.
Network Suite variables - Can be used to exchange data between all tests that participate in testing. To track the variables state, the
WaitForNetVarChangemethod and the
OnNetVarChangeevent can be used.
Azure DevOps: to implement interactions between TestComplete tests, you can use the following:
Use various parallel strategies to organize the pipeline run. You can command the pipeline to run several instances of a job simultaneously or you can run several instances of a job with various inputs.
Use secure files to store resources that the pipeline can access during the build run. To learn more about secure files, please see About pipeline resources in the Azure DevOps docs.
Use dependencies and conditions to organize the stage run order in your pipeline.
Use pipeline variables to store, send, and get data from TestComplete tests during the pipeline run.
Triggering test runs
To run TestComplete tests, you run your pipeline. You can initiate the pipeline run manually, or you can configure your pipeline to start automatically on a specific event, or you can schedule the pipeline run.
Monitoring testing process
TestComplete: you can track the current state of running tests in the Network Suite editor, or you can do this in your tests by checking the
State property of appropriate scripting objects.
Azure DevOps: you can track the test progress on the Build page.
Viewing test results
TestComplete: Network Suite can retrieve test results from remote computers automatically, or it can store them on remote machines (this was helpful if the test log was too large).
Azure DevOps: in general, when you run TestComplete tests on remote computers from a CI/CD system, the test results will be available only on those remote computers where they were run. In Azure DevOps, the TestComplete Test Adapter extension will accumulate your test results and upload them to the Azure server. You can view them on the Tests page of your pipeline results.
You can configure the test adapter to copy the following results:
- All test results
- Results of failed tests only
- Results of failed tests and tests completed with warnings
If you have questions or need assistance with migration, please contact our Customer Care Team.