Migrating Distributed Tests to Azure DevOps

Applies to TestComplete 15.40, last modified on March 25, 2022

Starting from TestComplete version 14.81, the Network Suite functionality is deprecated. Though it will not be removed from the product, we do not recommend that you use it to manage distributed test runs. We recommend that you use a CI/CD system instead.

This section explains how to migrate your distributed test runs from Network Suite to Azure DevOps.

In This Document

Prerequisite: TestComplete Test Adapter extension for Azure DevOps/Team Foundation Server

To run TestComplete tests from Azure DevOps pipelines easily, you can use the TestComplete test adapter. The adapter must be installed on all remote computers (agents in Azure DevOps terms), where your tests will run. To get the adapter, install the TestComplete Test Adapter extension to your Azure DevOps server. You can do it from the Microsoft Marketplace or, if you have a private Team Foundation Server, you can install the extension locally:

From Microsoft Marketplace (for Azure DevOps)

Locally (for Team Foundation Server 2015–2018)

Possible alternatives

There are other ways to control TestComplete test runs from Azure DevOps, for example, you can run tests by using the TestComplete command line or the Session Creator utility. We will not examine these alternatives here.

How Network Suite elements translate to Azure DevOps

The image below shows an example of migrating TestComplete tests from Network Suite to Azure DevOps pipeline:

Migrating TestComplete Network Suite to Azure DevOps

Click the image to enlarge it.

Managing project

TestComplete: To manage test runs by using the Network Suite functionality, you use a special project (so-called “master”). It stores the list of computers where your tests will run (hosts), and specifies jobs and tasks to be run on hosts. It also defines synchronization points and Network Suite variables. You use this project to start and monitor the test run and to gather results from remote machines.

Azure DevOps: In Azure DevOps (and other CI/CD systems), a special TestComplete project is not required. You configure and manage test runs directly from Azure DevOps pipelines.

If you are new to Azure DevOps and pipelines, we recommend that you read the pipeline documentation on the Microsoft website:

docs.microsoft.com/en-us/azure/devops/pipelines

Hosts

TestComplete: In Network Suite, the remote computers, on which your tests are running, are called hosts.

Azure DevOps: Azure DevOps (and other CI/CD systems) support managing a pool of computers on which various tasks are run. Such computers are called agents. In Azure DevOps, agents can be Microsoft-hosted and self-hosted.

For TestComplete tests to be able to simulate user actions over the tested application user interface, the agent must be configured to run in the interactive mode.

For instructions on how to configure agents, see below.

Jobs

TestComplete: In Network Suite, a job is a collection of tasks that run in parallel on specified hosts. Network Suite can have one or several jobs. Jobs are run in a series.

Azure DevOps: Azure DevOps pipelines also have jobs. A single job runs on a single agent and executes one or several steps. The job always executes the steps in a series, one after another. Jobs can be grouped into stages.

To run tasks simultaneously, you can configure parallel jobs by using various parallel strategies. The pipeline will create several instances of a job and run them concurrently.

To organize parallel job runs, your Azure DevOps organization must have parallel jobs purchased. To learn more, see Configure and pay for parallel jobs in the Azure DevOps documentation.

If you are not interested in parallel runs, you can run your jobs and tasks in a series.

Tasks

TestComplete: a Network Suite task is a test to be run on a host as part of a job. Task properties specify the TestComplete project that contains the test you want to run, the test to run, the computer on which you want to run this test, and the test runner tool (TestComplete or TestExecute), and the way you would like to handle test results.

Tasks that belong to the same job run in parallel.

Azure DevOps: In Azure DevOps pipelines, a job consists of steps: tasks or scripts that run one after another. To run TestComplete tests as part of a pipeline, use the Visual Studio Test task.

Crucial differences from the Network Suite tasks:

  • You cannot assign the Visual Studio Test task to an agent directly. To do this, you use capabilities and demands. See below.

  • Unlike the Network Suite tasks that run in parallel, the Azure pipeline runs the tasks one after another in the order the pipeline specifies them. To have several tasks run in parallel, you can use parallel jobs. You can also configure the pipeline to distribute tests among several agents.

Distributed test in Azure DevOps – Example

To create and run a distributed test, you perform the following tasks:

  1. Configure remote computers (establish the connection between the CI/CD server and the remote computer, install the tested application and test runner on the remote machine, and so on).

  2. Copy test projects and data files to remote machines.

  3. Run the test runner on each remote computer.

  4. Gather results.

The following sections describe how to perform these tasks in Azure DevOps.

About the TestComplete project to run

Suppose we have TestComplete tests that we run on remote computers by using the Network Suite functionality. These projects are organized in the following way:

A distributed test managed by the Network Suite

Click the image to enlarge it.

  • The project called Main controls the test runs:

    • It has the Hosts collection that includes two hosts called RemoteAgent1 and RemoteAgent2. For them, it specifies credentials used to open interactive user sessions.

      Hosts

      Click the image to enlarge it.

    • Has one Job collection called Job1. The collection includes two tasks that specify the tests to execute on each of the hosts.

    • Task1 runs on RemoteAgent1. It executes Test1 that belongs to TestProject1.

      Task2 runs on RemoteAgent2. It executed Test2 that belongs to TestProject2.

      Job and tasks of the Network Suite

      Click the image to enlarge it.

  • Two test projects to be run on remote computers. These projects are called TestProject1 and TestProject2 respectively.

Let’s examine how you can migrate this project set up to Azure DevOps.

1. Prepare and configure Azure DevOps

1.1. Install the TestComplete Test Adapter extension

To enable Azure DevOps pipelines to run TestComplete tests, install the TestComplete Test adapter extension to Azure DevOps. For detailed instructions, see above.

1.2. Create an Azure DevOps project

In Azure DevOps, if you still do not have a project to which you want to add your TestComplete automated tests, create one.

To learn how to do this, see About projects in the Azure DevOps documentation.

1.3. Configure pools and agents

An agent is a remote computer (physical or virtual), where Azure pipelines run jobs and tasks. To run automated tests, you will need agents that meet the following requirements:

Requirements
  • Agents must run as an interactive process with the auto-logon enabled. See below to learn how to configure Azure DevOps agents.

  • Agents must have TestComplete or TestExecute installed on them. The TestComplete (TestExecute) instance must also be licensed.

  • Agents must have access to your tested application and all other files your tests use.

Register an agent pool

In Azure DevOps, agents are organized into pools. When you run a pipeline, it runs on an agent it selects from a pool that meets the required capabilities (see below).

If you still do not have a pool, create one, or ask your organization administrator to create a pool that you will use to run automated tests. To learn about creating and managing pools, please see Agent pools in the Azure DevOps documentation.

Download and configure agents

For Azure DevOps to be able to access computers where your tests will run, the agent must be running on them. To configure and run the agent:

  1. Log in to the computer that you want to use to run your TestComplete tests.

  2. There, launch a web browser, navigate to your Azure DevOps project, and then go to Project settings > Agent pools:

    Migrating TestComplete distributed tests to Azure DevOps: Configuring agent pools

    Click the image to enlarge it.

  3. Click New agent.

  4. In the resulting Get the agent dialog, switch to the Windows tab and click Download to get the agent installation files.

  5. After downloading the files, unpack them and run the config.cmd file.

  6. Follow the instructions to configure the agent. For detailed instructions, please see Download and configure the agent in the Azure DevOps documentation.

    If your TestComplete tests interact with the GUI, make sure to configure the agent to run as an interactive process.

    In our example, we will add RemoteAgent1 to the Test environments pool and configure the agent to run as an interactive process.

  7. After the agent is configured, run it by running the run.cmd file. While the agent is running, it is available to Azure DevOps and can run jobs and tasks.

Repeat the steps to configure other agents.

Install TestComplete (TestExecute) on agents

On every agent you will use to run tests, install TestComplete or TestExecute. These TestComplete (TestExecute) instances must be licensed. The easiest way is to use a Floating user license.

Assign capabilities to agents

For the Azure DevOps pipeline to select proper agents to execute jobs and tasks, capabilities and demands are used. Capabilities are a collection of name-value pairs that specify what an agent can do or what software it has installed. A demand is a condition that specifies on which agent the task or a job can run.

To specify agents on which your TestComplete tests will run, you can create a capability named TestRunner and set its value either to TestComplete or TestExecute (depending on the appropriate software installed):

  1. In Azure DevOps, open your project and click Project settings.

  2. Click Agent pools and then click the pool that contains agents where your tests will run.

  3. Click an agent and switch to the Capabilities tab.

  4. In the User-defined capabilities section, click the Add a new capability link.

  5. Enter the capability name and value and click Add. In our example, we will create the TestRunner capability and set its value to TestExecute.

Repeat the steps to add the capability to other agents.

2. Place TestComplete projects to your Azure DevOps repository

For Azure DevOps agents to be able to access your TestComplete projects to run them, add the project files to your Azure project’s repository. How you can do it, depends on the repository you use. For detailed instructions on adding files, please see Supported source repositories in the Azure DevOps documentation.

3. Create an Azure pipeline to run the test

Note: In Azure DevOps, you can create and configure the pipeline either by using the visual editor or as a YAML file.

  1. In your Azure DevOps project, go to Pipelines and then click New pipeline.

  2. Specify the repository where your TestComplete project files reside.

  3. Specify jobs that will run your TestComplete tests. In our example, we will have Job1:

    YAML

    name: Build_$(Date:yyyy_MM_dd)
    jobs:
      - job: Job1
        displayName: Job1
        continueOnError: true

    Visual pipeline editor

    Migrating TestComplete distributed tests to Azure DevOps: Adding a job to Azure pipeline

    Click the image to enlarge it.

  4. Specify the agent pool and agents on which you want to run tests. In our example, we will run the job on agents that have the TestRunner capability set to TestExecute. It is the capability we have assigned to agents earlier:

    YAML


    jobs:
    - job: Job1
      displayName: Job1
      pool:
        name: Test environments
        demands: TestRunner -equals TestExecute


    Visual pipeline editor

    Migrating TestComplete distributed tests to Azure DevOps: Specifying agents on which the job will run

    Click the image to enlarge it.

  5. If you do not have Visual Studio installed on your target agents, you can add the Install Visual Studio Test Platform task to the pipeline. The task will install the test platform that allows running tests on computers where Visual Studio is not installed:

    YAML


    jobs:
      - job: Job1
        displayName: Job1
        steps:
        - task: VisualStudioTestPlatformInstaller@1
        inputs:
          packageFeedSelector: 'nugetOrg'
          versionSelector: 'latestStable'

    Visual pipeline editor

    Migrating TestComplete distributed tests to Azure DevOps: Installing Visual Studio Test Platform

    Click the image to enlarge it.

  6. Add the Enable support for TestComplete tests task to the job. This task installs the TestComplete test adapter to your agent and enables the TestComplete test support.

    If needed, set the preferred test runner to use to run tests.

    Specify how you want to handle test logs: upload them only when the tests fails, completed with warnings, or always.

    YAML


    jobs:
      - job: Job1
      displayName: Job1
      steps:
        - task: VisualStudioTestPlatformInstaller@1
        …
        - task: InstallTestCompleteAdapter@1
          inputs:
            logsLevel: '0'

    Visual pipeline editor

    Migrating TestComplete distributed tests to Azure DevOps: Installing TestComplete test adapter

    Click the image to enlarge it.

  7. Add the Visual Studio Test task to the pipeline and configure it to run TestComplete tests:

    • Configure the task to select tests by using assemblies.

    • Specify the project suite file (.pjs) that contains your tests and that the task will run.

    • Specify the folder on your agent where the task will search for TestComplete project files. By default, it will search the files in the agent’s working folder.

    • If needed, you can use a filter expression to specify test items of your TestComplete project that you want to run.

      In our example, we will use the expression to run Test1 test item of the TestComplete project.

    • You may also specify the test run name to which test results will be assigned.

    YAML

    jobs:
      - job: Job1
        displayName: Job1
        steps:
        …
        - task: VSTest@2
          inputs:
            testSelector : 'testAssemblies'
            testAssemblyVer2: '**\*.pjs'
            searchFolder: '$(System.DefaultWorkingDirectory)'
            testFiltercriteria: 'Name=Test1'
            vsTestVersion: 'toolsInstaller'
            testRunTitle: 'Automated tests'

    Visual pipeline editor

    Migrating TestComplete distributed tests to Azure DevOps: Configuring VIsual Studio Test task to run TestComplete tests

    Click the image to enlarge it.

  8. To run other test projects, you can add more Visual Studio Test task to the job. To run tests in parallel, you can enable the Parallelism for the created Job1.

    Your organization must have parallel jobs. To learn more about purchasing parallel jobs, see Configure and pay for parallel jobs in the Azure DevOps documentation.

    After enabling parallelism for Job1, configure each instance of Job1 to run a separate TestComplete test:

    1. Create a variable (or variables) that will store values for each Job1 instances:

      YAML

      jobs:
      - job: Job1
        displayName: Job1
        strategy:
          matrix:
            Test1:
              test_name: 'Test1'
            Test2:
              test_name: 'Test2'
          maxParallel: 2


      Visual pipeline editor

      Migrating TestComplete distributed tests to Azure DevOps: Configuring a job to run in parallel

      Click the image to enlarge it.

    2. Use the created variable to specify the TestComplete test that the job should run:.

      YAML

      jobs:
      - job: Job1
        displayName: Job1
        strategy:
          matrix:
            Test1:
              test_name: 'Test1'
            Test2:
              test_name: 'Test2'
          maxParallel: 2
        steps:
        …
          - task: VSTest@2
            inputs:
              testSelector: 'testAssemblies'
              testAssemblyVer2: '**\*.pjs'
              searchFolder: '$(System.DefaultWorkingDirectory)'
              testFiltercriteria : 'Name=$(test_name)'
              vsTestVersion: 'toolsInstaller'
              testRunTitle: 'Automated tests'

      Visual pipeline editor

      Migrating TestComplete distributed tests to Azure DevOps: Specifying tests that a job instance will run

      Click the image to enlarge it.

Complete pipeline example (YAML)

The overall pipeline (with parallelism) will look as follows:

YAML

# A sample pipeline that shows how to run TestComplete tests
name: Build_$(Date:yyyy_MM_dd)
trigger:
- main
stages:
- stage: QA
  jobs:
  - job: Job1
    displayName: Job1
    pool:
      name: Test environments
      demands: TestRunner -equals TestExecute
  continueOnError: true
  strategy:
      matrix:
        Test1:
          test_name: 'Test1'
        Test2:
          test_name: 'Test2'
      maxParallel: 2
    steps:
    - checkout: self
      clean: true
      path: AutomatedTests
    - task: VisualStudioTestPlatformInstaller@1
      inputs:
        packageFeedSelector: 'nugetOrg'
        versionSelector: 'latestStable'
    - task: InstallTestCompleteAdapter@1
      inputs:
        logsLevel: '0'
    - task: VSTest@2
      inputs:
        testSelector: 'testAssemblies'
        testAssemblyVer2: '**\*.pjs'
        searchFolder: '$(System.DefaultWorkingDirectory)'
          testFiltercriteria: 'Name=$(test_name)'
        vsTestVersion: 'toolsInstaller'
        testRunTitle: 'Automated tests'

4. Run the test

Typically, you configure the pipeline to run on a specific event. For example, when new file versions are uploaded to the project repository:

YAML


trigger:
  - main

Visual pipeline editor

Migrating TestComplete distributed tests to Azure DevOps: Setting pipeline triggers

Click the image to enlarge it.

You can also run the pipeline manually.

5. Get test results

After the pipeline run is over, you can view the test results on the Tests page:

Migrating TestComplete distributed tests to Azure DevOps: Viewing test results

Click the image to enlarge it.

Typical tasks

This section describes how you can solve typical tasks that you may face when migrating your distributed tests to Azure DevOps.

Copying test projects to test agents

Before the testing starts, your test projects must be placed in a location where the test runner (TestComplete or TestExecute) can access them from the remote computers.

TestComplete: this task can be done automatically — you can configure your tasks to copy projects to hosts.

Azure DevOps: Typically, you place your TestComplete projects to your Azure project’s repository and then configure your pipeline to take the project files from that repository and place them to a target location on agents.

Opening user sessions

For your tests to be able to simulate user actions over the tested application GUI, agents, where the tests will run, must have an interactive user session opened.

TestComplete: you can store user credentials that will be used to open user sessions in Network Suite. For each host you add to your Network Suite, you can specify how user sessions should be opened on them.

Azure DevOps: configure test agents to run as an interactive process, and enable the auto-logon. When the agent starts, it will open the session automatically.

To learn more about configuring agents with the auto-logon enabled, see Self-hosted Windows agents in the Azure DevOps documentation.

Synchronization

In distributed testing, it may be crucial that tests that run on several remote computers interact with each other.

TestComplete: the Network Suite functionality offers the following means of interaction between projects:

  • Synchpoints - Can be used to pause the test execution on remote computers until a specific point is reached and then resume the test execution on all remote computers simultaneously.

  • Network Suite variables - Can be used to exchange data between all tests that participate in testing. To track the variables state, the WaitForNetVarChange method and the OnNetVarChange event can be used.

Azure DevOps: to implement interactions between TestComplete tests, you can use the following:

  • Use various parallel strategies to organize the pipeline run. You can command the pipeline to run several instances of a job simultaneously or you can run several instances of a job with various inputs.

  • Use secure files to store resources that the pipeline can access during the build run. To learn more about secure files, please see About pipeline resources in the Azure DevOps docs.

  • Use dependencies and conditions to organize the stage run order in your pipeline.

  • Use pipeline variables to store, send, and get data from TestComplete tests during the pipeline run.

Triggering test runs

To run TestComplete tests, you run your pipeline. You can initiate the pipeline run manually, or you can configure your pipeline to start automatically on a specific event, or you can schedule the pipeline run.

Monitoring testing process

TestComplete: you can track the current state of running tests in the Network Suite editor, or you can do this in your tests by checking the State property of appropriate scripting objects.

Azure DevOps: you can track the test progress on the Build page.

Viewing test results

TestComplete: Network Suite can retrieve test results from remote computers automatically, or it can store them on remote machines (this was helpful if the test log was too large).

Azure DevOps: in general, when you run TestComplete tests on remote computers from a CI/CD system, the test results will be available only on those remote computers where they were run. In Azure DevOps, the TestComplete Test Adapter extension will accumulate your test results and upload them to the Azure server. You can view them on the Tests page of your pipeline results.

You can configure the test adapter to copy the following results:

  • All test results
  • Results of failed tests only
  • Results of failed tests and tests completed with warnings

Questions?

If you have questions or need assistance with migration, please contact our Customer Care Team.

Highlight search results