Running Cross-Platform Web Tests in Parallel by Using Azure DevOps

Applies to TestExecute 14.72, last modified on January 12, 2021

Azure DevOps (and Team Foundation Server) is a collection of development, build, test and deploy tools and services by Microsoft. For example, they include source control repositories to store, manage and share source code, build and release pipelines to implement continuous integration, and test plans to manage manual and exploratory testing.

If you use Azure DevOps to manage your workflow, you can include your cross-platform web tests into it and configure the workflow to run these tests in parallel.


Team Server

You can use one of the following server versions:

  • Azure DevOps Services (formerly known as Visual Studio Team Services or VSTS).

    Note: Use self-hosted (private) agents to run cross-platform web tests. Running tests on Microsoft-hosted agents is not supported.

  • Azure DevOps Server 2019.

  • Team Foundation Server 2015–2018.

Test Agents

Your test agents (remote computers from which your cross-platform web tests will connect to your device cloud) must meet the following requirements:

  • The test agents must allow running cross-platform web tests in parallel. To learn how to get the test runner tool that supports parallel test runs, see Getting Installation Package.

    Note: You do not need to have the tool installed on devices in your device cloud.

  • The test agents must have access to your license manager.

  • The test agents must have access to your device cloud where you will run your cross-platform web tests.

  • The test agents must have access to your TestComplete project that contains your cross-platform web tests.

How it works

Cross-platform web tests you create with TestComplete can run in device clouds managed with Selenium Grid. It can be:

  • SmartBear device cloud solution —

  • A private hub with Selenium Grid configured on your local computer or on a remote computer in your network. To learn more about installing and configuring Selenium Grid, see

To control parallel test runs from your pipeline in Azure DevOps, you use the TestComplete test adapter that you get with the TestComplete test adapter extension and a test runner tool that allows running tests in parallel. Both the adapter and the tool should reside on your test agents.

The TestComplete test adapter enables the Visual Studio Test task to run TestComplete projects the same way it runs native Visual Studio tests. When the task runs a TestComplete project, the test adapter automatically detects all test cases that the project contains and runs them in parallel. You can set the maximum number of parallel tests that can run on a single agent when configuring your pipeline (see below).

After the test run is over, the adapter gets the test results and publishes them to your Azure DevOps Server.

Running cross-platform web tests from Azure DevOps

Click the image to enlarge it.

Note on exporting test results

By default, TestComplete tests report their status (passed or failed) to the build results automatically. In addition, you can configure your pipeline to include a detailed test report (.mht) to the build results (see below).

1. Prepare TestComplete project for running in parallel

1.1 Specify the tests you want to run

  1. Open your project in TestComplete.

  2. On the Test Items page, create test cases that will run your cross-platform web tests:

    Specifying tests to run in parallel

    Click the image to enlarge it.

1.2 Add your TestComplete project to the repository of your team project

We recommend that you do it directly from TestComplete IDE. This way, only the necessary files will be added. How to add the project suite depends on the type of the repository you use to store team files. To learn how to add TestComplete projects to various source control systems, see the Integration With Source Control Systems section in the TestComplete documentation.

2. Install the TestComplete Test Adapter extension

TestComplete test adapter manages TestComplete test runs on your agents. To install the adapter to the agents, you add the Enable TestComplete tests task to your pipeline. To make the task available to your team project, your Azure DevOps or Team Foundation Server must have the TestComplete Test Adapter extension installed.

From Microsoft Marketplace (for Azure DevOps)

  1. In a web browser, navigate to your team project web portal.

  2. Click , then click Browse marketplace.

  3. In the marketplace, locate the TestComplete Test Adapter extension.

  4. Click Get if free.

  5. Select the team project collection for which you want to make the extension available and click Install.

Locally (for Team Foundation Server 2015–2018)

  1. Copy the extension file (smartbearsoftware.install-testcomplete-adapter-task-for-tfs-14-72.NNN.vsix ) to a location available to your server.

    By default, the extension resides in the TestComplete folder:

    <TestComplete 14>\VS Integration\smartbearsoftware.install-testcomplete-adapter-task-for-tfs-14-72.NNN.vsix

  2. Connect to your team server. Click , and then click Manage extensions.

  3. On the Extensions page, click Browse local extensions.

  4. Click Manage Extensions. On the resulting Manage Extensions page, click Upload extension.

  5. In the resulting dialog, select the TestComplete test adapter extension file and click Upload.

  6. Click the extension in the list, and then click Get it free.

  7. To make the extension available for your team project, select the collection to which your team project belongs, and then click Install.

3. Configure the pipeline to run cross-platform web tests

3.1 Open your pipeline for editing

  1. Navigate the web portal of your team project. You can navigate to it directly in your web browser, or you can open it from Team Explorer in Visual Studio.

  2. Go to the Pipelines > Builds page.

  3. Open an existing pipeline or click New to create a new pipeline.

    You can create a YAML script that describes your pipeline, or you can click Use the visual designer to create a pipeline via a wizard. This topic describes how to configure the build pipeline using the visual designer.

3.2 Configure the pipeline to run your TestComplete project

  1. Configure your build pipeline to get your TestComplete project from your team repository.

  2. Select an agent job that you want to run your tests or add a new job. Specify agents on which your tests will run. Make sure that the agents meet all the needed requirements.

  3. Add the Enable support for TestComplete tests task. This task installs the TestComplete test adapter to your agent and enables the TestComplete test support. Configure the task:

    1. Set TestExecute Lite (Device Cloud Parallel) as a test runner.

    2. Set the maximum number of parallel test runs.

    3. If needed, specify when you want to retrieve test logs: only for failed tests, for failed tests and for tests with warnings, or for all tests.

    Configuring the pipeline to use TestExecute Lite (Device Cloud Parallel) as a test runner

    Click the image to enlarge it.

  4. Add the Visual Studio Test task to your definition and configure it to run your TestComplete tests:

    1. In the Select tests using drop-down list box, select Test assemblies.

    2. In the Test files text box, specify your TestComplete project suite (a .pjs file) to run.

    3. In the Search folder text box, specify the folder on your build machine or test agent where the task should search for the TestComplete project suite.

      Note: If you specify any Search folder other than the default working folder, in the Path to custom test adapters text box, specify the path to the TestComplete test adapter: $(System.DefaultWorkingDirectory)\TestCompleteAdapter.

    4. In the Test filter criteria text box, you can enter a filter expression that will specify one or several test items of your TestComplete project that you want to run.

    Configuring the pipeline to run TestComplete tests

    Click the image to enlarge it.

3.3 Complete the pipeline configuration

  1. If needed, specify other pipeline properties.

  2. Save the changes.

4. Run the pipeline

  1. On the Builds page of your project, queue the created pipeline.

  2. The build will be added to the build queue, and the Build Explorer will inform you about the build progress.

During the build run, the test adapter will locate the specified TestComplete project, detect its test items and run them in parallel.

5. View test results

  1. On the Builds page, double-click the needed build to open its results.

  2. Switch to the Tests section. It shows a summary report of a test run: the number of failed and passed tests and the status of each of them.

  3. If a test has failed, you can view information on the occurred error in the Debug section.

  4. To get the test log, download the log in the .mht format from the Attachments section. Depending on the Save logs for setting value, logs could be available only for failed tests, for failed tests and for tests with warnings, or for all tests.

Viewing test results

Click the image to enlarge it.

See Also

Running Cross-Platform Web Tests in Parallel

Highlight search results