Run TestComplete Tests as Part of a Build

Applies to TestComplete 14.70, last modified on April 22, 2021

Video Tutorial

 

Prerequisites

Use self-hosted (private) agents to run TestComplete tests. Running TestComplete tests on Microsoft-hosted agents is not supported.

Configure a Build Pipeline

  1. Open the web portal of your team project. You can navigate to it directly in your web browser, or you can open it from Team Explorer in Visual Studio.

  2. Navigate to the Pipelines > Builds page.

  3. Open an existing build definition or click New to create a new build definition:

    TestComplete integration with Azure DevOps: Creating a new build pipeline

    Click the image to enlarge it.

    You can create a YAML script that describes your pipeline, or you can click Use the visual designer to create a pipeline via a wizard. This topic describes how to configure the build pipeline using the visual designer.

  4. Configure your build pipeline to get your TestComplete project files from your team repository.

    For example, if you are using Team Foundation Version Control, click the ellipsis button of the Server path edit box and locate the repository directory that stores your TestComplete project suite. Then, in the Local path under $(build.sourcesDirectory) text box, specify the local folder on your test agents where your TestComplete project suite will be placed:

    TestComplete integration with Azure DevOps: Getting TestComplete project suite from a repository

    Click the image to enlarge it.

  5. If needed, configure your build pipeline to place your tested application and other files required for testing to the location where your TestComplete tests will be able to access them. For example, if they are stored in your team project’s repository, configure your pipeline to get them from there or if they are on another agent, configure your pipeline to copy them to your test agent.

  6. Add an agent job to the pipeline. Specify agents on which your tests will run. Make sure that the agents meet all the requirements.

    If needed, configure the job to distribute the tests among multiple agents. To learn how to do it, see Run tests in parallel using the Visual Studio Test task. Note: Your tests must be able to run independently from each other.

  7. If your test agents do not have Visual Studio installed, you can add the Install Visual Studio Test Platform task to the stage. The task will install the test platform that allows running tests on computers where Visual Studio is not installed. To learn about the task, see the task description:

    TestComplete integration with Azure DevOps: Adding the Install Visual Studio Test Platform task

    Click the image to enlarge it.

  8. Add the Enable support for TestComplete tests task. This task installs the TestComplete test adapter to your agent and enables the TestComplete test support:

    TestComplete integration with Azure DevOps: Adding the Enable TestComplete tests task

    Click the image to enlarge it.

    The task is available only if you have the TestComplete Test Adapter extension installed on your server. To learn how to install it, see Install TestComplete Test Adapter Extension.

  9. If needed, select an application that you want to use to run tests:

    • To run ordinary desktop and web tests sequentially, select TestComplete or TestExecute.

    • To run cross-platform web tests in parallel, select TestExecute Lite (Device Cloud Parallel). If needed, set the maximum number of tests that can run in parallel.To learn more about running tests in parallel, see About Running Tests in Parallel.

    TestComplete integration with Azure DevOps: Selecting a preferred test runner

    Click the image to enlarge it.

    Note: If the selected application is not installed on the agent, the one that is installed will be used instead.

  10. If needed, specify when you want to retrieve test logs: only for failed tests, for failed tests and for tests with warnings, or for all tests:

    TestComplete integration with Azure DevOps: Selecting when to retrieve test logs

    Click the image to enlarge it.

  11. Add the Visual Studio Test task to your definition.

    Configure the added task to run your TestComplete tests:

    By specifying a TestComplete project suite directly

    In the Select tests using drop-down list box, select Test assemblies.

    In the Test files text box, specify your TestComplete project suite (a .pjs file) to run.

    In the Search folder text box, specify the folder on your build machine or test agent where the task should search for the TestComplete project suite.

    Note: If you specify any Search folder other than the default working folder, in the Path to custom test adapters text box, specify the path to the TestComplete test adapter: $(System.DefaultWorkingDirectory)\TestCompleteAdapter.

    In the Test filter criteria text box, you can enter a filter expression that will specify one or several test items of your TestComplete project that you want to run.

    Filter syntax

    In the Test run title text box, you can enter the name of the test run to which TestComplete test results will be assigned.

    If necessary, in the Test platform version drop-down list, select the needed version of your Visual Studio Test Platform.

    TestComplete integration with Azure DevOps: Configuring a Visual Studio Test task to run TestComplete tests

    Click the image to enlarge it.

    If you are using version 1 of the Visual Studio Test task or if you have set the Search folder value to any value other than the default working folder, in the Path to Custom Test Adapters text box, specify the path to the TestComplete test adapter: $(System.DefaultWorkingDirectory)\TestCompleteAdapter:

    TestComplete integration with Azure DevOps: Configuring a Visual Studio Test task to use TestComplete test adapter

    Click the image to enlarge it.

    By specifying a test plan

    To run TestComplete tests as part of a test plan, your TestComplete project must be bound to your team project, and your TestComplete test cases must be linked to the team project’s test cases. See Link TestComplete Test Cases to Azure DevOps Test Cases.

    In the Select tests using drop-down list, select Test plan.

    In the Test plan drop-down list, select the test plan to which the tests you want to run belong.

    In the Test suite drop-down list, select the test suites that include the needed tests.

    In the Test configuration drop-down list, select the test environment where your tests will run.

    In the Search folder text box, specify the folder on your agent where the task should search for TestComplete tests.

    Note: If you specify any Search folder other than the default working folder, in the Path to custom test adapters text box, specify the path to the TestComplete test adapter: $(System.DefaultWorkingDirectory)\TestCompleteAdapter.

    If needed, in the Test run title text box, you can enter the name of the test run to which TestComplete test results will be assigned.

    TestComplete integration with Azure DevOps: Configuring a Visual Studio Test task to run test cases to which TestComplete tests are assigned

    Click the image to enlarge it.

  12. If needed, specify other build properties. You can learn more at docs.microsoft.com/en-us/azure/devops/pipelines/index.

  13. Save the changes.

Run a Build

  1. On the Builds page of your project, queue the created build.

  2. The build will be added to the build queue, and the Build Explorer will inform you about the build progress.

During the build run, the Visual Studio Test task will locate the specified TestComplete project suite and run the tests.

View Test Results

  1. On the Builds page, double-click the needed build to open its results.

  2. Switch to the Tests section:

    TestComplete integration with Azure DevOps: Test run summary

    Click the image to enlarge it.

    It shows a summary report of a test run: the number of failed and passed tests and the status of each of them.

  3. If a test has failed, you can view information on the occurred error in the Debug section:

    TestComplete integration with Azure DevOps: Error message

    Click the image to enlarge it.

  4. To get the test log, download the log in the .mht format from the Attachments section:

    TestComplete integration with Azure DevOps: TestComplete test log

    Click the image to enlarge it.

    Depending on the Save logs for setting value, logs could be available only for failed tests, for failed tests and for tests with warnings, or for all tests.

  5. To get a log generated for the entire test run, click the test run, then switch to the Attachments section and download the .trx file:

    TestComplete integration with Azure DevOps: Test run results

    Click the image to enlarge it.

See Also

Integration With Azure DevOps and Team Foundation Server via TestComplete Test Adapter
Install TestComplete Test Adapter Extension

Highlight search results