Run TestComplete Tests as Part of a Build

Use self-hosted (private) agents to run TestComplete tests. Running TestComplete tests on Microsoft-hosted agents is not supported.

• Your TestComplete projects must be prepared as described in Prepare TestComplete Project for Integration and added to your team project’s source control repository.

• Your test agents on which you will run tests must have TestComplete (or TestExecute) installed.

• Your team project must have the TestComplete test adapter extension installed. See Install TestComplete Test Adapter Extension.

1. Open the web portal of your team project. You can navigate to it directly in your web browser, or you can open it from Team Explorer in Visual Studio.

2. Navigate to the Pipelines > Builds page.

3. Open an existing build definition or click New to create a new build definition:

   

   Click the image to enlarge it.

   You can create a YAML script that describes your pipeline, or you can click Use the visual designer to create a pipeline via a wizard. This topic describes how to configure the build pipeline using the visual designer.

4. Configure your build pipeline to get your TestComplete project files from your team repository.

   For example, if you are using Team Foundation Version Control, click the ellipsis button of the Server path edit box and locate the repository directory that stores your TestComplete project suite. Then, in the Local path under $(build.sourcesDirectory) text box, specify the local folder on your test agents where your TestComplete project suite will be placed:

   

   Click the image to enlarge it.

5. If needed, configure your build pipeline to place your tested application and other files required for testing to the location where your TestComplete tests will be able to access them. For example, if they are stored in your team project’s repository, configure your pipeline to get them from there or if they are on another agent, configure your pipeline to copy them to your test agent.

6. Add an agent job to the pipeline. Specify agents on which your tests will run. Make sure that the agents meet all the requirements.

   If needed, configure the job to distribute the tests among multiple agents. To learn how to do it, see Run tests in parallel using the Visual Studio Test task. Note: Your tests must be able to run independently from each other.

7. If your test agents do not have Visual Studio installed, you can add the Install Visual Studio Test Platform task to the stage. The task will install the test platform that allows running tests on computers where Visual Studio is not installed. To learn about the task, see the task description:

   

   Click the image to enlarge it.

8. Add the Enable support for TestComplete tests task. This task installs the TestComplete test adapter to your agent and enables the TestComplete test support:

   

   Click the image to enlarge it.

   The task is available only if you have the TestComplete Test Adapter extension installed on your server. To learn how to install it, see Install TestComplete Test Adapter Extension.

9. If needed, select an application that you want to use to run tests:

   • To run ordinary desktop and web tests sequentially, select TestComplete or TestExecute.

   • To run cross-platform web tests in parallel, select TestExecute Lite (Device Cloud Parallel). If needed, set the maximum number of tests that can run in parallel.To learn more about running tests in parallel, see About Running Tests in Parallel.

   

   Click the image to enlarge it.

   Note: If the selected application is not installed on the agent, the one that is installed will be used instead.

10. If needed, specify when you want to retrieve test logs: only for failed tests, for failed tests and for tests with warnings, or for all tests:

    

    Click the image to enlarge it.

11. Add the Visual Studio Test task to your definition.

    Configure the added task to run your TestComplete tests:

    By specifying a TestComplete project suite directly

    By specifying a test plan

    By specifying a TestComplete project suite directly

    In the Select tests using drop-down list box, select Test assemblies.

    In the Test files text box, specify your TestComplete project suite (a .pjs file) to run.

    In the Search folder text box, specify the folder on your build machine or test agent where the task should search for the TestComplete project suite.

    Note: If you specify any Search folder other than the default working folder, in the Path to custom test adapters text box, specify the path to the TestComplete test adapter: $(System.DefaultWorkingDirectory)\TestCompleteAdapter.

    In the Test filter criteria text box, you can enter a filter expression that will specify one or several test items of your TestComplete project that you want to run.

    Filter syntax

    Filter syntaxFilter syntax

    <parameter><operator><value>

    For TestComplete tests, the following parameters are available:

    • Name - The name of the test item as it is specified in your TestComplete project, on the Test Items page.

    • Project - The name of the TestComplete project that belongs to the specified project suite and that includes the test items you want to run.

    • FullyQualifiedName - The name of the test item in the following format: testitem:<project_name>\\<test_item_name>.

    • Enabled - Specifies whether the test item or a group is enabled in the TestComplete project.

    • Group - The name of the group to which the needed test items belong as it is specified in your TestComplete project, on the Test Items page.

    If the parameter value includes a special character, use character escape sequences. For the list of available escape sequences, see the filter description at github.com/microsoft/vstest-docs/blob/master/docs/filter.md.

    Available operators are:

    =	Checks if the parameter equals the value.
    !=	Checks if the parameter does not equal the value.
    ~	Checks if the parameter contains the value.

    You can also combine several expressions by using the & (and) and | (or) operators.

    For example:

    Name=ProjectTestItem1

    Project=TestProject1

    Group=Orders&Enabled=True

    Name!=Test2

    Name~Test

    Group=Group1|Group=Group3

    In the Test run title text box, you can enter the name of the test run to which TestComplete test results will be assigned.

    If necessary, in the Test platform version drop-down list, select the needed version of your Visual Studio Test Platform.

    

    Click the image to enlarge it.

    If you are using version 1 of the Visual Studio Test task or if you have set the Search folder value to any value other than the default working folder, in the Path to Custom Test Adapters text box, specify the path to the TestComplete test adapter: $(System.DefaultWorkingDirectory)\TestCompleteAdapter:

    

    Click the image to enlarge it.

    By specifying a test plan

    To run TestComplete tests as part of a test plan, your TestComplete project must be bound to your team project, and your TestComplete test cases must be linked to the team project’s test cases. See Link TestComplete Test Cases to Azure DevOps Test Cases.

    In the Select tests using drop-down list, select Test plan.

    In the Test plan drop-down list, select the test plan to which the tests you want to run belong.

    In the Test suite drop-down list, select the test suites that include the needed tests.

    In the Test configuration drop-down list, select the test environment where your tests will run.

    In the Search folder text box, specify the folder on your agent where the task should search for TestComplete tests.

    Note: If you specify any Search folder other than the default working folder, in the Path to custom test adapters text box, specify the path to the TestComplete test adapter: $(System.DefaultWorkingDirectory)\TestCompleteAdapter.

    If needed, in the Test run title text box, you can enter the name of the test run to which TestComplete test results will be assigned.

    

    Click the image to enlarge it.

12. If needed, specify other build properties. You can learn more at docs.microsoft.com/en-us/azure/devops/pipelines/index.

13. Save the changes.

1. On the Builds page of your project, queue the created build.

2. The build will be added to the build queue, and the Build Explorer will inform you about the build progress.

During the build run, the Visual Studio Test task will locate the specified TestComplete project suite and run the tests.

1. On the Builds page, double-click the needed build to open its results.

2. Switch to the Tests section:

   

   Click the image to enlarge it.

   It shows a summary report of a test run: the number of failed and passed tests and the status of each of them.

3. If a test has failed, you can view information on the occurred error in the Debug section:

   

   Click the image to enlarge it.

4. To get the test log, download the log in the .mht format from the Attachments section:

   

   Click the image to enlarge it.

   Depending on the Save logs for setting value, logs could be available only for failed tests, for failed tests and for tests with warnings, or for all tests.

5. To get a log generated for the entire test run, click the test run, then switch to the Attachments section and download the .trx file:

   

   Click the image to enlarge it.