Run TestComplete Tests as Part of a Release

Use self-hosted (private) agents to run TestComplete tests. Running TestComplete tests on Microsoft-hosted agents is not supported.

• Your TestComplete projects must be prepared as described in Prepare TestComplete Project for Integration and added to your team project’s source control repository.

• Your test agents on which you will run tests must have TestComplete (or TestExecute) installed.

• Your team project must have the TestComplete test adapter extension installed. See Install TestComplete Test Adapter Extension.

To run TestComplete tests during release deployment, you need to put your TestComplete test files into the folder where your release pipeline can access them. The easiest way to do it is to place them into a build output folder along with other artifacts:

1. Open the web portal of your team project. You can navigate to it directly in your web browser, or you can open it from Team Explorer in Visual Studio.

2. Navigate to the Pipelines > Builds page of your team project.

3. Open an existing build definition or click New to create a new build definition.

   

   Click the image to enlarge it.

   You can create a YAML script that describes your pipeline, or you can click Use the visual designer to create a pipeline via a wizard. This topic describes how to configure a build pipeline using the visual designer.

4. Configure your build pipeline to get your TestComplete project files from your team repository.

   For example, if you are using Team Foundation Version Control, click the ellipsis button of the Server path edit box and locate the repository directory that stores your TestComplete project suite. Then, in the Local path under $(build.sourcesDirectory) text box, specify the local folder on your test agents to which your TestComplete project suite will be placed:

   

   Click the image to enlarge it.

5. Add an agent job to the pipeline. Specify agents on which your tests will run. Make sure that the agents meet all the requirements.

   If needed, configure the job to distribute the tests among multiple agents. To learn how to do it, see Run tests in parallel using the Visual Studio Test task. Note: Your tests must be able to run independently from each other.

6. Add a task that will copy your TestComplete project files to an artifact staging folder on your build computer:

   

   Click the image to enlarge it.

7. Add a task that will publish your build artifacts to the needed location:

   

   Click the image to enlarge it.

8. If needed, add other tasks to the build. You can learn more at docs.microsoft.com/en-us/azure/devops/pipelines/index.

9. Save the changes.

Run the created build definition to create a build that you will release:

1. On the Builds page, click Queue.

2. In the Queue Build dialog, configure the build run and click OK.

1. Navigate to the Pipelines > Releases page of your team project.

2. Select an existing release pipeline and open it for editing or create a new pipeline:

   

   Click the image to enlarge it.

3. Add an artifact that includes your TestComplete project files to the release:

   

   Click the image to enlarge it.

4. If needed, add an artifact that includes your tested application and other files required for testing. Make sure that your TestComplete tests will be able to access them (see Prepare TestComplete Project for Integration).

5. Select a stage that will run your TestComplete tests or create a new stage.

6. Specify agents on which your tests will run. Make sure that the agents meet all the requirements.

7. If your test agents do not have Visual Studio installed, you can add the Install Visual Studio Test Platform task to the stage. The task will install a test platform that allows running tests on computers where Visual Studio is not installed. To learn about the task, see the task description:

   

   Click the image to enlarge it.

8. Add the Enable support for TestComplete tests task. This task installs the TestComplete test adapter to your agent and enables the TestComplete test support:

   

   Click the image to enlarge it.

   The task is available only if you have the TestComplete Test Adapter extension installed on your server. To learn how to install it, see Install TestComplete Test Adapter Extension.

9. If needed, select an application that you want to use to run tests:

   • To run ordinary desktop and web tests sequentially, select TestComplete or TestExecute.

   • To run cross-platform web tests in parallel, select TestExecute Lite (Device Cloud Parallel). If needed, set the maximum number of tests that can run in parallel.To learn more about running tests in parallel, see About Running Tests in Parallel.

   

   Click the image to enlarge it.

   Note: If the selected application is not installed on the agent, the one that is installed will be used instead.

10. If needed, specify when you want to retrieve test logs: only for failed tests, for failed tests and for tests with warnings, or for all tests:

    

    Click the image to enlarge it.

11. Add the Visual Studio Test task to the stage and configure the task to run your TestComplete tests:

    By specifying a TestComplete project suite directly

    By specifying a test plan

    By specifying a TestComplete project suite directly

    In the Select tests using drop-down list box, select Test assemblies.

    In the Test files text box, specify your TestComplete project suite (a .pjs file) to run.

    In the Search folder text box, specify the folder on your agent where the task should search for the specified TestComplete project suite.

    Note: If you specify any Search folder other than the default working folder, in the Path to custom test adapters text box, specify the path to the TestComplete test adapter: $(System.DefaultWorkingDirectory)\TestCompleteAdapter.

    In the Test filter criteria text box, you can enter a filter expression that will specify one or several test items of your TestComplete project that you want to run.

    Filter syntax

    Filter syntaxFilter syntax

    <parameter><operator><value>

    For TestComplete tests, the following parameters are available:

    • Name - The name of the test item as it is specified in your TestComplete project, on the Test Items page.

    • Project - The name of the TestComplete project that belongs to the specified project suite and that includes the test items you want to run.

    • FullyQualifiedName - The name of the test item in the following format: testitem:<project_name>\\<test_item_name>.

    • Enabled - Specifies whether the test item or a group is enabled in the TestComplete project.

    • Group - The name of the group to which the needed test items belong as it is specified in your TestComplete project, on the Test Items page.

    If the parameter value includes a special character, use character escape sequences. For the list of available escape sequences, see the filter description at github.com/microsoft/vstest-docs/blob/master/docs/filter.md.

    Available operators are:

    =	Checks if the parameter equals the value.
    !=	Checks if the parameter does not equal the value.
    ~	Checks if the parameter contains the value.

    You can also combine several expressions by using the & (and) and | (or) operators.

    For example:

    Name=ProjectTestItem1

    Project=TestProject1

    Group=Orders&Enabled=True

    Name!=Test2

    Name~Test

    Group=Group1|Group=Group3

    In the Test run title text box, you can enter the name of the test run to which TestComplete test results will be assigned.

    In the Test platform version drop-down list, select the version of your Visual Studio Test Platform.

    

    Click the image to enlarge it.

    If you are using version 1 of the Visual Studio Test task or if you have set the Search folder value to any value other than the default working folder, in the Path to Custom Test Adapters text box, specify the path to TestComplete test adapter: $(System.DefaultWorkingDirectory)\TestCompleteAdapter:

    

    Click the image to enlarge it.

    By specifying a test plan

    To run TestComplete tests as part of a test plan, your TestComplete project must be bound to your Azure DevOps team project, and your TestComplete test cases must be linked to the team project’s test cases. See Link TestComplete Test Cases to Azure DevOps Test Cases.

    In the Select tests using drop-down list, select Test plan.

    In the Test plan drop-down list, select the test plan to which the tests you want to run belong.

    In the Test suite drop-down list, select the test suites that include the needed tests.

    In the Test configuration drop-down list, select the test environment where your tests will run.

    In the Search folder text box, specify the folder on your agent where the task should search for the tests.

    Note: If you specify any Search folder other than the default working folder, in the Path to custom test adapters text box, specify the path to the TestComplete test adapter: $(System.DefaultWorkingDirectory)\TestCompleteAdapter.

    If needed, in the Test run title text box, you can enter the name of the test run to which TestComplete test results will be assigned.

    

    Click the image to enlarge it.

12. Save the changes.

After you have configured a release pipeline, you can deploy the release. Your TestComplete tests will be run during the release deployment.

1. On the Pipelines > Releases page of your team project, click Create a release.

2. In the resulting dialog, select the build that you want to deploy. Click Create to create a release.

3. On the Releases page, view the release status. You can click the release to view detailed information on the deployment status of each of its stages:

   

   Click the image to enlarge it.

1. After a release is deployed, you can view its results:

   

   Click the image to enlarge it.

2. To view test results, click the needed stage, and then switch to the Tests section:

   

   Click the image to enlarge it.

   It shows the summary report on the test run: the number of failed and passed tests and the status of each of them.

3. If a test has failed, you can view information on the occurred error in the Debug section:

   

   Click the image to enlarge it.

4. To get the test log, download the log in the .mht format from the Attachments section:

   

   Click the image to enlarge it.

   Depending on the Save logs for setting value, logs could be available only for failed tests, for failed tests and for tests with warnings, or for all tests.

5. To get the log generated for the entire test run, click the test run, then switch to the Attachments section and download the .trx file:

   

   Click the image to enlarge it.