Run TestComplete Tests as Part of a Release

Use self-hosted (private) agents to run TestComplete tests. Running TestComplete tests on Microsoft-hosted agents is not supported.

• Your test agents on which you will run tests must have TestComplete (or TestExecute) installed.

• Your team project must have the TestComplete test adapter extension installed. See Install TestComplete Test Adapter Extension.

To run TestComplete tests during release deployment, you need to put your TestComplete test files into the folder where your release pipeline can access them. The easiest way to do it is to place them into a build output folder along with other artifacts:

1. Open the web portal of your team project. You can navigate to it directly in your web browser, or you can open it from Team Explorer in Visual Studio.

2. Navigate to the Pipelines > Builds page of your team project.

3. Open an existing build definition or click New to create a new build definition.

   

   Click the image to enlarge it.

   You can create a YAML script that describes your pipeline, or you can click Use the visual designer to create a pipeline via a wizard. This topic describes how to configure a build pipeline using the visual designer.

4. Configure your build pipeline to get your TestComplete project files from your team repository.

   For example, if you are using Team Foundation Version Control, click the ellipsis button of the Server path edit box and locate the repository directory that stores your TestComplete project suite. Then, in the Local path under $(build.sourcesDirectory) text box, specify the local folder on your test agents to which your TestComplete project suite will be placed:

   

   Click the image to enlarge it.

5. Add an agent job to the pipeline. Specify agents on which your tests will run. Make sure that the agents meet all the requirements.

   If needed, configure the job to distribute the tests among multiple agents. To learn how to do it, see Run tests in parallel using the Visual Studio Test task. Note: Your tests must be able to run independently from each other.

6. Add a task that will copy your TestComplete project files to an artifact staging folder on your build computer:

   

   Click the image to enlarge it.

7. Add a task that will publish your build artifacts to the needed location:

   

   Click the image to enlarge it.

8. If needed, add other tasks to the build. You can learn more at docs.microsoft.com/en-us/azure/devops/pipelines/index.

9. Save the changes.

Run the created build definition to create a build that you will release:

1. On the Builds page, click Queue.

2. In the Queue Build dialog, configure the build run and click OK.

1. Navigate to the Pipelines > Releases page of your team project.

2. Select an existing release pipeline and open it for editing or create a new pipeline:

   

   Click the image to enlarge it.

3. Add an artifact that includes your TestComplete project files to the release:

   

   Click the image to enlarge it.

4. Select a stage that will run your TestComplete tests or create a new stage.

5. Specify agents on which your tests will run. Make sure that the agents meet all the requirements.

6. If your test agents do not have Visual Studio installed, you can add the Install Visual Studio Test Platform task to the stage. The task will install a test platform that allows running tests on computers where Visual Studio is not installed. To learn about the task, see the task description:

   

   Click the image to enlarge it.

7. Add the Enable support for TestComplete tests task. This task installs the TestComplete test adapter to your agent and enables the TestComplete test support:

   

   Click the image to enlarge it.

   The task is available only if you have the TestComplete Test Adapter extension installed on your server. To learn how to install it, see Install TestComplete Test Adapter Extension.

8. If needed, select an application that you want to use to run tests: TestComplete or TestExecute:

   

   Click the image to enlarge it.

   Note: If the selected application is not installed on the agent, the one that is installed will be used instead.

9. Add the Visual Studio Test task to the stage and configure the task to run your TestComplete tests:

   By specifying a TestComplete project suite directly

   By specifying a test plan

   By specifying a TestComplete project suite directly

   In the Select tests using drop-down list box, select Test assemblies.

   In the Test files text box, specify your TestComplete project suite (a .pjs file) to run.

   In the Search folder text box, specify the folder on your agent where the task should search for the specified TestComplete project suite.

   In the Test filter criteria text box, you can enter a filter expression that will specify one or several test items of your TestComplete project that you want to run.

   Filter syntax

   Filter syntaxFilter syntax

   <parameter><operator><value>

   For TestComplete tests, the following parameters are available:

   • Name - The name of the test item as it is specified in your TestComplete project, on the Test Items page.

   • Project - The name of the TestComplete project that belongs to the specified project suite and that includes the test items you want to run.

   • FullyQualifiedName - The name of the test item in the following format: testitem:<project_name>\<test_item_name>.

   • Enabled - Specifies whether the test item or a group is enabled in the TestComplete project.

   • Group - The name of the group to which the needed test items belong as it is specified in your TestComplete project, on the Test Items page.

   Available operators are:

   =	Checks if the parameter equals the value.
   !=	Checks if the parameter does not equal the value.
   ~	Checks if the parameter contains the value.

   You can also combine several expressions by using the & (and) and | (or) operators.

   For example:

   Name=ProjectTestItem1

   Project=TestProject1

   Group=Orders&Enabled=True

   Name!=Test2

   Name~Test

   Group=Group1|Group=Group3

   In the Test run title text box, you can enter the name of the test run to which TestComplete test results will be assigned.

   In the Test platform version drop-down list, select the version of your Visual Studio Test Platform.

   

   Click the image to enlarge it.

   If you are using version 1 of the Visual Studio Test task, in the Path to Custom Test Adapters text box, specify the path to TestComplete test adapter: $(System.DefaultWorkingDirectory)\TestCompleteAdapter:

   

   Click the image to enlarge it.

   By specifying a test plan

   To run TestComplete tests as part of a test plan, your TestComplete project must be bound to your Azure DevOps team project, and your TestComplete test cases must be linked to the team project’s test cases. See Link TestComplete Test Cases to Azure DevOps Test Cases.

   In the Select tests using drop-down list, select Test plan.

   In the Test plan drop-down list, select the test plan to which the tests you want to run belong.

   In the Test suite drop-down list, select the test suites that include the needed tests.

   In the Test configuration drop-down list, select the test environment where your tests will run.

   In the Search folder text box, specify the folder on your agent where the task should search for the tests.

   If needed, in the Test run title text box, you can enter the name of the test run to which TestComplete test results will be assigned.

   

   Click the image to enlarge it.

10. Save the changes.

After you have configured a release pipeline, you can deploy the release. Your TestComplete tests will be run during the release deployment.

1. On the Pipelines > Releases page of your team project, click Create a release.

2. In the resulting dialog, select the build that you want to deploy. Click Create to create a release.

3. On the Releases page, view the release status. You can click the release to view detailed information on the deployment status of each of its stages:

   

   Click the image to enlarge it.

1. After a release is deployed, you can view its results:

   

   Click the image to enlarge it.

2. To view test results, click the needed stage, and then switch to the Tests section:

   

   Click the image to enlarge it.

   It shows the summary report on the test run: the number of failed and passed tests and the status of each of them.

3. If a test has failed, you can view information on the occurred error in the Debug section:

   

   Click the image to enlarge it.

   To get the log generated for the failed test, download the log in the .mht format from the Attachment section:

   

   Click the image to enlarge it.

4. To get the log generated for the entire test run, click the test run, then switch to the Attachments page and download the .trx file:

   

   Click the image to enlarge it.