Run TestComplete Tests as Part of a Test Plan

• Use self-hosted (private) agents to run TestComplete tests. Running TestComplete tests on Microsoft-hosted agents is not supported.

• Your team server must have the Test Manager extension installed and enabled.

• In your team project, you must have a test plan specified and test cases added to the plan. To learn how to create a test plan and manage test cases, see docs.microsoft.com/en-us/azure/devops/test-plans.

• Your test agents on which you will run tests must have TestComplete (or TestExecute) installed.

• Your team project must have the TestComplete test adapter extension installed. See Install TestComplete Test Adapter Extension.

• Your TestComplete test project must have test cases specified in its Test Items project.

• Your TestComplete project’s test cases must be linked to test cases of your team project’s test plan. To learn how to do it, see Link TestComplete Test Cases to Azure DevOps Test Cases.

• Your TestComplete project must reside in your team project’s repository. Otherwise, your test agents will not be able to access it.

1. If your TestComplete project still does not have test cases specified, specify them. To learn how to specify test cases, see the description of the Execution Plan editor.

2. If your TestComplete project is not added to your team project’s repository, add it. We recommend that you do it directly from TestComplete IDE. This way, only the necessary project files will be added to the repository. See the instructions in the Integration With Source Control Systems section.

If your TestComplete test cases specified in the Execution Plan editor of the project are not linked to test cases of your team test plan, link them. For detailed instructions, see:

Link TestComplete Test Cases to Azure DevOps Test Cases

Connect to your team project. If you still do not have a build for which you will run tests, create it. To learn how to create builds, see docs.microsoft.com/en-us/azure/devops/pipelines.

1. Open the web portal of your team project. You can navigate to it directly in your web browser, or you can open it from Team Explorer in Visual Studio.

2. Navigate to the Pipelines > Releases page of your team project.

3. Select an existing release pipeline and open it for editing or create a new pipeline:

   

   Click the image to enlarge it.

4. Add an artifact that includes your TestComplete project files to the release:

   

   Click the image to enlarge it.

5. Select a stage that will run your TestComplete tests or create a new stage.

6. Add an agent job to the pipeline. Specify agents on which your tests will run. Make sure that the agents meet all the requirements.

   If needed, configure the job to distribute the tests among multiple agents. To learn how to do it, see Run tests in parallel using the Visual Studio Test task. Note: Your tests must be able to run independently from each other.

7. Add the Enable support for TestComplete tests task. This task installs the TestComplete test adapter to your agent and enables the TestComplete test support:

   

   Click the image to enlarge it.

   The task is available only if you have the TestComplete Test Adapter extension installed on your server. To learn how to install it, see Install TestComplete Test Adapter Extension.

8. If needed, select an application that you want to use to run tests:

   • To run ordinary desktop and web tests sequentially, select TestComplete or TestExecute.

   • To run cross-platform web tests in parallel, select TestExecute Lite (Device Cloud Parallel). If needed, set the maximum number of tests that can run in parallel.To learn more about running tests in parallel, see About Running Tests in Parallel.

   

   Click the image to enlarge it.

   Note: If the selected application is not installed on the agent, the one that is installed will be used instead.

9. If needed, specify when you want to retrieve test logs: only for failed tests, for failed tests and for tests with warnings, or for all tests:

   

   Click the image to enlarge it.

10. Add the Visual Studio Test task to the stage.

    In the Select tests using drop-down box, select Test run.

    

    Click the image to enlarge it.

11. Specify other release properties, if needed.

12. Save the changes.

1. Navigate to the Test Plans page of your team project.

2. Locate the test case that has a TestComplete test assigned to it as automation:

   

   Click the image to enlarge it.

3. Click Run > Run with options.

4. In the Run with options dialog, do the following:

   • In the Test type and runner drop-down list, select Automated tests using release state.

   • In the Build text box, select a build for which you want to run tests.

   • In the Release pipeline drop-down list, select the pipeline you have configured on step 4.

   • In the Stage drop-down list, select a stage that will run your TestComplete tests.

   

   Click the image to enlarge it.

5. Click OK to save the changes and start the run.

6. To monitor the testing status, click View Test Run.

1. After a test run is over, you can view the test results:

   

   Click the image to enlarge it.

2. If a test has failed, you can double-click it to view the error message:

   

   Click the image to enlarge it.

3. To get the test log, download the log in the .mht format from the Attachments section:

   Depending on the Save logs for setting value, logs could be available only for failed tests, for failed tests and for tests with warnings, or for all tests.