Requirements and prerequisites
Your TestComplete projects must be prepared as described in Prepare TestComplete Project for Integration and added to your team project’s source control repository.
You must use the SmartBear ID-based license to manage TestComplete and TestExecute instances.
If you do not have the SmartBear ID-based license yet and want to try it, please contact our Customer Care team or our Sales team.
You must have the access key assigned to your SmartBear account. To get the access key, log in to the SmartBear Licensing Portal using your SmartBear ID (email and password) and select the following menu item:
You may need to set the screen resolution on the agents for your tests to be able to simulate user actions as expected. To set the screen resolution, you can use the Screen Resolution Utility extension provided by Microsoft. Install the extension and make it available for your team project. See the extension description for detailed instructions.
You can find more information on configuring Microsoft-hosted agents for UI-based testing at docs.microsoft.com/en-us/azure/devops/pipelines/test/ui-testing-considerations.
Configure a pipeline to run tests on Microsoft-hosted agents
Open the Azure DevOps project to which you want to add TestComplete tests.
Navigate to the Pipelines page of the Azure DevOps project.
Open an existing pipeline or click New to create a new one. You can create a YAML script that describes your pipeline, or you can create a pipeline via the visual editor.
Specify the repository that stores your TestComplete project files:
Add stages and jobs that will run TestComplete tests to the pipeline.
Specify the agent pool where your tests will run. To run TestComplete tests on Microsoft-hosted agents, use the Azure Pipelines pool. Specify a Windows-based virtual machine image that will be used to create agents:
Select the application that you want to use to run tests: TestComplete or TestExecute.
Configure the task to install the specified application on the agents.
If needed, configure the task to update the selected application. The task will update the application to match the extension version. If any patches or updates are available for the product, the task will install and apply them too.
Specify the access key assigned to your SmartBear account. This key will be used to license the TestComplete (TestExecute) instances running on the agents.
Specify when you want to retrieve test logs: only for failed tests, for failed tests and for tests with warnings, or for all tests.
By default, Microsoft-hosted agents have their screen resolution set to 1024 × 768. If your TestComplete tests interact with UI, you may need to configure the screen resolution on the agents to meet your testing goals. To do this, you can use the Screen Resolution Utility extension provided by Microsoft.
Install the extension and add the Set Screen Resolution task to the pipeline. Configure the task to set the screen resolution you need on the agents:
Add the Visual Studio Test task to your pipeline and configure the added task to run your TestComplete tests:
Configure the task to select tests by using assemblies.
Specify the project suite file (.pjs) that contains your tests and that the task will run.
Specify the folder on your agent where the task will search for TestComplete project files. By default, it will search for the files in the agent’s working folder.
Note: If you specify any Search folder other than the default working folder, in the Path to custom test adapters text box, specify the path to the TestComplete test adapter: $(System.DefaultWorkingDirectory)\TestCompleteAdapter.
If needed, you can use a filter expression to specify test items of your TestComplete project that you want to run.Filter syntaxFilter syntax
For TestComplete tests, the following parameters are available:
Parameter name Description
The name of the test as it is specified in your TestComplete project, on the Execution Plan page, or in the Project Explorer or in code, depending on where the test adapter searches for tests.
The name of the TestComplete project that belongs to the specified project suite and that includes the tests you want to run.
Applicable if the test adapter searches for tests in the project’s Execution Plan:
The name of the test item in the following format:
By default, the test adapter runs all available tests. To run only those tests that are enabled in the Execution Plan, set
The name of the group to which the needed test items belong as it is specified in your TestComplete project, on the Execution Plan page.
To run a group with all its child groups, use the
To run a group skipping all its child groups, use the
To run a child group without its parent group, specify its name using the
Suppose in the Execution Plan of your project, you have a group called Group1. In turn, this group has a child group called Group2.View imageHide image
Group=Group1- All tests that belong to Group1 will run. Tests that belong to the child group Group2 will not be run.
Group~Group1- All tests that belong to Group1 and all tests that belong to its child group Group2 will run.
Group~Group2- All tests that belong to the child group Group2 will run.
Applicable if the test adapter searches for tests among project items:
If set to
true, the test adapter will run all tests that have the tag_name tag assigned to them. If set to false, it will run tests that do not have such a tag assigned.
|to specify tag expressions.
@WebTests=true & @IE=false
If the parameter value includes a special character, use character escape sequences. For the list of available escape sequences, see the filter description at github.com/microsoft/vstest-docs/blob/master/docs/filter.md.
Available operators are:
Checks if the parameter equals the value.
Checks if the parameter does not equal the value.
Checks if the parameter contains the value.
You can also combine several expressions by using the
- stage: QA
- job: Job1
- task: InstallTestCompleteAdapter@1
- task: VSTest@2
testRunTitle: 'Automated tests'
If needed, specify other pipeline properties. You can learn more at docs.microsoft.com/en-us/azure/devops/pipelines/index.
Save the changes and run the pipeline.