Run TestComplete Tests on Microsoft-Hosted Agents

Open the Azure DevOps project to which you want to add TestComplete tests.

Navigate to the Pipelines page of the Azure DevOps project.

Open an existing pipeline or click New to create a new one. You can create a YAML script that describes your pipeline, or you can create a pipeline via the visual editor.

Specify the repository that stores your TestComplete project files:

YAML

Visual editor

Copy Code

YAML

name: Build1
trigger:
  - main
resources:
  repositories:
    - repository: AutomatedTestsRepo
      type : github
      endpoint: connection
      name: MyGitHubRepo/AutomatedTests
…
  steps:
    - checkout: self
    - checkout: AutomatedTestsRepo
      path: AutomatedTests
      clean: true

Visual editor

Click the image to enlarge it.

Add stages and jobs that will run TestComplete tests to the pipeline.

Specify the agent pool where your tests will run. To run TestComplete tests on Microsoft-hosted agents, use the Azure Pipelines pool. Specify a Windows-based virtual machine image that will be used to create agents:

YAML

Visual editor

Copy Code

YAML

name: Build1
…
stages:
  - stage: QA
    displayName: QA
    jobs:
    - job: Job1
      displayName: Job1
      continueOnError: true
      pool:
        name: Azure Pipelines
        vmImage: 'windows-latest'

Visual editor

Click the image to enlarge it.

Add the Enable support for TestComplete tests task to the pipeline and configure it to install TestComplete (or TestExecute) to the agents:

• Select the application that you want to use to run tests: TestComplete or TestExecute.

• Configure the task to install the specified application on the agents.

• If needed, configure the task to update the selected application. The task will update the application to match the extension version. If any patches or updates are available for the product, the task will install and apply them too.

• Specify the access key assigned to your SmartBear account. This key will be used to license the TestComplete (TestExecute) instances running on the agents.

• Specify when you want to retrieve test logs: only for failed tests, for failed tests and for tests with warnings, or for all tests.

YAML

Visual editor

Copy Code

YAML

name: Build1
…
stages :
  - stage: QA
  …
    jobs:
    - job: Job1
    …
      - task: InstallTestCompleteAdapter@1
        inputs:
          installExecutor:true
          updateExecutor: true
          accessKey: '123-456-789'
          logsLevel: '0'
          preferredExecutor: TE

…

Visual editor

Click the image to enlarge it.

By default, Microsoft-hosted agents have their screen resolution set to 1024 × 768. If your TestComplete tests interact with UI, you may need to configure the screen resolution on the agents to meet your testing goals. To do this, you can use the Screen Resolution Utility extension provided by Microsoft.

Install the extension and add the Set Screen Resolution task to the pipeline. Configure the task to set the screen resolution you need on the agents:

YAML

Visual editor

Copy Code

YAML

name: Build1
…
stages:
  - stage QA
  …
    jobs:
    - job: Job1
    …
      - task: ScreenResolutionUtility@1
        inputs:
          displaySettings: specific
          width: 1920
          height: 1080
…

Visual editor

Click the image to enlarge it.

Add the Visual Studio Test task to your pipeline and configure the added task to run your TestComplete tests:

1. Configure the task to select tests by using assemblies.

2. Specify the project suite file (.pjs) that contains your tests and that the task will run.

3. Specify the folder on your agent where the task will search for TestComplete project files. By default, it will search for the files in the agent’s working folder.

   Note: If you specify any Search folder other than the default working folder, in the Path to custom test adapters text box, specify the path to the TestComplete test adapter: $(System.DefaultWorkingDirectory)\TestCompleteAdapter.

4. If needed, you can use a filter expression to specify test items of your TestComplete project that you want to run.

   Filter syntax

   Filter syntaxFilter syntax

   <parameter><operator><value>

   For TestComplete tests, the following parameters are available:

   Parameter name	Description
   Name	The name of the test as it is specified in your TestComplete project, on the Execution Plan page, or in the Project Explorer or in code, depending on where the test adapter searches for tests. For example: Name=Test1
   Project	The name of the TestComplete project that belongs to the specified project suite and that includes the tests you want to run. For example: Project=Orders_Project
   Applicable if the test adapter searches for tests in the project’s Execution Plan:
   FullyQualifiedName	The name of the test item in the following format: testitem:<project_name>\\<test_item_name>. For example: testitem:MyProject\\Test1
   Enabled	By default, the test adapter runs all available tests. To run only those tests that are enabled in the Execution Plan, set Enabled=true. For example: Enabled=true Group=Group1&Enabled=true
   Group	The name of the group to which the needed test items belong as it is specified in your TestComplete project, on the Execution Plan page. To run a group with all its child groups, use the ~ condition. To run a group skipping all its child groups, use the = condition. To run a child group without its parent group, specify its name using the ~ condition. For example: Suppose in the Execution Plan of your project, you have a group called Group1. In turn, this group has a child group called Group2. View image View imageHide image Group=Group1 - All tests that belong to Group1 will run. Tests that belong to the child group Group2 will not be run. Group~Group1 - All tests that belong to Group1 and all tests that belong to its child group Group2 will run. Group~Group2 - All tests that belong to the child group Group2 will run.
   Applicable if the test adapter searches for tests among project items:
   @tag_name	If set to true, the test adapter will run all tests that have the tag_name tag assigned to them. If set to false, it will run tests that do not have such a tag assigned. Use & and | to specify tag expressions. For example: @MyTag=true @WebTests=true & @IE=false

   If the parameter value includes a special character, use character escape sequences. For the list of available escape sequences, see the filter description at github.com/microsoft/vstest-docs/blob/master/docs/filter.md.

   Available operators are:

   =	Checks if the parameter equals the value.
   !=	Checks if the parameter does not equal the value.
   ~	Checks if the parameter contains the value.

   You can also combine several expressions by using the & (and) and | (or) operators.

   For example:

   Group=Orders&Enabled=True

   @Tag1=true&@Tag2=false

   Group=Group1|Group=Group3

YAML

Visual editor

Copy Code

YAML

name: Build1
…
stages:
  - stage: QA
    displayName: QA
    jobs:
    - job: Job1
      displayName: Job1
      …
      steps:
      …
      - task: InstallTestCompleteAdapter@1
      …
      - task: VSTest@2
        inputs:
          testSelector: 'testAssemblies'
          testAssemblyVer2: '**\*.pjs'
          searchFolder: '$(System.DefaultWorkingDirectory)'
          testFiltercriteria: 'Name=Test1'
          vsTestVersion: 'toolsInstaller'
          testRunTitle: 'Automated tests'
…

Visual editor

Click the image to enlarge it.

If needed, specify other pipeline properties. You can learn more at docs.microsoft.com/en-us/azure/devops/pipelines/index.

Save the changes and run the pipeline.