Run TestComplete Tests on Microsoft-Hosted Agents

Applies to TestComplete 15.40, last modified on March 25, 2022

Requirements and prerequisites

Configure a pipeline to run tests on Microsoft-hosted agents

  1. Open the Azure DevOps project to which you want to add TestComplete tests.

  2. Navigate to the Pipelines page of the Azure DevOps project.

  3. Open an existing pipeline or click New to create a new one. You can create a YAML script that describes your pipeline, or you can create a pipeline via the visual editor.

  4. Specify the repository that stores your TestComplete project files:

    YAML

    name: Build1
    trigger:
      - main
    resources:
      repositories:
        - repository: AutomatedTestsRepo
          type : github
          endpoint: connection
          name: MyGitHubRepo/AutomatedTests

      steps:
        - checkout: self
        - checkout: AutomatedTestsRepo
          path: AutomatedTests
          clean: true


    Visual editor

    TestComplete integration with Azure DevOps: Getting TestComplete project suite from a repository

    Click the image to enlarge it.

  5. Add stages and jobs that will run TestComplete tests to the pipeline.

  6. Specify the agent pool where your tests will run. To run TestComplete tests on Microsoft-hosted agents, use the Azure Pipelines pool. Specify a Windows-based virtual machine image that will be used to create agents:

    YAML

    name: Build1

    stages:
      - stage: QA
        displayName: QA
        jobs:
        - job: Job1
          displayName: Job1
          continueOnError: true
          pool:
            name: Azure Pipelines
            vmImage: 'windows-latest'

    Visual editor

    TestComplete integration with Azure DevOps: Specifying Microsoft-hosted agents where TestComplete tests will run

    Click the image to enlarge it.

  7. Add the Enable support for TestComplete tests task to the pipeline and configure it to install TestComplete (or TestExecute) to the agents:

    • Select the application that you want to use to run tests: TestComplete or TestExecute.

    • Configure the task to install the specified application on the agents.

    • If needed, configure the task to update the selected application. The task will update the application to match the extension version. If any patches or updates are available for the product, the task will install and apply them too.

    • Specify the access key assigned to your SmartBear account. This key will be used to license the TestComplete (TestExecute) instances running on the agents.

    • Specify when you want to retrieve test logs: only for failed tests, for failed tests and for tests with warnings, or for all tests.

    YAML

    name: Build1

    stages :
      - stage: QA
      …
        jobs:
        - job: Job1
        …
          - task: InstallTestCompleteAdapter@1
            inputs:
              installExecutor:true
              updateExecutor: true
              accessKey: '123-456-789'
              logsLevel: '0'
              preferredExecutor: TE


    Visual editor

    TestComplete integration with Azure DevOps: Configuring the Enable TestComplete tests task

    Click the image to enlarge it.

  8. By default, Microsoft-hosted agents have their screen resolution set to 1024 × 768. If your TestComplete tests interact with UI, you may need to configure the screen resolution on the agents to meet your testing goals. To do this, you can use the Screen Resolution Utility extension provided by Microsoft.

    Install the extension and add the Set Screen Resolution task to the pipeline. Configure the task to set the screen resolution you need on the agents:

    YAML

    name: Build1

    stages:
      - stage QA
      …
        jobs:
        - job: Job1
        …
          - task: ScreenResolutionUtility@1
            inputs:
              displaySettings: specific
              width: 1920
              height: 1080

    Visual editor

    TestComplete integration with Azure DevOps: Configuring the Set Screen Resolution tests task

    Click the image to enlarge it.

  9. Add the Visual Studio Test task to your pipeline and configure the added task to run your TestComplete tests:

    1. Configure the task to select tests by using assemblies.

    2. Specify the project suite file (.pjs) that contains your tests and that the task will run.

    3. Specify the folder on your agent where the task will search for TestComplete project files. By default, it will search for the files in the agent’s working folder.

      Note: If you specify any Search folder other than the default working folder, in the Path to custom test adapters text box, specify the path to the TestComplete test adapter: $(System.DefaultWorkingDirectory)\TestCompleteAdapter.

    4. If needed, you can use a filter expression to specify test items of your TestComplete project that you want to run.

      Filter syntax

    YAML

    name: Build1

    stages:
      - stage: QA
        displayName: QA
        jobs:
        - job: Job1
          displayName: Job1
          …
          steps:
          …
          - task: InstallTestCompleteAdapter@1
          …
          - task: VSTest@2
            inputs:
              testSelector: 'testAssemblies'
              testAssemblyVer2: '**\*.pjs'
              searchFolder: '$(System.DefaultWorkingDirectory)'
              testFiltercriteria: 'Name=Test1'
              vsTestVersion: 'toolsInstaller'
              testRunTitle: 'Automated tests'

    Visual editor

    TestComplete integration with Azure DevOps: Configuring a Visual Studio Test task to run TestComplete tests

    Click the image to enlarge it.

  10. If needed, specify other pipeline properties. You can learn more at docs.microsoft.com/en-us/azure/devops/pipelines/index.

  11. Save the changes and run the pipeline.

See Also

Integration With Azure DevOps and Team Foundation Server via TestComplete Test Adapter
Install TestComplete Test Adapter Extension

Highlight search results