Run TestComplete Tests as Part of a Pipeline

• Your TestComplete projects must be prepared as described in Prepare TestComplete Project for Integration and added to your team project’s source control repository.

• Your test agents on which you will run tests must have TestComplete (or TestExecute) installed.

  Note: You can configure the pipeline to install TestComplete (or TestExecute) automatically. See below.

• Your team project must have the TestComplete test adapter extension installed. See Install TestComplete Test Adapter Extension.

1. Open the web portal of your team project. You can navigate to it directly in your web browser, or you can open it from Team Explorer in Visual Studio.

2. Navigate to the Pipelines page.

3. Open an existing pipeline or click New pipeline to create a new one.

   You can create a YAML script that describes your pipeline, or you can create a pipeline with the visual editor. To create a pipeline with the visual editor, click Use the classical editor:

   

   Click the image to enlarge it.

4. If needed, specify the repository that stores your TestComplete project files:

   For example:

   YAML

   Visual editor

   Copy Code

   YAML

   name: Build1
   trigger:
     - main
   resources:
     repositories:
       - repository: AutomatedTestsRepo
         type : github
         endpoint: connection
         name: MyGitHubRepo/AutomatedTests
   …
     steps:
       - checkout: self
       - checkout: AutomatedTestsRepo
         path: AutomatedTests
         clean: true

   
   

   Visual editor

   

   Click the image to enlarge it.

5. Add stages and jobs that will run your TestComplete tests.

6. Specify agents on which your tests will run:

   • If you use Microsoft-hosted agents, specify a Windows-based virtual machine that will be used to create agents. For example:

     YAML

     Visual editor

     Copy Code

     YAML

     name: Build1
     …
     stages:
       - stage: QA
         displayName: QA
         jobs:
         - job: Job1
           displayName: Job1
           continueOnError: true
           pool:
             name: Azure Pipelines
             vmImage: 'windows-latest'

     Visual editor

     

     Click the image to enlarge it.

     The agents must have TestComplete (or TestExecute) installed on them, and the installed product must use the SmartBear ID-based license. To learn how to install the product on the agents and how to specify the SmartBear ID, see below.

     Note: Running web tests in Edge on Microsoft-hosted agents is not supported. As a workaround, you can use another web browser, or you can run your web tests on self-hosted agents instead.

   • If you use self-hosted (private) agents, you can do one of the following:

     • Use demands to specify agents that already have TestComplete (or TestExecute) installed:

       YAML

       Visual editor

       Copy Code

       YAML

       name: Build1
       …
       stages:
         - stage: QA
           displayName: QA
           jobs:
           - job: Job1
             displayName: Job1
             continueOnError: true
             pool:
               name: QA
               demands: TestRunner -equals TestExecute

       Visual editor

       

       Click the image to enlarge it.

     • Configure the pipeline to install TestComplete (or TestExecute) to the agents. See below.

   If needed, configure the job to distribute the tests among multiple agents. To learn how to do it, see Run tests in parallel using the Visual Studio Test task. Note: Your tests must be able to run independently from each other.

7. If your test agents do not have Visual Studio installed, you can add the Install Visual Studio Test Platform task to the stage. The task will install the test platform that allows running tests on computers where Visual Studio is not installed. To learn about the task, see the task description.

   YAML

   Visual editor

   Copy Code

   YAML

   name: Build1
   …
   stages:
     - stage: QA
       displayName: QA
       jobs:
       - job: Job1
         …
         steps:
         …
         - task: VisualStudioTestPlatformInstaller@1
           inputs:
             packageFeedSelector: 'nugetOrg'
             versionSelector: 'latestStable'
   …

   Visual editor

   

   Click the image to enlarge it.

8. Add the Enable support for TestComplete tests task.

   Note: If the task is not available, make sure that you have the TestComplete Test Adapter extension installed on your server. See Install TestComplete Test Adapter Extension.

   Configure the task:

   • Select the application that you want to use to run tests: TestComplete or TestExecute.

     Note: If the selected application is not installed on the agent, the one that is installed will be used instead.

   • If needed, configure the task to install the specified test runner on agents. If neither TestComplete nor TestExecute is installed on the agent, the task will install the selected preferred test runner to the agent.

   • If needed, configure the task to update the test runner. The task will update the installed product to match the extension version. If any patches or updates, for instance, Google Chrome Support update, are available for the product, the task will install them either.

   • If you use a SmartBear ID-based license for your TestComplete (TestExecute) instance, specify the access key assigned to your SmartBear account. This key will be used to license the TestComplete (TestExecute) instances running on agents.

     How to get the access key from the Licensing Portal

     How to get the access key from the Licensing PortalHow to get the access key from the Licensing Portal

     Log in to the SmartBear Licensing Portal using your SmartBear ID (email and password) and select the following menu item:

     

     Click the image to enlarge it.

     If you are already logged in with your SmartBear account on the test agent, you can skip this step.

     If you use a key-based license, skip this step.

     If you do not have the SmartBear ID-based license yet and want to try it, please contact our Customer Care team or our Sales team.

   • Specify how the test adapter will search for tests to run:

     • In Execution Plan - The test adapter will search for test cases in the Execution Plan of your TestComplete projects. It will run the following items:

       • Separate keyword tests and script routines marked as test cases.

       • Separate BDD scenarios.

       • Scenario outlines. The Treat each example in Scenario Outline as a separate test case option of the project must be disabled.

     • Among project items - The test adapter will search for test cases among all items of TestComplete projects. It will run the following:

       • Keyword tests and script tests that have any tag assigned to them.

         By default, the test adapter treats all tests with tags assigned to them as test cases and will run all these tests. To run only tests that have a specific tag assigned, use the filter expression.

       • BDD scenarios.

       • Scenario outlines, if the Treat each example in Scenario Outline as a separate test case option of the project is disabled.

       • Script routines marked as test cases in code. To mark a routine as a test case, add a comment in the // @"test_case_name" format right before the routine declaration. Do not omit double quotes.

     You can also specify a filter expression to run only those found tests that match it. See below.

   • Specify when you want to retrieve test logs: only for failed tests, for failed tests and for tests with warnings, or for all tests.

   • If needed, configure the task to pass additional command-line parameters to TestComplete (or TestExecute).

     You can, for example, use the /EnableModule parameter to disable redundant TestComplete modules and addons on your test agents. You can also pass pipeline and task variables to your TestComplete tests by using the /PrjVar and /PSVar parameters or by using custom parameters. See Pass Environment Variables to TestComplete Tests.

   YAML

   Visual editor

   Copy Code

   YAML

   name: Build1
   …
   stages :
     - stage: QA
     …
       jobs:
       - job: Job1
       …
         - task: InstallTestCompleteAdapter@1
           inputs:
             installExecutor:true
             updateExecutor: true
             : '123-456-789'
             searchMode: ExecutionPlan
             logsLevel: '0'
             cmdLineParams: '/PrjVar:UserName="adam.smith"'
             preferredExecutor: TE

   
   …

   Visual editor

   

   Click the image to enlarge it.

9. By default, Microsoft-hosted agents have their screen resolution set to 1024 × 768. This may affect automated tests that interact with GUI. To avoid possible issues, set a proper screen resolution on the agents before running tests. You can do this by using the Screen Resolution Utility extension provided by Microsoft. See Run TestComplete Tests on Microsoft-Hosted Agents.

10. Add the Visual Studio Test task to your pipeline and configure the added task to run your TestComplete tests:

    By specifying a TestComplete project suite directly

    By specifying a test plan

    By specifying a TestComplete project suite directly

    1. Configure the task to select tests by using assemblies.

    2. Specify the project suite file (.pjs) that contains your tests and that the task will run.

    3. Specify the folder on your agent where the task will search for TestComplete project files. By default, it will search for the files in the agent’s working folder.

       Note: If you specify any Search folder other than the default working folder, in the Path to custom test adapters text box, specify the path to the TestComplete test adapter: $(System.DefaultWorkingDirectory)\TestCompleteAdapter.

    4. If needed, you can use a filter expression to specify test items of your TestComplete project that you want to run.

       Filter syntax

       Filter syntaxFilter syntax

       <parameter><operator><value>

       For TestComplete tests, the following parameters are available:

       Parameter name	Description
       Name	The name of the test as it is specified in your TestComplete project, on the Execution Plan page, or in the Project Explorer or in code, depending on where the test adapter searches for tests. For example: Name=Test1
       Project	The name of the TestComplete project that belongs to the specified project suite and that includes the tests you want to run. For example: Project=Orders_Project
       Applicable if the test adapter searches for tests in the project’s Execution Plan:
       FullyQualifiedName	The name of the test item in the following format: testitem:<project_name>\\<test_item_name>. For example: testitem:MyProject\\Test1
       Enabled	By default, the test adapter runs all available tests. To run only those tests that are enabled in the Execution Plan, set Enabled=true. For example: Enabled=true Group=Group1&Enabled=true
       Group	The name of the group to which the needed test items belong as it is specified in your TestComplete project, on the Execution Plan page. To run a group with all its child groups, use the ~ condition. To run a group skipping all its child groups, use the = condition. To run a child group without its parent group, specify its name using the ~ condition. For example: Suppose in the Execution Plan of your project, you have a group called Group1. In turn, this group has a child group called Group2. View image View imageHide image Group=Group1 - All tests that belong to Group1 will run. Tests that belong to the child group Group2 will not be run. Group~Group1 - All tests that belong to Group1 and all tests that belong to its child group Group2 will run. Group~Group2 - All tests that belong to the child group Group2 will run.
       Applicable if the test adapter searches for tests among project items:
       @tag_name	If set to true, the test adapter will run all tests that have the tag_name tag assigned to them. If set to false, it will run tests that do not have such a tag assigned. Use & and | to specify tag expressions. For example: @MyTag=true @WebTests=true & @IE=false

       If the parameter value includes a special character, use character escape sequences. For the list of available escape sequences, see the filter description at github.com/microsoft/vstest-docs/blob/master/docs/filter.md.

       Available operators are:

       =	Checks if the parameter equals the value.
       !=	Checks if the parameter does not equal the value.
       ~	Checks if the parameter contains the value.

       You can also combine several expressions by using the & (and) and | (or) operators.

       For example:

       Group=Orders&Enabled=True

       @Tag1=true&@Tag2=false

       Group=Group1|Group=Group3

    5. You may also specify the test run name to which test results will be assigned.

    YAML

    Visual editor

    Copy Code

    YAML

    name: Build1
    …
    stages:
      - stage: QA
        displayName: QA
        jobs:
        - job: Job1
          displayName: Job1
          …
          steps:
          …
          - task: InstallTestCompleteAdapter@1
          …
          - task: VSTest@2
            inputs:
              testSelector: 'testAssemblies'
              testAssemblyVer2: '**\*.pjs'
              searchFolder: '$(System.DefaultWorkingDirectory)'
              testFiltercriteria: 'Name=Test1'
              vsTestVersion: 'toolsInstaller'
              testRunTitle: 'Automated tests'
    …

    Visual editor

    

    Click the image to enlarge it.

    If you are using version 1 of the Visual Studio Test task or if you have set the Search folder value to any value other than the default working folder, in the Path to Custom Test Adapters text box, specify the path to the TestComplete test adapter: $(System.DefaultWorkingDirectory)\TestCompleteAdapter.

    Visual editor

    Copy Code

    Visual editor

    

    Click the image to enlarge it.

    By specifying a test plan

    To run TestComplete tests as part of a test plan, your TestComplete project must be bound to your team project, and your TestComplete test cases must be linked to the team project’s test cases. See Link TestComplete Test Cases to Azure DevOps Test Cases.

    1. Configure the task to run tests of a test plan.

    2. Specify the test plan and a test suite that includes the TestComplete tests.

    3. Specify the test environments where you want to run tests.

    4. Specify the folder on your agent where the task will search for TestComplete project files. By default, it will search for the files in the agent’s working folder.

       Note: If you specify any Search folder other than the default working folder, in the Path to custom test adapters text box, specify the path to the TestComplete test adapter: $(System.DefaultWorkingDirectory)\TestCompleteAdapter.

    5. You may also specify the test run name to which test results will be assigned.

    YAML

    Visual editor

    Copy Code

    YAML

    name: Build1
    …
      stages:
        - stage: QA
          displayName: QA
          jobs:
          - job: Job1
            displayName: Job1
            …
            steps:
              …
            - task: InstallTestCompleteAdapter@1
            …
            - task: VSTest@2
              inputs:
                testSelector: 'testPlan'
                testPlan: 'My test plan 1'
                testSuite: 'My test suite 1'
                testConfiguration: '8'
                searchFolder: '$(System.DefaultWorkingDirectory)'
                vsTestVersion: 'toolsInstaller'
                testRunTitle: 'Automated tests'

    
    …

    Visual editor

    

    Click the image to enlarge it.

11. If needed, specify other pipeline properties. You can learn more at docs.microsoft.com/en-us/azure/devops/pipelines/index.

12. Save the changes.

Queue the created pipeline, for example, by clicking Run. On the Build page of the pipeline, you can monitor the pipeline progress.

The Visual Studio Test task will locate the specified TestComplete tests and run them.

1. Switch to the Tests section of the pipeline. It shows a summary report of a test run: the number of failed and passed tests and the status of each of them:

   

   Click the image to enlarge it.

2. If a test has failed, you can view information on the occurred error in the Debug section:

   

   Click the image to enlarge it.

3. To get the detailed test log, you can download the log in the .mht format from the Attachments section:

   

   Click the image to enlarge it.

   Depending on the Save logs for setting value, logs could be available only for failed tests, for failed tests and for tests with warnings, or for all tests.

4. To get a log generated for the entire test run, click the test run, then switch to the Attachments section and download the .trx file:

   

   Click the image to enlarge it.