Configure Build Pipeline (vNext Builds) to Run TestComplete Tests

Applies to TestComplete 14.50, last modified on April 22, 2021
This approach to running TestComplete tests as part of a build pipeline is obsolete. We do not recommend using it. Starting from version 14.20, to run TestComplete tests as part of a build, you can use the TestComplete test adapter. To learn more, see Integration With Azure DevOps and Team Foundation Server.

Define a build process that will run TestComplete tests. Depending on whether you use Azure DevOps Services, Azure DevOps Server or Team Foundation Server to manage your builds, and on your Team Foundation Server version, you can use either the Visual Studio Test task or Run Functional Tests tasks to run TestComplete tests on your build computer or on remote test agents.

Requirements

You must have access to your team project:

  • To access the project from Visual Studio Team Explorer, you must have Microsoft Visual Studio 2015–2019 (Community, Enterprise or Professional) with the Team Explorer component.

– or –

  • To access the project directly from your web browser by using Web Portal:

    • For Azure DevOps Services (formerly known as Visual Studio Team Services or VSTS):

      https://dev.azure.com/<Your_Azure_Account_Name>/<Project_Name>

      – or –

      https://<Your_VSO_Account_Name>.visualstudio.com/<Project_Name> (if your organization has not moved to the dev.azure.com domain yet)

    • For On-Premises Azure DevOps Server and Team Foundation Server:

      http://<Your_Server_Name>:8080/tfs/<Collection_Name>/<Project_Name>

    You can find information on Web Portal at docs.microsoft.com/en-us/azure/devops/project/navigation/.

The computer, where your TestComplete tests will run, must have the following software installed:

  • If you use Visual Studio Test tasks to run tests:

    Build and Release Agent for Windows.

    If you use Run Functional Tests tasks to run tests (available for Team Foundation Server 2015 Update 2–2017 On-Premises):

    Test Agent for Microsoft Visual Studio.

    Note: You can use the Visual Studio Test Agent Deployment task to install the test agent. To learn about the requirements the remote computers where the test engine will be installed must meet, see the task description.

    The build agent or test agent must be running as an interactive process so that TestComplete can interact with the desktop and with the tested application UI.

    An interactive user session must be opened on the computer where the agent is running. You can open the session manually, or, if you use Team Foundation Server 2018 or later, you can configure the agent to run with the auto-logon enabled so that it opens a user session automatically when it starts.

  • TestComplete 14 (or TestExecute 14).

  • TestComplete 14 Visual Studio Integration Package.

    The Integration Package installer is shipped along with TestComplete. You can find it in the TestComplete folder after installing the product:

    <TestComplete 14>\VS Integration\VSIntegration.exe

    Run this installer after installing TestComplete on computers where tests will run. In the installation wizard, select a Test Agent Integration feature appropriate to your agent version.

Configure a Build Definition

  1. Open the web portal of your team project. You can navigate to it directly in your web browser, or you can open it from Team Explorer in Visual Studio.

  2. If you use Azure DevOps or Team Foundation Server 2018, navigate to the Pipelines > Builds page.

    If you use Team Foundation Server 2015 Update 2–2017, navigate to the Build & Release > Builds page.

    If you use Team Foundation Server 2015 Update 1, navigate to the Build hub.

    TestComplete integration with Visual Studio: Builds hub in Web Portal

    Click the image to enlarge it.

  3. Open an existing build definition or click New to create a new build definition.

    TestComplete integration with Visual Studio: Create a new build definition or open an exisitng one

    Click the image to enlarge it.

    If you use Azure DevOps or Team Foundation Server 2018, you can create a YAML script that describes your pipeline, or you can click Use the visual designer to create a pipeline via a wizard.

    In the wizard, configure the build process (the way the wizard looks depends on whether you use Azure DevOps Services, Azure DevOps Server 2019 or one of the supported Team Foundation Server versions):

    TestComplete integration with Visual Studio: Configure a new build pipeline using the visual designer

    Click the image to enlarge it.

  4. Configure your build process to get the needed files from your source control.

    For example, if you use Git, specify the repository and branches that store your team project files:

    TestComplete integration with Visual Studio:  Get project files from the Azure Repos Git

    Click the image to enlarge it.

    Keep in mind that the Visual Studio test project stores the relative path to TestComplete tests. When mapping the TestComplete source control path, make sure to select a local folder to keep the relative path specified in the test project valid.
  5. Specify the details of the build process (what functions the build will perform):

    • Add a task that will build your Visual Studio test project:

      TestComplete integration with Visual Studio: Add a build task that will build your Visual Studio test project

      Click the image to enlarge it.

      In the Solution box, click the ellipsis button and browse for the Visual Studio test project that contains your TestComplete tests:

      TestComplete integration with Visual Studio: Select Visual Studio test project including TestComplete Test items

      Click the image to enlarge it.

    • Add tasks that will run your TestComplete tests. Depending on your server version, you can use one of the following tasks:

      Visual Studio Test Task

      Add a Visual Studio Test task to your definition:

      Add a test step

      Click the image to enlarge it.

      Configure the added task to run your TestComplete tests:

      In the Select tests using drop-down list box, select Test assemblies.

      In the Test files (or Test Assembly) text box, specify the TestComplete test items (.tctest files) to run.

      In the Search folder text box, specify the folder on your build machine or test agent where the task should search for TestComplete tests..

      In the Test run title text box, enter the name of the test run to which TestComplete test results will be assigned.

      TestComplete integration with Visual Studio: Select TestComplete Test Items

      Click the image to enlarge it.

      Note: If you use task version 1, TestComplete test results will not be published to the server. To learn how to include TestComplete test results in build reports, see Publish TestComplete Test Results.

      Run Functional Tests Task

      Note: This task is deprecated in Azure DevOps Services, Azure DevOps Server 2019 and Team Foundation Server 2018. You can use it if you use Team Foundation Server 2015 Update 2–2017 On-Premises.

      1. Share a folder on remote computers where tests will run. To that folder, the test files will be placed.

      2. Configure your definition to copy your TestComplete tests from build output folder to computers where tests will run. For example, you can use the Windows Machine File Copy task to do this:

        TestComplete integration with Visual Studio: Copying test files to remote agents

        Click the image to enlarge it.

        Note:Make sure that the path to TestComplete projects or project suites that the TestComplete test items store remains valid.
      3. Configure your definition to deploy Test Agent for Visual Studio on computers where tests will run:

        TestComplete integration with Visual Studio: Deploying agents on remote computers

        Click the image to enlarge it.

        Specify the computers, to which the agent will be deployed. Specify the login and password of the user account that will be used to deploy agents.

        Important: Configure the agent to run UI tests. Otherwise, the agent will launch TestComplete (or TestExecute) in non-interactive mode, and it will not be able to access UI elements to simulate user actions.

      4. Make sure that TestComplete (or TestExecute) and the TestComplete Integration package are installed on remote test agents before you run tests.

        You can configure your definition to install them by using the Run PowerShell Script on Target Machines task. Configure the task to run a script that will install TestComplete (or TestExecute) and the Integration package via command line in silent mode.

        To learn more about installing TestComplete from the command line, see Installing TestComplete in Silent Mode. To learn about installing TestExecute in silent mode, see the appropriate topic in the TestExecute documentation.

        To learn more about installing the Integration package via command line, see Installing Integration Package.

      5. Add a Run Functional Tests task to the definition:

        TestComplete integration with Visual Studio: Adding a Run Functional Tests task to a team build

        Click the image to enlarge it.

        Configure the added task to run your TestComplete tests:

        In the Machines text box, enter the network names or IP addresses of test agents where tests will run.

        In the Test Drop Location text box, enter the path where your tests reside.

        In the Test Selection drop-down list box, select Test Assembly.

        TestComplete integration with Visual Studio: Configuring the Run Functional Tests task to run TestComplete tests

        Click the image to enlarge it.

        To include TestComplete test results into a build report and assign them to a test run, in the Test Run Title text box, enter the needed test run name.

    You can find information about tasks in the Build and release tasks article in the MSDN Library.

  6. If needed, specify other build properties. You can learn more at docs.microsoft.com/en-us/azure/devops/pipelines/index.

  7. Save the changes.

Prev     Next

See Also

Defining a Build Process
Running TestComplete Tests as Part of a Build and Viewing Test Results

Highlight search results