Migrating Distributed Tests to Jenkins

Applies to TestComplete 15.40, last modified on March 25, 2022

Starting from TestComplete version 14.81, the Network Suite functionality is deprecated. Though it will not be removed from the product, we do not recommend that you use it to manage distributed test runs. We would recommend that you use a CI/CD system instead.

This section explains how to migrate your distributed test runs from Network Suite to Jenkins.

In This Document

Prerequisite: TestComplete Support plugin for Jenkins

The most convenient way to run TestComplete tests from Jenkins projects is to use the TestComplete support plugin. The plugin appends the TestComplete Test step to the list of available build steps. You use this step to run TestComplete tests.

To install the TestComplete Support plugin to your Jenkins server:

  1. Log in to your Jenkins instance as an administrator.

  2. Go to Manage Jenkins > Manage Plugins.

  3. Switch to the Available tab, and in the Search field, type TestComplete (you can type part of this phrase). Jenkins will update the list of available plugins that match the typed text.

  4. Select the check box next to the TestComplete support plugin and click Install without restart.

Possible alternatives

There are other ways to control TestComplete test runs from Jenkins projects, for example, you can run TestComplete from the command line or run the TestComplete SessionCreator utility from the command line. We will not examine these alternatives here because using the TestComplete support plugin is an easier and more straightforward way to run TestComplete tests.

How Network Suite elements translate to Jenkins

The image below shows an example of migrating TestComplete tests from the Network Suite to Jenkins Pipeline:

Migrating TestComplete Network Suite to Jenkins

Click the image to enlarge it.

Managing project

TestComplete: To manage test runs by using the Network Suite, you use a special project (so-called “master”). It stores the list of computers where your tests will run (hosts), and specifies jobs and tasks to be run on hosts. It also defines synchronization points and Network Suite variables. You use this project to start and monitor the test run and to gather results from remote machines.

Jenkins: In Jenkins and other CI/CD systems, a special project is not required. You configure and manage test runs directly from your CI/CD.

Hosts

TestComplete: In Network Suite, the remote computers, on which your tests are running, are called hosts.

Jenkins: Jenkins and other CI/CD systems also support managing a pool of computers on which various tasks are run. In Jenkins, such computers are called nodes. These are computers on which Jenkins agents are running and that are connected to your Jenkins server. You can assign individual tasks, or stages, or the entire project to a node or a group of nodes.

You will need to configure nodes in a special way for running interactive user sessions and TestComplete tests on them. See below for details.

Jobs

TestComplete: In Network Suite, a job is a collection of tasks that run in parallel on specified hosts. The Network Suite can have one or several jobs. Jobs are run in a series.

Jenkins: to manage tasks you want to run in parallel, you use the parallel keyword in your Pipeline. It allows specifying stages or steps to run in parallel. For detailed information on using the Pipeline keywords, see the Pipeline syntax reference.

Note: To run TestComplete tests from Jenkins, it is also possible to use freestyle Jenkins projects. However, such projects do not provide a straightforward way to run tasks in parallel. If you are not interested in parallel runs, you can use freestyle Jenkins projects to run your TestComplete tests.

Tasks

TestComplete: a Network Suite task is a test to be run on a host as part of a job. Task properties specify the TestComplete project that contains the test you want to run, the test to run, the computer on which you want to run this test, and the test runner tool (TestComplete or TestExecute), and the way you would like to handle test results.

Jenkins: to specify a task that will run a TestComplete test, you use the TestComplete Test build step. Configuring the step is very similar to configuring Network Suite tasks in TestComplete: you specify the TestComplete project that contains the test you want to run, the test to run, the test runner tool, and the way to handle results. However, there are some differences:

  • You don’t specify the host for the run directly in step properties. In Jenkins, you assign steps to nodes by using labels. You can assign them to stages or to the entire Pipeline or project.

  • You may need to configure the step to open an interactive user session on the host. For detailed information on this, see Storing user credentials and opening user sessions.

If you use freestyle projects, you configure the TestComplete Test step visually.

If you use Pipelines, you configure the step by specifying its parameters in your Pipeline code directly. You can find the full list of supported parameters in the TestComplete documentation: TestComplete Test step Pipeline syntax. You can also use the Snippet Generator to configure the step using the visual editor and then generate the code.

Distributed test in Jenkins – Example

To create and run a distributed test, you perform the following tasks:

  1. Configure remote computers (establish the connection between the CI/CD server and the remote computer, install the tested application and test runner on the remote machine, and so on).

  2. Copy test projects and data files to remote machines.

  3. Run the test runner on each remote computer.

  4. Gather results.

See below on how to perform these tasks in Jenkins.

About the TestComplete project to run

Suppose we have TestComplete tests that we run on remote computers by using the Network Suite. These projects are organized in the following way:

A distributed test managed by the Network Suite

Click the image to enlarge it.

  • The project called Main controls the test runs:

    • It has the Hosts collection that includes two hosts called RemoteAgent1 and RemoteAgent2. For them, it specifies credentials used to open interactive user sessions.

      Hosts

      Click the image to enlarge it.

    • Has one Job collection called Job1. The collection includes two tasks that specify the tests to execute on each of the hosts.

    • Task1 runs on RemoteAgent1. It executes Test1 that belongs to TestProject1.

      Task2 runs on RemoteAgent2. It executed Test2 that belongs to TestProject2.

      Job and tasks of the Network Suite

      Click the image to enlarge it.

  • Two test projects to be run on remote computers. These projects are called TestProject1 and TestProject2 respectively.

Let’s examine how you can migrate these projects to Jenkins.

1. Install and configure Jenkins

1.1. Install Jenkins

You can get the Jenkins installation package from the Jenkins website:

jenkins.io/download

For instructions on installing and configuring Jenkins, see the Jenkins documentation:

jenkins.io/doc/book/installing

1.2. Install the TestComplete support plugin

For instructions on how to install the plugin, see above.

1.3. Install other plugins

Your CI/CD job can require other plugins, for instance, it may need a plugin that provides support for your source control system. You can find the list of available plugins in the Jenkins documentation:

plugins.jenkins.io

2. Configure Jenkins nodes

A node is a remote machine (physical or virtual), where your test will run. This machine must meet the following requirements and must be configured for the run.

Requirements

A Jenkins node must:

  • Have a Jenkins agent installed on it.

  • Run an interactive user session or allow the TestComplete Test step to open an interactive user session (more on this below).

  • Have TestComplete (or TestExecute) installed.

    The TestComplete (TestExecute) instance must also be licensed.

  • Have access to your tested application and all other files your tests use.

2.1. Add a node to your Jenkins server

You need to register this machine in the list of available nodes in Jenkins:

  1. Log in to your Jenkins instance as an administrator.

  2. Go to the Manage Jenkins > Manage Nodes and Clouds page.

  3. Click New Node:

    Adding a Jenkins node

    Click the image to enlarge it.

  4. Enter the name for the created node. For example, RemoteAgent1.

  5. Click Permanent Agent.

  6. Click OK.

  7. In the Remote Root directory text box, enter the path to the folder that will be used as a working folder for the Jenkins agent on the created node. For example, C:\jenkins\agent.

  8. Enter labels for the created node. For example, you can use labels TestExecute and TestComplete to specify nodes on which TestComplete and TestExecute are installed.

  9. Select the launch method for the agent. In this example, we will select Launch agent by connecting it to the master from the drop-down list.

  10. Click Save.

Repeat the steps to add another node called RemoteAgent2.

2.2. Run Jenkins agents and connect them to your Jenkins server

For the Jenkins server to be able to communicate with a node, a piece of software called Jenkins agent must be running on that node. To install and run that agent on a node:

  1. Log in to a remote computer you are planning to use as RemoteAgent1.

  2. On the node computer, run a web browser and open the Manage Nodes page of your Jenkins server:

    http://<your_jenkins_server>/computer

  3. Click the needed node in the list.

  4. Follow the instructions to launch the Jenkins agent on the node.

Launching Jenkins agents

Click the image to enlarge it.

While the agent is running on the node, it is connected to your Jenkins server and you can assign and run tasks on this node.

Repeat the steps to install and launch the Jenkins agent on RemoteAgent2.

2.3. Add credentials

In the Jenkins server, go to the Manage Jenkins > Manage Credentials page, add credentials under which you will run tests on remote computers.

If needed, you can also add credentials to be used to access your source control repository, where your TestComplete projects are stored.

2.4. Open interactive user sessions on nodes

To be able to simulate user actions over a tested application during the test run, TestComplete requires an interactive user session to be opened on the computer, where the test is running. This means that there must be a user session opened on your Jenkins nodes when your test session starts.

In Jenkins, you can configure TestComplete Test steps in your projects to open an interactive user session on the nodes before the testing starts. However, your Jenkins nodes must be configured to allow opening sessions on them. To learn how to configure nodes, please see TestComplete documentation:

Integration With Jenkins - Requirements

To learn how to configure TestComplete Test steps to open sessions, see below.

2.5. Install TestComplete (TestExecute) on a node

On every node, you need to install TestComplete or TestExecute. These TestComplete (TestExecute) instances must be licensed. The easiest way is to use a Floating user license.

3. Place TestComplete projects to a source control repository

For your Jenkins nodes to be able to access your TestComplete projects, add the project files to a source control system. How you can do it, depends on the source control system you use.

This step is needed to simplify the deployment of the latest test versions to nodes: the Jenkins task will take the TestComplete project files from the source control to the remote machine.

4. Create a Pipeline to run the test

The Jenkins Pipeline will copy TestComplete projects and test files to nodes and run the test on the nodes.

4.1. Create the Pipeline
  1. In Jenkins, click New Item.

  2. Click Pipeline and enter its name.

  3. Click OK.

  4. In the Pipeline > Definition drop-down list, select whether you want to enter the Pipeline code directly or to get it from a .jenkinsfile. In the latter case, specify the source control system that stores the file and the path to the file.

4.2. Add instructions to get test files from the repository and place to nodes

To your Pipeline code, add instructions to get your TestComplete project files from your source control repository and to place them to configured nodes. The instructions to use depend on your source control repository and on plugins you use to access it from Jenkins. We recommend that you see your plugin documentation.

4.3. Add stages in which your tests will run

In our example, we will have a stage called QA that will run our TestComplete tests in parallel:

Pipeline

pipeline{
  agent none
  stages{
    stage('QA'){
      parallel{
        stage('Tests on RemoteAgent1'){
          agent{
              label 'RemoteAgent1'
            }
            ...
        }
        stage('Tests on RemoteAgent2'){
          agent{
            label 'RemoteAgent2'
          }
          ...
        }
      }
    }
  }
}

To specify nodes where each stage will run, we use the node name as a label.

4.4. Add instructions to run tests

To run tests, we will use the TestComplete Test step.

For each step, we will specify:

  • The path to the TestComplete project suite (suite) and the name of the project that contains the tests to run (project).

  • The type of the test to run (launchType). It can be an entire project or a single test.

  • The test to run. For a script test, we specify the name of the unit (unit) and the routine to run (routine).

  • That a user session should be opened on the remote computer (useTCService). We also specify the ID of the Credentials entry that stores the user data with which the session will be opened (credentialsId).

Pipeline

stage('Tests on RemoteAgent1'){
  agent{
    label 'RemoteAgent1'
  }
  steps{
    testcompletetest credentialsId: 'tester',
      executorType: 'TE',
      launchType: 'lcRoutine',
      project: 'TestProject',
      routine: 'Test1',
      suite: 'TestProject1\\TestProject1.pjs',
      unit: 'Unit1',
      useTCService: true
  }
}
Complete solution

The overall Pipeline code will look as follows:

Example

5. Running the test

Typically, you configure your Pipeline to start the tests automatically, for example, after a new build of your tested application is available. See below.

It is also possible to run a Pipeline manually. To do this, click Build Now on your Pipeline page:

Running Pipeline

Click the image to enlarge it.

6. Getting results

After the run is over, the TestComplete support plugin will collect the test results from all nodes and add them to the build result automatically. To view them, click TestComplete Test Results on the Status page of the needed build:

Viewing TestComplete test results in Jenkins

Click the image to enlarge it.

You can view the results of all tests run by your Jenkins project:

Viewing test logs in Jenkins

Click the image to enlarge it.

Typical tasks

This section describes how you can use Jenkins to solve typical tasks that concern distributed testing.

Copying test projects to test agents

Before the testing starts, your test projects must be placed in a location where the test runner (TestComplete or TestExecute) can access them from the remote computers.

TestComplete: this task can be done automatically — you can configure your tasks to copy projects to hosts.

Jenkins: you have to configure your build to do this. For instance, you can:

  • Place your TestComplete projects in a source control repository and then configure your Jenkins project or Pipeline to take the project files from that repository and place them in a needed location on your Jenkins nodes. There is a number of plugins that allow working with a source control system from Jenkins.

  • Place your TestComplete projects in a shared network location where all your Jenkins nodes can access them.

  • Configure your Jenkins project to copy your TestComplete projects to your nodes.

Storing user credentials and opening user sessions

Each remote computer, where you will run your tests, should have an interactive user session. Otherwise, TestComplete (or TestExecute) will not be able to simulate user actions during the run.

TestComplete: you can store user credentials that will be used to open user sessions in the Network Suite. For each host you add to your Network Suite, you can specify how user sessions should be opened on them.

Jenkins: the user credentials are stored separately in the Credentials storage. This way, you do not have to store the sensitive data in a plain view in your .jenkinsfile (if you use Pipeline) or in your project settings (if you use freestyle projects). To open user sessions on Jenkins nodes, you configure your TestComplete Test steps in your project. The useTCService property of the step specifies whether the step will open the session. The credentialsID property specifies the Credentials storage entry that stores the user data to be used to open the user session.

For example, the Pipeline code below shows how to configure the TestComplete Test step to open a user session on nodes before running tests:

Pipeline

pipeline {
  agent any
  stages{
    stage('Automated tests'){
      agent{
        label 'TestExecute'
      }
      steps{
        testcompletetest credentialsId: 'tester',
          executorType: 'TE',
          launchType: 'lcRoutine',
          project: 'TestProject1',
          routine: 'Test1',
          suite: 'ProjectSuite1\\ProjectSuite1.pjs',
          unit: 'Unit1',
          useTCService: true
        }

To learn more about handling credentials in Jenkins, see:

Credentials plugin description

To learn more about the TestComplete Test step and its properties, see:

TestComplete Test step pipeline syntax

Synchronization

In distributed testing, it may be crucial that tests that run on several remote computers interact with each other.

TestComplete: the Network Suite offers the following means of interaction between projects:

  • Synchpoints - Can be used to pause the test execution on remote computers until a specific point is reached and then resume the test execution on all remote computers simultaneously.

  • Network Suite variables - Can be used to exchange data between all tests that participate in testing. To track the variables state, the WaitForNetVarChange method and the OnNetVarChange event can be used.

Jenkins: to implement interactions between TestComplete tests you run as part of a Jenkins project or Pipeline, you can use the following:

  • (For Pipelines) Use the stage and parallel keywords to organize tests in your Pipeline. The stage keyword specifies a group of steps (tasks) to run together. The parallel keyword specifies the stages or steps to be run simultaneously.

  • (For Pipelines) You can implement custom code that will control the testing flow. For example, the code below shows how to create a custom SynchPoint class and then use it to control test runs:

    Pipeline

    pipeline{
      agent any
      stages{
        stage('Init'){
          steps{
            script{
              AB = new SynchPoint(2); // This synchpoint will synchronize up to two threads
            }
          }
        }
        stage('QA'){
          parallel{
              stage('Tests on RemoteAgent1'){
              agent{
                label 'RemoteAgent1'
                }
              steps{
                testcompletetest credentialsId: 'tester',
                  executorType: 'TE',
                  launchType: 'lcRoutine',
                  project: 'TestProject',
                  routine: 'Test1',
                  suite: 'TestProject\\TestProject.pjs',
                  unit: 'Unit1',
                  useTCService: true
                script{
                  AB.synchronize();
                }
                testcompletetest credentialsId: 'tester',
                  executorType: 'TE',
                  launchType: 'lcRoutine',
                  project: 'TestProject',
                  routine: 'Test2',
                  suite: 'TestProject\\TestProject.pjs',
                  unit: 'Unit1',
                  useTCService: true
              }
            }
            stage('Tests on RemoteAgent2'){
              agent{
                label 'RemoteAgent2'
              }
              steps{
                testcompletetest credentialsId: 'tester',
                  executorType: 'TE',
                  launchType: 'lcRoutine',
                  project: 'TestProject',
                  routine: 'Test1',
                  suite: 'TestProject\\TestProject.pjs',
                  unit: 'Unit1',
                  useTCService: true
                script{
                  AB.synchronize();
                }
                testcompletetest credentialsId: 'tester',
                  executorType: 'TE',
                  launchType: 'lcRoutine',
                  project: 'TestProject',
                  routine: 'Test2',
                  suite: 'TestProject\\TestProject.pjs',
                  unit: 'Unit1',
                  useTCService: true
                }
            }
          }
        }
      }
    }

    class SynchPoint{
      int _count;

      SynchPoint(count)
      {
        _count = count;
      }

      void synchronize(){
        _count --;
        while (_count > 0){
          sleep(0);
        }
      }
    }
  • You can use environment variables and Pipeline variables to send values to TestComplete tests that you run.

  • You can store shared data to an external file and place it in a location that your TestComplete tests can access to get data from the file or to put data to the file. For example, it can be a shared location in your network.

Triggering test runs

To run TestComplete tests from a CI/CD system, you need to run a task that controls the test run. For example, in Jenkins, you run a project (a free-style job or a Pipeline) that executes the TestComplete Test step. You can run the project manually or you can configure your project to start the test run automatically, for example, at time periods, or on a specific event (for example, when new TestComplete project files are added to the repository or when your tested application’s new build is available).

Monitoring testing process

TestComplete: you can track the current state of running tests in the Network Suite editor, or you can do this in your tests by checking the State property of appropriate scripting objects.

Jenkins: you can track the test progress on the build page of your project.

Viewing test results

TestComplete: Network Suite can retrieve test results from remote computers automatically, or it can store them on remote machines (this was helpful if the test log was too large).

Jenkins: in general, if you run your TestComplete tests from a CI/CD system, by default, test results will be available only on computers where they were run. In Jenkins, the TestComplete support plugin will accumulate your test results on your Jenkins server automatically and add them to your build results. You can view them on your build status page.

In addition, you can configure your TestComplete Test steps to:

  • Export Summary reports of your TestComplete tests to an external file in a JUnit format. Then, you can configure your Jenkins project to get the file and process it in a needed way.

  • Export test results to a JUnit-style report (different from the one described above).

  • Export the detailed test logs to a .tcLogX file, or .mht file, or both.

Questions?

If you have questions or need assistance with migration, please contact our Customer Care Team.

Highlight search results