Migrating Distributed Tests to Jenkins

Prerequisite: TestComplete Support plugin for Jenkins

How Network Suite elements translate to Jenkins

Distributed test in Jenkins – Example

Typical tasks

Questions?

There are other ways to control TestComplete test runs from Jenkins projects, for example, you can run TestComplete from the command line or run the TestComplete SessionCreator utility from the command line. We will not examine these alternatives here because using the TestComplete support plugin is an easier and more straightforward way to run TestComplete tests.

TestComplete: To manage test runs by using the Network Suite, you use a special project (so-called “master”). It stores the list of computers where your tests will run (hosts), and specifies jobs and tasks to be run on hosts. It also defines synchronization points and Network Suite variables. You use this project to start and monitor the test run and to gather results from remote machines.

Jenkins: In Jenkins and other CI/CD systems, a special project is not required. You configure and manage test runs directly from your CI/CD.

TestComplete: In Network Suite, the remote computers, on which your tests are running, are called hosts.

Jenkins: Jenkins and other CI/CD systems also support managing a pool of computers on which various tasks are run. In Jenkins, such computers are called nodes. These are computers on which Jenkins agents are running and that are connected to your Jenkins server. You can assign individual tasks, or stages, or the entire project to a node or a group of nodes.

You will need to configure nodes in a special way for running interactive user sessions and TestComplete tests on them. See below for details.

TestComplete: In Network Suite, a job is a collection of tasks that run in parallel on specified hosts. The Network Suite can have one or several jobs. Jobs are run in a series.

Jenkins: to manage tasks you want to run in parallel, you use the parallel keyword in your Pipeline. It allows specifying stages or steps to run in parallel. For detailed information on using the Pipeline keywords, see the Pipeline syntax reference.

Note: To run TestComplete tests from Jenkins, it is also possible to use freestyle Jenkins projects. However, such projects do not provide a straightforward way to run tasks in parallel. If you are not interested in parallel runs, you can use freestyle Jenkins projects to run your TestComplete tests.

TestComplete: a Network Suite task is a test to be run on a host as part of a job. Task properties specify the TestComplete project that contains the test you want to run, the test to run, the computer on which you want to run this test, and the test runner tool (TestComplete or TestExecute), and the way you would like to handle test results.

Jenkins: to specify a task that will run a TestComplete test, you use the TestComplete Test build step. Configuring the step is very similar to configuring Network Suite tasks in TestComplete: you specify the TestComplete project that contains the test you want to run, the test to run, the test runner tool, and the way to handle results. However, there are some differences:

• You don’t specify the host for the run directly in step properties. In Jenkins, you assign steps to nodes by using labels. You can assign them to stages or to the entire Pipeline or project.

• You may need to configure the step to open an interactive user session on the host. For detailed information on this, see Storing user credentials and opening user sessions.

If you use freestyle projects, you configure the TestComplete Test step visually.

If you use Pipelines, you configure the step by specifying its parameters in your Pipeline code directly. You can find the full list of supported parameters in the TestComplete documentation: TestComplete Test step Pipeline syntax. You can also use the Snippet Generator to configure the step using the visual editor and then generate the code.

Before the testing starts, your test projects must be placed in a location where the test runner (TestComplete or TestExecute) can access them from the remote computers.

TestComplete: this task can be done automatically — you can configure your tasks to copy projects to hosts.

Jenkins: you have to configure your build to do this. For instance, you can:

• Place your TestComplete projects in a source control repository and then configure your Jenkins project or Pipeline to take the project files from that repository and place them in a needed location on your Jenkins nodes. There is a number of plugins that allow working with a source control system from Jenkins.

• Place your TestComplete projects in a shared network location where all your Jenkins nodes can access them.

• Configure your Jenkins project to copy your TestComplete projects to your nodes.

Each remote computer, where you will run your tests, should have an interactive user session. Otherwise, TestComplete (or TestExecute) will not be able to simulate user actions during the run.

TestComplete: you can store user credentials that will be used to open user sessions in the Network Suite. For each host you add to your Network Suite, you can specify how user sessions should be opened on them.

Jenkins: the user credentials are stored separately in the Credentials storage. This way, you do not have to store the sensitive data in a plain view in your .jenkinsfile (if you use Pipeline) or in your project settings (if you use freestyle projects). To open user sessions on Jenkins nodes, you configure your TestComplete Test steps in your project. The useTCService property of the step specifies whether the step will open the session. The credentialsID property specifies the Credentials storage entry that stores the user data to be used to open the user session.

For example, the Pipeline code below shows how to configure the TestComplete Test step to open a user session on nodes before running tests:

To learn more about handling credentials in Jenkins, see:

Credentials plugin description

To learn more about the TestComplete Test step and its properties, see:

TestComplete Test step pipeline syntax

In distributed testing, it may be crucial that tests that run on several remote computers interact with each other.

TestComplete: the Network Suite offers the following means of interaction between projects:

• Synchpoints - Can be used to pause the test execution on remote computers until a specific point is reached and then resume the test execution on all remote computers simultaneously.

• Network Suite variables - Can be used to exchange data between all tests that participate in testing. To track the variables state, the WaitForNetVarChange method and the OnNetVarChange event can be used.

Jenkins: to implement interactions between TestComplete tests you run as part of a Jenkins project or Pipeline, you can use the following:

• (For Pipelines) Use the stage and parallel keywords to organize tests in your Pipeline. The stage keyword specifies a group of steps (tasks) to run together. The parallel keyword specifies the stages or steps to be run simultaneously.

• (For Pipelines) You can implement custom code that will control the testing flow. For example, the code below shows how to create a custom SynchPoint class and then use it to control test runs:

  Pipeline

  Copy Code

  Pipeline

  pipeline{
    agent any
    stages{
      stage('Init'){
        steps{
          script{
            AB = new SynchPoint(2); // This synchpoint will synchronize up to two threads
          }
        }
      }
      stage('QA'){
        parallel{
            stage('Tests on RemoteAgent1'){
            agent{
              label 'RemoteAgent1'
              }
            steps{
              testcompletetest credentialsId: 'tester',
                executorType: 'TE',
                launchType: 'lcRoutine',
                project: 'TestProject',
                routine: 'Test1',
                suite: 'TestProject\\TestProject.pjs',
                unit: 'Unit1',
                useTCService: true
              script{
                AB.synchronize();
              }
              testcompletetest credentialsId: 'tester',
                executorType: 'TE',
                launchType: 'lcRoutine',
                project: 'TestProject',
                routine: 'Test2',
                suite: 'TestProject\\TestProject.pjs',
                unit: 'Unit1',
                useTCService: true
            }
          }
          stage('Tests on RemoteAgent2'){
            agent{
              label 'RemoteAgent2'
            }
            steps{
              testcompletetest credentialsId: 'tester',
                executorType: 'TE',
                launchType: 'lcRoutine',
                project: 'TestProject',
                routine: 'Test1',
                suite: 'TestProject\\TestProject.pjs',
                unit: 'Unit1',
                useTCService: true
              script{
                AB.synchronize();
              }
              testcompletetest credentialsId: 'tester',
                executorType: 'TE',
                launchType: 'lcRoutine',
                project: 'TestProject',
                routine: 'Test2',
                suite: 'TestProject\\TestProject.pjs',
                unit: 'Unit1',
                useTCService: true
              }
          }
        }
      }
    }
  }
  
  class SynchPoint{
    int _count;
  
    SynchPoint(count)
    {
      _count = count;
    }
  
    void synchronize(){
      _count --;
      while (_count > 0){
        sleep(0);
      }
    }
  }

• You can use environment variables and Pipeline variables to send values to TestComplete tests that you run.

• You can store shared data to an external file and place it in a location that your TestComplete tests can access to get data from the file or to put data to the file. For example, it can be a shared location in your network.

To run TestComplete tests from a CI/CD system, you need to run a task that controls the test run. For example, in Jenkins, you run a project (a free-style job or a Pipeline) that executes the TestComplete Test step. You can run the project manually or you can configure your project to start the test run automatically, for example, at time periods, or on a specific event (for example, when new TestComplete project files are added to the repository or when your tested application’s new build is available).

TestComplete: you can track the current state of running tests in the Network Suite editor, or you can do this in your tests by checking the State property of appropriate scripting objects.

Jenkins: you can track the test progress on the build page of your project.

TestComplete: Network Suite can retrieve test results from remote computers automatically, or it can store them on remote machines (this was helpful if the test log was too large).

Jenkins: in general, if you run your TestComplete tests from a CI/CD system, by default, test results will be available only on computers where they were run. In Jenkins, the TestComplete support plugin will accumulate your test results on your Jenkins server automatically and add them to your build results. You can view them on your build status page.

In addition, you can configure your TestComplete Test steps to:

• Export Summary reports of your TestComplete tests to an external file in a JUnit format. Then, you can configure your Jenkins project to get the file and process it in a needed way.

• Export test results to a JUnit-style report (different from the one described above).

• Export the detailed test logs to a .tcLogX file, or .mht file, or both.