Configuring Jenkins Pipelines to Run TestComplete Tests

1. Specify nodes on which tests will run

2. Add steps that will prepare testing environments

3. Add script statements to run TestComplete tests

Examples

TestComplete Test step pipeline syntax

The node where you will run tests must have TestComplete or TestExecute installed. To make sure you will run your tests on the appropriate nodes only, add code to your Pipeline script (Jenkinsfile) that will specify the needed nodes. For example, you can use labels, tags or node names to specify where the tests should run:

Example

ExampleExample

Declarative Pipeline

pipeline {
  agent {
    label 'testagent'
  }
  …
}

Scripted Pipeline

node('testagent') {
  …
}

For details, see the Jenkins Pipeline documentation at jenkins.io/doc/book/pipeline/syntax/.

Prepare your nodes to run TestComplete tests:

1. Copy TestComplete project files, your tested application, and other test-related files to the nodes where the test will run.

   Note:

   TestComplete test project files must reside in the node’s working folder or in its subfolders.

2. Make sure TestComplete tests can access all the files they needed during the test run.

Otherwise, Jenkins will not be able to run your test.

You can copy all the needed files to the nodes in advance, or you can add steps to your Jenkins Pipeline to perform all the preparation tasks. To learn more about the steps you can use, see jenkins.io/doc/pipeline/steps/.

1. If you manage your Pipeline from the Jenkins UI, open the Pipeline section of your project and locate the Script box.

   If you use SCM, open your Jenkinsfile.

2. Add the code that will run TestComplete tests to your Pipeline script. You can write the code manually (see the Pipeline Syntax section below).

   – or –

   You can use the built-in Jenkins Snippet Generator utility to generate the code automatically:

   1. Open the Pipeline Syntax page of your Pipeline.

   2. On the resulting page, select testcomplete: TestComplete test from the Steps drop-down list:

      

      Click the image to enlarge it.

   3. Configure the step to run your TestComplete tests:

      View instructions

      View instructionsView instructions

      1. Specify the test project suite in the Project suite file field.

         Note:

         The path should be relative to the job’s working folder on the node.

      2. Select the test of the project suite to be run:

         • Entire suite - Runs the entire project suite.

         • Project test - Runs a specified project in the project suite.

         • Tags (TestComplete 14.20 or later is required) - Runs tests of the specified project that match a tag or tag expression.

           Tag expression syntax

           Tag expression syntaxTag expression syntax

           • Use @ before tag names, that is, use @tag1 rather than tag1.

           • To combine tags, use the and, or and not operations, for example:

             @tag1 or @tag2
             @tag1 and not @tag3
             not @tag5

             These operations don’t depend on the scripting language you use in your TestComplete project. That is, for example, in JavaScript projects you should use and, or and not rather than &&, || and !  —

             @tag1 and not @tag2  ← Correct!
             @tag1 && !@tag1      ← Incorrect!

           • If needed, you can use braces to group tags, for example:

             @tag1 and (@tag2 or @tag3)
             (@tag1 or not @tag2) and (not @tag3 or @tag4)

         • Script test - Runs a script routine.

         • Keyword test - Runs a keyword test.

         • Other - Runs any test your project contains: a project test item, a low-level procedure, a unit test, a network suite test, and so on. In the Project text box, enter the name of your project. In the Test name text box, enter the full path to the project item or test item to run. Use vertical bars ( | ) as separators.

           Example

           ExampleExample

           Test	Description
           ProjectTestItem1 ProjectTestItem1|ProjectChildTestItem1	The specified test item and all its child test items (if any). Only those test items that are enabled on the Test Items page of the project will be run.
           KeywordTests|Test1	Keyword test Test1.
           Script|Unit1|Main	Script function Main that is located in the Unit1 script unit.
           LLCollection1|LLP1	Low-level procedure LLP1 from the LLCollection1 project item.
           ManualTests|ManualTest1	Manual test ManualTest1.
           UnitTesting1|NUnit1	Unit test NUnit1 from the UnitTesting1 collection.
           ReadyAPI1|Test1	API test Test1 from the ReadyAPI1 collection.
           NetworkSuite|Jobs|Job1|Task1	Task1 from the Job1 job of the network suite.
           NetworkSuite|Jobs|Job1	Job1 of the network suite.

           The test path should contain collection names only. Do not include any project folders in the path.

         • CrossBrowserTesting tests (TestComplete 12.20 or later with the Web module license is required) – runs tests defined in the CrossBrowserTesting manager.

           This approach to run tests in CrossBrowserTesting environments is deprecated starting with TestComplete 14.40. We recommend using the Device Cloud add-on instead. See About Cross-Platform Web Tests for details.

           Note: Your slave nodes must have TestComplete installed. TestExecute cannot run tests defined in the CrossBrowserTesting manager of the project suite.
           TestComplete on your slave nodes must be able to access the CrossBrowserTesting cloud, and your project suite must include environments and tests assigned to them.
           To learn more, see Prepare Jenkins Nodes to Run Tests in CrossBrowserTesting Environments.

      3. Click Advanced to configure additional settings of the TestComplete Test step:

         • Test runner - Specifies whether the step will use TestComplete or TestExecute for the test run. If both are installed and (Any) is selected, TestExecute is run.

           Notes:

           • The selected product must be installed on the node. If the 64-bit version of the product is available on the node, it will be used to run tests. Otherwise, the 32-bit version will be used.

           • To run tests in CrossBrowserTesting environments, select TestComplete.

         • Version - Specifies the TestComplete or TestExecute version to be used in case you have several versions of these products installed on the node. Jenkins uses the latest installed version by default, and we recommend using the default setting to avoid problems with running your tests in the latest product versions.

         • Action on warnings - Specifies whether Jenkins should mark the entire build as failed or unstable when the TestComplete test log contains warnings.

         • Action on errors - Specifies whether Jenkins should mark the entire build as failed or unstable when the TestComplete test log contains errors.

         • Additional command-line arguments - Specifies arbitrary command-line arguments to be passed to TestComplete. To learn more about the TestComplete command line, see TestComplete Command Line.

         • Use test timeout - Specifies the maximum test execution time in seconds.

           The timeout includes both the startup time of TestComplete (TestExecute) and the test run time. If you run tests in the CrossBrowserTesting cloud, the timeout will also include the startup time of CrossBrowserTesting environments.

           If the test is not finished until the specified period elapses, Jenkins will mark the entire build as failed.

         • Run interactive user session - Specifies whether your TestComplete test will run in an interactive session.

           This option is effective only if you use TestComplete or TestExecute 10.60 or later.

           If this option is disabled and Jenkins is running as a service, your TestComplete tests will fail.

           For information on when you need to use this option, see Requirements.

           • User - The user profile that will be used to run the test.

           • Password - The password used to log in to the user account.

           • Use active session - Specifies whether an existing user session will be used to run a TestComplete test. If it is selected, TestComplete will run in the current session. Otherwise, Jenkins will close all the applications, terminate the currently active session, start a new session under the specified user account and then start the test.

             Note:

             Jenkins will not stop the user session if another user is logged on.

           • Session screen resolution - Select a screen resolution for the user session in which your TestComplete test will run. The default value is 1280×1024.

             If you use a TestComplete version earlier than 12.30, the specified screen resolution will be ignored and the default screen resolution (1280×1024) will be used.

             If the Use active session option is enabled, the specified screen resolution will be ignored and the actual screen resolution of the corresponding user session will be used.

             Note:

             The maximum resolution that can be set for a user session depends on the operating system version and on the Remote Desktop Protocol version installed on the target computer. To learn more, see Remote Desktop Protocol Maximum Supported Resolutions.

         • Generate MHT log file - If this option is enabled, TestComplete exports test results to a .mht file after the build is over. You can download this file from the Jenkins page displaying test results.

           If this option is disabled, TestComplete does not export test results in the MHT format, but you can still examine results in Jenkins. See Viewing TestComplete Test Results in Jenkins.

         • Generate JUnit-style report - Specifies whether Jenkins will generate JUnit-style reports. For more information on how JUnit reports are represented in Jenkins, see Viewing TestComplete Test Results in Jenkins.

           Note: This JUnit-style report slightly differs from standard JUnit reports. To generate JUnit reports, use the /ExportSummary command line argument. You can specify it in the Additional command-line arguments text box of the step. To learn more about the argument, see TestComplete Command Line.

      Script tags
      Keyword test tags
      BDD tags

   4. Click Generate Pipeline Script:

      

      Click the image to enlarge it.

   5. Copy the generated code, and then paste it to your Pipeline script (Jenkinsfile):

      

      Click the image to enlarge it.

• The following sample script runs a TestComplete project suite (all the enabled test items in all the enabled projects that belong to that project suite):

  testcompletetest suite: 'Projects\\JenkinsTests.pjs'

• The following sample script runs the MyProj project of the project suite:

  testcompletetest launchType: 'lcProject', project: 'MyProj', suite: 'Projects\\JenkinsTests.pjs'

• The following sample script runs the ProjectTestItem3 test item that is a child item of the ProjectTestItem1 test item of the MyProj project:

  testcompletetest launchType: 'lcItem', project: 'MyProj', suite: 'Projects\\JenkinsTests.pjs', test: 'ProjectTestItem1|ProjectTestItem3'

• The following sample script runs the MyKDT keyword test of the MyProj project:

  testcompletetest launchType: 'lcKdt', project: 'MyProj', suite: Projects\\JenkinsTests.pjs', test: 'MyKDT'

• The following sample script runs the Main script routine located in the Unit1 unit of the MyProj project:

  testcompletetest launchType: 'lcRoutine', project: 'MyProj', routine: 'Main', suite: 'Projects\\JenkinsTests.pjs', unit: 'Unit1'

• The following sample script runs the MyProj project’s tests that match the @tag1 or (@tag2 and @tag3) tag expression:

  testcompletetest launchType: 'lcTags', project: 'MyProj', suite: 'Projects\\JenkinsTests.pjs', tags: '@tag1 or (@tag2 and @tag3)'

• The following sample script launches TestComplete and commands it to run tests defined in the CrossBrowserTesting manager:

  testcompletetest executorType: 'TC', launchType: 'lcCBT', suite: 'Projects\\JenkinsTests.pjs'

• The following sample script opens an interactive user session on the node and runs the MyProj project:

  testcompletetest launchType: 'lcProject', project: 'MyProj', suite: 'Projects\\JenkinsTests.pjs',
  useTCService: true, userName: 'Tester', userPassword: 'password'

• The following sample script runs the MyProj project of the project suite and exports the test results to a .mht file:

  testcompletetest generateMHT: true, launchType: 'lcProject', project: 'MyProj', suite: 'Projects\\JenkinsTests.pjs'