Running ReadyAPI Tests From Jenkins

We need to create a Jenkins job that will run ReadyAPI functional tests from Jenkins agent machines (or Jenkins nodes). To run functional tests, you need to install ReadyAPI on each Jenkins node and use the ReadyAPI Functional Testing plugin or ReadyAPI command-line runners to run the tests.

The ReadyAPI Functional Testing plugin provides you with an easy way to run functional tests from Jenkins jobs and Jenkins Pipelines. If you need to run load or secure tests, or if you need to specify advanced options for a functional test run, use the command-line runners.

In order to run ReadyAPI tests, you need to activate the appropriate license on each node. The easiest way to do this is to install ReadyAPI on a node and activate the license through the ReadyAPI user interface. However, it is considered to be a good practice to clean up the test environment after the test run is over. In other words, you may want to configure nodes automatically before each run, and this means you may want to automate the ReadyAPI license activation. ReadyAPI supports a special command-line tool to automate activation (see below).

Which license to use: Floating or Fixed? Floating licenses are required for dynamic testing environments. You can activate the license from SmartBear License Servers or install it offline on a computer within your network, which can serve as a license server. You can then configure ReadyAPI instances on Jenkins nodes to connect to this server and obtain a license from it.

• One of the supported Java versions installed.

  Note:

  ReadyAPI is updated to support Java 17.0.12 in ReadyAPI 3.56.0. The Installers automatically install the appropriate Java version and we recommend using the Java version bundled with the product. Support for this version will also be required with your Virtual Machines and Docker instances.

• Access to the ReadyAPI installation program (you can copy it to your Jenkins nodes, or you can put it to a shared network folder).

• On Windows Jenkins nodes, disable the User Account Control (UAC). You can do this in the Control Panel. Otherwise, Windows can display various system confirmation dialogs, which require user interaction.

If you are using File-based licenses

• Network access to the license server with the Floating license activated (by default, the server works on port 1099).

• Access to the ReadyAPI License Manager. You can download it in the .zip format from our website. Unpack the downloaded .zip archive and copy its content to your Jenkins node or put it to a shared network folder, to which the node has access.

If you are using SmartBear ID-based licenses, please refer to 3.1

To make the communication between Jenkins and ReadyAPI easier, consider installing the ReadyAPI Functional Testing plugin. To install it:

1. Open Jenkins and click Manage Jenkins:

   

2. Open Manage Plugins:

   

   Click the image to enlarge it.

3. On the Available page, type ReadyAPI in the Filter field and select ReadyAPI Functional Testing.

   

   Click the image to enlarge it.

4. Click Install without restart or Download now and install after restart:

   

   Click the image to enlarge it.

Jenkins documentation includes a detailed guide on configuring Jenkins nodes. Follow the instructions on this page:

https://wiki.jenkins.io/display/JENKINS/Step+by+step+guide+to+set+up+master+and+agent+machines+on+Windows

Tip: We recommend using Java Web Start for launching nodes in most scenarios. The optimal method depends on your environment.

To install ReadyAPI automatically:

1. Use the Execute batch command build step. The way you add it depends on the operating system of the Jenkins node:

   • Windows Jenkins nodes: select Add build step > Execute Windows batch command.

   • Linux or macOS nodes: select Add build step > Execute shell.

   

   Click the image to enlarge it.

2. Configure the step to run the ReadyAPI installer. Add the -q argument to the installer command line. This way, the installation will run in silent mode, that is, without any user interaction, and will use the default values for all options.

   To change the installation folder, you can use the -dir argument. -varfile lets you specify additional installation parameters. You can find more information in the install4j documentation.

   Here is a sample command line that installs ReadyAPI in silent mode:

   "c:\sample\ReadyAPI-x64-3.56.0.exe" -q -dir "c:\readyapi-folder" -varfile response.varfile

ReadyAPI Jenkins plugins are updated to support SmartBear ID-based license options, both online and offline. The plugin now supports two types of license:

SmartBear ID-based licenses – These licenses eliminate the need for license keys or files. Instead, you can simply provide your access credentials to use the product. For offline or on-premise use with headless test runs, we recommend using an access key and a floating license server. Below are the specific details.

File-based licenses – These licenses were previously used and required a SmartBear license file. Please be aware that this type of license will be phased out in October 2023.

3.1 When using ID-Based Licensing

3.2 When using File-Based Licensing

To run ReadyAPI tests from a Jenkins node, you need to copy your ReadyAPI project to this node. You also need to copy the data files your tests use.

If your tests use custom script libraries, copy the libraries to the <ReadyAPI installation>/bin/scripts folder on the Jenkins node.

To copy the files, use the Jenkins Copy file build step. A possible alternative is to place the project and data file to some shared network folder to which Jenkins nodes have access.

By using the ReadyAPI Functional Testing plugin

To run functional tests in Jenkins:

1. Make sure the ReadyAPI Functional Testing plugin is installed.

2. In Jenkins, select a job and click Configure.

3. In the Build section, click Add build step > ReadyAPI Test: Run functional Test:

   

4. Jenkins will add the step to the build:

   

   Click the image to enlarge it.

5. Configure the following build step options:

   Option	Description
   Path to TestRunner	Specifies the fully-qualified path to the test runner binary (testrunner.bat or testrunner.sh). By default, you can find it in the <ReadyAPI installation>/bin directory.
   Path to ReadyAPI project	Specifies the fully-qualified path to the ReadyAPI project you want to run.
   Test Suite	Specifies the test suite to run. To run all the test suites of your project, leave the field blank.
   Test Case	Specifies the test case to run. If you leave the field blank, the runner will execute all the test cases of the specified test suite, or, if you have not specified a test suite, all the test cases of your project.
   Test Suite Tags	Specifies which tags the test suite must contain to be run. To create complex conditions, use the || (logical OR), && (logical AND) and ! (logical NOT) operators.
   Test Case Tags	Specifies which tags the test case must contain to be run. To create complex conditions, use the || (logical OR), && (logical AND) and ! (logical NOT) operators.
   Project Password	Specifies the encryption password, if you encrypted the entire project or some of its custom properties. See Protecting Sensitive Data.
   Environment	Specifies the environment configuration for the test run.

   If you want to specify additional arguments for the test runner, call the test runner using the command line – see below.

Analyze results

If you use the ReadyAPI Functional Testing plugin, the functional test runner exports a JUnit-style HTML report to your Jenkins workspace directory.

You can also view a PDF version of the report directly from Jenkins. To do that, open the needed project and build, then click the ReadyAPI Test Results link to the left:

Click the image to enlarge it.

By using Jenkins Pipeline

The ReadyAPI Functional Testing plugin supports launching tests from Jenkins Pipeline. To run a test, you need to add the SoapUI Pro Pipeline step to your Pipeline script:

1. Make sure the ReadyAPI Functional Testing and Jenkins Pipeline plugins are installed.

2. In Jenkins, select a Pipeline and click Configure.

3. In the Pipeline section, add the SoapUI Pro Pipeline step to your Pipeline script:

   

   Click the image to enlarge it.

   Below is an example of a Pipeline script with a SoapUI Pro Pipeline step:

   View example

   View exampleHide example

   Scripted Pipeline

   Scripted Pipeline

   node('master') {
     stage('Build') {
       sh 'make'
     }
    
     stage('Test') {
       SoapUIPro ( pathToTestrunner: 'SmartBear/ReadyAPI-3.56.0/bin/testrunner.sh', pathToProjectFile: 'ReadyAPI-Projects/sample-soapui-test.xml', testSuite: '', testCase: '', testSuiteTags: '(tag1 || tag2) && !tag3', projectPassword: '', environment: '' )
     }
    
     stage('Deploy') {
       sh 'make publish'
     }
   }
   

   Tip:	To learn more about the Pipeline syntax, see https://jenkins.io/doc/book/pipeline/syntax/.

4. Click Save.

Note:

This instruction describes adding a SoapUI Pro Pipeline step to a Pipeline using the classic Jenkins UI. You can also do that in Blue Ocean, or, if you store the Jenkinsfile with the script in a source control system, you can just edit it manually.

SoapUI Pro Pipeline step

The format of the Pipeline step is as follows:

SoapUIPro (
    pathToTestrunner: '<path>',
    pathToProjectFile: '<path>',
    testSuite: '[<testSuiteName>]',
    testCase: '[<testCaseName>]',
    testSuiteTags: '<test suite tags>',
    testCaseTags: '<test case tags>'
    projectPassword: '[<password>]',
    environment: '[<environmentName>]' )

pathToTestrunner

Required. Specifies the fully-qualified path to the test runner binary (testrunner.bat or testrunner.sh). By default, you can find it in the <ReadyAPI installation>/bin directory.

pathToProjectFile

Required. Specifies the fully-qualified path to the ReadyAPI Test project you want to run.

testSuite

Optional. Specifies the test suite to run. If not specified, the runner will execute all the test suites of your project.

testCase

Optional. Specifies the test case to run. If not specified, the runner will execute all the test cases of the specified test suite, or, if you have not specified a test suite, all the test cases of your project.

testSuiteTags

Optional. Specifies which tags must contain the test suite to be run. To create complex conditions, use the || (logical OR), && (logical AND) and ! (logical NOT) operators.

testCaseTags

Optional. Specifies which tags must contain the test case to be run. To create complex conditions, use the || (logical OR), && (logical AND) and ! (logical NOT) operators.

projectPassword

Optional. Specifies the encryption password, if you encrypted the entire project or some of its custom properties. See Protecting Sensitive Data.

environment

Optional. Specifies the environment configuration for the test run.

If you want to specify additional arguments for the test runner, call the test runner using the command line – see below.

By using command-line runners

• By default, Jenkins runs jobs by using its own user profile. In this case, if you use a Fixed license, ReadyAPI will not be able to find it. To learn how to fix this issue, see Jenkins: "License Not Found" Issue.

• The ReadyAPI Functional Testing plugin ver. 1.4 fixed a security vulnerability in project password storage. If you update to ver. 1.4 from an earlier version, to ensure the security of your passwords, you need to do the following for all the jobs that use the plugin:

  1. Select a job and click Configure.
  2. Save the configuration without making any changes by clicking Save or Apply.