Running ReadyAPI Tests From Jenkins

Applies to ReadyAPI 3.51, last modified on March 04, 2024

This topic describes a general approach to running ReadyAPI functional tests from Jenkins.

Overview

We need to create a Jenkins job that will run ReadyAPI functional tests from Jenkins agent machines (or Jenkins nodes). To run functional tests, you need to install ReadyAPI on each Jenkins node and use the ReadyAPI Functional Testing plugin or ReadyAPI command-line runners to run the tests.

The ReadyAPI Functional Testing plugin provides you with an easy way to run functional tests from Jenkins jobs and Jenkins Pipelines. If you need to run load or secure tests, or if you need to specify advanced options for a functional test run, use the command-line runners.

Note: To run ReadyAPI tests in TestEngine, use the ReadyAPI TestEngine Jenkins Plugin.
To deploy, start, stop or update virtual services on VirtServer, use the Jenkins VirtServer plugin.

Required license

In order to run ReadyAPI tests, you need to activate the appropriate license on each node. The easiest way to do this is to install ReadyAPI on a node and activate the license through the ReadyAPI user interface. However, it is considered to be a good practice to clean up the test environment after the test run is over. In other words, you may want to configure nodes automatically before each run, and this means you may want to automate the ReadyAPI license activation. ReadyAPI supports a special command-line tool to automate activation (see below).

Which license to use: Floating or Fixed? Floating licenses are required for dynamic testing environments. You can activate the license from SmartBear License Servers or install it offline on a computer within your network, which can serve as a license server. You can then configure ReadyAPI instances on Jenkins nodes to connect to this server and obtain a license from it.

Requirements to Jenkins nodes

  • One of the supported Java versions installed.

    Note:

    ReadyAPI is updated to support Java 17.0.7 in ReadyAPI 3.51. The Installers automatically install the appropriate Java version and we recommend using the Java version bundled with the product. Support for this version will also be required with your Virtual Machines and Docker instances.

  • Access to the ReadyAPI installation program (you can copy it to your Jenkins nodes, or you can put it to a shared network folder).

  • On Windows Jenkins nodes, disable the User Account Control (UAC). You can do this in the Control Panel. Otherwise, Windows can display various system confirmation dialogs, which require user interaction.

If you are using File-based licenses

  • Network access to the license server with the Floating license activated (by default, the server works on port 1099).

  • Access to the ReadyAPI License Manager. You can download it in the .zip format from our website. Unpack the downloaded .zip archive and copy its content to your Jenkins node or put it to a shared network folder, to which the node has access.

If you are using SmartBear ID-based licenses, please refer to 3.1

ReadyAPI Functional Testing plugin

To make the communication between Jenkins and ReadyAPI easier, consider installing the ReadyAPI Functional Testing plugin. To install it:

  1. Open Jenkins and click Manage Jenkins:

  2. Open Manage Plugins:

    Automating ReadyAPI: Managing Jenkins plugins

    Click the image to enlarge it.

  3. On the Available page, type ReadyAPI in the Filter field and select ReadyAPI Functional Testing.

    Automating ReadyAPI: Select ReadyAPI Functional Testing Plugin

    Click the image to enlarge it.

  4. Click Install without restart or Download now and install after restart:

    Install ReadyAPI Test Pro Functional Testing Plugin

    Click the image to enlarge it.

1. Create and configure Jenkins nodes

Jenkins documentation includes a detailed guide on configuring Jenkins nodes. Follow the instructions on this page:

https://wiki.jenkins.io/display/JENKINS/Step+by+step+guide+to+set+up+master+and+agent+machines+on+Windows

Tip: We recommend using Java Web Start for launching nodes in most scenarios. The optimal method depends on your environment.

2. Install ReadyAPI on Jenkins nodes

To install ReadyAPI automatically:

  1. Use the Execute batch command build step. The way you add it depends on the operating system of the Jenkins node:

    • Windows Jenkins nodes: select Add build step > Execute Windows batch command.

    • Linux or macOS nodes: select Add build step > Execute shell.

    Execute batch command build step

    Click the image to enlarge it.

  2. Configure the step to run the ReadyAPI installer. Add the -q argument to the installer command line. This way, the installation will run in silent mode, that is, without any user interaction, and will use the default values for all options.

    To change the installation folder, you can use the -dir argument. -varfile lets you specify additional installation parameters. You can find more information in the install4j documentation.

    Here is a sample command line that installs ReadyAPI in silent mode:

    "c:\sample\ReadyAPI-x64-3.51.0.exe" -q -dir "c:\readyapi-folder" -varfile response.varfile

3. Get a ReadyAPI license

ReadyAPI Jenkins plugins are updated to support SmartBear ID-based license options, both online and offline. The plugin now supports two types of license:

SmartBear ID-based licenses – These licenses eliminate the need for license keys or files. Instead, you can simply provide your access credentials to use the product. For offline or on-premise use with headless test runs, we recommend using an access key and a floating license server. Below are the specific details.

File-based licenses – These licenses were previously used and required a SmartBear license file. Please be aware that this type of license will be phased out in October 2023.

3.1 When using ID-Based Licensing

We have updated the ReadyAPI Jenkins plugin that now enables ReadyAPI tests to be run on Jenkins in three different ways:

  • ReadyAPI Functional Testing plugin can now be used as a new Build step > ReadyAPI Test: Run functional Test (available from ReadyAPI 3.46.0 onwards).

  • Jenkins Pipeline can now be used with the SoapUI Pro Pipeline step (available from ReadyAPI 3.46.0 onwards).

  • Command-line runners can still be used, as before.

Note: Starting from version 3.46.0, you can pass the Access Key as a "-K" parameter for the first two options listed above.

In ReadyAPi 3.46.1, the Jenkins Plugin has extended to support the same license authentication as recent changes to the Azure Plugin. We have added the following parameters:

  • License authentication method (Select one of the following options: "File based license", "API KEY", "User and Password", or "Access for everyone")

  • SLM License API Host

  • SLM License API Port

License authentication methods
  • File based license – ReadyAPI authenticates the testrunner process using a file-based license (.key). The test runner is executed without additional license-related parameters, following the existing implementation.

  • API KEY – Testrunner will attempt to obtain a license using the access key (from the SLM license server). Testrunner command line includes -K {API_KEY} parameter - it is already implemented in Jenkins Plugin v1.8.

    Jenkins API Key

    Click the image to enlarge it.

    If the license host and port are added, they will be added to the testrunner command line as -DlicenseApiHost and -DlicenseApiPort. License host and port are required if using an on-premise license server (for offline access) with authentication configured. It is the recommended approach to acquiring licenses when using the Jenkins plugin.

  • User and Password – Testrunner will attempt to acquire a license using a username and password for the SLM license server. It is executed with the -U {username} and -V {password} parameters.

    Jenkins Username and Password

    Click the image to enlarge it.

    Currently, the Jenkins Plugin requires the SLM License API Host and Port as mandatory parameters. These parameters will be added to the test runner command line as -DlicenseApiHost and -DlicenseApiPort, specifying where your SLM on-premise license server is installed. This configuration requires an SLM on-premise SLM license server with LDAP properly configured, where the username and password match the SLM license server/LDAP configuration. If you are using SmartBear-hosted ID-based licenses, we recommend using headless commands with an access key as outlined here.

    Note: SmartBear License Management's license authentication and request flows were revised in November 2023. Authentication using the username and password is no longer supported for test execution via TestRunners, Jenkins or Azure plugins, and SmartBear-hosted licenses. Username and password are limited to on-prem-hosted licenses only. Access Key is the required option for authentication if using SmartBear-hosted licenses headlessly.
  • Access for everyone – Testrunner will attempt to acquire a license as an anonymous user. This option is only available on an SLM on-premise license server. The testrunner command line includes the -DlicenseApiAccessForEveryone=true parameter.

    Additionally, in this case, the SLM License API Host and Port are required parameters. They will be included in the testrunner command line as -DlicenseApiHost and -DlicenseApiPort.

3.2 When using File-Based Licensing

  1. On your Jenkins node, create a text file that will contain the code of the ReadyAPI application, whose license you want to consume:

    1 ReadyAPI Test
    2 ReadyAPI Performance
    3 Secure (if you have an old license)
    4 ReadyAPI Virtualization
    5 VirtServer
    6 ReadyAPI Bundle
    7 TestEngine
    8 SwaggerHub
    9 SwaggerHub 2
    Tip: If you use the ReadyAPI Functional Testing plugin, specify 1.
  2. To get the ReadyAPI license, run the license-manager.bat (on Windows node) or .sh file (on Linux and macOS nodes) from the command line. To do this, add another Execute batch command step to your Jenkins job. Use the -s license-server-IP:port < file argument to specify the license server from which the License Manager will consume a Floating license:

    license-manager.bat -s 10.0.81.128:1099 < c:\readyapi-folder\MyCode.txt

    Here, MyCode.txt is the file that contains the code of the licensed ReadyAPI application (see above).

    Note: The License Manager requires Java 12. You can use Java shipped with ReadyAPI or download the required version from https://jdk.java.net/archive/.

4. Copy test projects and data files to nodes

To run ReadyAPI tests from a Jenkins node, you need to copy your ReadyAPI project to this node. You also need to copy the data files your tests use.

If your tests use custom script libraries, copy the libraries to the <ReadyAPI installation>/bin/scripts folder on the Jenkins node.

To copy the files, use the Jenkins Copy file build step. A possible alternative is to place the project and data file to some shared network folder to which Jenkins nodes have access.

5. Run tests

By using the ReadyAPI Functional Testing plugin

To run functional tests in Jenkins:

  1. Make sure the ReadyAPI Functional Testing plugin is installed.

  2. In Jenkins, select a job and click Configure.

  3. In the Build section, click Add build step > ReadyAPI Test: Run functional Test:

  4. Jenkins will add the step to the build:

    Automating ReadyAPI: Run functional tests from Jenkins

    Click the image to enlarge it.

  5. Configure the following build step options:

    Option Description
    Path to TestRunner Specifies the fully-qualified path to the test runner binary (testrunner.bat or testrunner.sh). By default, you can find it in the <ReadyAPI installation>/bin directory.
    Path to ReadyAPI project Specifies the fully-qualified path to the ReadyAPI project you want to run.
    Test Suite Specifies the test suite to run. To run all the test suites of your project, leave the field blank.
    Test Case Specifies the test case to run. If you leave the field blank, the runner will execute all the test cases of the specified test suite, or, if you have not specified a test suite, all the test cases of your project.
    Test Suite Tags Specifies which tags the test suite must contain to be run. To create complex conditions, use the || (logical OR), && (logical AND) and ! (logical NOT) operators.
    Test Case Tags Specifies which tags the test case must contain to be run. To create complex conditions, use the || (logical OR), && (logical AND) and ! (logical NOT) operators.
    Project Password Specifies the encryption password, if you encrypted the entire project or some of its custom properties. See Protecting Sensitive Data.
    Environment Specifies the environment configuration for the test run.
    If you want to specify additional arguments for the test runner, call the test runner using the command line – see below.
Analyze results

If you use the ReadyAPI Functional Testing plugin, the functional test runner exports a JUnit-style HTML report to your Jenkins workspace directory.

You can also view a PDF version of the report directly from Jenkins. To do that, open the needed project and build, then click the ReadyAPI Test Results link to the left:

The SoapUI Pro Test Results link

Click the image to enlarge it.

By using Jenkins Pipeline

The ReadyAPI Functional Testing plugin supports launching tests from Jenkins Pipeline. To run a test, you need to add the SoapUI Pro Pipeline step to your Pipeline script:

  1. Make sure the ReadyAPI Functional Testing and Jenkins Pipeline plugins are installed.

  2. In Jenkins, select a Pipeline and click Configure.

  3. In the Pipeline section, add the SoapUI Pro Pipeline step to your Pipeline script:

    Automating ReadyAPI: Run functional tests from a Jenkins Pipeline

    Click the image to enlarge it.

    Below is an example of a Pipeline script with a SoapUI Pro Pipeline step:

    View example

    Tip: To learn more about the Pipeline syntax, see https://jenkins.io/doc/book/pipeline/syntax/.
  4. Click Save.

Note: This instruction describes adding a SoapUI Pro Pipeline step to a Pipeline using the classic Jenkins UI. You can also do that in Blue Ocean, or, if you store the Jenkinsfile with the script in a source control system, you can just edit it manually.
SoapUI Pro Pipeline step

The format of the Pipeline step is as follows:

SoapUIPro (
    pathToTestrunner: '<path>',
    pathToProjectFile: '<path>',
    testSuite: '[<testSuiteName>]',
    testCase: '[<testCaseName>]',
    testSuiteTags: '<test suite tags>',
    testCaseTags: '<test case tags>'
    projectPassword: '[<password>]',
    environment: '[<environmentName>]' )

pathToTestrunner

Required. Specifies the fully-qualified path to the test runner binary (testrunner.bat or testrunner.sh). By default, you can find it in the <ReadyAPI installation>/bin directory.

pathToProjectFile

Required. Specifies the fully-qualified path to the ReadyAPI Test project you want to run.

testSuite

Optional. Specifies the test suite to run. If not specified, the runner will execute all the test suites of your project.

testCase

Optional. Specifies the test case to run. If not specified, the runner will execute all the test cases of the specified test suite, or, if you have not specified a test suite, all the test cases of your project.

testSuiteTags

Optional. Specifies which tags must contain the test suite to be run. To create complex conditions, use the || (logical OR), && (logical AND) and ! (logical NOT) operators.

testCaseTags

Optional. Specifies which tags must contain the test case to be run. To create complex conditions, use the || (logical OR), && (logical AND) and ! (logical NOT) operators.

projectPassword

Optional. Specifies the encryption password, if you encrypted the entire project or some of its custom properties. See Protecting Sensitive Data.

environment

Optional. Specifies the environment configuration for the test run.

If you want to specify additional arguments for the test runner, call the test runner using the command line – see below.

To run tests, use one of the ReadyAPI command-line test runners:

To run any of these tools, add an Execute batch command step to your Jenkins job. As we wrote above, the way you do this depends on the operating system of your Jenkins node.

Below is a sample command line that starts a functional test on a Windows Jenkins node:

"c:\readyapi-folder\bin\testrunner.bat" "-sTestSuite 1" -cSimpleRequest "-RTestCase Report" "-EDefault environment" "c:\work\projects\my-project.xml"

For complete information on the command-line arguments, follow the links above.

Possible issues

  • By default, Jenkins runs jobs by using its own user profile. In this case, if you use a Fixed license, ReadyAPI will not be able to find it. To learn how to fix this issue, see Jenkins: "License Not Found" Issue.

  • The ReadyAPI Functional Testing plugin ver. 1.4 fixed a security vulnerability in project password storage. If you update to ver. 1.4 from an earlier version, to ensure the security of your passwords, you need to do the following for all the jobs that use the plugin:

    1. Select a job and click Configure.
    2. Save the configuration without making any changes by clicking Save or Apply.

See Also

Integrations

Watch the video
 
Highlight search results