Run ReadyAPI Tests from Azure DevOps (VSTS)

The easiest way to run ReadyAPI tests from Azure DevOps is to use the ReadyAPI Test for Azure DevOps task. This topic describes how to do that.

Alternatively, you can run ReadyAPI tests by using command-line runners. See Command-Line Runners Specifics for details.

Also, you can run ReadyAPI tests in TestEngine. To learn about it, see Run ReadyAPI Tests in TestEngine from Azure DevOps.

Supported Azure DevOps versions

You can use the ReadyAPI Test for Azure DevOps extension with the following versions of Azure DevOps:

  • Azure DevOps Services (formerly Visual Studio Team Services)

  • Azure DevOps Server 2019

  • Team Foundation Server 2018

  • Team Foundation Server 2017

  • Team Foundation Server 2015 Update 4

There is not much difference between configuring Azure DevOps pipelines and Team Foundation Server build definitions. In this topic, we will show you how to configure only Azure DevOps pipeline.

Requirements for Azure DevOps agents

  • Important

    Use self-hosted agents to run ReadyAPI tests. Running tests on Microsoft-hosted agents is not supported. (To learn more about agents, see the Build and Release Agents article in the MSDN Library.)

  • The agent must have activated Pro license to run ReadyAPI tests.

  • ReadyAPI Azure DevOps plugins are updated to support SmartBear ID-based license. We now support two license types:

    • SmartBear ID-based licenses– These licenses don't require license keys or files. Users can specify their access credentials to use the product. For offline or on-premises use with headless test runs, we recommend the use of an access key and a floating license server. Below are the specific details.

    • File-based licenses– These are the licenses you previously used and require a SmartBear license file. Please be aware that this type of license will be retired in October 2023.

Getting Started

For detailed information on how to install ReadyAPI, see Install Information. To learn how to activate ReadyAPI licenses, see SmartBear ID-based Licenses.

  • The agent must have access to your team project and be registered in your agent pool. See Deploy an agent on Windows in Azure documentation.

    Note

    You cannot use the ReadyAPI Test for Azure DevOps extension on agents deployed on Linux or macOS. To run ReadyAPI tests on such agents, use command-line runners.

  • In addition, if you are going to export test logs, make sure your build agent has access to the folder to which the logs will be exported.

1. Prepare ReadyAPI project

  • Make your ReadyAPI project available to the agents.

    Typically, you store your project in a source control system (like Git or Team Foundation Version Control) and then configure your build to get the project from source control and place it in the needed folder on the computer where tests will run.

    As an alternative, you can place your project in a shared network folder. Make sure your agent has access to that folder.

  • In addition, if your project uses external files as data sources for your tests (see Data Sources and Data-Driven Testing), make sure your build agent has access to these files.

2. Install ReadyAPI Test for Azure DevOps extension

  1. Go to the Visual Studio Marketplace.

    Tip

    You can open the marketplace directly from Azure DevOps by selecting Browse Marketplace in the top right corner:

    Integration with Team Foundation Server

  2. Search for the ReadyAPI Test for Azure DevOps extension.

  3. The way you install the extension depends on the Azure DevOps version you use:

3. Configure build definition

To run ReadyAPI tests from Azure DevOps, perform the following steps:

go.gif3.1. Create a build definition

go.gif3.2. Configure pipeline to run tests

go.gif3.3. (Optional) Get the test results

go.gif3.4. (Optional) Select agents to run

go.gif3.5. Save build definition

go.gifNo valid ReadyAPI Test license

go.gifThe "UseBiasedLocking was deprecated" warning

3.1. Create a build definition

  1. Open your Azure DevOps project’s web portal and navigate to the Pipeline hub.

  2. Open an existing pipeline:

    Open an existing pipeline

    – or –

    Create a new one:

    Create a new pipeline

  3. If your ReadyAPI project is stored in a source control system, configure your build definition to copy it to the folder to which an agent has access:

    Get files from source control

3.2. Configure pipeline to run tests

To run ReadyAPI tests, use the ReadyAPI Test for Azure DevOps task. It allows you to easily configure your pipeline to run ReadyAPI Test tests.

Alternatively, you can run ReadyAPI tests by using command-line runners. In this case, you must add one of the built-in tasks and specify command-line arguments. To learn more, see Command-Line Runners Specifics.

Add the ReadyAPI Test for Azure DevOps task and specify its parameters:

Add a ReadyAPI Test for Azure DevOps task

Tip

By default, Azure DevOps adds version 2 of the extension. If you want to use version 1, select it from the Version drop-down list.

Option

Description

Path to ReadyAPI Test Pro Project

Required. The path to your ReadyAPI project.

Note

Specify a dot (.) if your composite project is stored in the root folder of the repository. For example, if you use Git integration.

Path to Functional TestRunner

Fully-qualified path to the particular version of Functional TestRunner. Leave the field blank to use the recent version of ReadyAPI installed on an agent.

Root Folder

Required. Fully-qualified path to the directory on the agent where ReadyAPI will store test reports. The default value is $(Common.TestResultsDirectory).

Test Suite

A test suite to run. To run all the test suites in the project, leave the field blank.

Test Case

A test case to run. Leave the field blank to run all the test cases of the specified test suite, or, if you have not specified a test suite, all the test cases of your project.

Password

The encryption password. It is required if your project is encrypted.

Environment

The environment configuration that the task will use for the test run. See Environments.

Test Suite Tags

The tags of the test suites you want to run.

Note

The parameter is available in the extension version 2.0.

Test Case Tags

The tags of the test cases you want to run.

Note

The parameter is available in the extension version 2.0.

Authentication Required

Required. Allows one of following 4 options:

  • legacy (File based license) – ReadyAPI will attempt to authenticate testrunner process using file based(.key) license.

    Azure-legacy

  • anonymous (Anonymous access) – Testrunner will attempt to obtain license as anonymous user. Is available only for SLM on-prem license server:

    • Available with testrunner only with ReadyAPI 3.46.0 or newer

      Azure-anonymous
      Azure-anonymous

  • accessKey (Access key) – Testrunner will attempt to obtain license using access key (from SLM license server).

    Azure-accesskey

    • maps to -K command line argument.

    • Recommended – user must provide accessKey argument to pipeline for this authentication method to work correctly.

  • credentials (Username and password) – Testrunner will attempt to obtain license using username and password for SLM license server.

    Azure-credentials

    • will work only for SLM on-prem license server with LDAP configured correctly.

    • maps to respectively -U and -V command line arguments for username and password.

    • user must provide username and password arguments to pipeline for this authentication method to work correctly.

  • client credentials (clientId and clientSecret) – When using SLM on-premise license server 2.0 or later, and clientId and clientSecret are added as arguments, ReadyAPI will retrieve the token from SLM. This token is then used to fetch, lock, and release licenses.

    If the Client Credentials option is elected, clientId and clientSecret are expected to be entered into input boxes in the Azure plugin.

    Example:

    testrunner.bat "C:\RAPI_Project\REST-Sample-Project-readyapi-project.xml" --clientId "**********" --clientSecret "**********" -DlicenseApiHost=https://********** -DlicenseApiPort=*****

licenseApiHost (License Server Configuration → Host)

Host on which the SLM license server resides:

  • maps to -DlicenseApiHost command line argument

licenseApiPort (License Server Configuration → Port)

Port on which the slm license server accepts connections:

  • maps to -DlicenseApiPort command line argument

Note

licenseApiHost and licenseApiPort will most likely need to be provided together.

Configure a ReadyAPI Test for Azure DevOps task

3.3. (Optional) Get the test results

Important

Skip this step if you use the ReadyAPI Test for Azure DevOps extension version 2.0. This version of the extension publishes test results automatically.

After the test run is over, the ReadyAPI Test for Azure DevOps extension exports the test results in a JUnit-style format, so you can publish the test results to your team project.

Note

If you use command-line runners, the way you export test results depends on the arguments you pass to the runner. If you export test results in a format different from a JUnit-style report, you can use the Copy Files or Publish Artifacts task to fetch the exported files and place them in the desired folder (for example, in your build’s drop folder).

To publish test results to your team project:

  1. Add the Publish Test Results task to your pipeline.

  2. In the Test result format drop-down list, select JUnit.

  3. In the Test results files text box, specify the **\*.xml minimatch pattern to search for the exported JUnit-style report files.

  4. In the Search Folder text box, specify the folder in which your ReadyAPI test results are stored.

  5. To merge several reports into a single test result, select the Merge test results check box.

  6. In the Test run title text box, enter the name of the test run that will be assigned to the test results. This name will be used in the test run statistics (you can view it in the Test hub).

Publish test results

3.4. (Optional) Select agents to run

If you have multiple Azure DevOps agents, you can configure your pipeline to run ReadyAPI tests only on those agents that have ReadyAPI installed. To do this, for the needed agents, register custom capabilities that will specify that the agents have ReadyAPI installed, and then specify demands for your pipelines to run only on the needed agents:

1. Define custom capacities for your agents

  1. Open your Azure DevOps project’s web portal. Navigate to the Agents Pool page (you must have an appropriate permission to access the page and manage agent queues):

    For Azure DevOps Services:

    go.gifhttps://<Your_VSO_Account_Name>.visualstudio.com/_admin/_AgentPool

    For On-Premises Azure DevOps Server:

    go.gifhttps://<Your_Server_Name>/tfs/<Collection_Name>/_admin/_AgentPool

  2. Select the pool to which your agent belongs:

    Select an agent pool

  3. Switch to the Agents page and select the needed agent:

    Select an agent

  4. Switch to the Capabilities page.

  5. Click add-b.gif Add a new capability and create a custom capability that will specify whether ReadyAPI is installed on the agent:

    Adding custom capabilities

  6. Click Add to apply the changes.

2. Specify build and release demands

  1. Open a pipeline that will run your ReadyAPI tests.

  2. Switch to the Options tabbed page.

  3. In the Demands section, click Add.

  4. Enter the needed capability name, condition and values:

    Adding demands of agents to a build defintion

  5. Save the changes.

Your pipeline will run only on those agents that have ReadyAPI installed.

3.5. Save build definition

To save the build, select Save & queue > Save:

VSTS integration: Save a build

4. Run build and view test result

To run the build:

  • Select it on the Builds page of the Pipelines hub and then click queue-build-b.gif Queue.

– or –

  • Click queue-b.gif Queue on the Edit Build Definition page.

Switch to the Jobs page to view the build progress:

Monitoring the build progress

The build will run the task that executes your ReadyAPI tests. It will also fetch and publish the test results if configured.

After the build is over, you can view the test results. The Tests page of the build report shows the number of failed and passed tests:

Test results in the Build Summary

Click the needed test to view detailed test results:

Detailed test results

On the Test hub, you can view the test run statistics for your team project:

Test run statistics

Click a test run to view detailed information.

Possible issues

No valid ReadyAPI Test license

When you configure an Azure agent to run as a service, you need to run the service as the same user who activated the ReadyAPI Test license. Otherwise, ReadyAPI will not be able to find it:

Azure integration: License not found issue

You can do it both when configuring an agent and when the agent service is already created:

Specify a user while configuring an agent

When you create an agent, you need to configure it by using the utility shipped with the agent binaries. See the Microsoft documentation. You can specify the needed user during this process. To do this:

  1. After you specify that you are going to run the agent as a service, the utility asks you which user account you want to use:

    Azure integration: Register new agent as a service

  2. Specify the user account that you used to activate the ReadyAPI Test license and enter its password:

    Azure integration: Register new agent as a service

Specify a user when a service exists

  1. Make sure you have a valid license activated. If you do not have a valid license, activate a ReadyAPI Test license (see License Activation).

  2. Open the Control Panel.

  3. Select System and Security and then Administrative Tools.

  4. Run the Services utility.

  5. Search for the agent service. Its name looks something like VSTS Agent (<agent name>).

  6. Right-click the service and select Properties.

  7. On the Log On tab, select This account and enter the credentials of the Windows user account you will use to run ReadyAPI Test.

    Note

    This is the account that you used when activating the ReadyAPI Test license.

    Agent service properties

  8. Restart the service to apply the changes.

The "UseBiasedLocking was deprecated" warning

ReadyAPI uses the UseBiasedLockin JVM option. Since this option was deprecated, your build may fail because of the following warning:

OpenJDK 64-Bit Server VM warning: Option UseBiasedLocking was deprecated in version 15.0 and will likely be removed in a future release

To avoid the warning, remove the -XX:+UseBiasedLocking JVM option. To do this, open the runner executable file in any text editor and remove the option from the list of JVM parameters:

.bat file
set JAVA_OPTS=-Xms128m -Xmx%READY_XMX% -XX:+UseBiasedLocking -Dtest.history.disabled=true -Dsoapui.properties=soapui.properties -Dgroovy.source.encoding=iso-8859-1 "-Dsoapui.home=%READY_API_HOME%\"
.sh file
JAVA_OPT="-Xms128m -Xmx$READY_XMX -XX:+UseBiasedLocking -Dtest.history.disabled=true -Dsoapui.properties=soapui.properties -Dgroovy.source.encoding=iso-8859-1 -Dsoapui.home=$READY_API_HOME/bin"

To learn more, see Modifying JVM Settings.

See Also

In this section:
© 2025
Publication date: