Run ReadyAPI Tests from Azure DevOps (VSTS)

Applies to ReadyAPI 3.52, last modified on April 25, 2024

The easiest way to run ReadyAPI tests from Azure DevOps is to use the ReadyAPI Test for Azure DevOps task. This topic describes how to do that.

Alternatively, you can run ReadyAPI tests by using command-line runners. See Command-Line Runners Specifics for details.

Also, you can run ReadyAPI tests in TestEngine. To learn about it, see Run ReadyAPI Tests in TestEngine from Azure DevOps.

Supported Azure DevOps versions

You can use the ReadyAPI Test for Azure DevOps extension with the following versions of Azure DevOps:

  • Azure DevOps Services (formerly Visual Studio Team Services)

  • Azure DevOps Server 2019

  • Team Foundation Server 2018

  • Team Foundation Server 2017

  • Team Foundation Server 2015 Update 4

There is no much difference between configuring Azure DevOps pipelines and Team Foundation Server build definitions. In this topic, we will show you how to configure only Azure DevOps pipeline.

Requirements for Azure DevOps agents

  • Use self-hosted agents to run ReadyAPI tests. Running tests on Microsoft-hosted agents is not supported. (To learn more about agents, see the Build and Release Agents article in the MSDN Library.)

  • The agent must have activated Pro license to run ReadyAPI tests.

  • ReadyAPI Azure DevOps plugins are updated to support SmartBear ID-based license. We now support two license types:

    • SmartBear ID-based licenses– These licenses don't require license keys or files. Users can simply specify their access credentials to use the product. For offline or on-premise use with headless test runs, we recommend the use of an access key and a floating license server. Below are the specific details.

    • File-based licenses– These are the licenses you previously used and require a SmartBear license file. Please be aware that this type of license will be retired in October 2023.

Getting Started

For detailed information on how to install ReadyAPI, see Install Information. To learn how to activate ReadyAPI licenses, see SmartBear ID-based Licenses.

  • The agent must have access to your team project and be registered in your agent pool. See Deploy an agent on Windows in Azure documentation.

    Note: You cannot use the ReadyAPI Test for Azure DevOps extension on agents deployed on Linux or macOS. To run ReadyAPI tests on such agents, use command-line runners.

  • In addition, if you are going to export test logs, make sure your build agent has access to the folder to which the logs will be exported.

1. Prepare ReadyAPI project

  • Make your ReadyAPI project available to the agents.

    Typically, you store your project in a source control system (like Git or Team Foundation Version Control) and then configure your build to get the project from source control and place it to the needed folder on the computer where tests will run.

    As an alternative, you can place your project in a shared network folder. Make sure your agent has access to that folder.

  • In addition, if your project uses external files as data sources for your tests (see Data Sources and Data-Driven Testing), make sure your build agent has access to these files.

2. Install ReadyAPI Test for Azure DevOps extension

  1. Go to the Visual Studio Marketplace.

    Tip: You can open the marketplace directly from Azure DevOps by selecting Browse Marketplace in the top right corner:
    Integration with Team Foundation Server

  2. Search for the ReadyAPI Test for Azure DevOps extension.

  3. The way you install the extension depends on the Azure DevOps version you use:

    Azure DevOps Services

    Connected Azure DevOps Server

    Disconnected Azure DevOps Server

3. Configure build definition

To run ReadyAPI tests from Azure DevOps, perform the following steps:

3.1. Create a build definition

  1. Open your Azure DevOps project’s web portal and navigate to the Pipeline hub.

  2. Open an existing pipeline:

    Open an existing pipeline

    Click the image to enlarge it.

    – or –

    Create a new one:

    Create a new pipeline

    Click the image to enlarge it.

  3. If your ReadyAPI project is stored in a source control system, configure your build definition to copy it to the folder to which an agent has access:

    Get files from source control

    Click the image to enlarge it.

3.2. Configure pipeline to run tests

To run ReadyAPI tests, use the ReadyAPI Test for Azure DevOps task. It allows you to easily configure your pipeline to run ReadyAPI Test tests.

Alternatively, you can run ReadyAPI tests by using command-line runners. In this case, you must add one of the built-in tasks and specify command-line arguments. To learn more, see Command-Line Runners Specifics.

Add the ReadyAPI Test for Azure DevOps task and specify its parameters:

Add a ReadyAPI Test for Azure DevOps task

Click the image to enlarge it.

Tip:

By default, Azure DevOps adds version 2 of the extension. If you want to use version 1, select it from the Version drop-down list.
Option Description
Path to ReadyAPI Test Pro Project Required. The path to your ReadyAPI project.

Note: Specify a dot (.) if your composite project is stored in the root folder of the repository. For example, if you use Git integration.
Path to Functional TestRunner Fully-qualified path to the particular version of Functional TestRunner. Leave the field blank to use the recent version of ReadyAPI installed on an agent.
Root Folder Required. Fully-qualified path to the directory on the agent where ReadyAPI will store test reports. The default value is $(Common.TestResultsDirectory).
Test Suite A test suite to run. To run all the test suites in the project, leave the field blank.
Test Case A test case to run. Leave the field blank to run all the test cases of the specified test suite, or, if you have not specified a test suite, all the test cases of your project.
Password The encryption password. It is required if your project is encrypted.
Environment The environment configuration that the task will use for the test run. See Environments.
Test Suite Tags The tags of the test suites you want to run.
Note: The parameter is available in the extension version 2.0.
Test Case Tags The tags of the test cases you want to run.
Note: The parameter is available in the extension version 2.0.
Authentication Required Required. Allows one of following 4 options:

  • legacy (File based license) – ReadyAPI will attempt to authenticate testrunner process using file based(.key) license.

    Azure-legacy

    Click the image to enlarge it.

  • anonymous (Anonymous access) – Testrunner will attempt to obtain license as anonymous user. Is available only for SLM on-prem license server:

    • Available with testrunner only with ReadyAPI 3.46.0 or newer

      Azure-anonymous

      Click the image to enlarge it.

      Azure-anonymous

      Click the image to enlarge it.

  • accessKey (Access key) – Testrunner will attempt to obtain license using access key (from SLM license server).

    Azure-accesskey

    Click the image to enlarge it.

    • maps to -K command line argument.

    • Recommended– user must provide accessKey argument to pipeline for this authentication method to work correctly.

  • credentials (Username and password) – Testrunner will attempt to obtain license using username and password for SLM license server.

    Azure-credentials

    Click the image to enlarge it.

    • will work only for SLM on-prem license server with LDAP configured correctly.

    • maps to respectively -U and -V command line arguments for username and password.

    • user must provide username and password arguments to pipeline for this authentication method to work correctly.
licenseApiHost (License Server Configuration → Host)   Host on which the slm license server resides:

  • maps to -DlicenseApiHost command line argument
licenseApiPort (License Server Configuration → Port)   Port on which the slm license server accepts connections:

  • maps to -DlicenseApiPort command line argument

Note: licenseApiHost and licenseApiPort will most likely need to be provided together.
Configure a ReadyAPI Test for Azure DevOps task

Click the image to enlarge it.

3.3. (Optional) Get the test results

Skip this step if you use the ReadyAPI Test for Azure DevOps extension version 2.0. This version of the extension publishes test results automatically.

After the test run is over, the ReadyAPI Test for Azure DevOps extension exports the test results in a JUnit-style format, so you can publish the test results to your team project.

Note: If you use command-line runners, the way you export test results depends on the arguments you pass to the runner. If you export test results in a format different from a JUnit-style report, you can use the Copy Files or Publish Artifacts task to fetch the exported files and place them in the desired folder (for example, in your build’s drop folder).

To publish test results to your team project:

  1. Add the Publish Test Results task to your pipeline.

  2. In the Test result format drop-down list, select JUnit.

  3. In the Test results files text box, specify the **\*.xml minimatch pattern to search for the exported JUnit-style report files.

  4. In the Search Folder text box, specify the folder in which your ReadyAPI test results are stored.

  5. To merge several reports into a single test result, select the Merge test results check box.

  6. In the Test run title text box, enter the name of the test run that will be assigned to the test results. This name will be used in the test run statistics (you can view it in the Test hub).

Publish test results

Click the image to enlarge it.

3.4. (Optional) Select agents to run

If you have multiple Azure DevOps agents, you can configure your pipeline to run ReadyAPI tests only on those agents that have ReadyAPI installed. To do this, for the needed agents, register custom capabilities that will specify that the agents have ReadyAPI installed, and then specify demands for your pipelines to run only on the needed agents:

  1. Open your Azure DevOps project’s web portal. Navigate to the Agents Pool page (you must have an appropriate permission to access the page and manage agent queues):

  2. Select the pool to which your agent belongs:

    Select an agent pool

    Click the image to enlarge it.

  3. Switch to the Agents page and select the needed agent:

    Select an agent

    Click the image to enlarge it.

  4. Switch to the Capabilities page.

  5. Click Add a new capability and create a custom capability that will specify whether ReadyAPI is installed on the agent:

    Adding custom capabilities

    Click the image to enlarge it.

  6. Click Add to apply the changes.

  1. Open a pipeline that will run your ReadyAPI tests.

  2. Switch to the Options tabbed page.

  3. In the Demands section, click Add.

  4. Enter the needed capability name, condition and values:

    Adding demands of agents to a build defintion

    Click the image to enlarge it.

  5. Save the changes.

Your pipeline will run only on those agents that have ReadyAPI installed.

3.5. Save build definition

To save the build, select Save & queue > Save:

VSTS integration: Save a build

Click the image to enlarge it.

4. Run build and view test result

To run the build:

  • Select it on the Builds page of the Pipelines hub and then click Queue.

– or –

  • Click Queue on the Edit Build Definition page.

Switch to the Jobs page to view the build progress:

Monitoring the build progress

Click the image to enlarge it.

The build will run the task that executes your ReadyAPI tests. It will also fetch and publish the test results if configured.

After the build is over, you can view the test results. The Tests page of the build report shows the number of failed and passed tests:

Test results in the Build Summary

Click the image to enlarge it.

Click the needed test to view detailed test results:

Detailed test results

Click the image to enlarge it.

On the Test hub, you can view the test run statistics for your team project:

Test run statistics

Click the image to enlarge it.

Click a test run to view detailed information.

Possible issues

When you configure an Azure agent to run as a service, you need to run the service as the same user who activated the ReadyAPI Test license. Otherwise, ReadyAPI will not be able to find it:

Azure integration: License not found issue

Click the image to enlarge it.

You can do it both when configuring an agent and when the agent service is already created:

When you create an agent, you need to configure it by using the utility shipped with the agent binaries. See the Microsoft documentation. You can specify the needed user during this process. To do this:

  1. After you specify that you are going to run the agent as a service, the utility asks you which user account you want to use:

    Azure integration: Register new agent as a service

    Click the image to enlarge it.

  2. Specify the user account that you used to activate the ReadyAPI Test license and enter its password:

    Azure integration: Register new agent as a service

    Click the image to enlarge it.

  1. Make sure you have a valid license activated. If you do not have a valid license, activate a ReadyAPI Test license (see License Activation).

  2. Open the Control Panel.

  3. Select System and Security and then Administrative Tools.

  4. Run the Services utility.

  5. Search for the agent service. Its name looks something like VSTS Agent (<agent name>).

  6. Right-click the service and select Properties.

  7. On the Log On tab, select This account and enter the credentials of the Windows user account you will use to run ReadyAPI Test.

    Note: This is the account that you used when activating the ReadyAPI Test license.
    Agent service properties

    Click the image to enlarge it.

  8. Restart the service to apply the changes.

ReadyAPI uses the UseBiasedLockin JVM option. Since this option was deprecated, your build may fail because of the following warning:

OpenJDK 64-Bit Server VM warning: Option UseBiasedLocking was deprecated in version 15.0 and will likely be removed in a future release

To avoid the warning, remove the -XX:+UseBiasedLocking JVM option. To do this, open the runner executable file in any text editor and remove the option from the list of JVM parameters:

.bat file
.sh file

.bat file

set JAVA_OPTS=-Xms128m -Xmx%READY_XMX% -XX:+UseBiasedLocking -Dtest.history.disabled=true -Dsoapui.properties=soapui.properties -Dgroovy.source.encoding=iso-8859-1 "-Dsoapui.home=%READY_API_HOME%\"

.sh file

JAVA_OPT="-Xms128m -Xmx$READY_XMX -XX:+UseBiasedLocking -Dtest.history.disabled=true -Dsoapui.properties=soapui.properties -Dgroovy.source.encoding=iso-8859-1 -Dsoapui.home=$READY_API_HOME/bin"

To learn more, see Modifying JVM Settings.

See Also

Azure DevOps (formerly VSTS) Integration
Command-Line Runners Specifics
ReadyAPI Documentation

Highlight search results