To run ReadyAPI tests from Azure DevOps, we recommend that you use the ReadyAPI Test for Azure DevOps extension.

Besides the extension, you can use one of the built-in Azure Pipeline tasks to run ReadyAPI command-line runners. It may be useful in the following cases:

If you need to run load or security tests.
If you need to specify additional options for Functional TestRunner (for example, you can pass project, system or global properties to the test, or export test results to a PDF file).
If you want to use the Azure DevOps agent deployed on Linux or macOS.

Instructions

To run ReadyAPI tests by using command-line runners, perform the steps described in the Run ReadyAPI Tests from Azure DevOps (VSTS) topic. On step 3.2, instead of adding and configuring the ReadyAPI Test for Azure DevOps task, add one of the following tasks and configure it:

Command Line
Bash Script
Batch Script
PowerShell Script

Command Line task (for Windows OS)

After you add the Command Line task to your pipeline, configure it:

In the Tool text box, specify the path to the command-line runner that will run your tests. For example –

C:\Program Files\SmartBear\ReadyAPI-3.56.0\bin\testrunner.bat

The runner must be installed on the build agent, and the appropriate license must be activated. (See Required license.)
In the Arguments text box, specify the test runner command-line arguments. For detailed information on the command-line arguments, see the appropriate topics.

Make sure the path to your ReadyAPI project is valid and the runner on your build agent can access it. You can use build variables to specify the path if needed.

For example –

-f$(Common.TestResultsDirectory) "-RJUnit-Style HTML Report" $(Build.SourcesDirectory)\sample-project.xml

These arguments will command the test runner utility to run functional tests of the ReadyAPI project stored in the specified folder on the build agent and to export the test results to the TestResults folder.

Click the image to enlarge it.
If you have configured the test runner to export test results, you can configure your build definition to get the test results. If the test results are exported to JUnit-style reports, you can also publish them to your team build. See Get the test results.

Bash Script/Batch Script/PowerShell Script tasks

1. Prepare a script

1.1 Create a script that will run your tests

The script must call the desired ReadyAPI test runner with the needed parameters set for it. To learn more about command-line parameters, see the related topics.

If you store your script in a source control repository, when you run your build, the script will be placed in the specified folder on your build agent. Make sure the path to your ReadyAPI project specified in the script remains valid and the script can access the project file.

Below is a sample script:

Bash

Batch

PowerShell

Bash

#!/bin/bash
/data/app/ReadyAPI-3.56.0/bin/testrunner.sh -f$COMMON_TESTRESULTSDIRECTORY -RJUnit-Style HTML Report $BUILD_SOURCESDIRECTORY\sample-project.xml

Batch

CALL C:\Program Files\SmartBear\ReadyAPI-3.56.0\bin\testrunner.bat "-sTest Suite 1" "-cTestCase 1" -f%COMMON_TESTRESULTSDIRECTORY% "-RJUnit-Style HTML Report" %BUILD_SOURCESDIRECTORY%\sample-project.xml

PowerShell

$testrunner = 'C:\Program Files\SmartBear\ReadyAPI-3.56.0\bin\testrunner.bat'
$project = '$(Build.SourcesDirectory)\sample-project.xml'
$arguments = $project '"-sTest Suite 1"' '"-cTestCase 1"' '-f$(Common.TestResultsDirectory)' '"-RJUnit-Style HTML Report"'
Start-Process -FilePath $testrunner -ArgumentList $arguments -Wait

You can use build variables in your script. To learn more, see the Build Variables article in the MSDN Library.

1.2 Make your script is available for the agent

You can do it in one of the following ways:

(Recommended) Place your script to a source control repository (for example, the Team Foundation Version Control repository or Git repository). Then configure your build process to get it from the repository and place it in the needed folder on your build agent.

– or –

Copy the script to the needed folder on the build agent manually.

– or –

Place the script in a shared network folder. Make sure your build agent has access to that folder.

2. Configure a task

After you prepare your script, add the needed task to your pipeline and configure it:

Bash script (for Linux and macOS)

In the Script Path text box, specify the path to the bash script that will run your tests:

Click the image to enlarge it.

If your bash script commands the test runner to export test results, you can configure your build definition to fetch and publish the test results. See Get the test results.

Batch script (for Windows OS)

In the Path text box of the task, specify the path to the batch script file that will command your ReadyAPI test runner on your build and release agent to run the needed tests:

Click the image to enlarge it.

Make sure the specified script file can access the needed ReadyAPI project and the appropriate test runner is installed on your build agent.

If your script commands your test runner to export test results, you can configure your build definition to get the results. If the test results are exported to JUnit-style reports, you can publish them to your team project. See Get the test results.

PowerShell script (for Windows OS)

In the Type drop-down list, select whether you want to run an inline script or a script file.
If you have selected an inline script, enter the needed script in the Inline Script text box:

Click the image to enlarge it.

For example –

$testrunner = 'C:\Program Files\SmartBear\ReadyAPI-3.56.0\bin\testrunner.bat'
$project = '$(Build.SourcesDirectory)\sample-project.xml'
$arguments = $project '-f$(Common.TestResultsDirectory)' '"-RJUnit-Style HTML Report"'
Start-Process -FilePath $testrunner -ArgumentList $arguments -Wait

You can use build variables in the script. To learn more about them, see Azure documentation.
If you have selected a script file, specify the path to it in the Script Path text box:

Click the image to enlarge it.
If needed, specify additional arguments to pass to PowerShell.
If your PowerShell script commands the test runner utility to export test results, you can configure your build definition to get them. If the test results are exported to JUnit-style reports, you can also publish them to your team project. See Get the test results.