Run ReadyAPI Tests in TestEngine from Azure DevOps

Applies to ReadyAPI 3.10, last modified on October 07, 2021

This topic describes how to run ReadyAPI projects in TestEngine as part of your Azure Pipeline by using the Run ReadyAPI projects in TestEngine extension.

Supported Azure DevOps versions

You can use the Run ReadyAPI projects in TestEngine extension with the following versions of Azure DevOps:

  • Azure DevOps Services (formerly Visual Studio Team Services)

  • Azure DevOps Server 2019

1. Prepare ReadyAPI project

  1. Make your ReadyAPI project available to the agents.

    Typically, you store your project in a source control system (like Git or Azure Repos) and then configure your build to get the project from source control and place it to the needed folder.

    As an alternative, you can place your project in a shared network folder. Make sure your agent has access to that folder.

  2. In addition, if your project uses external files as data sources for your tests (see Data Sources and Data-Driven Testing), make sure your build agent has access to these files.

2. Install the extension

  1. Go to the Azure Marketplace.

    Tip: You can open the marketplace directly from Azure DevOps by selecting Browse Marketplace in the top right corner:
    Integration with Azure DevOps
  2. Search for the Run ReadyAPI projects in TestEngine extension.

  3. The way you install the extension depends on the Azure DevOps version you use:

    Azure DevOps Services

    Connected Azure DevOps Server

    Disconnected Azure DevOps Server

3. Configure pipeline

To run ReadyAPI tests from Azure DevOps, perform the following steps:

3.1. Create a pipeline

  1. Open your Azure DevOps project’s web portal and navigate to the Pipeline hub.

  2. Open an existing pipeline:

    Open an existing pipeline

    Click the image to enlarge it.

    – or –

    Create a new one:

    Create a new pipeline

    Click the image to enlarge it.

  3. If your ReadyAPI project is stored in a source control system, configure your pipeline to copy it to the folder to which an agent has access:

    Get files from source control

    Click the image to enlarge it.

3.2. Configure pipeline to run tests

To run ReadyAPI tests, add the Run ReadyAPI projects in TestEngine task and specify its parameters:

Add a Run ReadyAPI projects in TestEngine task

Click the image to enlarge it.

Option Description

Path to project file or directory

Required. The path to your ReadyAPI project.
You can also specify the path to a folder with several ReadyAPI projects. In this case, the task will send all of them to TestEngine.

Test Type

Specifies whether you run a functional or security test.

Currently, the Run ReadyAPI projects in TestEngine extension does not support load tests.

Test suite name

A test suite to run. To run all the test suites in the project, leave the field blank.

Note: Available only if you run a functional test.

Test case name

The test case to run. Leave the field blank to run all the test cases of the specified test suite, or, if you have not specified a test suite, all the test cases of your project.

Note: Available only if you run a functional test.

Security test name

Required. The security test to run.

Note: Available only if you run a security test.

Active environment

The environment configuration that the task will use for the test run. See Environments.

Callback URL

The address where TestEngine will send a JSON report after the test finishes.

Priority Job

Enable this option to skip all non-priority jobs in the queue (requires administrator rights).

Server URL

Required. The address of TestEngine. Make sure it is available for the Azure agent.

Username

Required. The name of a TestEngine user, on behalf of which Azure DevOps runs the project.

Password

Required. The user’s password.

Tip: To protect your password, use Azure variables.

Project file password

The encryption password. It is required if your project is encrypted.

Endpoint

The endpoint against which the test will run.

Tags

The tags of the test cases you want to run.

Publish test results

Enable this option to publish test results in Azure DevOps.

Save JUnit Report

Enable this option to save the JUnit report as a file to the $(System.DefaultWorkingDirectory) directory.

Filename

It is shown only when the Save JUnit Report option is enabled. Specifies the name of the report file.

Configure a Run ReadyAPI projects in TestEngine task

Click the image to enlarge it.

3.3. Save pipeline

To save the build, select Save & queue > Save:

VSTS integration: Save a build

Click the image to enlarge it.

4. Run build and view test result

To run the build:

  • Select it on the Pipelines page and then click Run pipeline.

– or –

  • Click Queue on the Edit Pipeline page.

Switch to the Job page to view the build progress:

Monitoring the build progress

Click the image to enlarge it.

The build will run the task that executes your ReadyAPI projects in TestEngine. It will also fetch and publish the test results if the Publish test results option is enabled.

After the build is over, you can view the test results. On the Summary page of the build report, view the Test Results section. It shows the number of failed and passed tests:

Test results in the Build Summary

Click the image to enlarge it.

Switch to the Tests page to view detailed test results:

Detailed test results

Click the image to enlarge it.

On the Test hub, you can view the test run statistics for your team project:

Test run statistics

Click the image to enlarge it.

Click a test run to view detailed information.

See Also

Azure DevOps (formerly VSTS) Integration
Command-Line Runners Specifics
ReadyAPI Documentation

Highlight search results