Run ReadyAPI Tests in TestEngine from Azure DevOps

This topic describes how to run ReadyAPI projects in TestEngine as part of your Azure Pipeline by using the Run ReadyAPI projects in TestEngine extension.

Supported Azure DevOps versions

You can use the Run ReadyAPI projects in TestEngine extension with the following versions of Azure DevOps:

  • Azure DevOps Services (formerly Visual Studio Team Services)

  • Azure DevOps Server 2019

1. Prepare ReadyAPI project

  1. Make your ReadyAPI project available to the agents.

    Typically, you store your project in a source control system (like Git or Azure Repos) and then configure your build to get the project from source control and place it to the needed folder.

    As an alternative, you can place your project in a shared network folder. Make sure your agent has access to that folder.

  2. In addition, if your project uses external files as data sources for your tests (see Data Sources and Data-Driven Testing), make sure your build agent has access to these files.

2. Install the extension

  1. Go to the Azure Marketplace.

    Tip: You can open the marketplace directly from Azure DevOps by selecting Browse Marketplace in the top right corner:

    tfs-browse-marketplace.png

  2. Search for the Run ReadyAPI projects in TestEngine extension.

  3. The way you install the extension depends on the Azure DevOps version you use:

    • Azure DevOps Services

      1. Click Get it free.

      2. Select the organization for which you want to install the extension and click Install.

        To learn more, see Azure DevOps documentation.

    • Connected Azure DevOps Server

      1. Click Install.

      2. Select the team project collection to which you want to install the extension.

      3. Click Confirm.

        To learn more, see Azure DevOps documentation.

    • Disconnected Azure DevOps Server

      1. Click Download and save the extension file.

      2. Go to https://{your-azure-devops-server}/tfs/_gallery/manage.

      3. Click Upload new extension.

      4. Browse for the extension file you downloaded earlier. Click Upload.

        To learn more, see Azure DevOps documentation.

3. Configure pipeline

To run ReadyAPI tests from Azure DevOps, perform the following steps:

go.gif3.1. Create a pipeline

go.gif3.2. Configure pipeline to run tests

go.gif3.3. Save pipeline

3.1. Create a pipeline

  1. Open your Azure DevOps project’s web portal and navigate to the Pipeline hub.

  2. Open an existing pipeline:

    edit-existing-pipeline.png

    - or -

    Create a new one:

    create-new-pipeline.png

  3. If your ReadyAPI project is stored in a source control system, configure your pipeline to copy it to the folder to which an agent has access:

    get-sources-pipeline.png

3.2. Configure pipeline to run tests

To run ReadyAPI tests, add the Run ReadyAPI projects in TestEngine task and specify its parameters:

add-testengine-task.png

Option

Description

Path to project file or directory

Required. The path to your ReadyAPI project.

You can also specify the path to a folder with several ReadyAPI projects. In this case, the task will send all of them to TestEngine.

Test Type

Specifies whether you run a functional or security test.

Important

Currently, the Run ReadyAPI projects in the TestEngine extension do not support load tests.

Test suite name

A test suite to run. To run all the test suites in the project, leave the field blank.

Note

Available only if you run a functional test.

Test case name

The test case to run. Leave the field blank to run all the test cases of the specified test suite, or, if you have not specified a test suite, all the test cases of your project.

Note

Available only if you run a functional test.

Security test name

Required. The security test to run.

Note

Available only if you run a security test.

Active environment

The environment configuration that the task will use for the test run. See Environments.

Callback URL

The address where TestEngine will send a JSON report after the test finishes.

Priority Job

Enable this option to skip all non-priority jobs in the queue (requires administrator rights).

Server URL

Required. The address of TestEngine. Make sure it is available for the Azure agent.

Username

Required. The name of a TestEngine user, on behalf of which Azure DevOps runs the project.

Password

Required. The user’s password.

Tip

To protect your password, use Azure variables.

Project file password

The encryption password. It is required if your project is encrypted.

Endpoint

The endpoint against which the test will run.

Tags

The tags of the test cases you want to run.

Publish test results

Enable this option to publish test results in Azure DevOps.

Save JUnit Report

Enable this option to save the JUnit report as a file to the $(System.DefaultWorkingDirectory) directory.

Filename

It is shown only when the Save JUnit Report option is enabled. Specifies the name of the report file.

readyapi-project-in-testengine-task.png

3.3. Save Pipeline

To save the build, select Save & queue > Save:

save-build-pipeline.png

4. Run build and view test result

To run the build:

  • Select it on the Pipelines page and then click Run Pipeline.

    - or -

  • Click queue-b-gif.gif Queue on the Edit Pipeline page.

Switch to the Job page to view the build progress:

monitor-build-process-testengine-task.png

The build will run the task that executes your ReadyAPI projects in TestEngine. It will also fetch and publish the test results if the Publish test results option is enabled.

After the build is over, you can view the test results. On the Summary page of the build report, view the Test Results section. It shows the number of failed and passed tests:

build-summary-testengine.png

Switch to the Tests page to view detailed test results:

detailed-test-results-testengine.png

On the Test hub, you can view the test run statistics for your team project:

test-run-statistics-testengine.png

Click a test run to view detailed information.

See Also

In this section:
© 2025
Publication date: