Applies to ReadyAPI 2.7, last modified on June 25, 2019

About

Software developers can use an OpenAPI, Swagger, WADL, RAML or other specification to describe their REST web services. A specification defines paths (resources), methods, parameters and other items that make up a web service. Knowing this information makes it possible to check what parts of the service definition your tests cover.

If your RESTful services have an OpenAPI, Swagger or WADL specification, you can command ReadyAPI to collect coverage information when it is running functional tests. Coverage results will help you quickly understand what you have tested and what you have not, and estimate how complete your tests are.

Requirements

REST coverage testing is available if –

  • Your web service uses a WADL specification and passes data in the XML format.

  • Your web service uses an OpenAPI\Swagger specification and passes data in the XML or JSON format.

How It Works

  • To detect covered areas of the tested web service, ReadyAPI uses representations – data that describes the state of a resource (see the Web Service Architecture). Your definition can already have a representation. For example, here is a representation in a WADL specification:

    REST API coverage testing: Representations in WADL specification of a web service

    Click the image to enlarge it.

    – or you can create them in ReadyAPI (see below).

    You can find all the representations defined for a method of your web service on the Method-Name > Representations page of the REST service editor:

    REST API coverage testing: Method Representations page

    Click the image to enlarge it.

  • A representation has the following properties:

    Property Description
    Type The type of the message the representation matches:
    • REQUEST – Used for requests.

    • RESPONSE – Used for responses with the success (2xx) or redirection (3xx) response codes.

    • FAULT – Used for responses with the client error (4xx) or server error (5xx) response codes.

    Message Type The data type of the message. This is the value of the message’s Content-Type header.
    Status Codes Used for RESPONSE and FAULT representations only. Specifies one or multiple response codes of the message.
    QName The namespace and name of the root element of the response body data. For JSON data, ReadyAPI generates root elements automatically. They are either Fault, or Response. A namespace is typically the URL of the tested web service.
  • ReadyAPI collects coverage data during the test case, test suite or test project run.

    When ReadyAPI detects an incoming or outgoing message, it checks the content type of the message, its status code (if the message is a response) and the root element. If your project has a representation that matches the combination of these data, ReadyAPI considers the representation as covered. At the end of the run, ReadyAPI merges information on the representations covered by different test steps simulating the same API message and reports this info as coverage results for that message.

  • Besides tracking representations, ReadyAPI also collects information on the payload and parameters that were set during request simulation (see below). It reports used parameters and payload elements as covered.

Create Representations

If your web service specification has no representations defined, you can command ReadyAPI to create representations automatically every time you simulate a request. Follow these steps:

1. Enable the Auto-Create setting

  1. Switch to Projects.

  2. Find your request in the Navigator panel on the left and open the request for editing.

  3. In the Response part of the request editor, activate the Representations page.

  4. Select the Auto-Create check box.

REST API coverage testing: Enabling the Auto-Create setting in Projects

Click the image to enlarge it.

  1. Switch to SoapUI.

  2. Expand the test suite and test cases in the Navigator panel on the left to find the REST Request step of your service. Open the test step for editing.

  3. In the Response part of the request editor, activate the Representations page.

  4. Select the Auto-Create check box.

REST API coverage testing: Enabling the Auto-Create setting in Projects

Click the image to enlarge it.

When you enable or disable the Auto-Create setting of a request in Projects, it is automatically enabled (or disabled) for the REST Request test steps that simulate this request. Similarly, if you enable or disable the Auto-Create setting in the REST Request step editor, it is enabled (or disabled) in all other places.

2. Run requests

Run your request in any manner: run an individual request from Projects, run a test case or a test suite from the ReadyAPI UI or from the command line.

ReadyAPI will automatically create representations for those combinations of the content type, status code and root element that are not in the representation list yet.

It is not recommended to run coverage tests with the Auto-Create setting enabled (see below).

3. Restart ReadyAPI

After you add representations, restart ReadyAPI.

After you create representations, you can view them on the Method-Name > Representations page of your REST service editor (see above).

It is not recommended to run coverage tests with the Auto-Create setting enabled. That is, if you have coverage enabled, then  

  • Disable coverage.
  • Enable the Auto-Create setting and run requests to create representations.
  • Disable the Auto-Create setting.
  • Enable coverage and run your coverage tests.

Run Coverage Tests

ReadyAPI collects coverage data when executing a SoapUI test:

  1. By default, coverage testing is disabled. To enable it, open your test project, test suite or test case for editing, switch to the Coverage page and select the Enable Coverage check box:

    Note: ReadyAPI does not save the check box state between sessions. So, to run coverage tests after the product restarts, select the check box again.
  2. Run your project, test suite or test case. ReadyAPI will track the requests and responses and collect coverage information.

REST API coverage testing: Enable coverage testing and run tests

Click the image to enlarge it.

Coverage Results

You can see coverage results on the Coverage page of the test case, test suite or test project editor. Below is a sample view of test suite coverage results:

Coverage testing of web services: REST coverage testing results

Click the image to enlarge it.

The Coverage page displays the operations defined in your API specifications and the hierarchy of test suites and test cases you have in your project. You can expand the tree to see the coverage results of individual operations, requests and responses.

The page uses the green color for covered elements, and the pink color for uncovered items. SOAP coverage results also use dark green in certain cases, for REST services dark green is not used:

Coverage testing of REST web services: Color coding
Note: The green color indicates coverage. It does not mean the request passed successfully.
API Coverage

To see API coverage results, expand the API node in the tree. Method coverage results are grouped by representation types (see the image below). The tree also displays coverage results for method parameters. If a definition contains a payload schema for a representation, ReadyAPI shows how many elements of the payload were covered. The numbers are the number of covered elements against the total number of them. To see what elements were covered, select the representation and click Message Coverage:

Coverage testing of REST web services: Payload coverage

ReadyAPI counts each representation without payload data as 1.

Note that the results you see for each API message is the sum of coverage results of individual test steps that simulated this message.

The numbers of parent nodes are the sums of their respective child nodes:

Coverage testing of REST web services: Combined values

To check what exactly representation or parameter was covered, select the Parameters or Response > Status code node and open the Message Coverage page on the right:

Coverage testing of REST web services: Coverage details
Status code coverage
Coverage testing of REST web services: Coverage details
Parameter coverage
Request Coverage Results

To check coverage results of simulated REST requests, expand the node that corresponds to the test suite or test case you ran. The results you can see are similar to the results you see for API methods. The numbers indicate the representations and parameters covered by a request and the total number of defined representations and parameters. The numbers of parent nodes are the sums of their respective child values:

Coverage testing of REST web services: Coverage results of individual requests

To check what exactly representation or parameter was covered, select the Parameters, Response or Status code node and open the Message Coverage page on the right:

Coverage testing of REST web services: Coverage details
Coverage details

To view the contents of a message or parameter, select it in the tree and then switch to the Message Contents page on the right:

Coverage testing of RESTful web services: Request contents

Click the image to enlarge it.

The question marks indicate that ReadyAPI detected data for which your web service has no representation defined, for example, a response with an unexpected code, or a response with non-XML and non-JSON data:

Coverage testing of RESTful web services: Unknown data

Select the Response node and switch to the Message Contents page to view the response data. As you can see in the image below, in our example the response has no namespace defined, which caused ReadyAPI to fail to find a representation:

Coverage testing of RESTful web services: Problematic response contents

Click the image to enlarge it.

Notes

  • ReadyAPI clears coverage results before each test run. To clear the results during the run, click on the toolbar.

  • Double-click an item in the tree to jump to its editor (or viewer).

  • Regardless of the editor to which the Coverage page belongs, the page always displays all the APIs and all the test suites and test cases existing in your project. So, zero values and the pink color may not indicate bad coverage, but the fact that these or those test cases were not executed during the test run.

  • The Enable Coverage setting works for test runs. If you select the check box for a test suite, then during the test suite run, ReadyAPI will collect coverage information for all the test cases that belong to this suite. If you enable the setting for a project, ReadyAPI will collect coverage data for all the test suites of the project.

    However, if you enable coverage for a test suite and then run an individual test case of this suite while the Enable Coverage setting is off, then ReadyAPI will not collect coverage for that test case. Similarly, if you enable coverage for a test case, and then run the test suite to which this test case belongs, ReadyAPI will not collect coverage information for the test case unless the suite’s Enable Coverage setting is on.

See Also

SOAP Coverage Testing

Highlight search results