REST Coverage Testing

Software developers can use an OpenAPI, Swagger, WADL, RAML or other specification to describe their REST web services. A specification defines paths (resources), methods, parameters and other items that make up a web service. Knowing this information makes it possible to check what parts of the service definition your tests cover.

If your RESTful services have an OpenAPI, Swagger or WADL specification, you can command ReadyAPI to collect coverage information when it is running functional tests. Coverage results will help you quickly understand what you have tested and what you have not, and estimate how complete your tests are.

REST coverage testing is available if –

• Your web service uses a WADL specification and passes data in the XML format.

• Your web service uses an OpenAPI\Swagger specification and passes data in the XML or JSON format.

• To detect covered areas of the tested web service, ReadyAPI uses representations – data that describes the state of a resource (see the Web Service Architecture). Your definition can already have a representation. For example, here is a representation in a WADL specification:

  

  Click the image to enlarge it.

  – or you can create them in ReadyAPI (see below).

  You can find all the representations defined for a method of your web service on the Method-Name > Representations page of the REST service editor:

  

  Click the image to enlarge it.

• A representation has the following properties:

  Property	Description
  Type	The type of the message the representation matches: REQUEST – Used for requests. RESPONSE – Used for responses with the success (2xx) or redirection (3xx) response codes. FAULT – Used for responses with the client error (4xx) or server error (5xx) response codes.
  Message Type	The data type of the message. This is the value of the message’s Content-Type header.
  Status Codes	Used for RESPONSE and FAULT representations only. Specifies one or multiple response codes of the message.
  QName	The namespace and name of the root element of the response body data. For JSON data, ReadyAPI generates root elements automatically. They are either Fault, or Response. A namespace is typically the URL of the tested web service.

• ReadyAPI collects coverage data during the test case, test suite or test project run.

  When ReadyAPI detects an incoming or outgoing message, it checks the content type of the message, its status code (if the message is a response) and the root element. If your project has a representation that matches the combination of these data, ReadyAPI considers the representation as covered. At the end of the run, ReadyAPI merges information on the representations covered by different test steps simulating the same API message and reports this info as coverage results for that message.

• Besides tracking representations, ReadyAPI also collects information on the payload and parameters that were set during request simulation (see below). It reports used parameters and payload elements as covered.

If your web service specification has no representations defined, you can command ReadyAPI to create representations automatically every time you simulate a request. Follow these steps:

1. Enable the Auto-Create setting

1. Select an API request or a REST Request test step in the Navigator.

2. In the Response part of the editor, activate the Representations page.

3. Select the Auto-Create check box.

Click the image to enlarge it.

When you enable or disable the Auto-Create setting of an API request, it is automatically enabled (or disabled) for the REST Request test steps that simulate this request. Similarly, if you enable or disable the Auto-Create setting in the REST Request step editor, it is enabled (or disabled) in all other places.

2. Run requests

3. Restart ReadyAPI

After you create representations, you can view them on the Method-Name > Representations page of your REST service editor (see above).

It is not recommended to run coverage tests with the Auto-Create setting enabled. That is, if you have coverage enabled, then  

• Disable coverage.
• Enable the Auto-Create setting and run requests to create representations.
• Disable the Auto-Create setting.
• Enable coverage and run your coverage tests.

ReadyAPI collects coverage data when executing a SoapUI test:

1. By default, coverage testing is disabled. To enable it, open your test project, test suite or test case for editing, switch to the Coverage page and select the Enable Coverage check box:

   Note:

   ReadyAPI does not save the check box state between sessions. So, to run coverage tests after the product restarts, select the check box again.

2. Run your project, test suite or test case. ReadyAPI will track the requests and responses and collect coverage information.

Click the image to enlarge it.

You can see coverage results on the Coverage page of the test case, test suite or test project editor. Below is a sample view of test suite coverage results:

Click the image to enlarge it.

The Coverage page displays the operations defined in your API specifications and the hierarchy of test suites and test cases you have in your project. You can expand the tree to see the coverage results of individual operations, requests and responses.

The page uses the green color for covered elements, and the pink color for uncovered items. SOAP coverage results also use dark green in certain cases, for REST services dark green is not used:

API Coverage

To see API coverage results, expand the API node in the tree. Method coverage results are grouped by representation types (see the image below). The tree also displays coverage results for method parameters. If a definition contains a payload schema for a representation, ReadyAPI shows how many elements of the payload were covered. The numbers are the number of covered elements against the total number of them. To see what elements were covered, select the representation and click Message Coverage:

ReadyAPI counts each representation without payload data as 1.

Note that the results you see for each API message is the sum of coverage results of individual test steps that simulated this message.

The numbers of parent nodes are the sums of their respective child nodes:

To check what exactly representation or parameter was covered, select the Parameters or Response > Status code node and open the Message Coverage page on the right:

Status code coverage

Parameter coverage

Request Coverage Results

To check coverage results of simulated REST requests, expand the node that corresponds to the test suite or test case you ran. The results you can see are similar to the results you see for API methods. The numbers indicate the representations and parameters covered by a request and the total number of defined representations and parameters. The numbers of parent nodes are the sums of their respective child values:

To check what exactly representation or parameter was covered, select the Parameters, Response or Status code node and open the Message Coverage page on the right:

Coverage details

To view the contents of a message or parameter, select it in the tree and then switch to the Message Contents page on the right:

Click the image to enlarge it.

The question marks indicate that ReadyAPI detected data for which your web service has no representation defined, for example, a response with an unexpected code, or a response with non-XML and non-JSON data:

Select the Response node and switch to the Message Contents page to view the response data. As you can see in the image below, in our example the response has no namespace defined, which caused ReadyAPI to fail to find a representation:

Click the image to enlarge it.

• ReadyAPI clears coverage results before each test run. To clear the results during the run, click on the toolbar.

• Double-click an item in the tree to jump to its editor (or viewer).

• Regardless of the editor to which the Coverage page belongs, the page always displays all the APIs and all the test suites and test cases existing in your project. So, zero values and the pink color may not indicate bad coverage, but the fact that these or those test cases were not executed during the test run.

• The Enable Coverage setting works for test runs. If you select the check box for a test suite, then during the test suite run, ReadyAPI will collect coverage information for all the test cases that belong to this suite. If you enable the setting for a project, ReadyAPI will collect coverage data for all the test suites of the project.

  However, if you enable coverage for a test suite and then run an individual test case of this suite while the Enable Coverage setting is off, then ReadyAPI will not collect coverage for that test case. Similarly, if you enable coverage for a test case, and then run the test suite to which this test case belongs, ReadyAPI will not collect coverage information for the test case unless the suite’s Enable Coverage setting is on.