Software developers can use an OpenAPI, Swagger, WADL, RAML or other specification to describe their REST web services. A specification defines paths (resources), methods, parameters and other items that make up a web service. Knowing this information makes it possible to check what parts of the service definition your tests cover.
If your RESTful services have an OpenAPI, Swagger or WADL specification, you can command ReadyAPI to collect coverage information when it is running functional tests. Coverage results will help you quickly understand what you have tested and what you have not, and estimate how complete your tests are.
REST coverage testing is available if –
Your web service uses a WADL specification and passes data in the XML format.
Your web service uses an OpenAPI\Swagger specification and passes data in the XML or JSON format.
How It Works
To detect covered areas of the tested web service, ReadyAPI uses representations – data that describes the state of a resource (see the Web Service Architecture). Your definition can already have a representation. For example, here is a representation in a WADL specification:
– or you can create them in ReadyAPI (see below).
You can find all the representations defined for a method of your web service on the Method-Name > Representations page of the REST service editor:
Property Description Type The type of the message the representation matches:
REQUEST – Used for requests.
RESPONSE – Used for responses with the success (2xx) or redirection (3xx) response codes.
FAULT – Used for responses with the client error (4xx) or server error (5xx) response codes.
Message Type The data type of the message. This is the value of the message’s
Status Codes Used for RESPONSE and FAULT representations only. Specifies one or multiple response codes of the message. QName The namespace and name of the root element of the response body data. For JSON data, ReadyAPI generates root elements automatically. They are either Fault, or Response. A namespace is typically the URL of the tested web service.
ReadyAPI collects coverage data during the test case, test suite or test project run.
When ReadyAPI detects an incoming or outgoing message, it checks the content type of the message, its status code (if the message is a response) and the root element. If your project has a representation that matches the combination of these data, ReadyAPI considers the representation as covered. At the end of the run, ReadyAPI merges information on the representations covered by different test steps simulating the same API message and reports this info as coverage results for that message.
Besides tracking representations, ReadyAPI also collects information on the payload and parameters that were set during request simulation (see below). It reports used parameters and payload elements as covered.
If your web service specification has no representations defined, you can command ReadyAPI to create representations automatically every time you simulate a request. Follow these steps:
1. Enable the Auto-Create setting
Select an API request or a REST Request test step in the Navigator.
In the Response part of the editor, activate the Representations page.
Select the Auto-Create check box.
When you enable or disable the Auto-Create setting of an API request, it is automatically enabled (or disabled) for the REST Request test steps that simulate this request. Similarly, if you enable or disable the Auto-Create setting in the REST Request step editor, it is enabled (or disabled) in all other places.
2. Run requests
Run your request in any manner: run an individual request, run a test case or a test suite from the ReadyAPI UI or from the command line.
ReadyAPI will automatically create representations for those combinations of the content type, status code and root element that are not in the representation list yet.
It is not recommended to run coverage tests with the Auto-Create setting enabled (see below).
3. Restart ReadyAPI
After you add representations, restart ReadyAPI.
It is not recommended to run coverage tests with the Auto-Create setting enabled. That is, if you have coverage enabled, then
- Disable coverage.
- Enable the Auto-Create setting and run requests to create representations.
- Disable the Auto-Create setting.
- Enable coverage and run your coverage tests.
Run Coverage Tests
ReadyAPI collects coverage data when executing a SoapUI test:
By default, coverage testing is disabled. To enable it, open your test project, test suite or test case for editing, switch to the Coverage page and select the Enable Coverage check box:
Note: ReadyAPI does not save the check box state between sessions. So, to run coverage tests after the product restarts, select the check box again.
Run your project, test suite or test case. ReadyAPI will track the requests and responses and collect coverage information.
You can see coverage results on the Coverage page of the test case, test suite or test project editor. Below is a sample view of test suite coverage results:
The Coverage page displays the operations defined in your API specifications and the hierarchy of test suites and test cases you have in your project. You can expand the tree to see the coverage results of individual operations, requests and responses.
The page uses the green color for covered elements, and the pink color for uncovered items. SOAP coverage results also use dark green in certain cases, for REST services dark green is not used:
|Note:||The green color indicates coverage. It does not mean the request passed successfully.|
To see API coverage results, expand the API node in the tree. Method coverage results are grouped by representation types (see the image below). The tree also displays coverage results for method parameters. If a definition contains a payload schema for a representation, ReadyAPI shows how many elements of the payload were covered. The numbers are the number of covered elements against the total number of them. To see what elements were covered, select the representation and click Message Coverage:
ReadyAPI counts each representation without payload data as
Note that the results you see for each API message is the sum of coverage results of individual test steps that simulated this message.
The numbers of parent nodes are the sums of their respective child nodes:
To check what exactly representation or parameter was covered, select the Parameters or Response > Status code node and open the Message Coverage page on the right:
Status code coverage
Request Coverage Results
To check coverage results of simulated REST requests, expand the node that corresponds to the test suite or test case you ran. The results you can see are similar to the results you see for API methods. The numbers indicate the representations and parameters covered by a request and the total number of defined representations and parameters. The numbers of parent nodes are the sums of their respective child values:
To check what exactly representation or parameter was covered, select the Parameters, Response or Status code node and open the Message Coverage page on the right:
To view the contents of a message or parameter, select it in the tree and then switch to the Message Contents page on the right:
The question marks indicate that ReadyAPI detected data for which your web service has no representation defined, for example, a response with an unexpected code, or a response with non-XML and non-JSON data:
Select the Response node and switch to the Message Contents page to view the response data. As you can see in the image below, in our example the response has no namespace defined, which caused ReadyAPI to fail to find a representation:
ReadyAPI clears coverage results before each test run. To clear the results during the run, click on the toolbar.
Double-click an item in the tree to jump to its editor (or viewer).
Regardless of the editor to which the Coverage page belongs, the page always displays all the APIs and all the test suites and test cases existing in your project. So, zero values and the pink color may not indicate bad coverage, but the fact that these or those test cases were not executed during the test run.
The Enable Coverage setting works for test runs. If you select the check box for a test suite, then during the test suite run, ReadyAPI will collect coverage information for all the test cases that belong to this suite. If you enable the setting for a project, ReadyAPI will collect coverage data for all the test suites of the project.
However, if you enable coverage for a test suite and then run an individual test case of this suite while the Enable Coverage setting is off, then ReadyAPI will not collect coverage for that test case. Similarly, if you enable coverage for a test case, and then run the test suite to which this test case belongs, ReadyAPI will not collect coverage information for the test case unless the suite’s Enable Coverage setting is on.