SOAP Coverage Testing

Each SOAP web service has a WSDL specification that describes the methods, responses, faults and other items that make up a web service. SOAP uses XML for storing request and response data. A WSDL specification defines the elements and attributes that will be used in the request and response bodies. That is, it gives a well-defined description for everything included in your web service. This makes it possible to check if your tests cover all the parts of your service definition or not.

ReadyAPI collects coverage data for test cases, test suites and test projects (see below). It can also collect coverage data for SOAP virtual services that you create in ReadyAPI. This helps you estimate what virtual operations are covered by your tests and what are not. For more information on this, see Coverage Testing of Virtual Services. This topic contains information on coverage features of functional tests.

1. ReadyAPI collects coverage data during the test case, test suite or test project run. By default, coverage testing is disabled. To enable it, select the Enable Coverage check box on the Coverage page of your test case, suite or project editor:

   

   Click the image to enlarge it.

   Note:

   ReadyAPI does not save the check box state between sessions. So, to run coverage tests after the product restarts, select the check box again.

2. After you select the check box, run your test case, suite or project.

   ReadyAPI will check the contents of requests and responses and detect what elements and attributes they use. Request and response bodies can have a complex data structure with multiple elements on each level. ReadyAPI will check the elements on all levels.

   If your SOAP request has an XPath Match assertion, ReadyAPI will track the assertion result and will report what element in the response body the assertions checked (see below). That is, XPath Match assertions help you ensure that your tests “cover” specific element values.

You view coverage results on the Coverage page of the test case, test suite or test project editor. Below is a sample view of coverage results at the project level:

Click the image to enlarge it.

The page displays the operations defined in your WSDL specification, and the tree of test suites, test cases and SOAP Request test steps you have in your project. You can expand the tree to see the elements that an individual test step covers and to check what assertions passed successfully or failed.

The page uses the green color for covered elements and pink for uncovered. Dark green indicates that the element was covered and was checked with an XPath Match assertion:

The numbers indicate the number of checked XML elements (SOAP passes data in the XML format). For example, in the image below, 4/5 means the method body can have 5 elements in total (including the parent element), and the requests covered only 4 of them. If a request covers at least one child of some parent element, ReadyAPI will treat that parent element as covered:

The Coverage page also displays the percent of covered elements in requests and responses. The percent value in parentheses is the percent of response elements that were asserted successfully:

The result numbers of parent items are sums of their respective child values. For example, test suite and test case results combine the results of their respective child items. Coverage results that you see for operations under the API nodes combine the results of SOAP Request test steps that simulated these operations:

To see what exactly elements were covered by some request or response, select this request or response in the tree and switch to the Message Coverage page on the right. The page uses color coding to indicate covered (green) and non-covered (pink) elements:

Click the image to enlarge it.

As you can see, if a test covered some child element, the parent element is also considered covered.

To view the contents of a simulated request or response, select it in the tree and then switch to the Message Contents page on the right:

Click the image to enlarge it.

• ReadyAPI clears coverage results before each test run. To clear the results during the run, click on the toolbar.

• To track coverage for certain response values, create XPath Match assertions for simulated SOAP requests before running your test.

• Regardless of the editor to which the Coverage page belongs, it always displays all the APIs and all the test suites and test cases existing in your project. So, the pink color and zero results for some test suites and test cases may indicate that you did not execute these test suites or cases during the last test run.

• Double-click an item in the result tree to jump to its editor (or viewer).

• Use the Coverage Options dialog to specify how ReadyAPI should treat empty values and what elements it should exclude from testing.

• The Enable Coverage setting works for test runs. If you select the check box for a test suite, then during the test suite run, ReadyAPI will collect coverage information for all the test cases that belong to this suite. If you enable the setting for a project, ReadyAPI will collect coverage data for all the test suites of the project. However, if you enable coverage for a test suite and then run some test case of this suite, ReadyAPI will not collect coverage data for that test case (unless its Enable Coverage setting is enabled, of course).