TestNG Integration

Applies to Zephyr Scale Cloud, last modified on April 09, 2021.

Intro

TestNG is a framework similar to JUnit in style, but with additional functionality that improves the ease of use. To learn more, see here

The TestNG integration works through uploading to Zephyr Scale a JUnit XML results file generated by TestNG. Once the generated output results files are uploaded to Zephyr Scale, it is possible to generate reports and leverage all the other capabilities that Zephyr Scale offers.

Writing TestNG tests

TestNG tests are written in the form of Java classes and methods, for example:


public class ExempleTest {

    @Test(groups = { "group-1", "group-2" })
    public void DEV_T19_testMethod1() {
        assertThat(true).isTrue();
    }

    @Test()
    public void testMethod2_DEV_T21() {
        fail("failing test");
    }

    @Test(dependsOnMethods = {"testMethod2_DEV_T21"})
    public void testMethod3() {
        assertThat(true).isTrue();
    }
}

Note that the first method name begins with DEV_T19 and the second method name ends with DEV_T21. This particular prefix will map the result of each of those test methods to the corresponding test cases in Zephyr Scale if they exist, respectively test cases with key DEV-T19 and DEV-T21.

Also, note that the last method testMethod3 is not prefixed with a test case key. Zephyr Scale will try to find an existing test case that matches the package, class and method name, in this case com.example.testng.ExampleTest.testMethod3, and map the result of the test method to the test case. If these test cases don’t exist, an error will be returned when the results are uploaded to Zephyr Scale.

If any test case, prefixed with the test case key or not, is not found, an error will be returned when the results are uploaded to Zephyr Scale. In this case, if you intend to automatically create test cases that don’t exist previously, a flag autoCreateTestCases=true can be set when uploading test results to Zephyr Scale.

To summarize, the rules for matching a test method to a test case are:

  • Try to match by parsing a test case key from the test method name

  • If no test case is matched, then try to match a test case by using the method full qualified name

  • If no test case is matched, then:

    • If the parameter autoCreateTestCases is true, create the test case and create a new test execution for it

    • If the parameter autoCreateTestCases is false or not present, return an error - no test cases have been matched

Configuring TestNG to output JUnit XML results file

By default, TestNG should already generate JUnit XML results file. If the listener responsible for generating this file is disabled for some reason, it might be necessary to configure the listener org.testng.reporters.JUnitXMLReporter to make sure the JUnit XML will be generated. This configuration works either if you are using Maven or Gradle. Check our code sample in the next sections below for more information on how to configure this listener.

Uploading results to Zephyr Scale

Once the TestNG tests have executed, the results can be uploaded to Zephyr Scale. There are two options for uploading test results:

  • Using a single JUnit XML results file generated by TestNG

  • Using a zip file containing multiple JUnit XML results file generated by TestNG

Both options work the same way and can be uploaded using the same API endpoint: https://support.smartbear.com/zephyr-scale-cloud/api-docs/#operation/createJUnitExecutions.

Below, an example using curl of how to use the API endpoint for uploading one single file:


1curl -H "Authorization: BearerĀ ${TOKEN}" -F "file=@TestSuites.xml;type=application/xml" https://api.adaptavist.io/tm4j/v2/automations/executions/junit?projectKey="JQA"&autoCreateTestCases=true

…and for uploading a zip file containing multiple XML files:


1curl -H "Authorization: Bearer ${TOKEN}" -F "file=@TestSuites.zip;type=application/x-zip-compressed" https://api.adaptavist.io/tm4j/v2/automations/executions/junit?projectKey="JQA"&autoCreateTestCases=true

Note the query parameters on the URL. The projectKey specifies what project will be used to create the test cycle. The autoCreateTestCase will create a test case using the TestNG test method name when no matching test case is found.

Code samples

Please check our code example using TestNG for more technical information on how to write tests and configure TestNG to output JUnit XML results files:

SmartBear/zephyr-scale-testNG-example 

Suggested Improvements

Is something missing? If you’d like to suggest an improvement to this integration, take a look at our Ideas Portal

You can search for existing ideas or create your own. We review ideas regularly and will take them into account for future development. 

Highlight search results