Viewing Cross-Platform Web Test Results in Jenkins

Applies to TestExecute 14.72, last modified on January 12, 2021

If you use the TestComplete Support plugin to control your parallel test runs from Jenkins, after your Jenkins project run is over, the test results will be added to the build results automatically. You can view them in Jenkins:

TestComplete test results for a build

  1. Open the build page in your browser.

  2. Click TestComplete Test Results in the menu on the left:

    Running cross-platform web tests from Jenkins: Cross-platform web test results in Jenkins

    Click the image to enlarge it.

The TestComplete Test Results page shows a summary of test runs, information on what nodes tests ran, their run time and the number of logged errors and warnings.

Depending on the test results, the following statuses can be displayed:

Status Description
Failed to start The test did not start. Click the test name to find out what happened.
Failed Errors occurred during the test execution, or the test did not finish.
Warning Warning messages were logged, but no errors occurred and the test finished.
Passed No errors or warnings occurred during the test execution.
To view detailed test log

Click the test name. The TestComplete Test Result report for the run will be opened:

Running cross-platform web tests from Jenkins: Viewing cross-platform web test results in Jenkins

Click the image to enlarge it.

To get the detailed test log as a separate file

To download the test log in the .tcLogX format , click the Download Log File (tcLogX) button:

Running cross-platform web tests from Jenkins: Download test results in the .tcLogX format

Click the image to enlarge it.

To view test results in this file format, you have to export them to a TestComplete project.

If generating MHT logs is enabled for your tests (see the TestComplete Test Step description), you can click the Download Log File (MHT) button to get them:

Running cross-platform web tests from Jenkins: Download test results in the .mht format

Click the image to enlarge it.

Note: You can open .mht files only in Internet Explorer.

JUnit-style test reports

If generating JUnit-style reports is enabled for your tests (see the TestComplete Test Step description), the TestComplete Jenkins plugin will create reports similar to those generated by JUnit tests. However, this format differs from JUnit reports described below.

To view the results, click Test Result on the Status page or in the menu on the left:

Running cross-platform web tests from Jenkins: Jenkins project page

Click the image to enlarge it.

On the Test Result page, you can examine the results of a test run:

Running cross-platform web tests from Jenkins: Jenkins test result page

Click the image to enlarge it.

The All Tests section provides brief information on the executed TestComplete tests during the build process:

  • Package level results in this section represent project suite runs.

  • Class level results correspond to projects in a project suite.

  • Test level results correspond to individual test items or individual tests in a project.

Click items in the All Tests section to view details information about them.

If the test run failed, the page contains the All Failed Tests section. Click an item in this section to view information about the failure.

Adding Test Results to Test Result Trend Graph

The Status page of the build result shows a graph with test results for the builds. This graph shows the ratio of failed tests to the total number of tests. The graph counts only the builds for which JUnit-style reports were generated:

Running cross-platform web tests from Jenkins: Adding cross-platform web test results to the Test Result Trend graph in Jenkins

Click the image to enlarge it.

To include your cross-platform test results in the graph, configure your tests to generate JUnit-style reports.

Adding test results to test results history

The History page shows the statistics of test runs for your builds. To open the page, click History on the Test Results page of your build:

Running cross-platform web tests from Jenkins: Jenkins test history page

Click the image to enlarge it.

The graph at the top of the page shows the execution time of the tests and the ratio between the failed and passed tests. Click show count to view the number of executed tests rather than time.

The table below the graph contains information on each build: test duration, and the number of failed, skipped and total tests per build. The TestComplete Jenkins plugin does not skip tests, so the number of skipped tests is always 0.

The page only takes into account those builds for which JUnit-style reports were generated. To include your cross-platform test results in the test results history, configure your tests to generate JUnit-style reports.

JUnit reports

You can configure your tests to export the Summary report in the JUnit format. It will be similar to the reports generated by JUnit tests. However, this format differs from the JUnit-style report described above.

Unlike Junit-style reports described above, the plugin will not publish this report automatically. You will have to publish the report manually (you publish ordinary JUnit reports in the same way). For example, you can use the JUnit plugin to publish a report.

To generate a report:

  • If you use a freestyle job, in your TestComplete Test build step, specify the /ExportSummary command line argument in the Additional command-line arguments text box and specify the path to the folder where you want to save the generated test results:

    Running cross-platform web tests from Jenkins: Exporting cross-platform web test results to JUnit format

    Click the image to enlarge it.

  • If you use a pipeline, use the commandLineArguments parameter to run tests with the additional /ExportSummary argument:

    testcompletetest launchType: 'lcProject', project: 'MyProj', suite: 'Projects\\JenkinsTests.pjs',
    commandLineArguments: '/ExportSummary:"C:\workspace\Test\reports\result.xml"', executorType: 'TELite'

When you run your build, the TestComplete Test step will generate a report and save it to the specified location.

Access test results via API

The API through which you can access test results is available at:

http://<jenkins_server>:<port>/job/<job_name>/<job_run>/TestComplete/api/<format>

Where:

  • jenkins_server:port – The name and port of your Jenkins server. Instead of the name, you can also specify the server’s IP address.

  • job_name – The URL-encoded name of the job that ran your tests.

  • job_run – The identifier of the job run (decimal, integer).

  • format –  A constant that indicates the format of the resulting data. Possible values are:

    • xml
    • json
    • python

To get the results of your cross-platform web tests, send the GET request. For example:

curl -X GET http://builder:8080/job/MyJob1/241/TestComplete/api/xml
Result structure

Here is an example of XML data Jenkins returns. Results in the JSON and Python formats have similar elements:

XML

<tcSummaryAction _class="com.smartbear.jenkins.plugins.testcomplete.TcSummaryAction">
  <report>
    <agent>Jenkins</agent>
    <details>
      <duration>4806</duration>
      <errors>0</errors>
      <timestamp>1535544341163</timestamp>
      <warnings>0</warnings>
    </details>
    <error/>
    <exitCode>0</exitCode>
    <failedToStart>false</failedToStart>
    <success>true</success>
    <testName>
      Orders_C#_JavaScript/Orders_C#_JavaScript/Orders_JavaScript/Main
    </testName>
    <url>
      http://localhost:8080/job/TestJob/6/TestComplete/reports/3536754
    </url>
  </report>
<tcSummaryAction>

If you use the XML format, the root node is tcSummaryAction. The JSON and Python formats don’t use root nodes.

View child nodes description

See Also

Running Cross-Platform Web Tests in Parallel by Using Jenkins

Highlight search results