If you use the TestComplete Support plugin to control your parallel test runs from Jenkins, after your Jenkins project run is over, the test results will be added to the build results automatically. You can view them in Jenkins:
TestComplete test results for a build
Open the build page in your browser.
Click TestComplete Test Results in the menu on the left:
The TestComplete Test Results page shows a summary of test runs, information on what nodes tests ran, their run time and the number of logged errors and warnings.
Depending on the test results, the following statuses can be displayed:
|Failed to start||The test did not start. Click the test name to find out what happened.|
|Failed||Errors occurred during the test execution, or the test did not finish.|
|Warning||Warning messages were logged, but no errors occurred and the test finished.|
|Passed||No errors or warnings occurred during the test execution.|
To view detailed test log
Click the test name. The TestComplete Test Result report for the run will be opened:
To get the detailed test log as a separate file
To download the test log in the .tcLogX format , click the Download Log File (tcLogX) button:
To view test results in this file format, you have to export them to a TestComplete project.
If generating MHT logs is enabled for your tests (see the TestComplete Test Step description), you can click the Download Log File (MHT) button to get them:
Note: You can open .mht files only in Internet Explorer.
JUnit-style test reports
If generating JUnit-style reports is enabled for your tests (see the TestComplete Test Step description), the TestComplete Jenkins plugin will create reports similar to those generated by JUnit tests. However, this format differs from JUnit reports described below.
To view the results, click Test Result on the Status page or in the menu on the left:
On the Test Result page, you can examine the results of a test run:
The All Tests section provides brief information on the executed TestComplete tests during the build process:
Package level results in this section represent project suite runs.
Class level results correspond to projects in a project suite.
Test level results correspond to individual test items or individual tests in a project.
Click items in the All Tests section to view details information about them.
If the test run failed, the page contains the All Failed Tests section. Click an item in this section to view information about the failure.
Adding Test Results to Test Result Trend Graph
The Status page of the build result shows a graph with test results for the builds. This graph shows the ratio of failed tests to the total number of tests. The graph counts only the builds for which JUnit-style reports were generated:
To include your cross-platform test results in the graph, configure your tests to generate JUnit-style reports.
Adding test results to test results history
The History page shows the statistics of test runs for your builds. To open the page, click History on the Test Results page of your build:
The graph at the top of the page shows the execution time of the tests and the ratio between the failed and passed tests. Click show count to view the number of executed tests rather than time.
The table below the graph contains information on each build: test duration, and the number of failed, skipped and total tests per build. The TestComplete Jenkins plugin does not skip tests, so the number of skipped tests is always 0.
The page only takes into account those builds for which JUnit-style reports were generated. To include your cross-platform test results in the test results history, configure your tests to generate JUnit-style reports.
You can configure your tests to export the Summary report in the JUnit format. It will be similar to the reports generated by JUnit tests. However, this format differs from the JUnit-style report described above.
Unlike Junit-style reports described above, the plugin will not publish this report automatically. You will have to publish the report manually (you publish ordinary JUnit reports in the same way). For example, you can use the JUnit plugin to publish a report.
To generate a report:
If you use a freestyle job, in your TestComplete Test build step, specify the
/ExportSummarycommand line argument in the Additional command-line arguments text box and specify the path to the folder where you want to save the generated test results:
If you use a pipeline, use the commandLineArguments parameter to run tests with the additional
/ExportSummaryargument:testcompletetest launchType: 'lcProject', project: 'MyProj', suite: 'Projects\\JenkinsTests.pjs',
commandLineArguments: '/ExportSummary:"C:\workspace\Test\reports\result.xml"', executorType: 'TELite'
When you run your build, the TestComplete Test step will generate a report and save it to the specified location.
Access test results via API
The API through which you can access test results is available at:
jenkins_server:port – The name and port of your Jenkins server. Instead of the name, you can also specify the server’s IP address.
job_name – The URL-encoded name of the job that ran your tests.
job_run – The identifier of the job run (decimal, integer).
format – A constant that indicates the format of the resulting data. Possible values are:
To get the results of your cross-platform web tests, send the GET request. For example:
Here is an example of XML data Jenkins returns. Results in the JSON and Python formats have similar elements:
If you use the XML format, the root node is
tcSummaryAction. The JSON and Python formats don’t use root nodes.