A closer look at what happens when a test case is executed may give you a better understanding of how to build test cases so they run as you want them to regarding individual test steps, the test case log, and triggered events.
A test case execution consists of three main parts: startup, running test steps, and finishing.
When starting a test case, the following actions are taken internally:
Context initialization for a test case execution. It is passed to all scripts, Property Expansions, and other objects that will appear during the test case execution allowing you to share resources and various data between them.
For example, it is used internally to store the
HttpStateobject in the context when the Maintain HTTP Session option has been selected in the Test Case Options dialog.
Execution of a Setup Script for a test case.
Setting the maximum execution time for a test case. If you have specified the maximum execution time in the Test Case Options dialog and when the time is up, the test case will be terminated and fail if it has not finished in the specified time.
TestRunListenerswhich are defined either externally or as Event Handlers at the project level.
Performing any WS-RM or AMF-related session or transaction initializations that were configured under the corresponding tabs in the Test Case Options dialog.
Actions 2 and 4 can terminate the test case if any external conditions are not met by calling
When running from the UI, an entry marking the start of the test case will be written to the execution log.
Running Test Steps
Since actual testing takes place inside test steps, running a test case is mostly a case of running the contained test steps.
Execution starts with the first enabled test step (or with the currently selected test step if started with the Run from here test step context menu).
For each test step, the following happens:
beforeTestStepevent is triggered to all applicable
The test step itself is run and its
TestStepResultis added to the
The test step itself executes other test steps (for example a Run Test Case test step or Script test step calling
testRunner.runTestStep), they are recursively executed as described here.
afterTestStepevent is triggered to all applicable
If running in the UI, the corresponding log is updated with
TestStepResultsaved in (2):
Depending on the outcome and internal logic of the test step, the following happens:
- If the test step fails (due to assertion failures, internal errors, and so on) the test case is aborted if the Abort on error option is selected in the Test Case options (default setting).
- If the test step does not fail and if the Discard Successful Results option is selected in the Test Case Options dialog, the data contained in
TestStepResultis discarded from the memory. Selecting the Discard Successful Results option can save a lot of memory when executing long-running tests.
If the current test step transfers execution to another test step via the
testRunner.gotoTestStepByXXXmethod, that test step is executed next instead of the one "next in line". The following test steps do this internally:
Conditional GoTo if a condition is met.
DataSource Loop if more data is available.
Once the execution passes the last test step, the test case is finished, and the following events are triggered:
All test steps are internally "finished", which allows them to close any resources, write results to files, and so on.
The TearDown script of the test case is called, for example, for cleaning up any resources or doing final validations.
TestRunListener.afterRunevent is triggered for applicable listeners.
The Status of the test case will be set to
FAIL if one of the following conditions is met:
The test case was aborted due to an internal error.
The test case was aborted because of a failed test step, or if there were failed test steps and the Fail Test Case on Error option in the Test Case Options dialog is selected.
A script in any of the triggered events, scripts, or handlers failed the test case with
You can find the final state of the test case in the Run Log.
Test Cases in Load Tests
When a test case is run as a load test, a complete copy of the test case is created for each executing thread. Each thread then runs the test case according to the test case options.
Any test step result that does not lead to a load test assertion failure is always discarded from the memory if the Discard Successful Results option is set.
Any test step result that leads to the failure of a load test assertion is never discarded from the memory.
The context available in the test case contains the
LoadRunContext property holding the context of the load test. This can be used in scripts to share data or resources between executing threads in your load tests.
LoadTestRunListener events are triggered for
TestRunListener counterparts, which allows you to create handlers/listeners that operate on a load test level for all running test cases (or Threads).