Running Tests in Parallel From Command Line

Applies to TestExecute 14.72, last modified on January 12, 2021

Cross-platform web tests you create with TestComplete can be run in parallel in your device cloud. To do this, you use a command-line test runner tool that comes with a Device Cloud Parallel license and is called TestExecute Lite. The tool allows several instances of it running on the same workstation at the same time. You use it for parallel test runs: run multiple instances of the tool with each instance executing a separate test.

To control the tool, you can use the following command-line syntax:

TestExecuteLite.exe project_file <log> <log option> <test> <variables> <external reporting> <timeout> <licensing> <statistics>

Note: To learn about controlling sequential test runs by using the TestExecute command line, see TestExecute - Command Line.

Calling from CI systems

Continuous integration (CI) is a software development approach that implies merging changes made by developers several times a day and making frequent automated builds and tests. If your company uses continuous integration, then, most likely, it uses a special continuous integration system like Jenkins, Bamboo, Azure DevOps, or other. That system builds your software regularly and runs automated tests after each build on one or more test computers.

Command-line arguments

project_file

Launches the test runner tool, opens the specified project (.mds file) or project suite (.pjs), and runs the tests that the project suite or project contains. Which tests will be run depends on what other command-line arguments are specified.

If tests to run are not specified explicitly, the product will run all tests that the project or project suite contains.

If a project includes an item that cannot be run in parallel mode (for example, a Network Suite item or Manual Testing collection), it will not open the project and report an error.


Log

Specify how you want to store the test log:

/ExportLog:file_name (or /el:file_name)

Commands to export test results to the file specified by the file_name parameter after the test run is over.

The following file formats are supported:

  • .mht
  • .tcLogX
  • .html
  • .htm

If the parameter value does not include the folder path, the file will be saved to the current project folder.

Important notes:

  • The test results will not be saved to the TestComplete project.

  • file_name should specify the name of a non-existent file. If it specifies an existing file, the test won't start.


/ForceLogIntoProject (or /flitp)

Commands to add the test results to the project or project suite to which the test belongs.
Log options

Use the following options to manage additional log data:

/ErrorLog:file_name

Commands to save information on the errors that occurred during the run to the text file the file_name parameter specifies. This file will include information on the errors that occurred in your tests and on the errors that did not allow opening the project or running the test (for example, an incorrect project path).

If the parameter value does not include the folder path, the test engine will save the file to the current working folder (by default, it is the project folder).

file_name should specify a non-existent file. If you specify the name of an existing file, the test engine will fail to run the test.


/ExportSummary:file_name (or /es:file_name)

Commands to generate a Summary report for the current test run and to save it in the JUnit report format to the XML file specified by the file_name parameter.

If there are test cases specified for the run, the test engine will export their results. If no test cases are specified, the test engine will export the results of the executed tests.

The report will include the total number of run tests, the number of passed tests, and the number of failed tests.

We recommend that the file_name parameter specify a fully-qualified file name. If the parameter value does not include the path to a folder, the product saves the file to the current working folder (by default, it is the project folder).

If the file_name parameter specifies an existing file, the product will overwrite it.


/NoSummary

Commands not to generate the Summary report and not to include it to the log exported by the /ExportLog argument.

This argument does not affect the /ForceLogIntoProject and /ExportSummary arguments.

Test

Use the following options to specify the test you want to run:

/project:project_name (or /p:project_name)

All enabled test items of the specified project will run. Project_name is the name of the project as it is shown in the Project Explorer panel in TestComplete. You can view the list of test items on the Test Items page of the project.

Only the enabled test items will be run.

Note: Do not confuse test items with project items. Test items are a sequence of tests to run that you specify on the Test Items page of your project. Project items are elements that belong to your project (for instance, a keyword test) and that you can view in Project Explorer.

For example, the following command runs all test items of the MyProj project:

TestExecuteLite.exe "C:\My Projects\MySuite.pjs" /p:MyProj

/project:project_name /projectitem:item_name

Commands to run a test or (tests) provided by the specified project item. You can view project items in the Project Explorer panel in TestComplete. Project_name is the name of the project as it is shown in the Project Explorer panel. Item_name is the name of a project item.

Note: Do not confuse project items with test items. Test items are what is displayed on the Test Items page of the project editor.

For example, the following command runs a test associated with the KeywordTests>Test1 project item:

TestExecuteLite.exe "C:\Work\My Projects\MySuite.pjs" /p:MyTest /pi:KeywordTests|Test1

/project:project_name /test:test_name

Commands to run the specified test (or tests). This can be any of the following:
  • A test item you specify on the View > Organize Tests page of the project editor

  • A keyword test

  • A script routine

  • A BDD scenario or feature file

Project_name specifies the name of the project that holds the needed test (use the same name as that shown in the Project Explorer panel).

Test_name specifies the test to run. It can be either the full name of the needed test or a tag:

Full name

Tag

For example, the following command runs the Test1 keyword test of the MyProj project:

TestExecuteLite.exe "C:\Work\My Projects\MySuite.pjs" /p:MyProj /t:"KeywordTests|Test1"

The following command runs tests that match the Tag1 tag of the MyProj project:

TestExecuteLite.exe "C:\Work\My Projects\MySuite.pjs" /p:MyProj /t:"@Tag1"

The following command runs the ProjectItem1 test item of the MyProj project:

TestExecuteLite.exe "C:\Work\My Projects\MySuite.pjs" /p:MyProj /t:"ProjectItem1"

/project:project_name /unit:unit_name /routine:routine_name

Commands to run the specified script routine. Project_name specifies the name of the project to which the routine belongs. Unit_name specifies the name of the unit holding the desired routine. Routine_name is the name of the script routine to be called. The routine to be called must not use parameters or return any values. Project_name and unit_name should be the same as the project and unit names shown in the Project Explorer.

For example, the following command runs the Main script routine of the Unit1 unit of the MyProj project:

TestExecuteLite.exe "C:\Work\My Projects\MySuite.pjs" /p:MyProj /u:Unit1 /rt:Main

/project:project_name /tags:tag_expression

Commands to run all the script functions, keyword tests, BDD scenarios and BDD features that the specified project contains and that match the specified tag or tag expression. Project_name is the name of the project that holds the needed tests (use the same name as shown in the Project Explorer panel). Tag_expression specifies the needed tag or tag expression.

Tag expression syntax

For example:

/tags:@AllTests

/tags:"@low-priority and @normal-priority"

/tags:"(@high-priority or (@Web-Store-Tests and @FF))"

Notes:

  • The tests that match the specified tag or tag expression will be run in an arbitrary order.
  • If both the /test and /tags arguments are specified in the command line, the /test argument will be ignored.
  • If the tags argument specifies an empty tag expression, the command will execute all the tagged tests and scenarios of the specified project.

For example, the following command runs all the tests that match the Tag1 tag of the MyProj project:

TestExecuteLite.exe "C:\Work\My Projects\MySuite.pjs" /p:MyProj /tag:"@Tag1"
Variables

Use the following options to set values of project and project suite variables:

/PSVar:VarName=VarValue (or /psv)   /PrjVar:VarName=VarValue (or /pv)

Variable values you pass via the command line are temporary. At the beginning of the test run, they overwrite the initial value of appropriate variables of your project or project suite. After the test run is over, the variable values are restored to their initial state.
If the variable you set via the command line does not exist in your project or project suite, it will be created at the beginning of the test run and removed after the test run is complete.
If you run several projects that have a project variable with the same name, the command will set the variable value in all executed projects.
External reporting

Use the following options to control the export of test results to external test management systems:

/DisableExternalReporting

Disables sending test results to external test management systems (for instance, Zephyr for Jira).

/ZephyrVersion:version

If test items of your TestComplete project are bound to Zephyr tests in your Jira project, this argument specifies the release version of your Jira project to which test results will be added.

/ZephyrCycle:cycle

If test items of your TestComplete project are bound to Zephyr tests in your Jira project, this argument specifies the test cycle in your Jira project to which test results will be added.

Note: If neither a release version nor a test cycle is specified, the product will send the test results to the release version and test cycle that your project settings specify. If the project settings do not specify a release version or a test cycle, the product will send the test results to the default release version (Unscheduled ) and to the default test cycle (Ad hoc).

Timeout

/Timeout:time_in_seconds

Timeout for the testing session in seconds.

This timeout includes both the test engine startup time and the test run time.

If a test is still running after the timeout elapses, the test engine will post an error to the test log and stop the test run. All further tests will be canceled.

Possible values are 30…4 294 967 seconds (49.7 days). If the specified value is out of range, the closest in-range value is used.

Licensing

/ManageLicense

Shows the License Management dialog in which you can activate a new license and move or deactivate the existing one. See Activating Device Cloud Parallel License.
Statistics

/UsageStatistics:File_name (or /US:File_name)

Commands to collect statistics on usage of Name Mapping items during test runs. It will save the gathered data to the specified file. You can then use this data to find and remove unused objects to clear the Name Mapping repository.

If the file name includes spaces, enclose it in quotes:

TestExecuteLite.exe /UsageStatistics:"c:\some shared folder\Name Mapping Data.stat"

Custom parameters

It is possible to send custom parameters to the executable via command line. You can access these parameters from your tests by using the BuiltIn.ParamStr and BuildIn.ParamCount methods. To learn how to handle custom command-line parameters, see the Passing Test Parameters via Command Line section in the TestComplete documentation.

Examples

The examples below show how to use the command line to run cross-platform web tests in parallel.

Each instance of the test running utility allows only one test to run at a time, therefore, to run several tests in parallel, you need several instances of the utility to run simultaneously. To launch several utility instances at once, the examples below use the loop.

When running tests in parallel from the command line, you must specify the location to which you want to store the test results: in an external folder or in the project to which the test belongs. Both the examples below store the results to an external folder. To do this, they use the /ExportLog command-line parameter.
Before storing the test results, the examples clear the target folder contents. To do this, the examples use the Remove-Item and New-Item (a PowerShell script) and rmdir and mkdir (a batch file).

Example 1: Run several tests in parallel

The following example shows how to run several tests in parallel. The name of the tests to run is defined in a list (the PowerShell example) or in a file (the batch file example). The sample script launches several instances of the test runner utility (TestExecute Lite), each of the instances runs a separate test.

To iterate through the list of tests, the example uses the loop. To launch each instance of the utility immediately, without waiting until each preceding instance stops, the examples use Start-Process (in PowerShell) and start (in a batch file).

Batch file

@echo off
set telite="C:\Program Files (x86)\SmartBear\TestExecuteLite 14\x64\Bin\TestExecuteLite.exe"
set project=C:\tests\TestProject\TestProject.mds
set projectname=TestProject
set log=C:\tests\logs\
rmdir /q /s %log%
mkdir %log%
for /f "delims=* tokens=*" %%x in (tests.txt) do (
  start "" %telite% %project% /p:%projectname% /t:%%x /exportlog:%log%%%x.mht
)

PowerShell

$testexecutelite = 'C:\Program Files (x86)\SmartBear\TestExecuteLite 14\x64\Bin\TestExecuteLite.exe'
$project = "C:\tests\TestProject\TestProject.mds"
$projectname = 'TestProject'
$loglocation = 'C:\tests\logs\'
$testnames = @("Test1", "Test2", "Test3", "Test4", "Test5")
Remove-Item -path $loglocation -recurse
New-Item -path $loglocation -ItemType directory
foreach ($testname in $testnames)
{
  $logname = $loglocation + $testname + ".mht"
  $arguments = $project, "/p:$projectname", "/t:$testname", "/exportlog:$logname"
  Start-Process -FilePath $testexecutelite -ArgumentList $arguments
}
Example 2: Run a test in several environments in parallel

The following example shows how to run a test in several environments in parallel. The environments where the tests will run is specified in a list (the PowerShell example) or in a file (the batch file example). To iterate through the list of environments, the example uses the loop. To launch each instance of the test runner utility immediately, without waiting until the preceding instance stops, the example uses Start-Process (in PowerShell) and start (in a batch file).

To specify the testing environment (web browser) in which the test will run, the example uses the config project variable. To send the needed variable value to the test runner utility, the example uses the -pv command-line argument.

View test code and description

Batch file

@echo off
set telite="C:\Program Files (x86)\SmartBear\TestExecuteLite 14\x64\Bin\TestExecuteLite.exe"
set project=C:\tests\TestProject\TestProject.mds
set projectname=TestProject
set log=C:\tests\logs\
rmdir /q /s %log%
mkdir %log%
for /f "delims=* tokens=*" %%x in (configs.txt) do (
  start "" %telite% %project% /p:%projectname% /exportlog:%log%%%x.mht /pv:config=%%x
)

PowerShell

$testexecutelite = 'C:\Program Files (x86)\SmartBear\TestExecuteLite 14\x64\Bin\TestExecuteLite.exe'
$project = "C:\tests\TestProject\TestProject.mds"
$projectname = 'TestProject'
$loglocation = 'C:\tests\logs\'
$configs = @("Safari", "Chrome", "Firefox", "Edge", "IE")
Remove-Item -path $loglocation -recurse
New-Item -path $loglocation -ItemType Directory
foreach ($config in $configs)
{
  $logname = $loglocation + $config + ".mht"
  $arguments = $project, "/p:$projectname", "/exportlog:$logname", "/pv:config=$config"
  Start-Process -FilePath $testexecutelite -ArgumentList $arguments
}

See Also

Running Cross-Platform Web Tests in Parallel

Highlight search results