Running Cross-Platform Web Tests in Parallel

Applies to TestComplete 14.74, last modified on April 22, 2021

About

In TestComplete, you can create cross-platform web tests and run them in a device cloud managed by Selenium Grid. This way, you can easily check how your tested web application functions in as many browsers and browser versions and platforms as you need.

To manage resources of your device cloud more effectively, it would make much sense to run several cross-platform web tests at the same time. This would reduce the overall test run time and will help you decrease the cost of your testing efforts.

To run tests in parallel, you use the Device Cloud Parallel license and the TestExecute Lite tool. The tool comes as part of TestComplete and is installed automatically when you install the product. In addition, you get the standalone version of the tool as part of the Device Cloud Parallel license and can install it on computers where TestComplete is not installed.

A single instance of the tool can run a single test at a time. An unlimited number of instances of the tool can be running on a computer at the same time. By using instances running in parallel, you can execute your cross-platform web tests simultaneously.

You can control TestExecute Lite instances from TestComplete in two ways:

Running cross-platform web tests in parallel from TestComplete

Click the image to enlarge it.

After all test runs are over, the test results will be available in your TestComplete project:

Results of cross-platform web tests run in parallel from TestComplete

Click the image to enlarge it.

Requirements

  • TestComplete 14.70.

  • An active Device Cloud Parallel license.

    To get the license, please contact our Sales Team.

  • TestExecute Lite installed on your test workstation.

    If you have TestComplete version 14.71, TestExecute Lite comes with TestComplete and is installed together with the product automatically. To orchestrate cross-platform web tests from computers where TestComplete is not installed, you can use the standalone version of TestExecute Lite. It comes with the Device Cloud Parallel license by default. You can get the installation file in the My SmartBear > My Products section of our web site.

  • To run tests in various browsers and on various devices, you must have Selenium Grid that manages them. It can be:

    • A private hub with Selenium Grid configured. It can be Selenium Grid running on your local computer or on a remote computer in your network. To learn more about installing and configuring Selenium Grid, see selenium.dev.

    • Selenium Grid provided as a service by CrossBrowserTesting.

      Note: The Device Cloud add-on provides free access to one parallel test run and an unlimited number of automation minutes you can use to run your tests in CrossBrowserTesting.com environments.

  • Your Selenium Grid must be accessible from the computer where TestComplete is installed.

    If you use the CrossBrowserTesting device cloud as your Selenium Grid provider, the computer where TestComplete is installed must have access to CrossBrowserTesting.com. If there are any firewalls or proxies in your network, make sure that they allow traffic to that website. If needed, configure the firewall to allow outbound connections on ports 80 and 443 to crossbrowsertesting.com.

  • All cross-platform web tests you want to run must belong to the same TestComplete project from which you want to control the test runs. Running tests that belong to external projects is not supported.

    Note: To do this, you can control the test runs from TestExecute Lite directly. See Running Cross-Platform Web Tests in Parallel in the TestExecute Lite documentation on our website.

Supported features

Only cross-platform web tests can run in parallel. These tests can include the following:

  • Operations that simulate user actions over web elements.

  • Operations for connecting to the SmartBear device cloud and launching web browsers there.

  • Checkpoints.

  • Operations that get test input data from external files (data-driven tests).

  • aqNNN scripting objects, for example aqString or aqConvert.

  • Event handlers.

  • BDD features and scenarios.

Unsupported features

  • The TestExecute Lite utility does not support running projects that have parallel test runs defined in their execution plan or individual tests that use the Parallel object. You can only orchestrate parallel test runs from TestComplete or from a third-party CI/CD system.

  • Running cross-platform web tests in parallel is not supported on 32-bit operating systems.

  • Desktop tests. Tests that simulate user actions over Windows desktop applications cannot run in parallel. For example, the following is not supported:

    • Operations that simulate user actions over desktop applications.

    • Low-level procedures.

    • Operations that manage the Tested Applications collection.

  • Non-cross-platform web tests. Tests that simulate user actions over local web browsers accessed via ways other than by using Selenium Grid cannot run in parallel. For example, the following is not supported:

    • Accessing web browsers by using the Sys.Browser object.

    • The Run Browser and Browser Loop operation.

  • Mobile tests. Tests that simulate user actions over mobile devices connected to your local workstation cannot run in parallel. For example, the following is not supported:

    • Android Gesture collection.

    • Operations that access mobile devices connected to your workstation via the Mobile object.

  • Image-based and OCR-based tests.

  • Testing web services.

  • User-forms, Network Suites, Unit tests, manual tests, object-driven tests (ODT), ReadyAPI tests.

If your test uses an object or an operation that the Device Cloud Parallel license does not support, an error will occur. Depending on the way your project handle errors, the test run may be terminated.

Orchestration via the Execution Plan editor

The Execution Plan editor allows you to configure a list of cloud remote environments where the cross-platform web test will be run in parallel:

Running cross-platform web tests in parallel via the Execution Plan editor

Click the image to enlarge it.

Follow the steps below to manage your cross-platform web tests:

  1. Open the Execution Plan editor of your TestComplete project and add the test item that will run your cross-platform web test. To learn how to create test items, see the description of the project’s Execution Plan editor.

  2. In the Tested URL edit box of the Parallel Device Cloud Environments panel, specify the URL of the tested web page that the specified web browser on the remote platform will open upon launching.

    We recommend that you specify the complete URL of your target web page, including the protocol name.

    If your tested web application is behind a firewall in your local network or on your local computer, or if you want to route traffic to your tested application through a proxy server, you can configure a secure tunnel.

    If the tunnel has not been configured yet, click Select another location and then, in the resulting Choose Your Web Application Location dialog, specify the application location.

    View instructions

    The configured tunnel will be added to the project settings.

  3. Specify environment parameters (Platform, Operating System, Browser, and Resolution) and click Add:

    Running cross-platform web tests in parallel via the Execution Plan editor

    Click the image to enlarge it.

When you launch your project or a particular test item, TestComplete will run your cross-platform web test in parallel on added environments.

To cut, copy, or paste environments from one test item to another, use the Cut, Copy, Paste items of the panel’s context menu. Alternatively, you can save the list of environments to an external .xml file and then load it by using the Save to file and Load from file buttons respectively.

Additionally, you can execute your cross-platform tests in parallel on the same workstation (not in the cloud environments). In this case, TestExecute Lite will be used as a test runner as well. To do this, use the button on the toolbar of the Execution Plan editor. See Grouping test items executed in parallel.

Orchestration via the Parallel script object

Methods of the Parallel object allow launching several TestExecute Lite instances and commanding them to run tests in remote environments in a device cloud:

  • Parallel.RunEnvironments - Launches several TestExecute Lite instances that connect to a device cloud you specified and run the specified tests in the cloud in parallel.

  • Parallel.RunTests - Launches several TestExecute Lite instances that run the specified tests.

    Note: This method does not command TestExecute Lite to connect to the device cloud and does not specify target environments for the test to run. You have to do it manually in each test the method runs.

The examples below show how you can use the methods to run tests.

Example 1: Running several tests in specified environments in parallel

The sample code below runs the MyTest1 and MyTest2 cross-platform web tests in the Safari and Chrome web browsers that it launches in the SmartBear device cloud.

JavaScript, JScript
Python
VBScript
DelphiScript
C++Script, C#Script

JavaScript, JScript

function Main()
{
  // Set a list of tests to run in parallel
  var tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2'];
  // Set a list of test environments
  var environments = ['{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}',
          '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}'];

  // Set the URL of the device cloud server
  var serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub";

  var testedURL = "https://services.smartbear.com/samples/TestComplete14/smartstore/";

  // Run tests in parallel in specified environments
  Parallel.RunEnvironments(tests, environments, testedURL, serverURL);

}

function MyTest1()
{
  var browser = Sys.Browser();
  var page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

function MyTest2()
{
  var browser = Sys.Browser();
  var page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

Python

def Main():
  # Set a list of tests to run in parallel
  tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2']
  # Set a list of test environments
  environments = ['{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}', '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}']

  # Set the URL of the device cloud server
  serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub"

  testedURL = "https://services.smartbear.com/samples/TestComplete14/smartstore/"

  # Run tests in parallel in specified environments
  Parallel.RunEnvironments(tests, environments, testedURL, serverURL)

def MyTest1():
  browser = Sys.Browser()
  page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …

def MyTest2():
  browser = Sys.Browser()
  page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …

VBScript

Sub Main
  ' Set a list of tests to run in parallel
  Dim tests : tests = Array("Script|Unit1|MyTest1", "Script|Unit1|MyTest2")
  ' Set a list of test environments
  Dim environments : environments = Array(_
      "{""platform"": ""Mac OSX 10.14"", ""browserName"": ""Safari"", ""version"": ""12"", ""screenResolution"": ""1366x768""}",_
      "{""deviceName"": ""Pixel 4"", ""platformName"": ""Android"", ""platformVersion"": ""10.0"", ""browserName"": ""Chrome"", ""deviceOrientation"": ""portrait""}")

  ' Set the URL of the device cloud server
  serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub"

  testedURL = "https://services.smartbear.com/samples/TestComplete14/smartstore/"

  ' Run tests in parallel in specified environments
  Call Parallel.RunEnvironments(tests, environments, testedURL, serverURL)

End Sub

Sub MyTest1()
  Set browser = Sys.Browser()
  Set page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …
End Sub

Sub MyTest2()
  Set browser = Sys.Browser()
  Set page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …
End Sub

DelphiScript

procedure MyTest1();
var browser, page;
begin
  browser := Sys.Browser();
  page := browser.Page('*services.smartbear.com/samples/TestComplete*/smartstore/');
  …
end;
procedure MyTest2();
var browser, page;
begin
  browser := Sys.Browser();
  page := browser.Page('*services.smartbear.com/samples/TestComplete*/smartstore/');
  …
end;

procedure Main();
var serverURL, testedURL;
var tests : Array[0..1];
var environments : Array[0..1];
begin
  // Set a list of tests to run in parallel
  tests[0] : = 'Script|Unit1|MyTest1';
  tests[1] : = 'Script|Unit1|MyTest2';
  // Set a list of test environments
  environments[0] := '{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}';
  environments[1] := '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}';

  // Set the URL of the device cloud server
  serverURL := 'http://hub.crossbrowsertesting.com:80/wd/hub';

  testedURL := 'https://services.smartbear.com/samples/TestComplete14/smartstore/';

  // Run tests in parallel in specified environments
  Parallel.RunEnvironments(tests, environments, testedURL, serverURL);
end;

C++Script, C#Script

function Main()
{
  // Set a list of tests to run in parallel
  var tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2'];
  // Set a list of test environments
  var environments = ['{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}',
          '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}'];

  // Set the URL of the device cloud server
  var serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub";

  var testedURL = "https://services.smartbear.com/samples/TestComplete14/smartstore/";

  // Run tests in parallel in specified environments
  Parallel["RunEnvironments"](tests, environments, testedURL, serverURL);

}

function MyTest1()
{
  var browser = Sys["Browser"]();
  var page = browser["Page"]("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

function MyTest2()
{
  var browser = Sys["Browser"]();
  var page = browser["Page"]("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

Example 2: Running several tests in parallel

The sample code below runs MyTest1 and MyTest2 cross-platform web tests in parallel. Each test connects to a device cloud, launches a web browser there, and simulates user actions in that web browser.

JavaScript, JScript
Python
VBScript
DelphiScript
C++Script, C#Script

JavaScript, JScript

function Main()
{
  // Set a list of tests to run in parallel
  var tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2'];
  // Set a list of test environments
  var environments = ['{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}',
          '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}'];

  // Set the URL of the device cloud server
  var serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub";

  var testedURL = "https://services.smartbear.com/samples/TestComplete14/smartstore/";

  // Run tests in parallel in specified environments
  Parallel.RunEnvironments(tests, environments, testedURL, serverURL);

}

function MyTest1()
{
  var browser = Sys.Browser();
  var page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

function MyTest2()
{
  var browser = Sys.Browser();
  var page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

Python

def Main():
  # Set a list of tests to run in parallel
  tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2']
  # Set a list of test environments
  environments = ['{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}', '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}']

  # Set the URL of the device cloud server
  serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub"

  testedURL = "https://services.smartbear.com/samples/TestComplete14/smartstore/"

  # Run tests in parallel in specified environments
  Parallel.RunEnvironments(tests, environments, testedURL, serverURL)

def MyTest1():
  browser = Sys.Browser()
  page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …

def MyTest2():
  browser = Sys.Browser()
  page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …

VBScript

Sub Main
  ' Set a list of tests to run in parallel
  Dim tests : tests = Array("Script|Unit1|MyTest1", "Script|Unit1|MyTest2")
  ' Set a list of test environments
  Dim environments : environments = Array(_
      "{""platform"": ""Mac OSX 10.14"", ""browserName"": ""Safari"", ""version"": ""12"", ""screenResolution"": ""1366x768""}",_
      "{""deviceName"": ""Pixel 4"", ""platformName"": ""Android"", ""platformVersion"": ""10.0"", ""browserName"": ""Chrome"", ""deviceOrientation"": ""portrait""}")

  ' Set the URL of the device cloud server
  serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub"

  testedURL = "https://services.smartbear.com/samples/TestComplete14/smartstore/"

  ' Run tests in parallel in specified environments
  Call Parallel.RunEnvironments(tests, environments, testedURL, serverURL)

End Sub

Sub MyTest1()
  Set browser = Sys.Browser()
  Set page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …
End Sub

Sub MyTest2()
  Set browser = Sys.Browser()
  Set page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …
End Sub

DelphiScript

procedure MyTest1();
var browser, page;
begin
  browser := Sys.Browser();
  page := browser.Page('*services.smartbear.com/samples/TestComplete*/smartstore/');
  …
end;
procedure MyTest2();
var browser, page;
begin
  browser := Sys.Browser();
  page := browser.Page('*services.smartbear.com/samples/TestComplete*/smartstore/');
  …
end;

procedure Main();
var serverURL, testedURL;
var tests : Array[0..1];
var environments : Array[0..1];
begin
  // Set a list of tests to run in parallel
  tests[0] : = 'Script|Unit1|MyTest1';
  tests[1] : = 'Script|Unit1|MyTest2';
  // Set a list of test environments
  environments[0] := '{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}';
  environments[1] := '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}';

  // Set the URL of the device cloud server
  serverURL := 'http://hub.crossbrowsertesting.com:80/wd/hub';

  testedURL := 'https://services.smartbear.com/samples/TestComplete14/smartstore/';

  // Run tests in parallel in specified environments
  Parallel.RunEnvironments(tests, environments, testedURL, serverURL);
end;

C++Script, C#Script

function Main()
{
  // Set a list of tests to run in parallel
  var tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2'];
  // Set a list of test environments
  var environments = ['{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}',
          '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}'];

  // Set the URL of the device cloud server
  var serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub";

  var testedURL = "https://services.smartbear.com/samples/TestComplete14/smartstore/";

  // Run tests in parallel in specified environments
  Parallel["RunEnvironments"](tests, environments, testedURL, serverURL);

}

function MyTest1()
{
  var browser = Sys["Browser"]();
  var page = browser["Page"]("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

function MyTest2()
{
  var browser = Sys["Browser"]();
  var page = browser["Page"]("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

Number of parallel tests

By default, you can run any number of cross-platform web tests in parallel. The number is only limited by:

  • Your Device Cloud Parallel license. The number of concurrent instances cannot exceed the overall number of instances that your license allows.

  • Your workstation computing capacity: its processor, RAM, and so on. The more tests are running in parallel, the more workstation resources they are consuming.

  • You can limit the number of maximum tests running in parallel manually by using the LicenseCount parameter of the Parallel.RunTests and Parallel.RunEnvironments methods.

See Also

About Cross-Platform Web Tests

Highlight search results