Running Cross-Platform Web Tests in Parallel

Applies to TestComplete 14.80, last modified on May 04, 2021

About

In TestComplete, you can create cross-platform web tests and run them in a device cloud managed by Selenium Grid. This way, you can easily check how your tested web application functions in as many browsers and browser versions and platforms as you need.

To manage resources of your device cloud more effectively, it would make much sense to run several cross-platform web tests at the same time. This would reduce the overall test run time and will help you decrease the cost of your testing efforts.

You can control parallel runs from TestComplete in two ways:

Running cross-platform web tests in parallel from TestComplete

Click the image to enlarge it.

After all test runs are over, the test results will be available in your TestComplete project:

Results of cross-platform web tests run in parallel from TestComplete

Click the image to enlarge it.

Requirements

  • TestComplete 14.70.

  • To run tests in various browsers and on various devices, you must have Selenium Grid that manages them. It can be:

    • A private hub with Selenium Grid configured. It can be Selenium Grid running on your local computer or on a remote computer in your network. To learn more about installing and configuring Selenium Grid, see selenium.dev.

  • Your Selenium Grid must be accessible from the computer where TestComplete is installed.

    If you use the CrossBrowserTesting device cloud as your Selenium Grid provider, the computer where TestComplete is installed must have access to CrossBrowserTesting.com. If there are any firewalls or proxies in your network, make sure that they allow traffic to that website. If needed, configure the firewall to allow outbound connections on ports 80 and 443 to crossbrowsertesting.com.

  • All cross-platform web tests you want to run must belong to the same TestComplete project from which you want to control the test runs. Running tests that belong to external projects is not supported.

Supported features

Only cross-platform web tests can run in parallel. These tests can include the following:

  • Operations that simulate user actions over web elements.

  • Operations for connecting to the SmartBear device cloud and launching web browsers there.

  • Checkpoints.

  • Operations that get test input data from external files (data-driven tests).

  • aqNNN scripting objects, for example aqString or aqConvert.

  • Event handlers.

  • BDD features and scenarios.

Unsupported features

  • Nested parallel tests. Parallel tests that control other parallel tests are not supported.

  • Running cross-platform web tests in parallel is not supported on 32-bit operating systems.

  • Desktop tests. Tests that simulate user actions over Windows desktop applications cannot run in parallel. For example, the following is not supported:

    • Operations that simulate user actions over desktop applications.

    • Low-level procedures.

    • Operations that manage the Tested Applications collection.

  • Non-cross-platform web tests. Tests that simulate user actions over local web browsers accessed via ways other than by using Selenium Grid cannot run in parallel. For example, the following is not supported:

    • Accessing web browsers by using the Sys.Browser object.

    • The Run Browser and Browser Loop operation.

  • Mobile tests. Tests that simulate user actions over mobile devices connected to your local workstation cannot run in parallel. For example, the following is not supported:

    • Android Gesture collection.

    • Operations that access mobile devices connected to your workstation via the Mobile object.

  • Image-based and OCR-based tests.

  • Testing web services.

  • User-forms, Network Suites, Unit tests, manual tests, object-driven tests (ODT), ReadyAPI tests.

If your test uses an object or an operation that your current set of active licenses does not support, an error will occur. Depending on the way your project handle errors, the test run may be terminated.

Orchestration via the Execution Plan editor

The Execution Plan editor allows you to configure a list of cloud remote environments where the cross-platform web test will be run in parallel:

Running cross-platform web tests in parallel via the Execution Plan editor

Click the image to enlarge it.

The editor looks in this way if the Cross-Platform Web Testing Support plugin is enabled. This plugin is installed and enabled automatically as part of the TestComplete Web module.

If the panel is hidden, to make it visible, click the button in the right-top corner of the Execution Plan editor.

Follow the steps below to manage your cross-platform web tests:

  1. Open the Execution Plan editor of your TestComplete project and add the test item that will run your cross-platform web test. To learn how to create test items, see the description of the project’s Execution Plan editor.

  2. In the Tested URL edit box of the Parallel Device Cloud Environments panel, specify the URL of the tested web page that the specified web browser on the remote platform will open upon launching.

    Notes:

    • If the Parallel Device Cloud Environments panel is hidden, use the button in the right-top corner of the Execution Plan editor to make the panel visible.

    • We recommend that you specify the complete URL of your target web page, including the protocol name.

    If your tested web application is behind a firewall in your local network or on your local computer, or if you want to route traffic to your tested application through a proxy server, you can configure a secure tunnel.

    If the tunnel has not been configured yet, click Select another location and then, in the resulting Choose Your Web Application Location dialog, specify the application location.

    View instructions

    The configured tunnel will be added to the project settings.

  3. Specify environment parameters (Platform, Operating System, Browser, and Resolution) and click Add:

    Running cross-platform web tests in parallel via the Execution Plan editor

    Click the image to enlarge it.

When you launch your project or a particular test item, TestComplete will run your cross-platform web test in parallel on added environments.

To cut, copy, or paste environments from one test item to another, use the Cut, Copy, Paste items of the panel’s context menu. Alternatively, you can save the list of environments to an external .xml file and then load it by using the Save to file and Load from file buttons respectively.

Additionally, you can execute your cross-platform tests in parallel on the same workstation (not in the cloud environments). To do this, use the button on the toolbar of the Execution Plan editor. See Grouping test items executed in parallel.

Orchestration via the Parallel script object

Methods of the Parallel object allow running tests in remote environments in a device cloud:

  • Parallel.RunEnvironments - Connects to the device cloud you specified and runs the specified tests in the cloud in parallel.

  • Parallel.RunTests - Runs the specified tests.

    Note: This method does not connect to the device cloud and does not specify target environments for the test to run. You have to do it manually in each test the method runs.

The examples below show how you can use the methods to run tests.

Example 1: Running several tests in specified environments in parallel

The sample code below runs the MyTest1 and MyTest2 cross-platform web tests in the Safari and Chrome web browsers that it launches in the SmartBear device cloud.

JavaScript, JScript
Python
VBScript
DelphiScript
C++Script, C#Script

JavaScript, JScript

function Main()
{
  // Set a list of tests to run in parallel
  var tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2'];
  // Set a list of test environments
  var environments = ['{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}',
          '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}'];

  // Set the URL of the device cloud server
  var serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub";

  var testedURL = "https://services.smartbear.com/samples/TestComplete14/smartstore/";

  // Run tests in parallel in specified environments
  Parallel.RunEnvironments(tests, environments, testedURL, serverURL);

}

function MyTest1()
{
  var browser = Sys.Browser();
  var page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

function MyTest2()
{
  var browser = Sys.Browser();
  var page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

Python

def Main():
  # Set a list of tests to run in parallel
  tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2']
  # Set a list of test environments
  environments = ['{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}', '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}']

  # Set the URL of the device cloud server
  serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub"

  testedURL = "https://services.smartbear.com/samples/TestComplete14/smartstore/"

  # Run tests in parallel in specified environments
  Parallel.RunEnvironments(tests, environments, testedURL, serverURL)

def MyTest1():
  browser = Sys.Browser()
  page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …

def MyTest2():
  browser = Sys.Browser()
  page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …

VBScript

Sub Main
  ' Set a list of tests to run in parallel
  Dim tests : tests = Array("Script|Unit1|MyTest1", "Script|Unit1|MyTest2")
  ' Set a list of test environments
  Dim environments : environments = Array(_
      "{""platform"": ""Mac OSX 10.14"", ""browserName"": ""Safari"", ""version"": ""12"", ""screenResolution"": ""1366x768""}",_
      "{""deviceName"": ""Pixel 4"", ""platformName"": ""Android"", ""platformVersion"": ""10.0"", ""browserName"": ""Chrome"", ""deviceOrientation"": ""portrait""}")

  ' Set the URL of the device cloud server
  serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub"

  testedURL = "https://services.smartbear.com/samples/TestComplete14/smartstore/"

  ' Run tests in parallel in specified environments
  Call Parallel.RunEnvironments(tests, environments, testedURL, serverURL)

End Sub

Sub MyTest1()
  Set browser = Sys.Browser()
  Set page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …
End Sub

Sub MyTest2()
  Set browser = Sys.Browser()
  Set page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …
End Sub

DelphiScript

procedure MyTest1();
var browser, page;
begin
  browser := Sys.Browser();
  page := browser.Page('*services.smartbear.com/samples/TestComplete*/smartstore/');
  …
end;
procedure MyTest2();
var browser, page;
begin
  browser := Sys.Browser();
  page := browser.Page('*services.smartbear.com/samples/TestComplete*/smartstore/');
  …
end;

procedure Main();
var serverURL, testedURL;
var tests : Array[0..1];
var environments : Array[0..1];
begin
  // Set a list of tests to run in parallel
  tests[0] : = 'Script|Unit1|MyTest1';
  tests[1] : = 'Script|Unit1|MyTest2';
  // Set a list of test environments
  environments[0] := '{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}';
  environments[1] := '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}';

  // Set the URL of the device cloud server
  serverURL := 'http://hub.crossbrowsertesting.com:80/wd/hub';

  testedURL := 'https://services.smartbear.com/samples/TestComplete14/smartstore/';

  // Run tests in parallel in specified environments
  Parallel.RunEnvironments(tests, environments, testedURL, serverURL);
end;

C++Script, C#Script

function Main()
{
  // Set a list of tests to run in parallel
  var tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2'];
  // Set a list of test environments
  var environments = ['{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}',
          '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}'];

  // Set the URL of the device cloud server
  var serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub";

  var testedURL = "https://services.smartbear.com/samples/TestComplete14/smartstore/";

  // Run tests in parallel in specified environments
  Parallel["RunEnvironments"](tests, environments, testedURL, serverURL);

}

function MyTest1()
{
  var browser = Sys["Browser"]();
  var page = browser["Page"]("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

function MyTest2()
{
  var browser = Sys["Browser"]();
  var page = browser["Page"]("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

Example 2: Running several tests in parallel

The sample code below runs MyTest1 and MyTest2 cross-platform web tests in parallel. Each test connects to a device cloud, launches a web browser there, and simulates user actions in that web browser.

JavaScript, JScript
Python
VBScript
DelphiScript
C++Script, C#Script

JavaScript, JScript

function Main()
{
  // Set a list of tests to run in parallel
  var tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2'];
  // Set a list of test environments
  var environments = ['{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}',
          '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}'];

  // Set the URL of the device cloud server
  var serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub";

  var testedURL = "https://services.smartbear.com/samples/TestComplete14/smartstore/";

  // Run tests in parallel in specified environments
  Parallel.RunEnvironments(tests, environments, testedURL, serverURL);

}

function MyTest1()
{
  var browser = Sys.Browser();
  var page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

function MyTest2()
{
  var browser = Sys.Browser();
  var page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

Python

def Main():
  # Set a list of tests to run in parallel
  tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2']
  # Set a list of test environments
  environments = ['{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}', '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}']

  # Set the URL of the device cloud server
  serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub"

  testedURL = "https://services.smartbear.com/samples/TestComplete14/smartstore/"

  # Run tests in parallel in specified environments
  Parallel.RunEnvironments(tests, environments, testedURL, serverURL)

def MyTest1():
  browser = Sys.Browser()
  page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …

def MyTest2():
  browser = Sys.Browser()
  page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …

VBScript

Sub Main
  ' Set a list of tests to run in parallel
  Dim tests : tests = Array("Script|Unit1|MyTest1", "Script|Unit1|MyTest2")
  ' Set a list of test environments
  Dim environments : environments = Array(_
      "{""platform"": ""Mac OSX 10.14"", ""browserName"": ""Safari"", ""version"": ""12"", ""screenResolution"": ""1366x768""}",_
      "{""deviceName"": ""Pixel 4"", ""platformName"": ""Android"", ""platformVersion"": ""10.0"", ""browserName"": ""Chrome"", ""deviceOrientation"": ""portrait""}")

  ' Set the URL of the device cloud server
  serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub"

  testedURL = "https://services.smartbear.com/samples/TestComplete14/smartstore/"

  ' Run tests in parallel in specified environments
  Call Parallel.RunEnvironments(tests, environments, testedURL, serverURL)

End Sub

Sub MyTest1()
  Set browser = Sys.Browser()
  Set page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …
End Sub

Sub MyTest2()
  Set browser = Sys.Browser()
  Set page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …
End Sub

DelphiScript

procedure MyTest1();
var browser, page;
begin
  browser := Sys.Browser();
  page := browser.Page('*services.smartbear.com/samples/TestComplete*/smartstore/');
  …
end;
procedure MyTest2();
var browser, page;
begin
  browser := Sys.Browser();
  page := browser.Page('*services.smartbear.com/samples/TestComplete*/smartstore/');
  …
end;

procedure Main();
var serverURL, testedURL;
var tests : Array[0..1];
var environments : Array[0..1];
begin
  // Set a list of tests to run in parallel
  tests[0] : = 'Script|Unit1|MyTest1';
  tests[1] : = 'Script|Unit1|MyTest2';
  // Set a list of test environments
  environments[0] := '{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}';
  environments[1] := '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}';

  // Set the URL of the device cloud server
  serverURL := 'http://hub.crossbrowsertesting.com:80/wd/hub';

  testedURL := 'https://services.smartbear.com/samples/TestComplete14/smartstore/';

  // Run tests in parallel in specified environments
  Parallel.RunEnvironments(tests, environments, testedURL, serverURL);
end;

C++Script, C#Script

function Main()
{
  // Set a list of tests to run in parallel
  var tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2'];
  // Set a list of test environments
  var environments = ['{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}',
          '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}'];

  // Set the URL of the device cloud server
  var serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub";

  var testedURL = "https://services.smartbear.com/samples/TestComplete14/smartstore/";

  // Run tests in parallel in specified environments
  Parallel["RunEnvironments"](tests, environments, testedURL, serverURL);

}

function MyTest1()
{
  var browser = Sys["Browser"]();
  var page = browser["Page"]("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

function MyTest2()
{
  var browser = Sys["Browser"]();
  var page = browser["Page"]("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

Number of parallel tests

By default, you can run any number of cross-platform web tests in parallel. The number is only limited by:

  • Your workstation computing capacity: its processor, RAM, and so on. The more tests are running in parallel, the more workstation resources they are consuming.

  • You can limit the number of maximum tests running in parallel manually by using the LicenseCount parameter of the Parallel.RunTests and Parallel.RunEnvironments methods.

See Also

About Cross-Platform Web Tests

Highlight search results