Running Cross-Platform Web Tests in Parallel

Applies to TestComplete 15.40, last modified on July 05, 2022

About

In TestComplete, you can create cross-platform web tests and run them in a device cloud. This way, you can easily check how your tested web application functions in as many browsers and browser versions and platforms as you need.

To manage resources of your device cloud more effectively, it would make much sense to run several cross-platform web tests at the same time. This would reduce the overall test run time and will help you decrease the cost of your testing efforts.

You can control parallel runs from TestComplete in two ways:

Running cross-platform web tests in parallel from TestComplete

Click the image to enlarge it.

After all test runs are over, the test results will be available in your TestComplete project:

Results of cross-platform web tests run in parallel from TestComplete

Click the image to enlarge it.

Requirements

  • TestComplete 14.70.

  • To run tests in various browsers and on various devices, you must have a device cloud that manages them. It can be:

    • A private hub with Selenium Grid configured. It can be Selenium Grid running on your local computer or on a remote computer in your network. To learn more about installing and configuring Selenium Grid, see selenium.dev.

  • Your device cloud must be accessible from the computer where TestComplete is installed.

    If there are any firewalls or proxies in your network, make sure that they allow traffic to the device cloud. If needed, configure the firewall as described in TestComplete and Windows Firewall.

  • All cross-platform web tests you want to run must belong to the same TestComplete project from which you want to control the test runs. Running tests that belong to external projects is not supported.

Supported features

Tests that will run in parallel can include the following:

  • Operations that simulate user actions over web elements.

  • Operations for connecting to the SmartBear device cloud and launching web browsers there.

  • Checkpoints.

  • Operations that get test input data from external files (data-driven tests).

  • aqNNN scripting objects, for example aqString or aqConvert.

  • Event handlers.

  • BDD features and scenarios.

  • OCR-based actions and checkpoints.

Unsupported features

  • Nested parallel tests. Parallel tests that control other parallel tests are not supported.

  • Running parallel tests is not supported on 32-bit operating systems.

  • Desktop tests. Tests that simulate user actions over Windows desktop applications cannot run in parallel. For example, the following is not supported:

    • Operations that simulate user actions over desktop applications.

    • Low-level procedures.

  • Non-cross-platform web tests. Tests that simulate user actions over local web browsers accessed via ways other than by using Selenium Grid cannot run in parallel. For example, the following is not supported:

    • Accessing web browsers by using the Sys.Browser object.

    • The Run Browser and Browser Loop operation.

  • Legacy mobile tests. Tests that simulate user actions over physical mobile devices connected to your local workstation cannot run in parallel.

  • Image-based tests.

  • Testing web services.

  • User Forms, Network Suites, Unit tests, manual tests, object-driven tests (ODT), ReadyAPI tests.

If you run a test that uses an object or an operation that is not supported by parallel runs, the test engine will report an error. Depending on the way your project handles errors, the test run may be terminated.

Orchestration via the Execution Plan editor

The Execution Plan editor allows you to configure a list of cloud remote environments where the cross-platform web test will be run in parallel:

Running cross-platform web tests in parallel via the Execution Plan editor

Click the image to enlarge it.

If the panel is hidden, to make it visible, click the button in the right-top corner of the Execution Plan editor.

  1. Create a web test and prepare it for running in remote environments. See Creating and Running Cross-Platform Web Tests.

  2. Open the execution plan of your TestComplete project and add the created web test to it.

  3. Select the added test and, in the Parallel Device Cloud Environments panel, configure the parallel run:

    1. Select the web cloud provider you are going to use to run tests. Currently, only CrossBrowserTesting.com is supported.

      If this provider is not present on the panel, make sure that the Device Cloud > Web > Web cloud provider property of your project is set to CrossBrowserTesting.

    2. In the Tested URL text box, specify the URL of the tested web page that a web browser on a remote platform will open upon launching. You can type the URL manually or you can click the down-arrow button and select a project or project suite variable that stores the URL.

      We recommend that you specify the complete URL of your target web page, including the protocol name.

    3. If your tested web application is behind a firewall in your local network or on your local computer, or if you want to route traffic to your tested application through a proxy server, you can configure a secure tunnel.

      If the tunnel has not been configured yet, click Select another location and then, in the resulting Choose Your Web Application Location dialog, specify the application location.

      View instructions

      The configured tunnel will be added to the project settings.

    4. Click the New Environment button and specify environment parameters:

      Running cross-platform web tests in parallel via the Execution Plan editor

      Click the image to enlarge it.

    5. If needed, on the Custom parameters tab, you can specify additional capabilities to be used for the testing session. Specify the capabilities in the JSON format:

      Specifying custom parameters

      Click the image to enlarge it.

    6. Click OK. TestComplete will add the configured environment to the list.

When you launch your project or a particular test item, TestComplete will run your cross-platform web test in parallel on added environments.

To cut, copy, or paste environments from one test item to another, use the Cut, Copy, Paste items of the panel’s context menu. Alternatively, you can save the list of environments to an external .xml file and then load it by using the Export to file and Import from file buttons respectively.

For complete information about working with this panel, see Execution Plan Editor - Parallel Device Cloud Environments Panel.

Additionally, you can execute your cross-platform tests in parallel on the same workstation (not in the cloud environments). To do this, use the button on the toolbar of the Execution Plan editor. See Grouping test items executed in parallel.

Orchestration via the Parallel script object

Methods of the Parallel object allow running tests in a device cloud in parallel:

  • Parallel.RunWebTests - Connects to a device cloud and runs the specified tests in each of the specified target browsers in parallel.

  • Parallel.RunTests - Runs the specified tests in parallel.

The examples below show how you can use the methods to run tests.

Example 1: Running several tests in specified environments in parallel

The sample code below runs the MyTest1 and MyTest2 cross-platform web tests in the Safari and Chrome web browsers that it launches in the SmartBear device cloud.

JavaScript, JScript
Python
VBScript
DelphiScript
C++Script, C#Script
Copy Code

JavaScript, JScript

function Main()
{
  // Set a list of tests to run in parallel
  var tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2'];
  // Set a list of target web browsers
  var environments = [
  {
    "platform": "Mac OSX 10.14",
    "browserName": "Safari",
    "version": "12",
    "screenResolution": "1366x768"
    },
  {
    "deviceName": "Pixel 4",
    "platformName": "Android",
    "platformVersion": "10.0",
    "browserName": "Chrome",
    "deviceOrientation": "portrait"
    }
  ];

  // Set the URL of the device cloud server
  var serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub";

  var testedURL = "https://services.smartbear.com/samples/TestComplete15/smartstore/";

  // Run tests in parallel in specified web browsers
  Parallel.RunWebTests(tests, environments, testedURL, serverURL);

}

function MyTest1()
{
  var browser = Sys.Browser();
  var page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

function MyTest2()
{
  var browser = Sys.Browser();
  var page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

Python

def Main():
  # Set a list of tests to run in parallel
  tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2']
  # Set a list of target web browsers
  environments = [
    {
    "platform": "Mac OSX 10.14",
    "browserName": "Safari",
    "version": "12",
    "screenResolution": "1366x768"
    },
    {
    "deviceName": "Pixel 4",
    "platformName": "Android",
    "platformVersion": "10.0",
    "browserName": "Chrome",
    "deviceOrientation": "portrait"
    }
  ]

  # Set the URL of the device cloud server
  serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub"

  testedURL = "https://services.smartbear.com/samples/TestComplete15/smartstore/"

  # Run tests in parallel in specified web browsers
  Parallel.RunWebTests(tests, environments, testedURL, serverURL)

def MyTest1():
  browser = Sys.Browser()
  page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …

def MyTest2():
  browser = Sys.Browser()
  page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …

VBScript

Sub Main
  ' Set a list of tests to run in parallel
  Dim tests : tests = Array("Script|Unit1|MyTest1", "Script|Unit1|MyTest2")
  ' Set a list of target web browsers
  Dim environments : environments = Array(_
      "{""platform"": ""Mac OSX 10.14"", ""browserName"": ""Safari"", ""version"": ""12"", ""screenResolution"": ""1366x768""}",_
      "{""deviceName"": ""Pixel 4"", ""platformName"": ""Android"", ""platformVersion"": ""10.0"", ""browserName"": ""Chrome"", ""deviceOrientation"": ""portrait""}")

  ' Set the URL of the device cloud server
  serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub"

  testedURL = "https://services.smartbear.com/samples/TestComplete15/smartstore/"

  ' Run tests in parallel in specified web browsers
  Call Parallel.RunWebTests(tests, environments, testedURL, serverURL)

End Sub

Sub MyTest1()
  Set browser = Sys.Browser()
  Set page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …
End Sub

Sub MyTest2()
  Set browser = Sys.Browser()
  Set page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …
End Sub

DelphiScript

procedure MyTest1();
var browser, page;
begin
  browser := Sys.Browser();
  page := browser.Page('*services.smartbear.com/samples/TestComplete*/smartstore/');
  …
end;
procedure MyTest2();
var browser, page;
begin
  browser := Sys.Browser();
  page := browser.Page('*services.smartbear.com/samples/TestComplete*/smartstore/');
  …
end;

procedure Main();
var serverURL, testedURL;
var tests : Array[0..1];
var environments : Array[0..1];
begin
  // Set a list of tests to run in parallel
  tests[0] : = 'Script|Unit1|MyTest1';
  tests[1] : = 'Script|Unit1|MyTest2';
  // Set a list of target web browsers
  environments[0] := '{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}';
  environments[1] := '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}';

  // Set the URL of the device cloud server
  serverURL := 'http://hub.crossbrowsertesting.com:80/wd/hub';

  testedURL := 'https://services.smartbear.com/samples/TestComplete15/smartstore/';

  // Run tests in parallel in specified web browsers
  Parallel.RunWebTests(tests, environments, testedURL, serverURL);
end;

C++Script, C#Script

function Main()
{
  // Set a list of tests to run in parallel
  var tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2'];
  // Set a list of target web browsers
  var environments = ['{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}',
          '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}'];

  // Set the URL of the device cloud server
  var serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub";

  var testedURL = "https://services.smartbear.com/samples/TestComplete15/smartstore/";

  // Run tests in parallel in specified web browsers
  Parallel["RunWebTests"](tests, environments, testedURL, serverURL);

}

function MyTest1()
{
  var browser = Sys["Browser"]();
  var page = browser["Page"]("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

function MyTest2()
{
  var browser = Sys["Browser"]();
  var page = browser["Page"]("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

Example 2: Running several tests in parallel

The sample code below runs MyTest1 and MyTest2 cross-platform web tests in parallel. Each test connects to a device cloud, launches a web browser there, and simulates user actions in that web browser.

JavaScript, JScript
Python
VBScript
DelphiScript
C++Script, C#Script
Copy Code

JavaScript, JScript

function Main()
{
  // Set a list of tests to run in parallel
  var tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2'];
  // Set a list of target web browsers
  var environments = [
  {
    "platform": "Mac OSX 10.14",
    "browserName": "Safari",
    "version": "12",
    "screenResolution": "1366x768"
    },
  {
    "deviceName": "Pixel 4",
    "platformName": "Android",
    "platformVersion": "10.0",
    "browserName": "Chrome",
    "deviceOrientation": "portrait"
    }
  ];

  // Set the URL of the device cloud server
  var serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub";

  var testedURL = "https://services.smartbear.com/samples/TestComplete15/smartstore/";

  // Run tests in parallel in specified web browsers
  Parallel.RunWebTests(tests, environments, testedURL, serverURL);

}

function MyTest1()
{
  var browser = Sys.Browser();
  var page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

function MyTest2()
{
  var browser = Sys.Browser();
  var page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

Python

def Main():
  # Set a list of tests to run in parallel
  tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2']
  # Set a list of target web browsers
  environments = [
    {
    "platform": "Mac OSX 10.14",
    "browserName": "Safari",
    "version": "12",
    "screenResolution": "1366x768"
    },
    {
    "deviceName": "Pixel 4",
    "platformName": "Android",
    "platformVersion": "10.0",
    "browserName": "Chrome",
    "deviceOrientation": "portrait"
    }
  ]

  # Set the URL of the device cloud server
  serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub"

  testedURL = "https://services.smartbear.com/samples/TestComplete15/smartstore/"

  # Run tests in parallel in specified web browsers
  Parallel.RunWebTests(tests, environments, testedURL, serverURL)

def MyTest1():
  browser = Sys.Browser()
  page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …

def MyTest2():
  browser = Sys.Browser()
  page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …

VBScript

Sub Main
  ' Set a list of tests to run in parallel
  Dim tests : tests = Array("Script|Unit1|MyTest1", "Script|Unit1|MyTest2")
  ' Set a list of target web browsers
  Dim environments : environments = Array(_
      "{""platform"": ""Mac OSX 10.14"", ""browserName"": ""Safari"", ""version"": ""12"", ""screenResolution"": ""1366x768""}",_
      "{""deviceName"": ""Pixel 4"", ""platformName"": ""Android"", ""platformVersion"": ""10.0"", ""browserName"": ""Chrome"", ""deviceOrientation"": ""portrait""}")

  ' Set the URL of the device cloud server
  serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub"

  testedURL = "https://services.smartbear.com/samples/TestComplete15/smartstore/"

  ' Run tests in parallel in specified web browsers
  Call Parallel.RunWebTests(tests, environments, testedURL, serverURL)

End Sub

Sub MyTest1()
  Set browser = Sys.Browser()
  Set page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …
End Sub

Sub MyTest2()
  Set browser = Sys.Browser()
  Set page = browser.Page("*services.smartbear.com/samples/TestComplete*/smartstore/")
  …
End Sub

DelphiScript

procedure MyTest1();
var browser, page;
begin
  browser := Sys.Browser();
  page := browser.Page('*services.smartbear.com/samples/TestComplete*/smartstore/');
  …
end;
procedure MyTest2();
var browser, page;
begin
  browser := Sys.Browser();
  page := browser.Page('*services.smartbear.com/samples/TestComplete*/smartstore/');
  …
end;

procedure Main();
var serverURL, testedURL;
var tests : Array[0..1];
var environments : Array[0..1];
begin
  // Set a list of tests to run in parallel
  tests[0] : = 'Script|Unit1|MyTest1';
  tests[1] : = 'Script|Unit1|MyTest2';
  // Set a list of target web browsers
  environments[0] := '{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}';
  environments[1] := '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}';

  // Set the URL of the device cloud server
  serverURL := 'http://hub.crossbrowsertesting.com:80/wd/hub';

  testedURL := 'https://services.smartbear.com/samples/TestComplete15/smartstore/';

  // Run tests in parallel in specified web browsers
  Parallel.RunWebTests(tests, environments, testedURL, serverURL);
end;

C++Script, C#Script

function Main()
{
  // Set a list of tests to run in parallel
  var tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2'];
  // Set a list of target web browsers
  var environments = ['{"platform": "Mac OSX 10.14", "browserName": "Safari", "version": "12", "screenResolution": "1366x768"}',
          '{"deviceName": "Pixel 4", "platformName": "Android", "platformVersion": "10.0", "browserName": "Chrome", "deviceOrientation": "portrait"}'];

  // Set the URL of the device cloud server
  var serverURL = "http://hub.crossbrowsertesting.com:80/wd/hub";

  var testedURL = "https://services.smartbear.com/samples/TestComplete15/smartstore/";

  // Run tests in parallel in specified web browsers
  Parallel["RunWebTests"](tests, environments, testedURL, serverURL);

}

function MyTest1()
{
  var browser = Sys["Browser"]();
  var page = browser["Page"]("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

function MyTest2()
{
  var browser = Sys["Browser"]();
  var page = browser["Page"]("*services.smartbear.com/samples/TestComplete*/smartstore/");
  …
}

Number of parallel tests

  • Your workstation computing capacity: its processor, RAM, and so on. The more parallel tests TestComplete is controlling, the more workstation resources are consumed.

  • You can limit the number of maximum tests running in parallel manually by using the LicenseCount parameter of the Parallel object’s methods or by using the Options > Engine > Parallel > Parallel test count option.

Capturing images during parallel test runs

TestComplete can capture images of the tested web application or website during test runs. These images can help understand the tested application behavior if the test fails. By default, capturing images during parallel runs is disabled. To command TestComplete to capture images, enable the Capture images during parallel test runs project option.

See Also

About Cross-Platform Web Tests

Highlight search results