Running Mobile Tests in Parallel

Applies to TestComplete 15.31, last modified on March 17, 2022

In TestComplete, you can create mobile tests for applications running on devices managed by BitBar or by a private Appium server. To reduce the overall test run time and manage testing resources more efficiently, you can run several such tests concurrently.

You can control parallel runs from TestComplete by using the Execution Plan editor or Parallel object:

Running mobile tests in parallel

Click the image to enlarge it.

After all test runs are over, the test results will be available in your TestComplete project:

Results of mobile tests run in parallel

Click the image to enlarge it.

Requirements

  • TestComplete version 14.81.

  • An active license for the TestComplete Mobile module.

  • The Appium Support plugin must be enabled in TestComplete. The plugin is installed and enabled by default. You can check the plugin state in the File > Install Extensions dialog of TestComplete.

  • Mobile devices managed by Appium. It can be:

    • A mobile device cloud provided as a service by BitBar.

    – or –

    • A private mobile device cloud managed by Appium. It can run on your test workstation (a computer where TestComplete is running) or on a remote computer in your local network:

    For details, please see Testing Mobile Applications - Requirements.

  • The TestComplete workstation must have access to the server that controls the mobile devices.

  • The TestComplete project must be configured to support Appium mobile devices.

    By default, all new projects are configured this way. To make sure that your project is configured correctly, check its Mobile support type property on the Properties > Mobile Devices > General page. It must be set to Mobile device cloud (BitBar and local Appium).

  • Mobile tests you want to run in parallel must be Appium-compatible. Running legacy mobile tests in parallel is not supported.

  • Mobile tests you want to run in parallel must belong to the same TestComplete project from which you will control the test runs. Running tests that belong to external projects is not supported.

Supported features

Tests that will run in parallel can include the following:

  • Operations that simulate user actions over mobile applications.

  • Operations for connecting to the Appium server and opening testing sessions on mobile devices there.

  • Checkpoints.

  • Operations that get test input data from external files (data-driven tests).

  • aqNNN scripting objects, for example aqString or aqConvert.

  • Event handlers.

  • BDD features and scenarios.

Unsupported features

  • Nested parallel tests. Parallel tests that control other parallel tests are not supported.

  • Running parallel tests is not supported on 32-bit operating systems.

  • Desktop tests. Tests that simulate user actions over Windows desktop applications cannot run in parallel. For example, the following is not supported:

    • Operations that simulate user actions over desktop applications.

    • Low-level procedures.

  • Non-cross-platform web tests. Tests that simulate user actions over local web browsers accessed via ways other than by using Selenium Grid cannot run in parallel. For example, the following is not supported:

    • Accessing web browsers by using the Sys.Browser object.

    • The Run Browser and Browser Loop operation.

  • Legacy mobile tests. Tests that simulate user actions over physical mobile devices connected to your local workstation cannot run in parallel.

  • Image-based tests.

  • Testing web services.

  • User Forms, Network Suites, Unit tests, manual tests, object-driven tests (ODT), ReadyAPI tests.

  • OCR-based tests.

If you run a test that uses an object or an operation that is not supported by parallel runs, the test engine will report an error. Depending on the way your project handles errors, the test run may be terminated.

How to run tests in parallel

In the Execution Plan editor

In the Execution Plan editor, you can configure a list of remote environments where mobile tests will be run in parallel:

  1. Create a mobile test and prepare it for running in remote environments. See Creating and Running Mobile Tests.

  2. Open the execution plan of your TestComplete project and add the created test to it.

  3. Select the added test and, in the Parallel Device Cloud Environments panel, select the device cloud provider in the environments of which you will run the tests. For mobile tests, it can be either BitBar or a private Appium server.

  4. Click New Environment and, in the resulting dialog, specify the target testing environment:

    BitBar

    1. If the current TestComplete project does not have a BitBar API key specified, to connect to the BitBar device cloud, enter the API key.

      Learn how to get the API key

    2. Select the mobile device and a tested application for which you want to open a testing session:

      • You can select devices that have any status in the Status column. However, the selected device must be Available when the test run starts.

      • If the selected device is not available, TestComplete will not be able to connect to it. To command TestComplete to search for a replacement device, select the Connect to another device when the selected one is not available check box. On the BitBar & Local Appium page of your project, you can set the search criteria for the replacement devices.

      • In the Application drop-down list box, select a mobile application for which you want to open a testing session. The application file must be either stored in the BitBar Files Library or added to the Tested Applications of the current TestComplete project. In the latter case, click Upload to BitBar to upload the application file to the BitBar Files Library and then select it in the list.

        For iOS only: To be able to launch and stop the tested application from tests or by using the Mobile Device Screen window of TestComplete, specify the bundle ID of the application when connecting to the device. It is the bundle ID that was used to compile the specified .ipa file. Add the bundleId capability to the list of desired capabilities on the View more BitBar parameters > Custom Parameters tab.

    3. To get more advanced options, click View more BitBar parameters:

      • On the Parameters tab, you can filter the device list by the device type, the version of the operating system running on the devices, their screen resolution, and their location.

      • By default, when a device is disconnected, the testing session on it is closed, and the device is restored to its initial state. Until the device is restored, it will remain unavailable. To keep the session alive to be able to reconnect to it, select the Keep the device available for quick reconnection check box. The session will be available for reconnection for up to 60 seconds.

      • If needed, on the Custom parameters tab, you can specify additional capabilities to be used for the testing session. Specify the capabilities in the JSON format. For example:

        JSON
        Copy Code

        JSON

        {
          "bitbar_project": "Automated tests",
          "bitbar_description": "Testing the Orders app"
        }

        The JSON code will be validated as you are typing it.

        Note: The capabilities you specify on the Custom capabilities tab can overwrite values specified on the Parameters tab.

        For the list of available capabilities, see the BitBar Documentation - Desired Capabilities.

    Local Appium

    Note: You must have your local Appium server running and available from your TestComplete workstation. For information on how to configure a local Appium server, see Set up Appium Server.

    1. In the Server URL text box, enter the URL address of your Appium server.

    2. In the Application Path text editor, specify the full path to the application that you want to install on the target mobile device and from which you want to open a testing session. If your application is added to the list of tested applications in your TestComplete project, you can click the down-arrow button and select the application path from the drop-down list. You can also type the path manually, or you can click the ellipsis button and browse for the needed application file.

      Specify the application path relatively to the Appium server location.

      You can also specify the application path using the app capability in the Parameters editor. The app capability value will override the Application Path value.

    3. In the Parameters editor, enter the capabilities that describe a testing session you want to open, in the JSON format. The dialog will validate the JSON code as you are typing.

      View required capabilities

      For information on all available capabilities, see Appium Desired Capabilities.

When you run the TestComplete project, it will run your test in parallel in the specified environments in the selected device cloud.

By using the Parallel scripting object

  1. In your TestComplete project, create mobile tests that you will run in parallel in the device cloud managed by Appium. To learn more on creating cloud-compatible mobile tests, see Creating and Running Mobile Tests.

  2. Create a script test from which you will control the parallel test runs. Use methods of the Parallel object:

    • Parallel.RunMobileTests(Tests, Capabilities, ServerURL) - Connects to the device cloud specified by the ServerURL parameter, opens testing sessions described by the Capabilities parameter, and then runs mobile tests in these sessions in parallel.

    • Parallel.RunTests - Runs the specified tests in parallel. This method does not connect to the device cloud and does not specify target environments for the test to run. You have to do it manually in each test the method runs.

The examples below show how you can use the methods to run mobile tests in parallel.

Example 1: Running several tests on specified devices in parallel

The sample code below runs the MyTest1 and MyTest2 mobile tests on several mobile devices running in the BitBar device cloud.

JavaScript, JScript
Python
VBScript
DelphiScript
C++Script, C#Script
Copy Code

JavaScript, JScript

function Main_RunMobileTests_Parallel()
{
  var capabilities = [
    {
    "automationName": "UiAutomator2",
    "bitbar_app": "123",
    "bitbar_device": "Google Pixel 3a Android 10",
    "bitbar_findDevice": "false",
    "bitbar_target": "android",
    "deviceName": "Android Phone",
    "newCommandTimeout": "600",
    "platformName": "ANDROID"
    },
    {
    "automationName": "UiAutomator2",
    "bitbar_app": "123",
    "bitbar_device": "LG Google Nexus 5 6.0.1",
    "bitbar_findDevice": "false",
    "bitbar_target": "android",
    "deviceName": "Android Phone",
    "newCommandTimeout": "600",
    "platformName": "ANDROID"
    }
  ];

  var server = "https://appium.bitbar.com/wd/hub/";
  var tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2'];

  Parallel.RunMobileTests(tests, capabilities, server);

}

function MyTest1()
{
  var device = Mobile.Device();

  device.FindElementByXPath("//*[@text=\"Samuel Clemens\"]").Touch();
  device.FindElementById("smartbear.example.orders:id/editButton").Touch();

  // …
}
function MyTest2()
{
  var device = Mobile.Device();

  device.FindElementByXPath("//*[@text=\"Clare Jefferson\"]").Touch();
  device.FindElementById("smartbear.example.orders:id/editButton").Touch();

  // …
}

Python

def Main_RunMobileTests_Parallel():
  capabilities = [
    {
    "automationName": "UiAutomator2",
    "bitbar_app": "123",
    "bitbar_device": "Google Pixel 3a Android 10",
    "bitbar_findDevice": "false",
    "bitbar_target": "android",
    "deviceName": "Android Phone",
    "newCommandTimeout": "600",
    "platformName": "ANDROID"
    },
    {
    "automationName": "UiAutomator2",
    "bitbar_app": "123",
    "bitbar_device": "LG Google Nexus 5 6.0.1",
    "bitbar_findDevice": "false",
    "bitbar_target": "android",
    "deviceName": "Android Phone",
    "newCommandTimeout": "600",
    "platformName": "ANDROID"
    }
  ]

  server = "https://appium.bitbar.com/wd/hub/"
  tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2']

  Parallel.RunMobileTests(tests, capabilities, server)

def MyTest1():
  device = Mobile.Device()

  device.FindElementByXPath("//*[@text=\"Samuel Clemens\"]").Touch()
  device.FindElementById("smartbear.example.orders:id/editButton").Touch()

  # …
def MyTest2():
  device = Mobile.Device()

  device.FindElementByXPath("//*[@text=\"Clare Jefferson\"]").Touch()
  device.FindElementById("smartbear.example.orders:id/editButton").Touch()

  # …
}

VBScript

Sub Main
  Dim capabilities : capabilities = Array(_
      "{""automationName"": ""UiAutomator2"", ""bitbar_app"": ""123"", ""bitbar_device"": ""Google Pixel 3a Android 10"", ""bitbar_findDevice"": ""false"", ""bitbar_target"": ""android"", ""deviceName"": ""Android Phone"", ""newCommandTimeout"": ""600"", ""platformName"": ""ANDROID""}",_
      "{""automationName"": ""UiAutomator2"", ""bitbar_app"": ""123"", ""bitbar_device"": ""LG Google Nexus 5 6.0.1"", ""bitbar_findDevice"": ""false"", ""bitbar_target"": ""android"", ""deviceName"": ""Android Phone"", ""newCommandTimeout"": ""600"", ""platformName"": ""ANDROID""}")

  server = "https://appium.bitbar.com/wd/hub/"
  Dim tests : tests = Array("Script|Unit1|MyTest1", "Script|Unit1|MyTest2")
  Call Parallel.RunMobileTests(tests, capabilities, server)

End Sub

Sub MyTest1()
  Set device = Mobile.Device()

  device.FindElementByXPath("//*[@text=\"Samuel Clemens\"]").Touch
  device.FindElementById("smartbear.example.orders:id/editButton").Touch

  ' …
End Sub

Sub MyTest2()
  Set device = Mobile.Device

  device.FindElementByXPath("//*[@text=\"Clare Jefferson\"]").Touch
  device.FindElementById("smartbear.example.orders:id/editButton").Touch

  ' …
End Sub

DelphiScript

procedure MyTest1();
var device;
begin
  device := Mobile.Device();

  device.FindElementByXPath('//*[@text=\"Samuel Clemens\"]').Touch();
  device.FindElementById('smartbear.example.orders:id/editButton').Touch();

  // …
end;

procedure MyTest2();
var device;
bbegin
   device := Mobile.Device();

  device.FindElementByXPath('//*[@text=\"Clare Jefferson\"]').Touch();
  device.FindElementById('smartbear.example.orders:id/editButton').Touch();

  // …
end;

procedure Main_RunMobileTests_Parallel();
var capabilities : Array[0..1];
var server;
var tests : Array[0..1];
begin
  capabilities[0] := '{"automationName": "UiAutomator2", "bitbar_app": "123", "bitbar_device": "Google Pixel 3a Android 10", "bitbar_findDevice": "false", "bitbar_target": "android", "deviceName": "Android Phone", "newCommandTimeout": "600", "platformName": "ANDROID"}';
  capabilities[1] := '{"automationName": "UiAutomator2", "bitbar_app": "123", "bitbar_device": "LG Google Nexus 5 6.0.1", "bitbar_findDevice": "false", "bitbar_target": "android", "deviceName": "Android Phone", "newCommandTimeout": "600", "platformName": "ANDROID"}';

  server := 'https://appium.bitbar.com/wd/hub/';
  tests[0] := 'Script|Unit1|MyTest1';
  tests[1] := 'Script|Unit1|MyTest2';

  Parallel.RunMobileTests(tests, capabilities, server);

end;

C++Script, C#Script

function Main_RunMobileTests_Parallel()
{
  var capabilities = [
    '{"automationName": "UiAutomator2", "bitbar_app": "123", "bitbar_device": "Google Pixel 3a Android 10", "bitbar_findDevice": "false", "bitbar_target": "android", "deviceName": "Android Phone", "newCommandTimeout": "600", "platformName": "ANDROID"}',
    '{"automationName": "UiAutomator2", "bitbar_app": "123", "bitbar_device": "LG Google Nexus 5 6.0.1", "bitbar_findDevice": "false", "bitbar_target": "android", "deviceName": "Android Phone", "newCommandTimeout": "600", "platformName": "ANDROID"}'
  ];

  var server = "https://appium.bitbar.com/wd/hub/";
  var tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2'];

  Parallel["RunMobileTests"](tests, capabilities, server);

}

function MyTest1()
{
  var device = Mobile["Device"]();

  device["FindElementByXPath"]("//*[@text=\"Samuel Clemens\"]")["Touch"]();
  device["FindElementById"]("smartbear.example.orders:id/editButton")["Touch"]();

  // ...
}

function MyTest2()
{
  var device = Mobile["Device"]();

  device["FindElementByXPath"]("//*[@text=\"Clare Jefferson\"]").Touch();
  device["FindElementById"]("smartbear.example.orders:id/editButton").Touch();

  // ...
}

Example 2: Running several tests in parallel

The sample code below runs the MyTest1 and MyTest2 mobile tests in parallel. Each test connects to a mobile device, opens a testing session there, and simulates user actions:

JavaScript, JScript
Python
VBScript
DelphiScript
C++Script, C#Script
Copy Code

JavaScript, JScript

function Main()
{
  var tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2'];
  Parallel.RunTests(tests);
}

function MyTest1()
{
  var capabilities = {
    "automationName": "UiAutomator2",
    "bitbar_app": "123",
    "bitbar_device": "Google Pixel 3a Android 10",
    "bitbar_findDevice": "false",
    "bitbar_target": "android",
    "deviceName": "Android Phone",
    "newCommandTimeout": "600",
    "platformName": "ANDROID"
};

  var server = "https://appium.bitbar.com/wd/hub/";
  Mobile.ConnectDevice(server, capabilities);

  var device = Mobile.Device();

  device.FindElementByXPath("//*[@text=\"Samuel Clemens\"]").Touch();
  device.FindElementById("smartbear.example.orders:id/editButton").Touch();

  // …
}

function MyTest2()
{
  var capabilities = {
    "automationName": "UiAutomator2",
    "bitbar_app": "123",
    "bitbar_device": "LG Google Nexus 5 6.0.1",
    "bitbar_findDevice": "false",
    "bitbar_target": "android",
    "deviceName": "Android Phone",
    "newCommandTimeout": "600",
    "platformName": "ANDROID"
};

  var server = "https://appium.bitbar.com/wd/hub/";
  Mobile.ConnectDevice(server, capabilities);

  var device = Mobile.Device();

  device.FindElementByXPath("//*[@text=\"Clare Jefferson\"]").Touch();
  device.FindElementById("smartbear.example.orders:id/editButton").Touch();

  // …
}

Python

def Main():
  tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2']
  Parallel.RunTests(tests)

def MyTest1():
  capabilities = {
    "automationName": "UiAutomator2",
    "bitbar_app": "123",
    "bitbar_device": "Google Pixel 3a Android 10",
    "bitbar_findDevice": "false",
    "bitbar_target": "android",
    "deviceName": "Android Phone",
    "newCommandTimeout": "600",
    "platformName": "ANDROID"
}

  server = "https://appium.bitbar.com/wd/hub/"
  Mobile.ConnectDevice(server, capabilities)

  device = Mobile.Device()

  device.FindElementByXPath("//*[@text=\"Samuel Clemens\"]").Touch()
  device.FindElementById("smartbear.example.orders:id/editButton").Touch()

  # …

def MyTest2():
  capabilities = {
    "automationName": "UiAutomator2",
    "bitbar_app": "123",
    "bitbar_device": "LG Google Nexus 5 6.0.1",
    "bitbar_findDevice": "false",
    "bitbar_target": "android",
    "deviceName": "Android Phone",
    "newCommandTimeout": "600",
    "platformName": "ANDROID"
}

  server = "https://appium.bitbar.com/wd/hub/"
  Mobile.ConnectDevice(server, capabilities)

  device = Mobile.Device()

  device.FindElementByXPath("//*[@text=\"Clare Jefferson\"]").Touch()
  device.FindElementById("smartbear.example.orders:id/editButton").Touch()

  # …

VBScript

Sub Main
  DIm tests : tests = Array("Script|Unit1|MyTest1", "Script|Unit1|MyTest2")
  Call Parallel.RunTests(tests)
End Sub

Sub MyTest1()
  capabilities = "{""automationName"": ""UiAutomator2"",""bitbar_app"": ""123"",""bitbar_device"": ""Google Pixel 3a Android 10"",""bitbar_findDevice"": ""false"",""bitbar_target"": ""android"",""deviceName"": ""Android Phone"",""newCommandTimeout"": ""600"",""platformName"": ""ANDROID""}"

  server = "https://appium.bitbar.com/wd/hub/"
  Call Mobile.ConnectDevice(server, capabilities)

  Set device = Mobile.Device()

  device.FindElementByXPath("//*[@text=\"Samuel Clemens\"]").Touch()
  device.FindElementById("smartbear.example.orders:id/editButton").Touch()

  ' …
End Sub

Sub MyTest2()
  capabilities = "{""automationName"": ""UiAutomator2"",""bitbar_app"": ""123"",""bitbar_device"": ""LG Google Nexus 5 6.0.1"",""bitbar_findDevice"": ""false"",""bitbar_target"": ""android"",""deviceName"": ""Android Phone"",""newCommandTimeout"": ""600"",""platformName"": ""ANDROID""}"

  server = "https://appium.bitbar.com/wd/hub/"
  Call Mobile.ConnectDevice(server, capabilities)

  Set device = Mobile.Device()

  device.FindElementByXPath("//*[@text=\"Clare Jefferson\"]").Touch()
  device.FindElementById("smartbear.example.orders:id/editButton").Touch()

  ' …
End Sub

DelphiScript

procedure MyTest1();
var capabilities, server, device;
begin
  capabilities := '{"automationName": "UiAutomator2", "bitbar_app": "123", "bitbar_device": "Google Pixel 3a Android 10", "bitbar_findDevice": "false", "bitbar_target": "android", "deviceName": "Android Phone", "newCommandTimeout": "600", "platformName": "ANDROID"}';

  server := 'https://appium.bitbar.com/wd/hub/';
  Mobile.ConnectDevice(server, capabilities);

  device := Mobile.Device();

  device.FindElementByXPath('//*[@text=\"Samuel Clemens\"]').Touch();
  device.FindElementById('smartbear.example.orders:id/editButton').Touch();

  // …
end;

procedure MyTest2();
var capabilities, server, device;
begin
  capabilities := '{"automationName": "UiAutomator2", "bitbar_app": "123", "bitbar_device": "LG Google Nexus 5 6.0.1", "bitbar_findDevice": "false", "bitbar_target": "android", "deviceName": "Android Phone", "newCommandTimeout": "600", "platformName": "ANDROID"}';

  server := 'https://appium.bitbar.com/wd/hub/';
  Mobile.ConnectDevice(server, capabilities);

  device := Mobile.Device();

  device.FindElementByXPath('//*[@text=\"Clare Jefferson\"]').Touch();
  device.FindElementById('smartbear.example.orders:id/editButton').Touch();

  // …
end;

procedure Main();
var tests : Array[0..1];
begin
  tests[0] := 'Script|Unit1|MyTest1';
  tests[1] := 'Script|Unit1|MyTest2';
  Parallel.RunTests(tests);
end;

C++Script, C#Script

function Main()
{
  var tests = ['Script|Unit1|MyTest1', 'Script|Unit1|MyTest2'];
  Parallel["RunTests"](tests);
}

function MyTest1()
{
  var capabilities = {
    "automationName": "UiAutomator2",
    "bitbar_app": "123",
    "bitbar_device": "Google Pixel 3a Android 10",
    "bitbar_findDevice": "false",
    "bitbar_target": "android",
    "deviceName": "Android Phone",
    "newCommandTimeout": "600",
    "platformName": "ANDROID"
};

  var server = "https://appium.bitbar.com/wd/hub/";
  Mobile["ConnectDevice"](server, JSON["stringify"](capabilities));

  var device = Mobile["Device"]();

  device["FindElementByXPath"]("//*[@text=\"Samuel Clemens\"]")["Touch"]();
  device["FindElementById"]("smartbear.example.orders:id/editButton")["Touch"]();

  // …
}

function MyTest2()
{
  var capabilities = {
    "automationName": "UiAutomator2",
    "bitbar_app": "123",
    "bitbar_device": "LG Google Nexus 5 6.0.1",
    "bitbar_findDevice": "false",
    "bitbar_target": "android",
    "deviceName": "Android Phone",
    "newCommandTimeout": "600",
    "platformName": "ANDROID"
};

  var server = "https://appium.bitbar.com/wd/hub/";
  Mobile["ConnectDevice"](server, JSON["stringify"](capabilities));

  var device = Mobile["Device"]();

  device["FindElementByXPath"]("//*[@text=\"Clare Jefferson\"]")["Touch"]();
  device["FindElementById"]("smartbear.example.orders:id/editButton")["Touch"]();

  // …
}

Number of parallel tests

  • Your workstation computing capacity: its processor, RAM, and so on. The more parallel tests TestComplete is controlling, the more workstation resources are consumed.

  • You can limit the number of maximum tests running in parallel manually by using the LicenseCount parameter of the Parallel object’s methods or by using the Options > Engine > Parallel > Parallel test count option.

Capturing images during parallel test runs

TestComplete can capture images of the mobile device screen during test runs. These images can help understand the tested application behavior if the test fails. By default, capturing images during parallel runs is disabled. To command TestComplete to capture images, enable the Capture images during parallel test runs project option.

See Also

About Mobile Tests

Highlight search results