Last modified on October 21, 2020

This guide contains instructions for using the BitBar Run-in-Cloud plugin from a Pipeline script (Jenkinsfile). For instructions on how to use the UI build step in the same plugin, see Running Tests via the Jenkins Plugin UI.

Credentials setup

Before you can use the Pipeline plugin, you need to save your BitBar Cloud login details into Jenkins. To do this, from the Jenkins main page:

  1. Navigate to Credentials > System > Global credentials > Add Credentials.

  2. Select Username with password from the Kind dropdown.

  3. Enter your BitBar Cloud login details. Note that Username should be the BitBar Cloud email you use to login.

Credentials setup

Click the image to enlarge it.

Pipeline script - snippet generator

There is a convenient Jenkins pipeline syntax generator similar to the one that is used in the Jenkins Freestyle project:

  1. Create New Item. Choose Pipeline.

  2. Navigate to Configure.

  3. Click Pipeline Syntax.

    Pipeline syntax link

    Click the image to enlarge it.

  4. Choose Sample Step > runInCloud: Start a run in BitBar Cloud.

    Pipeline: selecting snippet generator

    Click the image to enlarge it.

  5. Set up parameters like in the Freestyle project.

    Pipeline: snippet generator

    Click the image to enlarge it.

  6. Use the generated snippet in your pipeline.

Pipeline script - simple use case

Read more about all possible parameters below. Note that the example is not a complete Jenkins file - just a snippet.

stage('Device tests') {
  steps {
    runInCloud(
      // parameters to start test with...
      cloudUrl: "https://cloud.bitbar.com",
      credentialsId: "my-bitbar-cloud-credentials-01",
      projectId: "12345",
      deviceGroupId: "110",
      appPath: "/path/to/app.apk",
      testPath: "/path/to/tests.zip",
      frameworkId: 252,

      // ... don't just start test, also wait for results!
      waitForResultsBlock:
         [
             downloadScreenshots: true,
             resultsPath: 'results',
             testRunStateCheckMethod: 'API_CALL'
         ]
    )
  }
}

Parameters

The list of parameters to go inside the runInCloud() method shown above.

appPath

String. The absolute or workspace relative path to the app executable (.apk or .ipa) that should be uploaded and targeted in this test run. For example, /path/to/app.ipa. Default value: the latest app uploaded for this project.

cloudUrl

String. The BitBar Cloud URL this run should target. For example, https://cloud.bitbar.com. The default value: the cloud URL that was specified in the Jenkins /config page BitBar Cloud section.

Also specify credentialsId if you specify this option.

credentialsId

String. The login credentials ID for the BitBar Cloud. The ID that you specified for your Username and password credentials (the guide on how to do this is above). For example, my~bitbar~cloud~credentials~01. The default value: use the credentials (email/password) that were specified in the Jenkins /config page BitBar Cloud section.

dataPath

String. The absolute or workspace-relative path to a .zip file that your test uses as additional data during a run. For example, /path/to/extra~data.zip.

Note: Do not use an additional data package.

deviceGroupId

String. Required. Used to specify which device(s) the test run will target. It's a number which you can find by visiting /#testing/devices in the BitBar Cloud and looking to the right where it says Device Groups. Under each "box" of devices, there is a number ID 12345 or such. That number, 12345 in this case, is the deviceGroupId. For example, 12345.

downloadScreenshots

Boolean. Specifies whether or not to download the screenshots (if any) after the test run completes. The default value: false; commands not to download screenshots (only logs) after the test run.

Also specify waitForResults: true if you set this option to true, otherwise it won't have any effect.

failBuildIfThisStepFailed

Boolean. Specifies whether the runInCloud build step should cause the whole Jenkins build to fail if there is an error. You can use this option to allow your Jenkins build to pass, even if an error occurred in the BitBarrunInCloud step. The default value: false; a build doesn't fail even if there is an error.

This has nothing to do with whether or not the tests in the cloud pass; it's only concerned with whether the step executes successfully or encounters exceptions.

forceFinishAfterBreak

Boolean. Whether or not to abort the BitBar Cloud run after waiting for the results for a set amount of time after the run started. If set to false, the cloud run is allowed to finish on its own even after the Jenkins job stopped waiting for it. The default value: false; commands not to abort the cloud run after waiting for it to finish.

If true, requires waitForResults: true and waitForResultsTimeout to be set.

hookURL

String. The URL to which the BitBar Cloud will send a POST request when the test run finishes. For obvious reasons, this address needs to be accessible by the global internet or have the BitBar Cloud in the allow list. This can be used by enterprise systems that need to react to completed test runs. For example, https://example.com/bitbar~cloud~endpoint.

The default value: JENKINS_URL/plugin/testdroid~run~in~cloud/api/json/cloud~webhook. This default value can be used if waitForResults: true is specified, and testRunStateCheckMethod: "HOOK_URL" is set. The hook URL info is then used by the BitBar Run~in~Cloud plugin to determine that a test run finished.

The data sent in the POST request is a short UTF8 snippet, in the following format:

testRunId=12345&projectId=45678&status=SUCCESS&deviceRunId=123,456,789

keyValuePairs

String. Additional environment variables provided to the test run. This can be used for advanced logic in test cases where the test might behave differently depending on some value. Key-value pairs should be semicolon-separated (;), and the key and value should be colon-separated (:). For example, SOME_PARAM:some~value;ANOTHER_PARAM:another~value. The default value: empty string.

language

Android only

String. The locale (language) that should be set on the phone when the run starts. Has to be a locale supported by Android. For example, zh-TW; Chinese, Taiwan. The default value: en-US; English, United States.

notificationEmail

String. The email address that'll receive notifications about the test run when it finishes. Emails are sent by the BitBar Cloud, not by Jenkins. For example, firstname.lastname@example.com. The default value: the BitBar Cloud account email that's being used for this Jenkins build.

notificationEmailType

String. Defines what types of events should trigger an email to the email address specified by notificationEmail.

Possible values:

  • ALWAYS – Sends an email after the test run, regardless of whether it passed or failed.

  • ON_FAILURE – Sends an email when a test run fails.

  • Empty string – The default value. Does not send emails.

projectId

String. Required. Defines which BitBar Cloud project this test run should be included in. By extension, this also selects the target platform Android/iOS. This value is a number that you can find by going to Projects in the BitBar Cloud and reading the URL when you select a project. For instance, if the URL in the browser is https://cloud.bitbar.com/#testing/projects/12345, then the project ID is 12345. For example, 12345.

resultsPath

String. The absolute or workspace-relative path to a folder where results should be downloaded after the run completes. For example, ./my~results. The default value: Jenkins workspace.

Requires waitForResults: true to be specified, or this field won't have any effect.

scheduler

String. Defines in what order tests should be run in the BitBar Cloud, in case multiple devices are targeted.

Possible values:

  • PARALLEL – The default value. Tests will execute on multiple devices in parallel.

  • SERIAL – Tests will run sequentially on devices, one device at a time.

  • SINGLE – The cloud will pick one device from the group and run the tests only on that device.

screenshotsDirectory

Android only

String. The location on device from where test screenshots should be pulled after a test run. For example, /sdcard/screenshots. The default value: /sdcard/test~screenshots.

testCasesSelect

Android only

String. Value specifies a way (package/class) to narrow down the test cases that should be run. This parameter applies only to projects of the “Android Instrumentation” type - Espresso tests.

Requires testCasesValue to be set to something meaningful.

Possible values:

  • PACKAGE – Default. Only executes test cases defined in a certain package.

  • CLASS – Only executes test cases defined in a certain class.

testCasesValue

String. If testCasesSelect is specified, this parameter defines the package/class by which to filter test cases. This parameter applies only to projects of the “Android Instrumentation” type - Espresso tests. For example, com.mycompany.espressotests.AuthenticationTest. An example of a class that might be defined in your tests.

testPath

String. The absolute or workspace-relative path to a test package that should be uploaded and run in the BitBar Cloud. Most often, a .zip or .apk file. For AppCrawler projects, this parameter should be omitted. For example, /path/to/my/tests.zip. The default value: use the latest test package that was uploaded to this project.

testRunName

String. Custom name for the test run in the BitBar Cloud. For example, My test run. By default, the cloud will assign a name like Test Run 45, depending on the number of previous runs in the cloud project.

testRunner

Android only

String. The test runner class that will execute Espresso tests. See InstrumentationTestRunner for details. For example, android.support.test.runner.AndroidJUnitRunner. The default value: android.test.InstrumentationTestRunner.

testRunStateCheckMethod

String. Specifies the method by which the plugin will check if the BitBar Cloud run has finished.

Requires waitForResults: true to be set.

Possible values:

  • API_CALL – The plugin will continuously pull the BitBar Cloud to see if the run has finished.

  • HOOK_URL – Read more about this option in the hookURL parameter description.

testTimeout

String. A value in seconds, how long the cloud test can run before it is forced to shut down. For example, 3600 (one hour). The default value: 600 (10 minutes).

waitForResults

Array. Specifies whether or not the build step should wait for results from the test run, or merely launch it. If present, the jenkins build step won't finish until the BitBar Cloud tests do.

waitForResultsTimeout

String. A value in seconds, specifies how long the build step should wait for cloud results. For example, 3600.

Requires waitForResults: true to be set. Also, unless forceFinishAfterBreak: true is set, setting waitForResultsTimeout won't abort the test run, it will just silently stop waiting.

withAnnotation

Android only

String. Runs only test cases that have a certain Java annotation specified. This parameter applies only to projects of the “Android Instrumentation” type - Espresso tests. For example, com.android.foo.MyAnnotation. Doesn't filter by annotations but instead runs all test cases.

withoutAnnotation

Android only

String. Does not run test cases that have a certain Java annotation specified. This parameter applies only to projects of the “Android Instrumentation” type - Espresso tests. For example, com.android.foo.MyAnnotation. Doesn't filter by annotations but instead runs all test cases.

See Also

Jenkins Run-in-Cloud Plugin
Integrations

Highlight search results