Running Tests With a Pipeline Script
This guide contains instructions for using the BitBar Run-in-Cloud plugin from a Pipeline script (Jenkinsfile). For instructions on how to use the UI build step in the same plugin, see Running Tests via the Jenkins Plugin UI.
Credentials setup
Before you can use the Pipeline plugin, you need to enter your BitBar API key into Jenkins. To do this, from the Jenkins main page:
Navigate to Credentials > System > Global credentials > Add Credentials.
Select BitBar API key from the Kind dropdown.
Enter your API key and ID.
Click Enter.
![]() |
Pipeline script - snippet generator
There is a convenient Jenkins pipeline syntax generator similar to the one that is used in the Jenkins Freestyle project
:
Create New Item. Choose Pipeline.
Navigate to Configure.
Click Pipeline Syntax.
Choose Sample Step > runInCloud: Start a run in BitBar Cloud.
Set up parameters like in the
Freestyle project
.Use the generated snippet in your pipeline.
Pipeline script - simple use case
Read more about all possible parameters below. Note that the example is not a complete Jenkins file - just a snippet.
stage('Device tests') { steps { runInCloud( // parameters to start test with... cloudUrl: "https://cloud.bitbar.com", credentialsId: "my-bitbar-cloud-credentials-01", projectId: "12345", deviceGroupId: "110", appPath: "/path/to/app.apk", testPath: "/path/to/tests.zip", frameworkId: 252, biometricInstrumentation: true, // ... don't just start test, also wait for results! waitForResultsBlock: [ downloadScreenshots: true, resultsPath: 'results', testRunStateCheckMethod: 'API_CALL' ] ) } }
Parameters
The list of parameters to go inside the runInCloud() method as displayed above.
appPath
String. The absolute or workspace relative path to the app executable (.apk
or .ipa
) that should be uploaded and targeted in this test run. For example, /path/to/app.ipa
. Default value: the latest app uploaded for this project.
biometricInstrumentation
Boolean. Specifies whether to enable instrumentation on the app in order to bypass biometric authentication during testing.
Important
Value is set to false when parameters do not pass.
cloudUrl
String. The BitBar Cloud URL this run should target. For example, https://cloud.bitbar.com
. The default value: the cloud URL that was specified in the Jenkins /config page BitBar Cloud section.
Important
Also specify credentialsId
if you specify this option.
credentialsId
String. The login credentials ID for the BitBar Cloud. The ID
that you specified for your Username and password credentials (the guide on how to do this is above). For example, my~bitbar~cloud~credentials~01
. The default value: use the credentials (email/password) that were specified in the Jenkins /config page BitBar Cloud section.
dataPath
String. The absolute or workspace-relative path to a .zip
file that your test uses as additional data during a run. For example, /path/to/extra~data.zip
.
Note
Do not use an additional data package.
deviceGroupId
String. Required. Used to specify which device(s) the test run will target. It's a number which you can find by visiting /#testing/devices
in the BitBar Cloud and looking to the right where it says Device Groups. Under each "box" of devices, there is a number or such. That number, 12345 in this case, is the deviceGroupId
. For example, 12345
.
downloadScreenshots
Boolean. Specifies whether or not to download the screenshots (if any) after the test run completes. The default value: false
; commands not to download screenshots (only logs) after the test run.
Important
Also specify waitForResults: true
if you set this option to true
, otherwise it won't have any effect.
failBuildIfThisStepFailed
Boolean. Specifies whether the runInCloud
build step should cause the whole Jenkins build to fail if there is an error. You can use this option to allow your Jenkins build to pass, even if an error occurred in the BitBarrunInCloud
step. The default value: false
; a build doesn't fail even if there is an error.
Important
This has nothing to do with whether or not the tests in the cloud pass; it's only concerned with whether the step executes successfully or encounters exceptions.
forceFinishAfterBreak
Boolean. Whether or not to abort the BitBar Cloud run after waiting for the results for a set amount of time after the run started. If set to false
, the cloud run is allowed to finish on its own even after the Jenkins job stopped waiting for it. The default value: false
; commands not to abort the cloud run after waiting for it to finish.
Important
If true
, requires waitForResults: true
and waitForResultsTimeout
to be set.
hookURL
String. The URL to which the BitBar Cloud will send a POST request when the test run finishes. For obvious reasons, this address needs to be accessible by the global internet or have the BitBar Cloud in the allow list. This can be used by enterprise systems that need to react to completed test runs. For example, https://example.com/bitbar~cloud~endpoint
.
The default value: JENKINS_URL/plugin/testdroid~run~in~cloud/api/json/cloud~webhook
. This default value can be used if waitForResults: true
is specified, and testRunStateCheckMethod: "HOOK_URL"
is set. The hook URL info is then used by the BitBar Run~in~Cloud plugin to determine that a test run finished.
The data sent in the POST request is a short UTF8 snippet, in the following format:
testRunId=12345&projectId=45678&status=SUCCESS&deviceRunId=123,456,789
keyValuePairs
String. Additional environment variables provided to the test run. This can be used for advanced logic in test cases where the test might behave differently depending on some value. Key-value pairs should be semicolon-separated (;
), and the key and value should be colon-separated (:
). For example, SOME_PARAM:some~value;ANOTHER_PARAM:another~value
. The default value: empty string.
language
Important
Android only
String. The locale (language) that should be set on the phone when the run starts. Has to be a locale supported by Android. For example, zh-TW
; Chinese, Taiwan. The default value: en-US
; English, United States.
notificationEmail
String. The email address that'll receive notifications about the test run when it finishes. Emails are sent by the BitBar Cloud, not by Jenkins. For example, [email protected]
. The default value: the BitBar Cloud account email that's being used for this Jenkins build.
notificationEmailType
String. Defines what types of events should trigger an email to the email address specified by notificationEmail
.
Possible values:
ALWAYS
– Sends an email after the test run, regardless of whether it passed or failed.ON_FAILURE
– Sends an email when a test run fails.Empty string – The default value. Does not send emails.
projectId
String. Required. Defines which BitBar Cloud project this test run should be included in. By extension, this also selects the target platform Android/iOS. This value is a number that you can find by going to Projects in the BitBar Cloud and reading the URL when you select a project. For instance, if the URL in the browser is https://cloud.bitbar.com/#testing/projects/12345
, then the project ID is 12345. For example, 12345
.
resultsPath
String. The absolute or workspace-relative path to a folder where results should be downloaded after the run completes. For example, ./my~results
. The default value: Jenkins workspace.
Important
Requires waitForResults: true
to be specified, or this field won't have any effect.
scheduler
String. Defines in what order tests should be run in the BitBar Cloud, in case multiple devices are targeted.
Possible values:
PARALLEL
– The default value. Tests will execute on multiple devices in parallel.SERIAL
– Tests will run sequentially on devices, one device at a time.SINGLE
– The cloud will pick one device from the group and run the tests only on that device.
screenshotsDirectory
Important
Android only
String. The location on device from where test screenshots should be pulled after a test run. For example, /sdcard/screenshots
. The default value: /sdcard/test~screenshots
.
testCasesSelect
Important
Android only
String. Value specifies a way (package/class) to narrow down the test cases that should be run. This parameter applies only to projects of the “Android Instrumentation” type - Espresso tests.
Important
Requires testCasesValue
to be set to something meaningful.
Possible values:
PACKAGE
– Default. Only executes test cases defined in a certain package.CLASS
– Only executes test cases defined in a certain class.
testCasesValue
String. If testCasesSelect
is specified, this parameter defines the package/class by which to filter test cases. This parameter applies only to projects of the “Android Instrumentation” type - Espresso tests. For example, com.mycompany.espressotests.AuthenticationTest
. An example of a class that might be defined in your tests.
testPath
String. The absolute or workspace-relative path to a test package that should be uploaded and run in the BitBar Cloud. Most often, a .zip
or .apk
file. For AppCrawler projects, this parameter should be omitted. For example, /path/to/my/tests.zip
. The default value: use the latest test package that was uploaded to this project.
testRunName
String. Custom name for the test run in the BitBar Cloud. For example, My test run
. By default, the cloud will assign a name like Test Run 45
, depending on the number of previous runs in the cloud project.
testRunner
Important
Android only
String. The test runner class that will execute Espresso tests. See InstrumentationTestRunner for details. For example, android.support.test.runner.AndroidJUnitRunner
. The default value: android.test.InstrumentationTestRunner
.
testRunStateCheckMethod
String. Specifies the method by which the plugin will check if the BitBar Cloud run has finished.
Important
Requires waitForResults: true
to be set.
Possible values:
API_CALL
– The plugin will continuously pull the BitBar Cloud to see if the run has finished.HOOK_URL
– Read more about this option in thehookURL
parameter description.
testTimeout
String. A value in seconds, how long the cloud test can run before it is forced to shut down. For example, 3600
(one hour). The default value: 600
(10 minutes).
waitForResults
Array. Specifies whether or not the build step should wait for results from the test run, or merely launch it. If present, the jenkins build step won't finish until the BitBar Cloud tests do.
waitForResultsTimeout
String. A value in seconds, specifies how long the build step should wait for cloud results. For example, 3600
.
Important
Requires waitForResults: true
to be set. Also, unless forceFinishAfterBreak: true
is set, setting waitForResultsTimeout
won't abort the test run, it will just silently stop waiting.
withAnnotation
Important
Android only
String. Runs only test cases that have a certain Java annotation specified. This parameter applies only to projects of the “Android Instrumentation” type - Espresso tests. For example, com.android.foo.MyAnnotation
. Doesn't filter by annotations but instead runs all test cases.
withoutAnnotation
Important
Android only
String. Does not run test cases that have a certain Java annotation specified. This parameter applies only to projects of the “Android Instrumentation” type - Espresso tests. For example, com.android.foo.MyAnnotation
. Doesn't filter by annotations but instead runs all test cases.