Running Tests With a Pipeline Script

Before you can use the Pipeline plugin, you need to enter your BitBar API key into Jenkins. To do this, from the Jenkins main page:

There is a convenient Jenkins pipeline syntax generator similar to the one that is used in the Jenkins Freestyle project:

Read more about all possible parameters below. Note that the example is not a complete Jenkins file - just a snippet.

stage('Device tests') {
  steps {
    runInCloud(
      // parameters to start test with...
      cloudUrl: "https://cloud.bitbar.com",
      credentialsId: "my-bitbar-cloud-credentials-01",
      projectId: "12345",
      deviceGroupId: "110",
      appPath: "/path/to/app.apk",
      testPath: "/path/to/tests.zip",
      frameworkId: 252,
      biometricInstrumentation: true,

      // ... don't just start test, also wait for results!
      waitForResultsBlock:
         [
             downloadScreenshots: true,
             resultsPath: 'results',
             testRunStateCheckMethod: 'API_CALL'
         ]
    )
  }
}

The list of parameters to go inside the runInCloud() method as displayed above.

appPath

String. The absolute or workspace relative path to the app executable (.apk or .ipa) that should be uploaded and targeted in this test run. For example, /path/to/app.ipa. Default value: the latest app uploaded for this project.

biometricInstrumentation

Boolean. Specifies whether to enable instrumentation on the app in order to bypass biometric authentication during testing.

cloudUrl

String. The BitBar Cloud URL this run should target. For example, https://cloud.bitbar.com. The default value: the cloud URL that was specified in the Jenkins /config page BitBar Cloud section.

credentialsId

String. The login credentials ID for the BitBar Cloud. The ID that you specified for your Username and password credentials (the guide on how to do this is above). For example, my~bitbar~cloud~credentials~01. The default value: use the credentials (email/password) that were specified in the Jenkins /config page BitBar Cloud section.

dataPath

String. The absolute or workspace-relative path to a .zip file that your test uses as additional data during a run. For example, /path/to/extra~data.zip.

deviceGroupId

String. Required. Used to specify which device(s) the test run will target. It's a number which you can find by visiting /#testing/devices in the BitBar Cloud and looking to the right where it says Device Groups. Under each "box" of devices, there is a number ID 12345 or such. That number, 12345 in this case, is the deviceGroupId. For example, 12345.

downloadScreenshots

Boolean. Specifies whether or not to download the screenshots (if any) after the test run completes. The default value: false; commands not to download screenshots (only logs) after the test run.

failBuildIfThisStepFailed

Boolean. Specifies whether the runInCloud build step should cause the whole Jenkins build to fail if there is an error. You can use this option to allow your Jenkins build to pass, even if an error occurred in the BitBarrunInCloud step. The default value: false; a build doesn't fail even if there is an error.

forceFinishAfterBreak

Boolean. Whether or not to abort the BitBar Cloud run after waiting for the results for a set amount of time after the run started. If set to false, the cloud run is allowed to finish on its own even after the Jenkins job stopped waiting for it. The default value: false; commands not to abort the cloud run after waiting for it to finish.

hookURL

String. The URL to which the BitBar Cloud will send a POST request when the test run finishes. For obvious reasons, this address needs to be accessible by the global internet or have the BitBar Cloud in the allow list. This can be used by enterprise systems that need to react to completed test runs. For example, https://example.com/bitbar~cloud~endpoint.

The default value: JENKINS_URL/plugin/testdroid~run~in~cloud/api/json/cloud~webhook. This default value can be used if waitForResults: true is specified, and testRunStateCheckMethod: "HOOK_URL" is set. The hook URL info is then used by the BitBar Run~in~Cloud plugin to determine that a test run finished.

The data sent in the POST request is a short UTF8 snippet, in the following format:

testRunId=12345&projectId=45678&status=SUCCESS&deviceRunId=123,456,789

keyValuePairs

String. Additional environment variables provided to the test run. This can be used for advanced logic in test cases where the test might behave differently depending on some value. Key-value pairs should be semicolon-separated (;), and the key and value should be colon-separated (:). For example, SOME_PARAM:some~value;ANOTHER_PARAM:another~value. The default value: empty string.

language

String. The locale (language) that should be set on the phone when the run starts. Has to be a locale supported by Android. For example, zh-TW; Chinese, Taiwan. The default value: en-US; English, United States.

notificationEmail

String. The email address that'll receive notifications about the test run when it finishes. Emails are sent by the BitBar Cloud, not by Jenkins. For example, [email protected]. The default value: the BitBar Cloud account email that's being used for this Jenkins build.

notificationEmailType

String. Defines what types of events should trigger an email to the email address specified by notificationEmail.

Possible values:

projectId

String. Required. Defines which BitBar Cloud project this test run should be included in. By extension, this also selects the target platform Android/iOS. This value is a number that you can find by going to Projects in the BitBar Cloud and reading the URL when you select a project. For instance, if the URL in the browser is https://cloud.bitbar.com/#testing/projects/12345, then the project ID is 12345. For example, 12345.

resultsPath

String. The absolute or workspace-relative path to a folder where results should be downloaded after the run completes. For example, ./my~results. The default value: Jenkins workspace.

scheduler

String. Defines in what order tests should be run in the BitBar Cloud, in case multiple devices are targeted.

Possible values:

screenshotsDirectory

String. The location on device from where test screenshots should be pulled after a test run. For example, /sdcard/screenshots. The default value: /sdcard/test~screenshots.

testCasesSelect

String. Value specifies a way (package/class) to narrow down the test cases that should be run. This parameter applies only to projects of the “Android Instrumentation” type - Espresso tests.

Possible values:

testCasesValue

String. If testCasesSelect is specified, this parameter defines the package/class by which to filter test cases. This parameter applies only to projects of the “Android Instrumentation” type - Espresso tests. For example, com.mycompany.espressotests.AuthenticationTest. An example of a class that might be defined in your tests.

testPath

String. The absolute or workspace-relative path to a test package that should be uploaded and run in the BitBar Cloud. Most often, a .zip or .apk file. For AppCrawler projects, this parameter should be omitted. For example, /path/to/my/tests.zip. The default value: use the latest test package that was uploaded to this project.

testRunName

String. Custom name for the test run in the BitBar Cloud. For example, My test run. By default, the cloud will assign a name like Test Run 45, depending on the number of previous runs in the cloud project.

testRunner

String. The test runner class that will execute Espresso tests. See InstrumentationTestRunner for details. For example, android.support.test.runner.AndroidJUnitRunner. The default value: android.test.InstrumentationTestRunner.

testRunStateCheckMethod

String. Specifies the method by which the plugin will check if the BitBar Cloud run has finished.

Possible values:

testTimeout

String. A value in seconds, how long the cloud test can run before it is forced to shut down. For example, 3600 (one hour). The default value: 600 (10 minutes).

waitForResults

Array. Specifies whether or not the build step should wait for results from the test run, or merely launch it. If present, the jenkins build step won't finish until the BitBar Cloud tests do.

waitForResultsTimeout

String. A value in seconds, specifies how long the build step should wait for cloud results. For example, 3600.

withAnnotation

String. Runs only test cases that have a certain Java annotation specified. This parameter applies only to projects of the “Android Instrumentation” type - Espresso tests. For example, com.android.foo.MyAnnotation. Doesn't filter by annotations but instead runs all test cases.

withoutAnnotation

String. Does not run test cases that have a certain Java annotation specified. This parameter applies only to projects of the “Android Instrumentation” type - Espresso tests. For example, com.android.foo.MyAnnotation. Doesn't filter by annotations but instead runs all test cases.