Last modified on October 21, 2020

Test planning checklist

Plan your testing process before you start testing:

  • Define the test goal

    Decide which application functionality you want to test. The clearer the goal and the simpler the test, the better. Large tests that deal with various aspects of the application’s behavior are hard to create and maintain. We recommend that you create a simple test that will test only certain functionality of your application.

  • Plan testing steps

    Decide which actions the test will perform. The testing steps depend on the test purpose and the nature of the application under test. Testing steps may include actions that prepare the application for the test (that is, they put the application to some initial state), or that enter some data in the application.

  • Plan checking actions

    Usually, the actions a user performs over an application cause some changes in the application. These can be visual changes (new values in forms, new windows or dialogs are displayed), or changes in the application's data (new objects, database entries, new or deleted files). You should decide what criteria should be used to check whether a test has passed or failed.

  • Log the results

    Make sure to save the full test results to a file or track the latest test run data. See Analyzing Results for details.

App requirements

The maximum app size that a user can upload is 500 MB.

Create iOS app package

Regardless of whether testing with XCTest or XCUITest, the application under test needs to be built and imported to BitBar Testing cloud. The below steps present how to create the appropriate Ad Hoc Distribution IPA build.

Create test IPA

  1. Open your project in Xcode. In the Xcode Product menu, select Build for, and then Testing.

    Creating a test .ipa
  2. The application still needs to be packaged into an IPA zip. For this the location of the app is needed. In Xcode, select the built .app, right click, and select Show it in Finder.

    Creating a test .ipa
  3. In Finder, right click the .app file and copy its location for later use. This location does not change from build to build so this step needs to be done only once.

    Creating a test .ipa
  4. The IPA package needed for XCTest in BitBar Testing is a zip package and its creation can be automated and made part of a CI integration. Open a terminal and create the IPA.

    $ mkdir /tmp/Payload
    $ cd /tmp/Payload
    $ cp -r /Users/username/Desktop/Build/Products/Debug-iphoneos/LocalizationDemo.app . # app path can be pasted
    $ cd ..
    $ zip --symlinks -qr "LocalizationDemo.ipa" Payload
    $ ls -lrt LocalizationDemo.ipa
    -rw-r--r-- 1 username staff 0 Dec 16 12:42 LocalizationDemo.ipa

Validate build

The easiest way to validate your build is using BitBar AppCrawler run.

  1. Go to AppCrawler option on BitBar Cloud.

  2. Upload your .ipa file.

  3. Select the devices.

  4. Run the test.

Symbolication

Symbolication is the process of translating iOS application crash logs to a human readable file using the dSYM file. dSYM or “debug symbols file” is a file generated during build of the application when the Strip Debug Symbols setting in Xcode is set to true. Without the dSYM file, any crash logs generated by a failing application will be unreadable by end users. If the dSYM file is part of the uploaded application package, then BitBar automatically processes the crash logs for the following frameworks: iOS XCUITEST, iOS Appium Client Side, iOS Appium Server Side.

The crash report is collected from the iOS device, it is symbolicated and sent to BitBar cloud - where it is made available via UI or API as file named symbolicated.crash.

Good to know

If building for iPhone 5 or iPhone 5C (having Armv7 32 bit processor) devices then an additional step is needed before creating the build. Starting from Xcode7 onwards armv7s is no more part of the default$(ARCHS_STANDARD) and so should be added as target build architecture.

Xcode Armv7 build option

Click the image to enlarge it.

By default Xcode sets the iOS target version to the latest available iOS version for new projects. For testing against devices running previous iOS versions the app and the tests need to be built for these separately.

For example, the project level Deployment Target can be set to 9.3.

Deployment target version

Click the image to enlarge it.

Then the same needs to be done for the app target (Calculator in these example screenshots)

Deployment info

Click the image to enlarge it.

and still for the tests (CalculatorUITests).

iOS Deployment Target

See Also

Testing With BitBar

Highlight search results