Running Flutter Tests for iOS

iOS simulator

iOS simulator test runs spend actual real device test time, the real iOS device is just idling while the simulator is running tests. iOS tests are run on the Mac OS. Flutter is pre-installed on virtual machines. Cocoapods need to be installed, it can be done using the run-tests.sh file.

Create a shell script file called run-tests.sh. This file will contain commands to execute the test.

  1. Unzip test files:

    unzip tests.zip
    
  2. Install cocoapods:

    sudo gem install cocoapods
    pod setup
    
  3. Run Flutter doctor:

    flutter doctor
    
  4. (Optional) Go to the flutter project directory (sample command):

    cd my_app
    
  5. (Optional) Clean the project folder (build/packages):

    flutter clean
    rm -r .packages
    rm pubspec.lock
    
  6. Install an iOS simulator:

    npm install -g ios-sim
    
  7. Start the simulator:

    ios-sim start --devicetypeid "iPhone-8"
    
  8. Find the simulator device from Flutter devices:

    flutter devices > devices.txt
    cat devices.txt | while read line
    do
    case "$line" in
    *simulator*)
      TEST_DEVICE=$line
      RUN_DEVICE=`echo ${TEST_DEVICE} | awk '{print $1, $2}'`
      echo $RUN_DEVICE > run_device.txt
      echo "device"
      cat run_device.txt
      ;;
      *)
      ;;
    esac
    done
    
  9. Run Flutter integration tests using the simulator (sample command):

    flutter drive -d "`cat run_device.txt`" --target=test_driver/main.dart
    
  10. (Optional) If test results (a file called TEST-all.xml) or screenshots should be displayed in the cloud UI:

    • Go back to the root directory (test result files are searched in the root directory):

      cd ...
      
    • Move TEST-all.xml to the root directory:

      mv my_app/TEST-all.xml TEST-all.xml
      
    • Create the screenshots directory and move screenshots there (sample command):

      mkdir -p screenshots
      mv /tmp/screenshots/test/ screenshots
      

Create a zip file containing the run-tests.sh file and your Flutter project directory. Note that the “run-tests.sh” file must be in the root of the zip file, otherwise, it is not found. Here is a sample command to create a zip file: zip -r ios-test-files run-tests.sh my_app. Upload this file to the cloud and use it as the “Use to run the test” file.

Create a Flutter iOS simulator test run

To create a Flutter iOS simulator test run in BitBar Cloud:

  1. Create a Appium iOS Server Side-type test run.

    iOS test run
    Appium iOS Server Side test run
  2. Upload the test zip file and the actual app .ipa file or a small sample app .ipa file (it needs to be able to get installed to the actual real iOS device, no simulator builds), the test does not use this app.

    Uploading files
  3. Select devices (the actually selected device is not used to run tests on).

  4. Set a test timeout period. Make sure it is long enough.

  5. Run tests.

iOS real device

BitBar Cloud supports running tests created with the following versions:

  • Flutter 1.22.6

  • Flutter 2.2.0

Before you start your testing, you need to build your iOS Flutter application locally, and then upload it to the BitBar cloud. You should also remember to move the app to the required directory during the test run.

Use steps from the Flutter user guide on how to integrate a Flutter module into your iOS project to add debug functionalities to the debug version of your app.

You can prepare the package in two ways.

Option 1

Create a shell script file called run-tests.sh. This file will contain commands to execute the test.

  1. Open your project in Xcode. In the product menu, select Build for > Testing.

  2. Locate your output from the compilation in the build/ios/Debug-iphoneos directory.

  3. Prepare the .ipa file using the following steps in the command line interface.

    cd build/ios/Debug-iphoneos
    mkdir Payload
    mv Runner.app Payload/
    zip --symlinks -qr application.ipa Payload
    
    
  4. Upload the app to the BitBar cloud with the default option Install on the device.

For detailed explanation, see how to create .ipa from .app.

Option 2

Run the integration test locally and archive your application:

  1. Test the execution example:

    flutter drive -v --target=test_driver/app.dart

    Note

    Remember that the target may be different and depends on your project naming convention.

  2. Archive your application using the following steps in the command-line interface.

    cd build/ios/iphoneos
    mkdir Payload
    mv Runner.app Payload/
    zip --symlinks -qr application.ipa Payload
    
    
  3. Upload the app to the BitBar cloud.

Prepare test.zip

Create a shell script file called run-tests.sh. This file contains commands to execute the test. Also, there are two Flutter versions pre-installed inside a VM. You should specify the version you want to use in the run-tests.shfile. See the examples of run-tests.sh:

  • Flutter 1.22.6

    Use the --no-build value to flutter drive command execution.

    #!/bin/bash#echo 'Extracting tests.zip...'
    yes | unzip tests.zip
    
    mv application.ipa application.zip
    
    unzip application.zip
    
    mv Payload/Runner.app Runner.app
    
    # make sure build folder exists
    mkdir -p build/ios/iphoneos
    
    # move app to build folder
    mv Runner.app build/ios/iphoneos/Runner.app
    
    echo 'starting flutter doctor ...'
    flutter-1.22.6 doctor
    
    echo 'Flutter devices ...'
    flutter-1.22.6 devices
    
    # Install flutter packages
    flutter-1.22.6 pub get
    
    flutter-1.22.6 drive -v --no-build --target=test_driver/app.dart
    
    
  • Flutter 2.2.0

    Use the --use-application-binary option to point to your Runner.app application:

    #!/bin/bash#echo 'Extracting tests.zip...'
    yes | unzip tests.zip
    
    mv application.ipa application.zip
    
    unzip application.zip
    
    mv Payload/Runner.app Runner.app
    
    # make sure build folder exists
    mkdir -p build/ios/iphoneos
    
    # move app to build folder
    mv Runner.app build/ios/iphoneos/Runner.app
    
    echo 'starting flutter doctor ...'
    flutter-2.2.0 doctor
    
    flutter-2.2.0 devices
    
    # Install flutter packages
    flutter-2.2.0 pub get
    
    flutter-2.2.0 drive -v --use-application-binary build/ios/iphoneos/Runner.app --target=test_driver/app.dart
    
    
  • Flutter 3.3.8

    The 3.3.8 Flutter version integration_test rebuilds an application from scratch. In the workspace, include all the needed dependencies to build the app for the chosen OS type. For more information, refer to the Flutter integration testing documentation page.

    #!/bin/bash#echo 'Extracting tests.zip...'
    yes | unzip tests.zip
    
    mv application.ipa application.zip
    
    unzip application.zip
    
    mv Payload/Runner.app Runner.app
    
    # make sure build folder exists
    mkdir -p build/ios/iphoneos
    
    # move app to build folder
    mv Runner.app build/ios/iphoneos/Runner.app
    
    echo 'starting flutter doctor ...'
    flutter-3.3.8 doctor
    
    flutter-3.3.8 devices
    
    # Install flutter packages
    flutter-3.3.8 pub get
    
    flutter-3.3.8 test integration_test/main.dart
    

Add the run-tests.sh to your Flutter project directory.

Create a tests.zip file containing run-tests.sh and the content of your Flutter project directory.

Important

Remember that therun-tests.sh file must be in the root of the zip file, otherwise, it is not found.

Ensure that you executed the flutter clean command before archive creation. See the example:

$ flutter clean
Cleaning Xcode workspace... 5.9s
Deleting build... 213ms
Deleting .dart_tool... 19ms
Deleting .packages... 2ms
Deleting Generated.xcconfig... 2ms
Deleting flutter_export_environment.sh... 2ms

The contents of the tests.zip archive should look something like this:

Test.zip contents

(Optional) View screenshots or TEST-all.xml file

If you want to display screenshots or the file TEST-all.xml in the cloud UI, follow the steps:

  1. Create the screenshots directory and move there your prepared screenshots.

  2. Generate TEST-all.xml based on the console output of flutter drive execution.

    Note

    You need changes in run-tests.sh starting from flutter drive execution, but remember that everything above the test execution must remain unchanged. Here is an example of how you can achieve it:

    # show logs on the cosnole output and additionally create log file, for parsing purposes.
    flutter-2.2.0 drive -v --use-application-binary build/ios/iphoneos/Runner.app --target=test_driver/app.dart 2>&1 | tee testconsole.log
    
    flutterExitCode=$?5
    while read line; do
    case "$line" in
    *setUpAll*)
    TEST_CLASS=$line
    ;;
    *)
    ;;
    esac
    case "'$line'" in
    *All*tests*passed*)
    TESTS_PASSED=true
    ;;
    *)
    ;;
    esac
    case "'$line'" in
    *Some*tests*failed*)
    TESTS_PASSED=false
    ;;
    *)
    ;;
    esac
    done <testconsole.log
    
    # parse results to TEST-all.xml
    CLASS_NAME_TEST=`echo ${TEST_CLASS} | awk -F"0m:" '/0m/{print $2}' | sed 's/[][]//g' | sed 's/(setUpAll)0m//'`
    
    if [ "$TESTS_PASSED" = "true" ]
    then
    sed -i.bu "s/default-testsuitename/$CLASS_NAME_TEST/" TEST-pass.xml
    sed -i.bu "s/default-testcase-class/$CLASS_NAME_TEST/" TEST-pass.xml
    sed -i.bu "s/default-testcase-1/All tests passed/" TEST-pass.xml
    sed -i.bu "s/default-testcase-result/All tests passed/" TEST-pass.xml
    mv TEST-pass.xml TEST-all.xml
    else
    sed -i.bu "s/default-testsuitename/$CLASS_NAME_TEST/" TEST-fail.xml
    sed -i.bu "s/default-testcase-class/$CLASS_NAME_TEST/" TEST-fail.xml
    sed -i.bu "s/default-testcase-1/Some tests failed/" TEST-fail.xml
    sed -i.bu "s/default-testcase-result/Some tests failed/" TEST-fail.xml
    for i in {1..4}
    do
    sed -i.bu "s/0/1/" TEST-fail.xml
    done
    mv TEST-fail.xml TEST-all.xml
    fi
    
    exit $flutterExitCode
    
    

Create a Flutter iOS real device test run

To create a Flutter iOS real device test run in BitBar cloud, follow these steps:

  1. Create the Flutter iOS test run.

    iOS test run
    Appium iOS Server Side test run
  2. Upload a test.zip file and the application.ipa file to the BitBar cloud and select them:

    Uploading files
  3. Select devices to run test on.

  4. Click Create and run automated test.

See Also

Publication date: