Running Flutter tests for iOS

iOS simulator test runs spend actual real device test time, the real iOS device is just idling while the simulator is running tests. iOS tests are run on the Mac OS. Flutter is pre-installed on virtual machines. Cocoapods need to be installed, it can be done using the run-tests.sh file.

Create a shell script file called run-tests.sh. This file will contain commands to execute the test.

1. Unzip test files:

   unzip tests.zip

2. Install cocoapods:

   sudo gem install cocoapods
   pod setup

3. Run Flutter doctor:

   flutter doctor

4. (Optional) Go to the flutter project directory (sample command):

   cd my_app

5. (Optional) Clean the project folder (build/packages):

   flutter clean
   rm -r .packages
   rm pubspec.lock

6. Install an iOS simulator:

   npm install -g ios-sim

7. Start the simulator:

   ios-sim start --devicetypeid "iPhone-8"

8. Find the simulator device from Flutter devices:

   flutter devices > devices.txt
   cat devices.txt | while read line
   do
   case "$line" in
   *simulator*)
     TEST_DEVICE=$line
     RUN_DEVICE=`echo ${TEST_DEVICE} | awk '{print $1, $2}'`
     echo $RUN_DEVICE > run_device.txt
     echo "device"
     cat run_device.txt
     ;;
     *)
     ;;
   esac
   done

9. Run Flutter integration tests using the simulator (sample command):

   flutter drive -d "`cat run_device.txt`" --target=test_driver/main.dart

10. (Optional) If test results (a file called TEST-all.xml) or screenshots should be displayed in the cloud UI:

    • Go back to the root directory (test result files are searched in the root directory):

      cd ...

    • Move TEST-all.xml to the root directory:

      mv my_app/TEST-all.xml TEST-all.xml

    • Create the screenshots directory and move screenshots there (sample command):

      mkdir -p screenshots
      mv /tmp/screenshots/test/ screenshots

Create a zip file containing the run-tests.sh file and your Flutter project directory. Note that the “run-tests.sh” file must be in the root of the zip file, otherwise, it is not found. Here is a sample command to create a zip file: zip -r ios-test-files run-tests.sh my_app. Upload this file to the cloud and use it as the “Use to run the test” file.

To create a Flutter iOS simulator test run in BitBar Cloud:

1. Create a Appium iOS Server Side-type test run.

   
   

2. Upload the test zip file and the actual app .ipa file or a small sample app .ipa file (it needs to be able to get installed to the actual real iOS device, no simulator builds), the test does not use this app.

   

   Click the image to enlarge it.

3. Select devices (the actually selected device is not used to run tests on).

4. Set a test timeout period. Make sure it is long enough.

5. Run tests.

BitBar Cloud supports running tests created with the following versions:

• Flutter 1.22.6

• Flutter 2.2.0

Before you start your testing, you need to build your iOS Flutter application locally, and then upload it to the BitBar cloud. You should also remember to move the app to the required directory during the test run.

Use steps from the Flutter user guide on how to integrate a Flutter module into your iOS project to add debug functionalities to the debug version of your app.

You can prepare the package in two ways.

Option 1

Option 2

Prepare test.zip

Create a shell script file called run-tests.sh. This file contains commands to execute the test. Also, there are two Flutter versions pre-installed inside a VM. You should specify the version you want to use in the run-tests.shfile. See the examples of run-tests.sh:

• Flutter 1.22.6

  Use the --no-build value to flutter drive command execution.

  #!/bin/bash
  
  #echo 'Extracting tests.zip...'
  yes | unzip tests.zip
  
  mv application.ipa application.zip
  
  unzip application.zip
  
  mv Payload/Runner.app Runner.app
  
  # make sure build folder exists
  mkdir -p build/ios/iphoneos
  
  # move app to build folder
  mv Runner.app build/ios/iphoneos/Runner.app
  
  echo 'starting flutter doctor ...'
  flutter-1.22.6 doctor
  
  echo 'Flutter devices ...'
  flutter-1.22.6 devices
  
  # Install flutter packages
  flutter-1.22.6 pub get
  
  flutter-1.22.6 drive -v --no-build --target=test_driver/app.dart
  

• Flutter 2.2.0

  Use the --use-application-binary option to point to your Runner.app application:

  #!/bin/bash
  
  #echo 'Extracting tests.zip...'
  yes | unzip tests.zip
  
  mv application.ipa application.zip
  
  unzip application.zip
  
  mv Payload/Runner.app Runner.app
  
  # make sure build folder exists
  mkdir -p build/ios/iphoneos
  
  # move app to build folder
  mv Runner.app build/ios/iphoneos/Runner.app
  
  echo 'starting flutter doctor ...'
  flutter-2.2.0 doctor
  
  flutter-2.2.0 devices
  
  # Install flutter packages
  flutter-2.2.0 pub get
  
  flutter-2.2.0 drive -v --use-application-binary build/ios/iphoneos/Runner.app --target=test_driver/app.dart
  

• Flutter 3.3.8

  The 3.3.8 Flutter version integration_test rebuilds an application from scratch. In the workspace, include all the needed dependencies to build the app for the chosen OS type. For more information, refer to the Flutter integration testing documentation page.

  #!/bin/bash
  
  #echo 'Extracting tests.zip...'
  yes | unzip tests.zip
  
  mv application.ipa application.zip
  
  unzip application.zip
  
  mv Payload/Runner.app Runner.app
  
  # make sure build folder exists
  mkdir -p build/ios/iphoneos
  
  # move app to build folder
  mv Runner.app build/ios/iphoneos/Runner.app
  
  echo 'starting flutter doctor ...'
  flutter-3.3.8 doctor
  
  flutter-3.3.8 devices
  
  # Install flutter packages
  flutter-3.3.8 pub get
  
  flutter-3.3.8 test integration_test/main.dart

Add the run-tests.sh to your Flutter project directory.

Create a tests.zip file containing run-tests.sh and the content of your Flutter project directory.

Remember that therun-tests.sh file must be in the root of the zip file, otherwise, it is not found. Ensure that you executed the flutter clean command before archive creation. See the example: $ flutter clean Cleaning Xcode workspace... 5.9s Deleting build... 213ms Deleting .dart_tool... 19ms Deleting .packages... 2ms Deleting Generated.xcconfig... 2ms Deleting flutter_export_environment.sh... 2ms

The contents of the tests.zip archive should look something like this:

Click the image to enlarge it.

(Optional) View screenshots or TEST-all.xml file

If you want to display screenshots or the file TEST-all.xml in the cloud UI, follow the steps:

1. Create the screenshots directory and move there your prepared screenshots.

2. Generate TEST-all.xml based on the console output of flutter drive execution.

   Note:

   You need changes in run-tests.sh starting from flutter drive execution, but remember that everything above the test execution must remain unchanged. Here is an example of how you can achieve it:

   # show logs on the cosnole output and additionally create log file, for parsing purposes.
   flutter-2.2.0 drive -v --use-application-binary build/ios/iphoneos/Runner.app --target=test_driver/app.dart 2>&1 | tee testconsole.log
   
   flutterExitCode=$?5
   while read line; do
   case "$line" in
   *setUpAll*)
   TEST_CLASS=$line
   ;;
   *)
   ;;
   esac
   case "'$line'" in
   *All*tests*passed*)
   TESTS_PASSED=true
   ;;
   *)
   ;;
   esac
   case "'$line'" in
   *Some*tests*failed*)
   TESTS_PASSED=false
   ;;
   *)
   ;;
   esac
   done <testconsole.log
   
   # parse results to TEST-all.xml
   CLASS_NAME_TEST=`echo ${TEST_CLASS} | awk -F"0m:" '/0m/{print $2}' | sed 's/[][]//g' | sed 's/(setUpAll)0m//'`
   
   if [ "$TESTS_PASSED" = "true" ]
   then
   sed -i.bu "s/default-testsuitename/$CLASS_NAME_TEST/" TEST-pass.xml
   sed -i.bu "s/default-testcase-class/$CLASS_NAME_TEST/" TEST-pass.xml
   sed -i.bu "s/default-testcase-1/All tests passed/" TEST-pass.xml
   sed -i.bu "s/default-testcase-result/All tests passed/" TEST-pass.xml
   mv TEST-pass.xml TEST-all.xml
   else
   sed -i.bu "s/default-testsuitename/$CLASS_NAME_TEST/" TEST-fail.xml
   sed -i.bu "s/default-testcase-class/$CLASS_NAME_TEST/" TEST-fail.xml
   sed -i.bu "s/default-testcase-1/Some tests failed/" TEST-fail.xml
   sed -i.bu "s/default-testcase-result/Some tests failed/" TEST-fail.xml
   for i in {1..4}
   do
   sed -i.bu "s/0/1/" TEST-fail.xml
   done
   mv TEST-fail.xml TEST-all.xml
   fi
   
   exit $flutterExitCode
   

To create a Flutter iOS real device test run in BitBar cloud, follow these steps:

1. Create the Flutter iOS test run.

   
   

2. Upload a test.zip file and the application.ipa file to the BitBar cloud and select them:

   

   Click the image to enlarge it.

3. Select devices to run test on.

4. Click Create and run automated test.