Custom Framework Support

With server-side tests, your tests need to be uploaded to the cloud for execution in a zip package. Here is an example content of such a zip package. Note that the run-tests.sh script must be at the root of the unzipped package.

$ ls -lrt
-rwxr-xr-x 1 user staff 60 Feb 28 16:31 requirements.txt
-rw-r--r-- 1 user staff 4667 Mar 9 15:38 TestdroidAppiumTest.py
-rw-r--r-- 1 user staff 2687 Mar 13 11:04 BitbarSampleAppTest.py
-rwxr-xr-x 1 user staff 1263 Mar 14 16:10 run-tests.sh

On the cloud servers, the run-tests.sh script is called by the test execution environment. You can edit this script when defining your required environment.

The uploaded test package and the uploaded application are always located in the working directory as /test/tests.zip and /test/application.apk (or .ipa for iOS applications). Make sure the path is reflected accordingly in your tests if needed.

When running a server-side test run, BitBar Cloud always seeks to execute the run-tests.sh script. This script contains the environment setup and finally the call to start the execution of your test suite. The below goes through a sample script with explanations of what is done.

A working sample is available in the BitBar GitHub repository.

# unzip the test package
echo "Extract the test package to the current directory"
unzip tests.zip

Your uploaded test suite becomes visible in the test run working directory as tests.zip. The above shell script lines unpack the test package you uploaded to the cloud into the current working directory.

Similarly, if you need it, the uploaded application is always present in your working directory as: /test/application.apk.

# The app file is in the present working directory
PWD=$(pwd)
ls -lrt ${PWD}/application.apk

Your application and test suite are available in your working directory. You can now install any additional libraries or frameworks to customize your test environment. If you don’t require any changes, you would instead start a test run.

As an example, you might want to use some specific Python libraries to post-process your test results, so you might need Pip. The best way to install Python libraries is to have them listed in the requirements.txt file that should be part of the uploaded test zip package. The below block shows how to install additional Appium Python dependencies.

echo "Installing pip for python"
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
sudo python get-pip.py

echo "Installing Appium Python Client 0.24 and xmlrunner 1.7.7"
chmod 0755 requirements.txt
sudo pip install -r requirements.txt

If you are using Node-based tools for app testing instead of Python or Java, to install Node tools, you need to use the npm tool.

echo "Trying out node"
node -v

echo "Trying out npm"
npm -v

In October 2017, the above prints out:

Trying out node
v6.11.2
Trying out npm
3.10.10

Installing Ruby is possible inside of the Appium (Android) server-side test run, using default Linux package management tools.

echo "Installing Ruby"
apt-get update
# When installing Ruby, tzdata also gets installed that requires your current
# location to set date and time preferences. To avoid questions during the
# installation:
export DEBIAN_FRONTEND=noninteractive
# now we can install Ruby
apt-get install -y ruby-full

Once you have all the required additional packages and tools installed, you are ready to start your test framework. In this case, the Appium server is started.

# start the Appium server
echo "Starting Appium ..."
appium-1.6 --log-no-colors --log-timestamp

# Export some desired capabilities in case they are not set in the test# The Appium server URL is the same for local & Cloud executions
export APPIUM_URL="http://localhost:4723/wd/hub"
export APPIUM_DEVICE="Local Device"
export APPIUM_PLATFORM="android"
export APPIUM_APPFILE="${PWD}/application.apk"

# Get the device API level with adb
APILEVEL=$(adb shell getprop ro.build.version.sdk)
APILEVEL="${APILEVEL//[$'\t\r\n']}"
echo "API level is: ${APILEVEL}"

# An older API level requires Selendroid for APPIUM_AUTOMATIONif [ "$APILEVEL" -gt "16" ]; then
  echo "Setting APPIUM_AUTOMATION=Appium"
  export APPIUM_AUTOMATION="Appium"
else
  echo "Setting APPIUM_AUTOMATION=selendroid"
  export APPIUM_AUTOMATION="Selendroid"
fi

Now everything is set up to launch a server-side test run. If your test suite is using Java, you would start testing by calling mvn test. With Python, the call would be python MyPythonTestSuite.py.

To get results visible in the BitBar Cloud UI, the test results need to be output using the Junit XML format. There are multiple libraries that produce this output format in any scripting or programming language. The location where results are generated varies between the libraries.

The xml test results need to be copied to the local directory as the test-all.xml file for the test post-processor to attach the results to the test run in the cloud UI. For this, we need to add one additional line to our run-tests.sh script.

# Make results available to the cloud UI
cp <test output dir>/*.xml TEST-all.xml

# E.g. Maven Surefire plugin generated results#cp target/surefire-reports/TEST-*.xml TEST-all.xml