Let’s talk Circle CI: How to run Android Tests & Code Coverage Reports

Rebecca Franks

DVT Android developer

Let’s talk Circle CI: How to run Android Tests & Code Coverage Reports

Tuesday, 12 April 2016 16:16

Do you have CI running for your android apps? Do you just build and deploy to the Google Play store from your own machine? Let’s hope not.

Setting up a build pipeline should be relatively simple. Unfortunately I have spent more time on this than I would have preferred (50+ builds later), which is why I decided to write a post about it.

We are going to take a look at how to set up Circle CI to build your app, run android tests and get code coverage reports published. In Part 2, we will dive into deploying to the Play Store.

What is CI?

Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.

What is Circle CI?

Circle CI is a continuous integration and delivery platform. Similar to Jenkins, Travis CI, Team City and the like. These platforms check out your code from your repository, build it and then test it.

How to set up Circle CI to build an Android app

First log in to Circle CI using your Github account. You will then select which projects from Github you wish to run Circle CI on.

By default, Circle CI will pick up what type of project you have and try to build it. For android projects, it will run ./gradlew test. But if you want to do anything else, you will need to create a circle.yml file in the root of your project.

Create a circle.yml file in your root folder:

1. Building your app on Circle CI

To build your app on Circle CI, you can simply add the following command to the test: section of your circle.yml file:


machine:
   environment:
     ANDROID_HOME: /usr/local/android-sdk-linux
test:
  override:
    - ./gradlew assembleDebug
   
  post:
    - cp -r app/build/outputs $CIRCLE_ARTIFACTS

This will call the gradle task assembleDebug when it builds your application. The post: section specifies to Circle CI that the files in the outputs directory should be archived and available for download after execution. You could also run any other gradle task you wish

2. Running Android Tests on Circle CI

In order to run tests on the Circle CI build server, you need to ensure that you have an emulator set up and running.

The test: section highlights what the build server needs to perform in order to run the tests. Firstly, I created a custom emulator (one with Google APIs) and then used the built in Circle CI function to wait for the device to boot up.

When trying to run the Android instrumentation tests, I was occasionally getting this error:


java.lang.RuntimeException: Waited for the root of the view hierarchy to have window focus and 
not be requesting layout for over 10 seconds. 
If you specified a non default root matcher, it may be picking a root that never takes focus.
Otherwise, something is seriously wrong. Selected Root:
...

After Googling and fighting for a couple of hours, a lot of different possible problems were highlighted from a couple of different sources. These problems include:

The emulator might not have Google APIs installed and is prompting to install them on the emulator, therefore blocking the UI of your app.
The emulator might be locked.
The screen might have gone off.
The animations are not turned off and are delaying tests. You need to disable these animations on your emulator to obtain reliable results : Window Animations, Transition Animation Scale, Animator Duration Scale.

In order to solve these issues, I tried a couple of things. For #1, I made sure I was running a custom emulator with Google APIs installed on it.


    - android create avd -n emulatorwithgoogleapi22 -t 12 --tag google_apis
    - echo 'vm.heapSize=512' >> ~/.android/avd/emulatorwithgoogleapi22.ini
    - echo 'hw.ramSize=1024' >> ~/.android/avd/emulatorwithgoogleapi22.ini
    - cat ~/.android/avd/emulatorwithgoogleapi22.ini
    - emulator -avd emulatorwithgoogleapi22 -no-audio -no-window :
        background: true
        parallel: true
    - circle-android wait-for-boot

For #2, I initially tried just unlocking the device before calling the gradle task with the following command:


adb shell input keyevent 82

This worked for the first few tests but the device may go to sleep if a test takes a long time to execute. In order to solve the issue of the device locking itself and to solve issues #3 and #4, I used a Custom Test Runner as suggested by this Gist (https://gist.github.com/danielgomezrico/9371a79a7222a156ddad).

First off, we need to define that the tests need permission to change settings on the device but we don’t want them to be changed in the prod version. For this, I added a file into mock/AndroidManifest.xml which will only be included when running the mock build.

The CustomTestRunner class:


import android.Manifest;
import android.app.KeyguardManager;
import android.content.Context;
import android.content.pm.PackageManager;
import android.os.Bundle;
import android.os.IBinder;
import android.os.PowerManager;
import android.support.test.runner.AndroidJUnitRunner;
import android.util.Log;

import java.lang.reflect.Method;

public final class CustomTestRunner extends AndroidJUnitRunner {

    private static final String TAG = "CustomTestRunner";

    @Override
    public void onStart() {

        runOnMainSync(new Runnable() {
            @Override
            public void run() {
                Context app = CustomTestRunner.this.getTargetContext().getApplicationContext();

                CustomTestRunner.this.disableAnimations(app);

                String name = CustomTestRunner.class.getSimpleName();
                unlockScreen(app, name);
                keepSceenAwake(app, name);
            }
        });

        super.onStart();
    }

  
    @Override
    public void finish(int resultCode, Bundle results) {
        super.finish(resultCode, results);
        enableAnimations(getContext());
    }
    private void keepSceenAwake(Context app, String name) {
        PowerManager power = (PowerManager) app.getSystemService(Context.POWER_SERVICE);
        power.newWakeLock(PowerManager.FULL_WAKE_LOCK | PowerManager.ACQUIRE_CAUSES_WAKEUP | PowerManager.ON_AFTER_RELEASE, name)
                .acquire();
    }

    private void unlockScreen(Context app, String name) {
        KeyguardManager keyguard = (KeyguardManager) app.getSystemService(Context.KEYGUARD_SERVICE);
        keyguard.newKeyguardLock(name).disableKeyguard();
    }

    void disableAnimations(Context context) {
        int permStatus = context.checkCallingOrSelfPermission(Manifest.permission.SET_ANIMATION_SCALE);
        if (permStatus == PackageManager.PERMISSION_GRANTED) {
            setSystemAnimationsScale(0.0f);
        }
    }

    void enableAnimations(Context context) {
        int permStatus = context.checkCallingOrSelfPermission(Manifest.permission.SET_ANIMATION_SCALE);
        if (permStatus == PackageManager.PERMISSION_GRANTED) {
            setSystemAnimationsScale(1.0f);
        }
    }

    private void setSystemAnimationsScale(float animationScale) {
        try {
            Class windowManagerStubClazz = Class.forName("android.view.IWindowManager$Stub");
            Method asInterface = windowManagerStubClazz.getDeclaredMethod("asInterface", IBinder.class);
            Class serviceManagerClazz = Class.forName("android.os.ServiceManager");
            Method getService = serviceManagerClazz.getDeclaredMethod("getService", String.class);
            Class windowManagerClazz = Class.forName("android.view.IWindowManager");
            Method setAnimationScales = windowManagerClazz.getDeclaredMethod("setAnimationScales", float[].class);
            Method getAnimationScales = windowManagerClazz.getDeclaredMethod("getAnimationScales");

            IBinder windowManagerBinder = (IBinder) getService.invoke(null, "window");
            Object windowManagerObj = asInterface.invoke(null, windowManagerBinder);
            float[] currentScales = (float[]) getAnimationScales.invoke(windowManagerObj);
            for (int i = 0; i < currentScales.length; i++) {
                currentScales[i] = animationScale;
            }
            setAnimationScales.invoke(windowManagerObj, new Object[]{currentScales});
            Log.d(TAG, "Changed permissions of animations");
        } catch (Exception e) {
            Log.e(TAG, "Could not change animation scale to " + animationScale + " :'(");
        }
    }
}

The CustomTestRunner class will first unlock the screen, then acquire a wakelock to keep the device awake and then it will disable animations on the device.

The next step is to configure gradle to use this test runner:


defaultConfig {
        applicationId "org.bookdash.android"
        minSdkVersion 16
        targetSdkVersion 23
        versionCode 17
        versionName "1.0.17"
        testInstrumentationRunner "org.bookdash.android.presentation.CustomTestRunner"
}

Then create a gradle task that grants permissions to the app before running the tests:


// Grant animation permissions to avoid test failure because of ui sync.
task grantAnimationPermissions(type: Exec, dependsOn: 'installMock') {
    group = 'test'
    description = 'Grant permissions for testing.'

    def absolutePath = file('..') // Get project absolute path
    commandLine "$absolutePath/set_animation_permissions.sh org.bookdash.android".split(" ")
}

// Source: http://stackoverflow.com/q/29908110/112705
afterEvaluate {
    tasks.each { task ->
        if (task.name.startsWith('assembleMockAndroidTest')) {
            task.dependsOn grantAnimationPermissions
        }
    }
}

The set_animation_permissions.sh bash script calls adb shell to grant permissions to the attached devices for the animations:


#!/bin/bash
#
# source https://github.com/zielmicha/adb-wrapper
#
# argument: apk package
# Set permission android.permission.SET_ANIMATION_SCALE for each device.
# ex: sh set_animation_permissions.sh 
#

adb=$ANDROID_HOME/platform-tools/adb
package=$1

if [ "$#" = 0 ]; then
    echo "No parameters found, run with sh set_animation_permissions.sh "
    exit 0
fi

# get all the devices
devices=$($adb devices | grep -v 'List of devices' | cut -f1 | grep '.')

for device in $devices; do
    echo "Setting permissions to device" $device "for package" $package
    $adb -s $device shell pm grant $package android.permission.SET_ANIMATION_SCALE
done

Now we should have fixed issues #1,#2, #3, #4. Whew!

Back to Circle CI config:

First the Instrumentation tests will be run, then the unit tests will be run:


./gradlew connectedAndroidTest
./gradlew test

Finally, we archive the results of the tests that ran. These will then be available for download later. We archive the results by recursively copying the files into the $CIRCLE_TEST_REPORTS folder.


cp -r app/build/outputs/androidTest-results/ $CIRCLE_TEST_REPORTS
cp -r app/build/test-results/mockDebug/ $CIRCLE_TEST_REPORTS

The best part about a CI tool like this? The awesome badge that you can put on your site to show your build is passing or failing:

3. Setting up Code Coverage Reports on Circle CI

What is Code Coverage? It is a measure used to describe the degree to which the source code of a program is tested by a particular test suite, it can reveal areas of the code that do not have tests and possibly uncover bugs.

In order to get a code coverage report of your androidTest folder, add the following to your app/build.gradle :


buildTypes {
    debug {
        // Run code coverage reports by default on debug builds.
        testCoverageEnabled = true
    }
}

Then run the following command:


./gradlew createMockDebugCoverageReport

This will run the test and get a coverage report. Then we should archive the results in Circle CI by doing the following:


 - cp -r app/build/reports/coverage/ $CIRCLE_TEST_REPORTS

You will then be able to see files like this being produced from your build server:

Final Circle CI Configuration:


machine:
   environment:
     ANDROID_HOME: /usr/local/android-sdk-linux

dependencies:
  pre:
    - touch app/google-services.json
    - echo $GOOGLE_SERVICES_JSON > app/google-services.json
    - echo y | android update sdk --no-ui --all --filter "tools,platform-tools,android-23"
    - echo y | android update sdk --no-ui --all --filter "build-tools-23.0.2"

test:
  pre:
    - android list targets
    - android create avd -n emulatorwithgoogleapi22 -t 12 --tag google_apis
    - echo 'vm.heapSize=512' >> ~/.android/avd/emulatorwithgoogleapi22.ini
    - echo 'hw.ramSize=1024' >> ~/.android/avd/emulatorwithgoogleapi22.ini
    - cat ~/.android/avd/emulatorwithgoogleapi22.ini
    - emulator -avd emulatorwithgoogleapi22 -no-audio -no-window :
        background: true
        parallel: true
    - circle-android wait-for-boot
    - adb shell input keyevent 82
    - adb shell svc power stayon true
    - adb shell settings put global window_animation_scale 0
    - adb shell settings put global transition_animation_scale 0
    - adb shell settings put global animator_duration_scale 0

  override:
    - ./gradlew createMockDebugCoverageReport
    - ./gradlew test

  post:
    - cp -r app/build/outputs $CIRCLE_ARTIFACTS
    - cp -r app/build/outputs/androidTest-results/ $CIRCLE_TEST_REPORTS
    - cp -r app/build/reports/coverage/ $CIRCLE_TEST_REPORTS
    - cp -r app/build/test-results/mockDebug/ $CIRCLE_TEST_REPORTS

If you want to see this project in action, checkout Book Dash Android App on GitHub and on Circle CI.

Next time we will look at more coverage reports for the unit tests and uploading an APK to the Google Play Store via Circle CI.

What CI do you use? What problems have you faced with running CI for your Android project? Let me know in the comments below!

Insights