Android application testing with the Android test framework

You can use the JUnit test framework to test Java classes directly which do not call the Android API. Android provides JUnit extensions to test Android API.

Tip

You can not use standard JUnit tests on the JVM, because all methods in the android.jar JAR file of your Android program are only placeholder. Calling them result in a RuntimeException.

This is because android.jar JAR file does not contain the Android framework code but only stubs for the type signatures, methods, types, etc. The android.jar JAR file is only used for the Java compiler before deployment on an Android device. It is not bundled with your application. Once you application is deployed on the device it will use the android.jar JAR file on the Android device.

Unfortunately this make it impossible to test the Android framework classes direct on the JVM without additional libraries.

To test Android classes you need to run them on an Android device or emulator. This unfortunately makes the execution of tests longer.

1.2. Unit tests vs functional tests

A unit test tests only the functionality of a certain component. For example if an action in an activity which starts another activity activity is tested, a unit test determine only of a the intent was issued, not if the activity was started. A functional test would also check if the activity was correctly started.

1.3. JUnit 3

Currently the Android testing API supports JUnit 3 and not JUnit 4. In JUnit 3 test methods must start with the test prefix. The setup method must be called setUp() and the final clean up method must be called tearDown().

1.4. Running tests on a server without display

To run test without a display (headless) specify the adb -no-window parameter.

2. Test hooks into the Android framework

2.1. Instrumentation

The Android testing API provides hooks into the Android component lifecycle. These hooks are called the instrumentation API and allow your tests to control the application lifecycle. .

The Android instrumentation API allows you to run the test project and the normal Android project in the same process so that the test project can call methods of the Android project directly.

For example you can call the getActivity() which starts an activity and returns the activity which is tested. . Afterwards you can call the finish() method, followed by getActivity() again and you can test if the application restored its state correctly.

2.2. How the Android system executes tests

The InstrumentationTestRunner is the base test runner for Android tests. This test runner starts and loads the test methods. Via the instrumentation API is communicates with the Android system. If you start a test for an Android application, the Android system kills any process of the application under test and then loads a new instance. It does not work the application this is done via the test methods. These test method control the lifecycle of the components of the application.

3. Android test classes and annotations

3.1. Android assertion and mock classes

The Android testing API provides in addition to the standard JUnit Assert class, the MoreAsserts and ViewAsserts class.

Mock classes allows you to isolate tests from a running system by stubbing out or overriding normal operations. Android provides mock classes for the Android framework in the android.test and android.test.mock packages.

3.2. Android test classes

The Android test framework is still based on JUnit3 hence your test needs to extend the TestCase class and all test methods must start with test.

You can use the AndroidTestCase to test Android components which have no visual parts, e.g. Applications, Services or ContentProvider.

Android provides the following test classes which extends the TestCase class.

Table 1. Test classes

Android Test class	Description
AndroidTestCase	Provides methods to test permissions.
ApplicationTestCase	Provides methods to test the setup and teardown of Application objects.
InstrumentationTestCase	Base class for using instrumentation methods.

Instrumentation allows to control a visible part of the application, e.g. an
activity. For this your testcase would extend ActivityUnitTestCase.

This instrumentation class allows you to start and stop
activities, run actions on the user interface thread, send key events and more.

The ActivityInstrumentationTestCase2 class can be used to test access several
Activities. ActivityInstrumentationTestCase2 allows to use the full Android system infrastructure. .

3.3. Android test groups

You can annotate tests with the @SmallTest, @MediumTest and @LargeTest annotation and decide which test group you want to run.

The following screenshot shows the selection in the Run Configuration of Eclipse.

This allows you to run for example only tests which do not run very
long in Eclipse. Long running tests could than run only in the
continuous integration server.

3.4. Flaky tests

Actions in Android are sometimes time dependent. To tell Android to repeat a test once if it fails, use the
@FlakyTest annotation. Via the tolerance
attribute of this annotation you can define how often the Android test
framework should try to repeat a test before marking it as failed.

4. Android test projects and running tests

4.1. Android test projects

Android organizes tests into separate Android test projects. Instead
of Android components, an Android test application contains one or more
test classes.

Note

The application which is tested is typically called the
application under test.

The Android development tools (ADT) provide support for the creation
of Android test projects. via a project creation wizard. This wizard can
be reached under
File → New →
Other... → Android →
Android Test Project.

Note

It is good practice to use the name of the project under test and add
Test or .test to it. We use the
.test notation to have the project name the same as the package name.

The project creation wizard adds the project which should be tested
as dependency to the test project. It also create a version of the
AndroidManifest.xml file which specifies that the android.test.runner test library should be used and it specifies an instrumentation.

A test project also specifies the package of the application to test in the AndroidManifest.xml file the under android:targetPackage attribute. The following listing shows an example
AndroidManifest.xml for a test project.

<?xml version="1.0" encoding="utf-8"?>

<manifest xmlns:android="http://schemas.android.com/apk/res/android"

      package="de.vogella.android.test.target.test"

      android:versionCode="1"

      android:versionName="1.0">

    <uses-sdk android:minSdkVersion="8" />

    <instrumentation

        android:targetPackage="de.vogella.android.test.target"

        android:name="android.test.InstrumentationTestRunner" />

    <application android:icon="@drawable/icon" android:label="@string/app_name">

    <uses-library android:name="android.test.runner" />

    </application>

</manifest>

4.2. Running tests

To start an test from Eclipse, right-click on the test class and select Run As → Andriod JUnit Test.

On the command line you can start test via the ant test command. This requires that you created the build.xml file for the test project with the android update test-project command.

# -p path to the test project

# -m path to the project under test

android update test-project -p . -m ../com.vogella.android.test.simpleactivity 

# Afterwards run the tests

ant test

5. Activity testing

5.1. ActivityUnitTestCase

To test an activity in isolation you can use the ActivityUnitTestCase class. This class allows you to check the layout of the activity and to check if intents are triggered as planned. The intents is not send to the Android system but you can use the getStartedActivityIntent() method to access a potential intent and validate its data.

ActivityUnitTestCase starts the activity in an IsolatedContext, i.e. mainly isolated from the Android system.

ActivityUnitTestCase can be used to test layouts and isolated methods in the activity.

As this test runs in an IsolatedContext the test must start the activity, i.e. it is not auto-started by the Android system.

Intent intent = new Intent(getInstrumentation().getTargetContext(),

        MainActivity.class);

startActivity(intent, null, null);

// After this call you can get the

// Activity with getActivity()

5.2. ActivityInstrumentationTestCase2

Functional tests for an activity can be written with the ActivityInstrumentationTestCase2 class. The communication with the Android infrastructure is done via the Instrumentation class which can be access via the getInstrumentation() method. This class allows you to send keyboard and click events.

If you prefer to set values directly you need to use the runOnUiThread() of the activity. If all statements in your method interact with the UI thread you can also use the @UiThreadTest annotation on the method. In this case you are not allowed to use methods which do not run in the main UI thread.

Only an instrumentation-based test class allows you to send key events (or touch events) to the application under test.

ActivityInstrumentationTestCase2 starts the activity in the standard Android context, similar as if a user would start the application.

If you planning to have user interface interaction, e.g. via touch, you should use ActivityInstrumentationTestCase2.

If you want to send touch or key events directly via your test you have turn them off in the emulator via setActivityInitialTouchMode(false) in your setup() method of the test.

5.3. Testing the initial state

It is good practice to test the initial state of the application before the main activity start to be sure that the test conditions for the activity are fulfilled.

5.4. State management tests

You should write tests which verifies that the state of an activity remains even if it is paused or terminated by the Android system.

The ActivityInstrumentationTestCase2 class uses the Instrumentation class, which allows you to call the lifecycle hooks of the activities directly.

6. Exercise: unit test for an activity

6.1. Create project which is tested

Create a new Android project called com.vogella.android.test.simpleactivity with the activity called MainActivity.

Add a second activity called SecondActivity to your project. This activity should use a layout with at least one TextView. The id of the TextView should be "resultText" and its text should be set to "Started".

Add a button to the layout used by MainActivity. If this button is clicked the second activity should be started. Put the "http://www.vogella.com" String as extra into thecom.vogella.android.test.simpleactivity.test intent, use the key "URL" for this.

Here is some example code for the MainActivity.

package com.vogella.android.test.simpleactivity;

import android.app.Activity;

import android.content.Intent;

import android.os.Bundle;

import android.view.View;

public class MainActivity extends Activity {

  @Override

  protected void onCreate(Bundle savedInstanceState) {

    super.onCreate(savedInstanceState);

    setContentView(R.layout.activity_main);

  }

  public void onClick(View view) {

    Intent intent = new Intent(this, SecondActivity.class);

    intent.putExtra("URL", "http://www.vogella.com");

    startActivity(intent);

  }

}

6.2. Create unit test for activity

Create a new test project called com.vogella.android.test.simpleactivity.test. Select com.vogella.android.test.simpleactivity as the project to test.

Create a test class called MainActivityUnitTest based on the superclass android.test.ActivityUnitTestCase. This class allows to test the activity.

package com.vogella.android.test.simpleactivity.test;

import android.content.Intent;

import android.test.TouchUtils;

import android.test.suitebuilder.annotation.SmallTest;

import android.widget.Button;

import com.vogella.android.test.simpleactivity.MainActivity;

public class MainActivityUnitTest extends

    android.test.ActivityUnitTestCase<MainActivity> {

  private int buttonId;

  private MainActivity activity;

  public MainActivityUnitTest() {

    super(MainActivity.class);

  }

  @Override

  protected void setUp() throws Exception {

    super.setUp();

    Intent intent = new Intent(getInstrumentation().getTargetContext(),

        MainActivity.class);

    startActivity(intent, null, null);

    activity = getActivity();

  }

  @SmallTest

  public void testLayout() {

    buttonId = com.vogella.android.test.simpleactivity.R.id.button1;

    assertNotNull(activity.findViewById(buttonId));

    Button view = (Button) activity.findViewById(buttonId);

    assertEquals("Incorrect label of the button", "Start", view.getText());

  }

  @SmallTest

  public void testIntentTriggerViaOnClick() {

    buttonId = com.vogella.android.test.simpleactivity.R.id.button1;

    Button view = (Button) activity.findViewById(buttonId);

    assertNotNull("Button not allowed to be null", view);

    // You would call the method directly via

    getActivity().onClick(view);

    // TouchUtils cannot be used, only allowed in

    // InstrumentationTestCase or ActivityInstrumentationTestCase2 

    // Check the intent which was started

    Intent triggeredIntent = getStartedActivityIntent();

    assertNotNull("Intent was null", triggeredIntent);

    String data = triggeredIntent.getExtras().getString("URL");

    assertEquals("Incorrect data passed via the intent",

        "http://www.vogella.com", data);

  }

  @Override

  protected void tearDown() throws Exception {

    super.tearDown();

  }

}

7. Exercise: functional test for activities

Create a new test class called MainActivityFunctionalTest which allows to test across activities.

package com.vogella.android.test.simpleactivity.test;

import android.app.Activity;

import android.app.Instrumentation;

import android.app.Instrumentation.ActivityMonitor;

import android.test.ActivityInstrumentationTestCase2;

import android.test.TouchUtils;

import android.test.ViewAsserts;

import android.view.KeyEvent;

import android.view.View;

import android.widget.Button;

import android.widget.TextView;

import com.vogella.android.test.simpleactivity.R;

import com.vogella.android.test.simpleactivity.MainActivity;

import com.vogella.android.test.simpleactivity.SecondActivity;

public class MainActivityFunctionalTest extends

    ActivityInstrumentationTestCase2<MainActivity> {

  private MainActivity activity;

  public MainActivityFunctionalTest() {

    super(MainActivity.class);

  }

  @Override

  protected void setUp() throws Exception {

    super.setUp();

    setActivityInitialTouchMode(false);

    activity = getActivity();

  }

  public void testStartSecondActivity() throws Exception {

    // Add monitor to check for the second activity

    ActivityMonitor monitor = getInstrumentation().addMonitor(SecondActivity.class.getName(), null, false);

    // Find button and click it

    Button view = (Button) activity.findViewById(R.id.button1);

    TouchUtils.clickView(this, view);

    // To click on a click, e.g. in a listview

    // listView.getChildAt(0);

    // Wait 2 seconds for the start of the activity

    SecondActivity startedActivity = (SecondActivity) monitor

        .waitForActivityWithTimeout(2000);

    assertNotNull(startedActivity);

    // Search for the textView

    TextView textView = (TextView) startedActivity.findViewById(R.id.resultText);

    // Check that the TextView is on the screen

    ViewAsserts.assertOnScreen(startedActivity.getWindow().getDecorView(),

        textView);

    // Validate the text on the TextView

    assertEquals("Text incorrect", "Started", textView.getText().toString());

    // Press back and click again

    this.sendKeys(KeyEvent.KEYCODE_BACK);

    TouchUtils.clickView(this, view);

  }

}

To test the direct modification of a view, create the following test class for the SecondActivity class.

package com.vogella.android.test.simpleactivity.test;

import android.app.Activity;

import android.app.Instrumentation;

import android.app.Instrumentation.ActivityMonitor;

import android.test.ActivityInstrumentationTestCase2;

import android.test.TouchUtils;

import android.test.UiThreadTest;

import android.test.ViewAsserts;

import android.view.KeyEvent;

import android.view.View;

import android.widget.Button;

import android.widget.TextView;

import com.vogella.android.test.simpleactivity.R;

import com.vogella.android.test.simpleactivity.MainActivity;

import com.vogella.android.test.simpleactivity.SecondActivity;

public class SecondActivityFunctionalTest extends

    ActivityInstrumentationTestCase2<SecondActivity> {

  private static final String NEW_TEXT = "new text";

  public SecondActivityFunctionalTest() {

    super(SecondActivity.class);

  }

  public void testSetText() throws Exception {

    SecondActivity activity = getActivity();

    // search for the textView

    final TextView textView = (TextView) activity

        .findViewById(R.id.resultText);

    // set text

    getActivity().runOnUiThread(new Runnable() {

      @Override

      public void run() {

        textView.setText(NEW_TEXT);

      }

    });

    getInstrumentation().waitForIdleSync();

    assertEquals("Text incorrect", NEW_TEXT, textView.getText().toString());

  }

  @UiThreadTest

  public void testSetTextWithAnnotation() throws Exception {

    SecondActivity activity = getActivity();

    // search for the textView

    final TextView textView = (TextView) activity

        .findViewById(R.id.resultText);

    textView.setText(NEW_TEXT);

    assertEquals("Text incorrect", NEW_TEXT, textView.getText().toString());

  }

}

8. Exercise: Run tests via Apache Ant

Update your test project called com.vogella.android.test.simpleactivity.test to have a build.xml file.

Afterwards run the tests with the ant test command.

9. Service testing

To test an service you use the ServiceTestCase class. It provides the startService() and bindService() methods to interact with the service. The bindService() return immediately a IBinder object without callback.

Testing asynchronous processing in services is a challenge as the duration of this processing may vary.

It is good practice to test if the service handles correctly multiple calls from startService(). Only the first call of startService() triggers the onCreate() of the service but all calls trigger a call to onStartCommand() of the service.

10. Content provider testing

To test an content provider you use the ProviderTestCase2 class. ProviderTestCase2 instantiates automatically the provider under test and inserts an IsolatedContext object which is isolated from the Android system but still allows file and database access.

The usage of the IsolatedContext object ensures that your provider test does not affect the real device.

ProviderTestCase2 provides also access to a MockContentResolver via the getMockCOnktentResolver() method.

You should test all operations of the provider and also what happens if the provider is called with an invalid URI or with an invalid projection.

11. User interface testing

11.1. Cross-component user interface testing

Functional or back-box user interface testing does test the complete application and not single components of your application.

11.2. uiautomator

The Android SDK contains the uiautomator Java library for creating user interface tests and provides an engine to run these user interface tests. Both tools work only as of API 16.

uiautomator test project are standalone Java projects which the JUnit3 library and the uiautomator.jar and android.jar files from the android-sdk/platforms/api-version directory added to the build path.

uiautomator provides the UiDevice class to communicate with the device, the UiSelector class to search for elements on the screen and the UiObject which presents an user interface elements and is created based on the UiSelector class. The UiCollection class allows to select a number of user interface elements at the same time and UiScrollable allows to scroll in a view to find an element.

The following coding shows an example test from the official Android developer side. The URL for this is Testing UI example.

package com.uia.example.my;

// Import the uiautomator libraries

import com.android.uiautomator.core.UiObject;

import com.android.uiautomator.core.UiObjectNotFoundException;

import com.android.uiautomator.core.UiScrollable;

import com.android.uiautomator.core.UiSelector;

import com.android.uiautomator.testrunner.UiAutomatorTestCase;

public class LaunchSettings extends UiAutomatorTestCase {

  public void testDemo() throws UiObjectNotFoundException {

    // Simulate a short press on the HOME button.

    getUiDevice().pressHome();

    // We???re now in the home screen. Next, we want to simulate

    // a user bringing up the All Apps screen.

    // If you use the uiautomatorviewer tool to capture a snapshot

    // of the Home screen, notice that the All Apps button???s

    // content-description property has the value ???Apps???. We can

    // use this property to create a UiSelector to find the button.

    UiObject allAppsButton = new UiObject(new UiSelector().description("Apps"));

    // Simulate a click to bring up the All Apps screen.

    allAppsButton.clickAndWaitForNewWindow();

    // In the All Apps screen, the Settings app is located in

    // the Apps tab. To simulate the user bringing up the Apps tab,

    // we create a UiSelector to find a tab with the text

    // label ???Apps???.

    UiObject appsTab = new UiObject(new UiSelector().text("Apps"));

    // Simulate a click to enter the Apps tab.

    appsTab.click();

    // Next, in the apps tabs, we can simulate a user swiping until

    // they come to the Settings app icon. Since the container view

    // is scrollable, we can use a UiScrollable object.

    UiScrollable appViews = new UiScrollable(new UiSelector().scrollable(true));

    // Set the swiping mode to horizontal (the default is vertical)

    appViews.setAsHorizontalList();

    // Create a UiSelector to find the Settings app and simulate

    // a user click to launch the app.

    UiObject settingsApp = appViews

        .getChildByText(new UiSelector()

            .className(android.widget.TextView.class.getName()),

            "Settings");

    settingsApp.clickAndWaitForNewWindow();

    // Validate that the package name is the expected one

    UiObject settingsValidation = new UiObject(new UiSelector()

        .packageName("com.android.settings"));

    assertTrue("Unable to detect Settings", settingsValidation.exists());

  }

}

You need to use Apache Ant to build and deploy the corresponding project.

<android-sdk>/tools/android create uitest-project -n <name> -t 1 -p <path>

# build the test jar

ant build

# push JAR to device

ant push output.jar  /data/local/tmp/

# Run the test

adb shell uiautomator runtest LaunchSettings.jar -c com.uia.example.my.LaunchSettings

11.3. uiautomatorviewer

Android provides the uiautomatorviewer tool, which allows you to analyze the user interface of an application. You can use this tool to find the index, text or attribute of the application.

This tools allows non programmers to analyze an application and develop tests for it via the uiautomator library.

The tool is depicted in the following screenshot.

12. Monkey

Monkey is a command line tool which sends random events to your device. You can restrict Monkey to run only for a certain package and therefore instruct Monkey to test only your application.

For example the following will send 2000 random events to the application with the de.vogella.android.test.target package.

adb shell monkey -p de.vogella.android.test.target -v 2000

Monkey sometimes causes problems with the adb server. Use the following commands to restart the adb server.

adb kill-server

adb start-server

You can use the -s [seed] parameter to ensure that the generated sequence of events is always the same.

For more info on Monkey please see Monkey description.

13. monkeyrunner

13.1. Testing with monkeyrunner

The monkeyrunner tool provides a Python API for writing programs that control an Android device or emulator from outside of Android code.

Via monkeyrunner you can complete script your test procedure. It run via the adb debug bright and allows you to install program, start them, control the flow and also take screenshots or your application.

To use monkeyrunner ensure that you have Python installed on your machine and in your path.

In monkeyrunner you have primary the following classes:

MonkeyRunner - allows to connect to devices
MonkeyDevice - allows to install and uninstall packages and to send keyboard and touch events to an application
MonkeyImage - allows to create screenshots, compare screenshots and save them

MonkeyImage can compare the screenshot with an existing image via the sameAs() method. A screenshot contains the Android notifcation bar, including time. You can enter a percentage as second parameter for sameAs() or use the getSubImage() method.

The API reference for monkeyrunner can be generated via the following command.

# outfile is the path qualified name

# of the output file

monkeyrunner help.py help <outfile>

13.2. monkeyrunner example

Ensure Python is installed and in your path. Also ensure the [android-sdk]/tools folder is in your path. Create a file for example called testrunner.py

from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice

import commands

import sys

import os

# starting the application and test

print "Starting the monkeyrunner script"

if not os.path.exists("screenshots"):

    print "creating the screenshots directory"

    os.makedirs("screenshots")

# connection to the current device, and return a MonkeyDevice object

device = MonkeyRunner.waitForConnection()

apk_path = device.shell('pm path com.vogella.android.test.simpleactivity')

if apk_path.startswith('package:'):

    print "application installed."

else:

    print "not installed, install APK"

    device.installPackage('com.vogella.android.test.simpleactivity.apk')

print "starting application...."

device.startActivity(component='com.vogella.android.test.simpleactivity/[...CONTINUE]

com.vogella.android.test.simpleactivity.MainActivity')

#screenshot

MonkeyRunner.sleep(1)

result = device.takeSnapshot()

result.writeToFile('./screenshots/splash.png','png')

print "screenshot taken and stored on device"

#sending an event which simulate a click on the menu button

device.press('KEYCODE_MENU', MonkeyDevice.DOWN_AND_UP)

print "Finishing the test"

You run this test via the monkeyrunner testrunner.py on the console.

14. Common Android testing requires and solution approaches

14.1. Common logging on a server

Frequently the log files of the tests should be stored on a server and not on the device. A good practice it to provide a server backend and post the result via an HTTP request to this server. The server logs it centrally and provides central access to it.

14.2. Triggering system changes via tests

During tests you sometimes want to change the system status, e.g. turn WIFI of for example. This typically cannot be done via the test directly, as the test only has the permissions of the application under test.

A good practice is to install another application on the device which has the required permission and trigger it via an intent from the test.

15. Robotium

See Robotiumfor user interface testing with the Robotium framework.

16. Robolectric

16.1. Overview

Robolectric is a framework which mocks part of the Android framework contained in the android.jar file and which allows you to run Android tests directly on the JVM with the JUnit 4 framework.

If Robolectric implements a method is forwards these method calls to shadow Android objects which behave like the objects of the Android SDK. If not implemented they simply returns a default value, e.g. null or 0.

Setting up Robolectric requires the Robolectric and the JUnit 4 pars in the classpath. You also need to add the android.jar and maps.jar from your Android SDK installation directory to the classpath of the test project.

Details on using Robolectric can be found on its home page: Robolectric Homepage.

16.2. Functionality

Robolectric is designed to allow you to test Android applications on the JVM. This enables you to run your Android tests in your continuous integration environment without any additional setup.

Robolectric supports resource handling, e.g. inflation of views. You can also use the findViewById() to search in the a view.

16.3. Installation

You need to download the robolectric-X.X.X-jar-with-dependencies.jar from Roboelectric from Sonatype.

16.4. Example

Checkout the example project available at Github under the following URL: RobolectricSample sample project.

17. RoboGuice

17.1. RoboGuice

Roboguice is a dependency injection framework for Android. Using it can reduce significantly the amount of code you write. You find this framework under the following URL: Roboguice homepage..

17.2. RoboGuice and testing

Testing RoboGuice is well supported. See for example this Tutorialfor a detailed test description.

18. Thank you

Please help me to support this article:

19. Questions and Discussion

Before posting questions, please see the vogella FAQ. If you have questions or find an error in this article please use the www.vogella.com Google Group. I have created a short list how to create good questionswhich might also help you.

20. Links and Literature

20.1. Source Code

Source Code of Examples

20.2. Android Resources

Android ListView and ListActivity

Android Location API and Google Maps

Android Intents

Android and Networking

Android Homepage

Android Developer Homepage

Android Issues / Bugs

Android Google Groups

Android Assets Studio

20.3. vogella Resources

vogella TrainingAndroid and Eclipse Training from the vogella team

Android TutorialIntroduction to Android Programming

GWT TutorialProgram in Java and compile to JavaScript and HTML

Eclipse RCP TutorialCreate native applications in Java

JUnit TutorialTest your application

Git TutorialPut everything you have under distributed version control system