+
Testing Mobile Apps
Joanne and Xidong
+
Content

Types of Application on Mobile

Browser Application and its testing strategy

Extra: current iOS and Android Automation Testing Software

Client Application and its testing strategy

Common Testing Technology
+
What is the focus of this topic?

To find an proper testing automation that significantly
reduces the overall time needed to test each software
release.
+
3 Categorized Applications

Client Application:


Messaging Application:


Applications are installed on a mobile device and run on that
device, may be written to look and feel like a native application
for specific phone models.
The servers need to receive and respond to the specifications of
SMS messages.
Browser Application:

Browser applications are server-based applications that can be
accessed through a web browser via a URL from a mobile device.
+
Aspects need to consider for
testing (in general)

The human languages (e.g., French, Kanji);

The locales (e.g., UK English, Australian, and US English),
which affect things like formatting numbers and currency
symbols;

Who pays for updates to be downloaded (users may be
unwilling to pay to download updates OTA);

How the software is installed on the device (e.g., in terms of
security permissions); and

The number of applications and versions you need to
support in parallel.
+
Goals when testing

To ensure we deliver attractive, easy-to-use, working
applications for as many users as practical. Lots of happy,
frequent users help show our software is successful and useful.

To have justified confidence in the quality of our software.
Ideally, most of the bugs would be found and fixed before the
software is widely used.

Fast feedback to developers. Fast feedback helps them to fix the
code while it is still “warm,” while they are still intimately
familiar with it.

To quickly detect issues so they can be addressed. This is
particularly relevant when the problem is related to external
factors (e.g., an operator’s network configuration or a specific
handset model).
+
Markup Language Application on
Mobile

Application is written HTML and run by browser in general.

But it has revolution for many years…
+
Markup Language Application on
Mobile (cont.)

Flash Lite is a system that enables devices like phones to
make use of a subset of the capabilities offered by Adobe
Flash. Flash Lite is used on some smartphones and feature
phones to enable things like embedded video viewing on
sites such as YouTube.

XHTML Mobile Profile (XHTML MP) is a hypertextual
computer language standard designed specifically for
mobile phones and other resource-constrained devices.

Comparison of XHTML Mobile profile and XHTML basic.
http://mobiforge.com/designing/story/comparison-xhtmlmobile-profile-and-xhtml-basic
+
Testing Markup Language
Application on Mobile

Strategy: Using a program or script since Markup languages
are designed to be processed by programs (such as the web
browser) rather than being directly interpreted by a human.

Problems: Cookies are sometimes intercepted by a gateway,
or a proxy server, provided by the carrier. Automated tests
may need to communicate across multiple distinct
connections, in parallel, probably across several devices or
computers, in order to trigger cookie sharing issues.
+
AJAX Mobile Application

Ajax (Asynchronous JavaScript and XML) is a group of
interrelated web development techniques used on the clientside to create asynchronous web applications. With Ajax,
web applications can send data to, and retrieve data from, a
server asynchronously (in the background) without
interfering with the display and behavior of the existing
page.
+
Testing Strategy for AJAX Mobile
Application

Be able to test rich UI interactions of the application running
within a web browser. (e.g., for an email application new
emails may arrive from time to time from the server without
any action by the user.)

Testing how well an application copes with intermittent
network connectivity can be convoluted and may require an
intermediate device. (e.g., to intercept and modify network
packets.)

Test specific features that are differentiated from applications
running on desktops. (e.g. rotation events)
+
What is Client Application?

Applications are installed onto mobile wireless devices.

They are able to provide users with more functionality, better
integration, and better performance than browser
applications.

Client applications can be divided into two categories:
portable applications and native applications.

Portable applications run in (or on) a virtual machine, such as the
Java Virtual Machine. (e.g. Android)

Native applications are written to run directly on particular
architecture or platform of the target devices. (e.g. iOS)
+
iOS Automation Testing

ios-driver


Test any IOS native, hybrid, or mobile web application using the
Selenium / Webdriver API. IOS automation is as easy as automation
for a browser, due to reuse of the well known API.
High level features :
 Implements the JSON wire protocol
 Runs on emulators and devices
 Can be run as a regular node in a Selenium Grid
 Can be used in an existing Selenium Grid to manage device /
simulator farm
 First class support for localized native apps ( 1 test works for all the
localized versions )
 No need to change the source
 No need to jailbreak the device
 No ios-driver app to install on the phone
+
iOS Automation Testing (cont.)

Appium is an open source test automation framework for
use with native and hybrid mobile apps. It drives iOS and
Android apps using the WebDriver JSON wire protocol.
+
Android Automation Testing


MonkeyRunner:

Written in Python;

Has APIs for connecting to a device, installing/uninstalling apps,
running apps, taking screenshots, comparing images.
Robotium:

A framework provides functional, system and acceptance tests
running across multiple activities; S

Supports Activities, Dialogs, Toasts, Menus and Context Menus,
and it is integrated with Maven and Ant for running continuous
integration tests.
+
Android Automation Testing (cont.)

Robolectric (taken a different path):


Uses shadow objects and runs the tests on a regular
workstation/server JVM avoiding dexing, packaging, deploying
and running the application tested on an emulator or a real
device, reducing the time needed to perform tests.
LessPainful (a step further):

Offering a service for automated testing of applications on
multiple real devices. The user uploads the application (*.apk)
and a test file written in Cucumber, a business-readable DSL,
selects the configuration of devices he wants the tests to run on,
and then the tests are automatically executed and a report is
generated. The list of devices supported include Garmin Asus,
several HTCs, LGs, Samsung Galaxy, Sony Xperia, and Motorola
Motodefy.
+
Techniques for Client Application

Automated Unit Tests

iOS application - unit testing framework is available on
code.google.com (http://code.google.com/p/google-toolboxfor-mac/) that even supports basic UI testing
(http://code.google.com/p/google-toolbox-formac/wiki/iPhoneUnitTesting).

Client applications are harder to automate

Ideally the test automation would include interaction with the
native system events (e.g., for keyboard and other inputs) and the
ability to query the GUI layer (e.g., to read the contents of a text
box). Extra features could be provided to enhance our ability to
test the application (e.g., where system calls can be intercepted
and modified to force certain conditions to be triggered.)
+
Techniques for Client Application

Automated Unit Tests (cont.)

Example of Unit Tests
protected void testBasicAuthForEmptyStrings() {
String authResult = BasicAuth.encode("", "");
assertEquals("EmptyStrings should result in 4 byte result", 4, authResult.length());
}

Running Unit Tests in Emulator


Challenging – capturing output, automating execution of the
tests in emulator
Code Injection

Used to obtain information, e.g., statement coverage, timing data,
etc., or it may include automated tests
+
Techniques for Client Application

Code Injection (cont.)


Unique features:

Source code is not required for some forms of code injection,
e.g., to add code coverage instrumentation.

The application’s source code does not need modifying
Injecting code requires:

The application,

The code to inject (which may be tightly coupled to the
implementation details of the application),

A way to inject the code, and

Instructions on where and how to inject the code.
+
Techniques for Client Application

Test Tools for Client Applications


On-Device debugging
 Not really a way to automate the testing
Emulators
 Blackberry Emulator: supports automation
 Command-line tools are used to enter commands, and to
create screenshots
 Challenges:
 Driving the user-interface.
 Interpreting screen-responses, including stitching together
images for pages that require scrolling, pattern matching
where the content varies from request to requests.
 Latency in the device interaction
+
Common Testing Techniques

GUI-level Automation


Challenges (in terms of making the tests relatively reliable):
 The need for high contrast, opaque, block colors for backgrounds,
etc., to make images
 easier to match for menus, etc.
 Deciding whether to use image pattern-matching or OCR (Optical
Character Recognition) .
 Structuring test resources, e.g., you may need a set of images per
device.
Image Stitching


Used to assemble a number of screenshots into a composite single
image that represents the original page.
Useful when the page is too large to be displayed on a physical phone
+
Common Testing Techniques

Image Stitching (cont.)

Step 1: Image retrieval:

Step 2: Image alignment:
+
Common Testing Techniques

Image Stitching (cont.)

Step 3: Image subdivision and election:

Step 4: Image reconstruction:
+
Common Testing Techniques

Image Stitching (cont.)

Challenges of image matching:

Captured images may vary significantly from device model to
model: Content varies across screen sizes and resolutions
+
Common Testing Techniques

Image Stitching (cont.)

Optical Character Recognition (OCR): to extract text from images

Encoding Data in Pixels:

Each character of ASCII: 8 bits =>HEX “0xNN” (N= 0~F)

E.g. “A” = 0x41, “a” = 0x61
+
Common Testing Techniques

“Contact Sheet”

multicolumn format of rows of small images: screenshots of results
generated by automated “user” input

Using Transparency Masking:
+
Common Testing Techniques

“Contact Sheet” (cont.)

Using Transparency Masking:
+
Common Testing Techniques

“Contact Sheet” (cont.)

Assisting test automation with human decision making:
+
Common Testing Techniques

Model Based Testing (MBT)

used to test other software domains such as web sites, desktop
applications, etc.

helps to automate longer-running automated tests

used MBT techniques to test a number of mobile wireless
applications, including a search engine and client applications.
+
When to test manually

UI and rendering problems:

alignment issues, flickering, etc.

When the UI is changing frequently, test scripts tend to break
and need repairing, requiring lots of work

When content is dynamic and hard to predict

When the cost of automating is likely to significantly exceed
the costs of manual testing
+
How to improve Testability

Generally:



Browser based application:


Add ID tags to the main elements the automated tests will interact
with
Client Applications


Provide a complete API to enable the test automation code to test
“below” (without) the GUI.
In programming languages such as Java consider making
methods and data protected.
Consider using high-contrast colors, visual markers, and even
pixel-encoding to make GUI-based automation more reliable
SMS Applications

Provide a scriptable library or interface (e.g., available from
Python) to make tests easier to create.
+
Reference

http://mobiforge.com/designing/story/comparison-xhtmlmobile-profile-and-xhtml-basic

http://www.mobileburn.com/definition.jsp?term=Flash+Lite

http://en.wikipedia.org/wiki/AJAX

http://ios-driver.github.io/ios-driver/index.html

http://appium.io

http://appium.io/getting-started.html#hacking-with-appiumios

http://www.infoq.com/news/2011/09/Android-Testing