GUI-based Testing
Alessandro Marchetto
Fondazione Bruno Kessler - IRST
GUI-based Testing
•
One of the practical methods commonly used to detect the
presence of errors (failures) in a computer program is to
exercise it by using its Graphical User Interface.
I1, I2, I3,
…, In, …
GUI
The output
is correct?
“Inputs”
Our program
Expected results
=?
Obtained results
2
GUI-based Testing: again four main questions

At which level conducting the testing?





How to choose inputs?




using the specifications/use cases/requirements
using the code
only considering the GUI (functionality, structure)
How to identify the expected output?


Unit
Integration
System
Regression
test oracles
How good test cases are?

when we can stop the testing activity
3
GUI …. what?






GUI as a means to use/interact with the software
systems
GUI are nowadays almost ubiquitous, even in safety
critical systems
Different types of device (web, pc, tablet, palm, mobile)
GUI interacts with the underlying code by method calls
or messages
GUI can exercise remote code
GUI responds to user events (e.g., mouse clicks)



GUIs are event-driven systems
Testing GUI correctness is critical for system usability,
robustness and safety
The whole system can be executed by means of the
GUI
4
GUI …. more formally
A GUI (Graphical User Interface) is a hierarchical,
graphical front end to a software system
A GUI contains graphical objects w, called widgets, each
with a set of properties p, which have discrete values
v at run-time.
At any time during the execution, the values of the
properties of each widget of a GUI define the GUI
state: {… (w, p, v), …}
A graphical event e is a state transducer, which yields the
GUI from a state S to the next state S’.
5
GUI-based Testing

Testing GUI software systems is different
from testing non-GUI software


Non-GUI testing: suites are composed of test cases
that invoke methods of the system and catch the
return value/s;
GUI-based testing: suites are composed of test
cases that are:






able to recognize/identify the components of a GUI;
able to exercise GUI events (e.g., mouse clicks);
able to provide inputs to the GUI components (e.g., filling text
fields);
able to test the functionality underlying a GUI set of components;
able to check the GUI representations to see if they are
consistent with the expected ones;
often, strongly dependent on the used technology;
6
GUI testing difficulties




GUI test automation is difficult
Often GUI test automation is technology-dependent
Observing and trace GUI states is difficult
UI state explosion problem


Controlling GUI events is difficult



A lot of possible states of the GUI
Explosion of the possible combinations of events to do
the same thing
GUI test maintenance is hard and costly
……
7
GUI testing advantages

Automation is feasible




Several frameworks and tools support it
Easy to conduct for non-expert people
It is funny to do
……
8
Which type of GUI-based testing?

System testing


Acceptance testing


Test the whole system
Accept the system
Regression testing

Test the system w.r.t. changes
9
GUI-based Acceptance Testing
Acceptance Tests are specified by the customer and analyst to test that the
overall system is functioning as required (Do developers build the right
system?).
How?




Manual Acceptance testing. User exercises the system manually
using his creativity.
Acceptance testing with “GUI Test Drivers” (at the GUI level).
These tools help the developer do functional/acceptance testing
through a user interface such as a native GUI or web interface.
Table-based acceptance testing. Starting from a user story (or
use case or textual requirement), the customer enters in a table
the expectations of the program’s behavior.
Black-Box (aka functional) approaches can be used to define
test specification then executed manually, by means of the GUI or
by table-based testing.
10
Approaches for GUI-based testing

Manual based


Capture and Replay


Based on the domain and application knowledge of the tester
Based on capture and replay of user sessions
Model-based testing

Based on the execution of user sessions selected from a
model of the GUI


Which type of model to use?
 Event-based model
 State-based model
 Domain model
How do obtain the model to be used?
 Specification-based model
 Model recovered from existing software systems
 Log-based model
11
Coverage criteria for GUI-based testing

Conventional code-based coverage cannot be
adequate;

GUIs are implemented in terms of event-based
system, hence, the abstraction level is different w.r.t.
the conventional system code. So mapping between
GUI events and system code can not be so easy.

Possible coverage criteria:



Event-coverage: all events of the GUI need to be executed at
least once
State-coverage: “all states” of the GUI need to be exercised
at least once
Functionality-coverage: .. using a functional point of view
12
Event-based Model
Model the space of GUI event interactions as a graph
Given a GUI:
1. create a graph model of all the possible sequences
that a user can execute
2. use the model to generate event sequences
13
Event-based Model
“Event-flow graph”
File
New
Open
Edit
Save
…
Cut
Copy
Editbox0
Find next
Paste
Editbox1
Replace
Replace
Undo
Match case
Replace all
Cancel
Top level
TC: <S0, event1, event2, …>
Oracle: <State1, State2, …> & !CRASH
14
…
Event-based Model
Model Type:


Complete event-model
Partial event-model
Event types:



Structural events (Edit, Replace)
Termination events (Ok, cancel)
System interaction events (Editbox0, Find next)
Coverage criteria





Event coverage
Event coverage according the exercised functionality
Coverage of semantically interactive events
2-way, 3-way coverage
….
15
State-based Model
Model the space of GUI event interactions as a state
model, e.g., by using a finite state machine (FSM):
•
•
States are screenshot/representation of the GUI
Transitions are GUI events that change the GUI state
Given a GUI:
1. create a FSM of the possible sequences that a user
can execute, considering the GUI state
2. use the FSM to generate event sequences
e1
e3
e4
e2
e5
16
State-based Model
<ok>
<fill>
<click>
<click>
<click>
<fill + ok>
<fill>
<ok>
<fill + ok>
FSM:
- State: GUI
screenshot
-Transition: event
that changes the
GUI state
17
..example
Problem: state explosion!
 Use of a state abstraction function
It maps concrete states into abstract states
(i.e., sets of concrete states)
18
Log-based Recovered Model

How do obtain the model?


starting from system specification or
requirements
starting from the system (i.e., reverse
engineering)
1.
2.
3.
trace some system executions (at method calls
level)
infer a model
refine it manually, if needed
19
Test oracles for GUI-based testing

It could be difficult to detect faults looking the GUI
Crash testing is often used;


In a GUI test case, an incorrect GUI states can take the
user to an unexpected/wrong interface screen or it
can make the user unable to do a specific action;
e.g., after the click of a button, we try to click the button again
but we fail since the button no longer exists, after the first click.


A GUI state can be “represented” by the components
expected to be part of the GUI in a give time and their
state/value

e.g., window position, GUI objects, GUI title, GUI content, GUI
screenshots,
20
GUI errors: examples









Incorrect functioning
Missing commands (e.g., GUI events)
Incorrect GUI screenshots/states
The absence of mandatory UI components (e.g., text
fields and buttons)
Incorrect default values for fields or UI objects
Data validation errors
Incorrect messages to the user, after errors
Wrong UI construction
….
21
GUI-based Testing: process
1.
2.
Identify the testing objective by defining a coverage
criteria
Generate test cases from GUI structure, specification,
model
 Generate sequences of GUI events
 Complete them with inputs and expected oracles
3.
Define executable test cases
4.
Run them and check the results
22
GUI-based Regression Testing

GUI-based testing means to execute the GUI of a
system exercising its GUI components;

A small changes in the GUI layout can make the GUI
test cases old and useless;
Hence, GUI-based test suite need to be maintained
and often chagned

 supporting tools are welcomed

Often, GUIs are realized by means of rapid prototyping
or automatic framework. This requires an efficient
approach to generate and maintain GUI test suite
 supporting tools are welcomed
23
Capture and Replay
A capture and replay testing tool captures user
sessions (user inputs and events) and
store them in scripts (one per session)
suitable to be used to replay the user
session.
An ad-hoc infrastructure is needed to intercept GUI events,
GUI states, thus storing user sessions and also to be
able to replay them.
- they can work at application or VM level
24
Recorded information

Inputs, outputs, and other information needed
to replay a user session need to be recorded
during the capture process.

Examples:



General information: date/time of recording, etc.
System start-up information
Events from test tool to system


Events from system to test tool


Point of control, event
Checkpoints / expected outputs
Time stamps
25
Capture and Replay: the process
1.
2.
The tester interacts with the system GUI to run the
system, thus generating sessions of sequence of
mouse clicks, UI and keyboard events;
The tool captures and stores the user events and the
GUI screenshots;

3.
The tester can automatically replay the execution by
running the script



4.
a script is produced per each user session
the script can be also changed by the tester
the script can be enriched with expected output,
checkpoints
the script can be replicated to generate many variants (e.g.,
changing the input values)
In case of GUI changes, the script must be updated
26
Tools for GUI-based testing







Marathon
Java
Abbot
Guitar
HtmlUnit, HttpUnit,JWebUnit
HtmlFixture
Selenium
….
Web
27
Running Example: Calculator
Calc_1 :

Logic mixed to GUI

GUI realized by using
Swing
Calc_2 :

Logic mixed to GUI

GUI realized by using AWT
Calc_3 :

Logic separated from the
GUI

GUI realized by using AWT
+Swing
These apps are from the net hence please refer to the author’s copyright
28
Marathon

Is a tool that helps writing System/GUI tests
for Java/Swing applications
 Consists of a recorder, player, and an editor
 Records tests script in Python


It is possible to use it without knowing Python …
Allows to insert assertions in the script easily

Using a specific GUI
Shows testing results using JUnit’s control bar



Red/green
http://www.marathontesting.com

download Marathon and its user manual
29
Marathon example (1)


A calculator application
was developed (Swing)
We want to write tests
for the calculator
 Ex. add two values
 Steps:
1.
2.
3.
create a new Marathon
project
create a new empty test
(a test is a Python script
file)
build the test (recording)
30
Marathon example (2)
Record the script
•
•
•
•
•
•
click on the record button in
the toolbar
– the red circle
enter “12 + 4 =”, the result is
displayed (16)
press control + right-click in
the text area, to open the
contextual menu
select Assert Text
stop recording (the button
with a red square), the script
now looks like:
save the script
31
Marathon example (3)
Run the script
•
•
•
•
•
Select the “JUnit” tab
Click the test view, you should
see your new created test
Click on the “Run all tests”
button, in the JUnit’s control
bar
and …
get the green bar …
32
Abbot






Is a tool that helps writing System/GUI tests for Java
AWT/Swing applications
Consists of a recorder, player, and an editor (via Costello, built
on top of Abbot)
Records tests script in Java
Allows to write test cases directly from Java code (named
programmatic GUI testing)
Allows to insert assertions in the script easily
Shows testing results using JUnit’s control bar


Two main building blocks:



Red/green
ComponentReferences to get a handle on a GUI component
Robot: to perform user-level actions on various GUI
components
http://abbot.sourceforge.net

download Marathon and its user manual
33
Abbot example (1)


A calculator application
was
developed
(AWT/Swing)
We want to write tests
for the calculator
 Ex. add two values

Steps:
1. create a new empty
Abbot test
2. Initialize the test with the
application under test
information
3. build the test (recording)
34
Abbot example (2)
Recording
1.
2.
3.
Start recording of all
actions
Execute the software
Add assertions
1.
2.
3.
4.
select the GUI element
with SHIFT+F1
select the property (e.g.,
(“Text”)
set the expected value
Stop recording
35
Guitar


A testing framework implementing the “Eventflow graph” approach
Four main components:




The GUIRipper: extract GUI information from a program
The GUIStructure2Graph: build a traversable graph
representation of the GUI elements
The TestCaseGenerator: create an extensive set of test
cases based on the graph
The GUIReplayer: run the program as instructed by these
tests
It is a research tool…
http://guitar.cs.umd.edu



It requires ANT to be executed
36
Guitar (1)
Four main components:
(1) GUIRipper: GUI information extraction
>ant -Dproperties=jfcripper.properties -f jfcripper.xml
(2) GUIStructure2Graph: Event-flow graph inference
>ant -v -f GUIStructure2GraphConvert.xml
37
Guitar (2)
(3) TestCaseGenerator: Test cases generation traversing the graph
>ant -v -f TestCaseGenerator.xml
(4) GUIReplayer: Run the program GUI
according to the generated test cases
>ant -Dproperties=jfcreplayer.properties -f jfcreplayer.xml
38
Tools for other GUI-based testing of Java




HtmlUnit, HttpUnit, JWebUnit
HtmlFixture
Selenium
….
39
XUnit Web tools
HtmlUnit, HTTPUnit, JWebUnit, etc.
- They are inspired by JUnit
- They let us to verify the functionalities of Web sites
- They emulate the relevant portions of browser behavior
form submission, JavaScript, http authentication, cookies, automatic page
redirection, etc.
- They allow Java test code to examine returned pages
in terms of text, DOM, or containers of forms, tables, and links
- They make it easy to write Junit tests that verify HTTP requests
and responses
40
… HttpUnit example
import junit.framework.TestCase;
import com.meterware.httpunit.WebResponse;
import com.meterware.httpunit.WebConversation;
import com.meterware.httpunit.WebForm;
import com.meterware.httpunit.WebRequest;
public class SearchExample extends TestCase {
Set the google page
public void testSearch() throws Exception {
WebConversation wc = new WebConversation();
WebResponse resp = wc.getResponse("http://www.google.com");
WebForm form = resp.getForms()[0];
Fill the google form
form.setParameter("q", "HttpUnit");
WebRequest req = form.getRequest("btnG");
Send the request
resp = wc.getResponse(req);
get the google response
assertNotNull(resp.getLinkWith("HttpUnit"));
resp = resp.getLinkWith("HttpUnit").click();
assertEquals(resp.getTitle(), "HttpUnit");
assertNotNull(resp.getLinkWith("User's Manual"));
verify the response
}
}
41
HtmlFixture
http://fitnesse.org/FitNesse.HtmlFixture
- It is used to exercise and test web pages.
- It allows us make assertions about the structure of a page and to
navigate between pages.
- We can also fire java script, submit forms, "click" links, etc.
- It is based on a fixture that lets us navigate to any element of the
page DOM.
To use this we need some knowledge about the DOM (Document
Object Model).
42
… example
43
Capture and Replay for Web
Selenium
http://seleniumhq.org/
Selenium IDE: a Firefox addon that will do simple recordand-playback of interactions
with the browser
Selenium WebDriver: a
collection of language specific
bindings to drive a browser -the way it is meant to be
driven.
44
Additional references
-Atif M. Memon. An event-flow model of GUI-based applications for testing.
Software Testing, Verification and Reliability, 2007, John Wiley & Sons
http://www.cs.umd.edu/~atif/papers/MemonSTVR2007-abstract.html
-Atif M. Memon, M.E. Pollack, Mary Lou Soffa. A planning-based Approach to GUI
testing. International Software/Internet Quality Week, May 2000
http://www.cs.umd.edu/~atif/papers/MemonSQW2000-abstract.html
-Marlon Vieira, Johanne Leduc, Bill Hasling, Rajesh Subramanyan, Juergen
Kazmeier. Automation of GUI Testing Using a Model-driven Approach AST’06
http://www.irisa.fr/lande/lande/icse-proceedings/ast/p9.pdf
-Jessica Chen and Suganthan Subramaniam. A GUI Environment to Manipulate
FSMs for Testing GUI-based Applications in Java. International Conference on
System Sciences, 2001
http://wenku.baidu.com/view/5f544c5377232f60ddcca145.html?from=related
-Alessandro Marchetto, Paolo Tonella and Filippo Ricca. State-based testing of
Ajax Applications. In Proceedings of the International Conference on Software
Testing, 2008 http://selab.fbk.eu/tonella/papers/icst2008.pdf
45