Graphical Test Planning
Left shift testing for real impact with GTP
David Bradley – Sr. Test Process Lead, DNA
April 2015
Bio - David Bradley
• David is the Senior Test Process Lead for the Desktops and Application Group at Citrix
Systems. In his current role he is responsible for ensuring the use of good test process
and practice, and alignment of these with the longer term test strategies and goals. He is
a proponent for the use of Graphical Test Planning* methodology in Citrix, providing
training sessions and support to teams around the globe.
• David has more than twenty five years in the software industry in a number of roles.
Initially as a software engineer he worked on a variety of embedded and distributed
systems, primarily for the defence sector, before becoming a Professional Tester at Citrix
in 2007. His main interests are improving his organisation’s test processes and
effectiveness, and changing current misconceptions about testing.
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
*Created by Hardeep Sharma in 2004
Methodologies Traditional vs. Modern
Changing how we think
Traditional Methods – How we used to do it
using System;
usingng
System.Collections.Generic;
Syst
using System.Configuration;
Syst
usingng
System.Data;
usingng
System.Linq;
Syst
using System.Threading.Tasks;
using System.Windows;
A
BBB
CCCCC
namespace WpfApplication1
{
/// <summary>
/// Interaction logic for App.xaml
/// </summary>
public partial class App : Application
{
}
}
Bottom-up
Start
FFB
RTT
Code
Write
RTM
Test
Release To Test
Final Function Build
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
Release To Manufacturing
Comparison of project methodologies
BUGS FOUND
Code
Release
To Test
600
400
Ineffective
use of Test
time
200
Traditional
0
1 2 3 4 5 6 7 8 9 10 11
12 13 14 15 16 17 18
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
Traditional
Scale of problem we solved
Desktop and Applications group only
10’s of products
100’s of testers
100’s of components
Multiple geographical locations
1000’s requirements
10,000’s of test cases
1,000,000’s lines of code
Many, many environment configurations
… and more
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
Early Test Involvement
Find and fix bugs at earliest opportunity
Fix Cost
10000
1
Traditionally we find bugs here
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
Science ... is a systematic enterprise
that builds and organizes knowledge in
the form of testable explanations and
predictions about the universe
from Wikipedia.
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
Graphical Test Planning – How we do it now
A
BBB
CCCCC
Top-down
Start
RTT
FFB
RTM
Code
Test
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
Release To Test
Final Function Build
Release To Manufacturing
Comparison of project methodologies
BUGS FOUND
Code
Release
To Test
600
400
GTP
improves
product
quality
Traditional
200
GTP
0
1 2 3 4 5 6 7 8 9 10
11 12 13 14 15 16 17 18
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
Traditional
Build a Model for Test
Why Model
Models enable people to understand and work with complex systems
• Electronic circuit diagrams
• Architectural blueprints
• CAD / CAM
• Software
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
How do we start
Traditional methods
GTP
Conceptual
System
Model
Requirements
Functional Spec
Test Plan
Test Cases
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
What is a conceptual model
• Example – World’s first hot drink vending system
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
From the idea build a Model of the Behaviour
Customer can
choose tea or
coffee
Conceptual
System Model
Customers can
choose what they
want to drink
Customer can
choose to add Milk
or cream
Customer can
choose to sweeten
drink
Customer can
provide correct
amount
Hot drinks can be
provided to paying
customers
Customers must
pay for drink
before receiving it
Change can be
given when
customer overpays
Drink is not
provided until
customer supplies
enough money
Hot drink is made
as requested
Customer can
enjoy the hot drink
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
Structured Relationship Diagram (SRD)
Customer can
choose tea or
coffee
Customers can
choose what they
want to drink
Customer can
choose to add Milk
or cream
Customer can
choose to sweeten
drink
Customer can
provide correct
amount
Hot drinks can be
provided to paying
customers
Customers must
pay for drink
before receiving it
Change can be
given when
customer overpays
Drink is not
provided until
customer supplies
enough money
Hot drink is made
as requested
Customer can
enjoy the hot drink
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
• Not a list of requirements
• Not a list of features
• Not a functional specification
• Not a flow chart
• It is a model of the behaviour
expected to be observed
Models are abstract from implementation
Customer can
choose tea or
coffee
Customers can
choose what they
want to drink
Customer can
choose to add Milk
or cream
Customer can
choose to sweeten
drink
Customer can
provide correct
amount
Hot drinks can be
provided to paying
customers
Customers must
pay for drink
before receiving it
Change can be
given when
customer overpays
Drink is not
provided until
customer supplies
enough money
Hot drink is made
as requested
Customer can
enjoy the hot drink
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
• Can be re-used for
different implementations
• Can be re-used for the
next release
Behaviours vs. Traditional Test Cases
Behaviours
• Describe the exact behaviour we expect to observe from the system
• Encourage understanding, research, investigation, more questions, etc.
Telephone alerts the
user of an incoming
call by ringing a bell
Traditional test cases
• Imply what might be covered
ᵒ Incoming phone call test
ᵒ Engine power limit test
ᵒ Input validation test
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
Engine power is
reduced when engine
RPM exceeds its
maximum limit
System should allow
valid input only
Machines can
have server OSes
Example
Behaviour
Model
Windows
machines can be
virtual machines
Windows
machines can be
physical machines
Full desktop
sessions can be
displayed
Application only
sessions can be
displayed
Machines will
be provisioned
when needed
Citrix Receiver is
required to
connect to a
session
Receiver can
be run on
many different
platforms
Access must
be secure
XenDesktop can
be managed easily
Session
experience must
be as good as
local experience
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
Machines can be
in a datacenter
Machines can be
in the cloud
XenDesktop allows
remote access to
Windows
machines
Actual model has:
10,000’s Behaviours,
1000’s pages,
100’s files, etc.
Machines can
have client OSes
Administration
can be
delegated
This example was created
solely for the purposes of this
presentation and is not
representative in any way of
the actual Citrix XenDesktop
model.
Having an impact
We find bugs from project conception
Conceptual
System Model
• Find bugs from project start
• No waiting for documents, code, etc.
• Be an integral part of the whole project lifecycle
Start
RTT
FFB
RTM
Code
Test
Release To Test
Final Function Build
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
Release To Manufacturing
We understand the system better
Customer can
choose tea or
coffee
Customers can
choose what they
want to drink
Customer can
choose to add Milk
or cream
Customer can
choose to sweeten
drink
Customer can
provide correct
amount
Hot drinks can be
provided to paying
customers
Customers must
pay for drink
before receiving it
Change can be
given when
customer overpays
Drink is not
provided until
customer supplies
enough money
Hot drink is made
as requested
Customer can
enjoy the hot drink
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
• Communicates knowledge and understanding
• Verifies the design
• Enables good Project Management & Control
• Provides flexibility and agility
We improve the system design
Customer can
choose tea or
coffee
Customers can
choose what they
want to drink
Customer can
choose other
drinks
Customer can
choose to add Milk
or cream
• Question the design
Customer can
choose to sweeten
drink
Customer can
provide correct
amount
Hot drinks can be
provided to paying
customers
Customers must
pay for drink
before receiving it
Change can be
given when
customer overpays
Drink is not
provided until
customer supplies
enough money
All drinks cost the
same
Hot drink is made
as requested
Customer can
enjoy the hot drink
Hot drink is
provided in a cup
Customer can
supply there own
cup
Customer can be
provided with a
cup
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
• Discover inaccuracies and ambiguities
Customer can get
a refund any time
before the drink is
made
Warned if not
enough change is
available
• Identify potential design issues
• Ensures the RIGHT product is built
• Minimises needless and costly bug
fixing and redesigning
Many people can contribute
• Model is based on facts, not speculation &
conjecture
Customer can
choose tea or
coffee
Customers can
choose what they
want to drink
Customer can
choose to add Milk
or cream
• We have all the information we need even
if we don’t get requirements, docs, etc.
Customer can
choose to sweeten
drink
Hot drinks can be
provided to paying
customers
Customers must
pay for drink
before receiving it
Customer can
provide correct
amount
Customer can get
a refund any time
before the drink is
made
Change can be
given when
customer overpays
Warned if not
enough change is
available
Drink is not
provided until
customer supplies
enough money
All drinks cost the
same
Hot drink is made
as requested
Customer can
enjoy the hot drink
Hot drink is
provided in a cup
Customer can
supply there own
cup
Customer can be
provided with a
cup
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
Customers
Business Analysts
Product Managers
Designers
Developers
Testers …
The model can help test early code drops
Customer can
choose tea or
coffee
Customers can
choose what they
want to drink
Customer can
choose to add Milk
or cream
• Can test directly from the model
Customer can
choose to sweeten
drink
Hot drinks can be
provided to paying
customers
Customers must
pay for drink
before receiving it
Customer can
provide correct
amount
Customer can get
a refund any time
before the drink is
made
Change can be
given when
customer overpays
Warned if not
enough change is
available
Drink is not
provided until
customer supplies
enough money
All drinks cost the
same
Hot drink is made
as requested
Customer can
enjoy the hot drink
Hot drink is
provided in a cup
Customer can
supply there own
cup
Customer can be
provided with a
cup
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
• Test against the confirmed
behaviour, not implementation
• Ensure the RIGHT product is
being built
Test Cases are derived from the Model
SRD - Structured Relationship Model
TCD – Test Case Diagram
Diesel engined car
will not accept a
petrol fuel pump
nozzle
M+ Test ID:
0
Cars go when
fuelled
A diesel engine
can be fitted
Engine can run
on normal
unleaded
Cars can use
an engine for
propulsion
Engine can run
on Super
unleaded
A petrol engine
can be fitted
Engine can be
mated to an
automatic
gearbox
Engine can be
mated to a
manual
gearbox
Cars go when
fuelled
M+ Test ID:
0
Petrol engined car
with Super
Unleaded petrol
M+ Test ID:
0
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
Car fuel tank
has correct
fuel in it
Car is on a
level flat
surface
Engine can run
on leaded
petrol
Petrol engined car
will not accept a
diesel fuel pump
nozzle
Press and hold
the foot brake
pedal to engage
the brakes
Petrol engined car
with Unleaded
petrol
Diesel engined car
with Diesel fuel
Release the
parking brake
Select “D” on an
automatic gearbox and
by using the cars
controls correctly check
that the car moves
Select “first” on a
manual gearbox and by
using the cars controls
correctly check that the
car moves
Parking brake
has been
applied and
car is
stationary
Engine stops
when fuel runs
out
Car is in
neutral and
engine is
running
Car comes to a
stop
We understand, estimate and plan better
Conceptual
System Model
Coverage
Test Plan
Derive
Test Cases
Estimates
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
Software testing is about adding value to
both the business and the customer
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
Next steps
• Contact me directly at [email protected] :
• For additional or more detailed information.
• If you are interested in a more hands on approach, we can make
arrangements to provide additional assistance.
• Documentation will be made publicly available in future.
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
Work better. Live better.
Additional Notes
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
SRD Symbols/Components
SRD Behaviour
Req ID:
SRD Requirement
SRD Testcase
High-Level Behaviour
Trivial Item List
Page Identifier
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
M+ Test I D:
Advanced Use – SRD Layout Heuristics
• Work to an imaginary grid of 7 x 8 (w x h) cells
• Work from left to right
• Towards the left capture the different behaviours
• Towards the right capture the differences in behaviour
• There is no vertical order imposed
• Always add the Page Identifier
• Always link any off-page references
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
TCD Layout
Test Case and Test Coverage Items
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
Pre-requisites
Validation Process
Example
Behaviour
Model
• Not a list of requirements
• Not a list of features
Kettle is manually
filled with water
Kettle heats water
to make hot
beverages
• It is a model of the behaviour
expected to be observed
© Copyright
2013 Citrix©| Citrix
Confidential
– Do
Not
Systems,
Inc.
AllDistribute
rights reserved.
Shows when kettle
is over filled
Shows how much
water is in the
kettle
Shows when kettle
is under filled
Can use filters to
improve the water
quality
Shows level on a
graduated scale
Kettle does not
leak
Heating of water is
manually started
Kettle can heat
water to boiling
point
• Not a functional specification
• Not a flow chart
Can be filled to
different levels
easily
Kettle can be filled
and heat other
liquids
Heating of water
can be manually
stopped at any
point
Heating of water
can be
automatically
stopped
Provides an
indication that the
water is being
heated
Kettle can be
handled safely,
even when hot
Water can be
emptied out of the
kettle
Water can be
safely and
accurately poured
Flow rate of water
coming out of the
kettle is restricted
Automatically
stops when water
has boiled for a
short period
Automatically
stops when very
little or no water is
present