1. No category

Basic Concepts
• Input domain – set of all possible inputs
• Failure-causing inputs – inputs that exhibit
failures
• Exhaustive testing –
Adaptive Random Testing
T. Y. Chen
– to test the program with the entire input
domain
– it is practically infeasible
[email protected]
Centre for Software Analysis and Testing
Swinburne University of Technology
Melbourne, Australia
1
2
Random Testing
failure-causing inputs
• Random Testing
– selects test cases from the entire input domain
randomly and independently
• Advantages:
– Intuitively simple
– Allows statistical quantitative estimation of the
software’s reliability
3
4
How to improve random testing
How to improve random testing
• Any common information or characteristics
to all faulty programs?
• Any common information or characteristics
to all faulty programs?
Failure-causing inputs
5
6
Patterns of failure-causing inputs
Types of Failure Patterns
Strip Pattern
• Strip Pattern
• Block Pattern
• Point Pattern
Block Pattern
Pattern
Point
7
Strip Pattern
Two Dimensional Input
Domain
8
Block Pattern
Two Dimensional Input
Domain
If (2*x – y > 10)
/* the correct statement is
If (2*x – y > 20) */
z = x/2 *y;
else
z := x*y;
If (x >= 4 and x <=6)
and
(y >= 4 and y <= 6)
z := x + y;
/* the correct statement is
z := x - y; */
else
z := 100;
9
Point Pattern
Two Dimensional
input domain
10
Which pattern occurs more frequently?
If ((x mod 10) = 0)
and
((y mod 10) = 0)
and
( x > 2) then
z:= f(x, y);
/* should be
z:= g(x,y); */
else
z := f(x, y);
block and strip patterns
11
12
Intuition of ART
Intuition for ART
Failure-causing pattern
fixed but unknown
r
r
o
o
t
13
14
Intuition of ART
Adaptive Random Testing
for non-point failure patterns –
an even spread of random test cases will enhance
the fault detection capabilities
r
o
r
t
15
Adaptive Random Testing
16
How to achieve “even spread”?
Simulation and empirical results showed that
as compared with random testing,
fewer test cases needed to detect the first failure
(smaller F-measure)
F-measure of ART ≅ 50-60% of that of RT
17
1)
2)
3)
4)
5)
notion of distance
notion of exclusion
notion of partitioning
notion of perturbation
…
18
ART by distance
ART by distance
T1
T1
C1
C3
C2
19
ART by distance
20
ART by exclusion
T1
T1
T2
21
ART by exclusion
22
ART by exclusion
C1
C1
T1
T1
C2
C3
23
24
ART by exclusion
ART by exclusion
T1
T1
T2
T2
25
ART by partitioning
26
ART by partitioning
Region 1
T1
Region 2
T1
Region 3
Region 4
27
ART by partitioning
Region 1
28
ART by partitioning
Region 2
Region 1
T1
Region 2
T1
Region 4.1
T2
Region 4.2
T2
Region 3
Region 3
Region 4.3
29
Region 4.4
30
Various Even Spread Approaches
Categories of Failure Patterns
Different advantages and disadvantage
Different applications
Different research interests
• Classification of failure patterns into block,
strip and point is too coarse
•
•
•
–
–
Distance - distance measures, size of candidate set, ..
Exclusion –exclusion amount, shape of exclusion region
31
32
Number of Failure Patterns
ART F-ratio = F A R T / F R T
=
1.1
+… +
+
1
0.9
0.8
0.7
0.6
0
10
20
30
40
50
60
70
80
90
100
Number of equally sized 2D square failure regions
failure rate = 0.005
failure rate = 0.001
failure rate = 0.0005
33
34
Compactness of Failure Pattern
ART F-ratio = F A R T / F R T
1.00
a
r
1
=
a
0.95
0.90
0.85
0.80
0.75
0.70
0.65
0.60
0
10
20
30
40
50
60
70
80
90
100
r
0.005
35
0.001
0.0005
36
Distributions of Size of Failure Patterns
+
ART F-ratio = F ART / FRT
=
1.1
+ … +
1
0.9
0.8
0.7
0.6
0
10
20
30
40
50
60
70
80
90
100
Number of failure regions
equal
uniform
50% dominant
80% dominant
37
Categories of Failure Patterns
38
Theoretical Analysis
• Number of failure patterns
• Compactness of failure patterns
• Distributions of sizes of failure patterns
• Theoretical analysis is very difficult
39
Theoretical Analysis
40
Example
• Assumptions:
– size, shape and orientation of failure patterns
are known (but not their locations)
• Theoretically proved
– The best possible test case selection scheme has
an expected F-measure equal to half that of
random testing
41
42
Partition Testing versus Random Testing
Partition Testing and Random Testing
Suppose the domain is divided into two
partitions of different sizes.
Assume one test case is selected from each
partition for the case of partition testing,
and two test cases are selected for the case
of random testing.
• Sufficient conditions for partition testing to
perform better than random testing
– P-measure: probability of detecting at least one
failure
– E-measure: expected number of failures
detected
Which method is more effective?
43
44
Conclusion
Proportional Sampling Strategy
• Use ART instead of RT
• All partitions have the same sampling rates
• Random selection of test cases
45
46