1. No category

Systematic Review of Empirical Literature
in Search Based Test Case Optimization
A Thesis Presented to
Department of Computer Sciences
Faculty of Basic &Applied Sciences
By
Beenish Fatima
293-FBAS/MSSE-F09
International Islamic University Islamabad
(2013)
1
Abstract
As the software is modified, new test cases are added to the test suite that is designed for
initial version of software, the size of test suite grows and cost of regression testing increases.
In order to decrease the cost of regression testing, researchers have focused on test case
optimization (TCO) techniques, which identify a subset of test cases that provide same
coverage of software as the original test suite. There are many techniques for test case
optimization but there the main focus is on search based techniques as they find the optimal
solutions for problems where analytical algorithms are either infeasible or too complex with
huge solution space. The overall objective of performing systematic review is to identify the
impact of search based algorithms on test case optimization approaches (prioritization,
selection, reduction and optimized test case generation) by scope, performance and cost of
software testing. This SLR is conducted on 62 articles, and find the algorithms used in each
approach and group them according to their similarities. Also extract the factors that affect
the performance of algorithms and the comparison of the algorithms on the bases of coverage
of SUT, execution time, error finding rate, stability of results, and minimum resulted test
suite produce, within each optimization approach.
2
Table of Content
Chapter 1
1.
Introduction……………………………………………………………………………...7
1.1. Test Case Optimization .................................................................................................8
1.1.1. Test Case Prioritization......................................................................................8
1.1.2. Test Case Selection ...........................................................................................9
1.1.3. Test Case Reduction ..........................................................................................9
1.1.4. Test Case Generation ......................................................................................10
1.2. Search Based Software Testing…………………………………………………….10
1.3. Aims and Objectives ……………………………………………………………….11
1.4. Expected outcome ………………………………………………………………….12
1.5. Research Methodology……………………………………………………………..12
Chapter 2
2.
Protocol Definition..........................................................................................................14
2.1. Research Questions......................................................................................................14
2.2. Detail of Questions .....................................................................................................14
2.3. Search Strategy ...........................................................................................................15
2.3.1. Strategy for Search terms ................................................................................15
2.3.2. Resources to be searched .................................................................................17
2.3.3. Other sources to be searched ...........................................................................17
2.4. Study selection criteria ................................................................................................17
2.4.1. Study of inclusion and Exclusion Criteria .......................................................17
2.4.2. Quality Instrument for assessing validity.........................................................18
Chapter 3
3.
Protocol Executions.........................................................................................................20
3.1. Search String Results..................................................................................................20
3.2. After Exclusion Filter 1...............................................................................................27
3.3. After Exclusion Filter 2...............................................................................................27
Chapter 4
4.
Result for Research Question.........................................................................................33
4.1. Research Question 1....................................................................................................33
3
4.2. Research Question 2.....................................................................................................36
4.3. Research Question 3....................................................................................................37
Chapter 5
5.
Analysis for Questions....................................................................................................43
5.1. Research Question 1.....................................................................................................43
5.2. Research Question 2.....................................................................................................48
5.3. Research Question 3....................................................................................................51
Chapter 6
6.
Conclusion.......................................................................................................................57
References…………………………………………………………………………………..59
Appendix
Appendix A…………………………………………………………………………………67
Appendix B…………………………………………………………………………………88
4
List of Figures
Figure 1: Number of Algorithm used in each Approach…………………………………….43
Figure 2: Algorithms used in Prioritization ………………………………………………….44
Figure 3: Algorithms used in Selection……………………………………………………....45
Figure 4: Algorithms used in Reduction …………………………………………………….45
Figure 5: Algorithms used in Optimized Generation ………………………………………..46
5
List of Tables
Table 1: Quality Instrument Checklist for Assessing the Articles…………………………...19
Table 2: Search string result of ACM………………………………………………………..20
Table 3: Search string result of IEEE………………………………………………………..22
Table 4: Search string result of Science Direct………………………………………………23
Table 5: Search string result of Springerlink…………………………………………………25
Table 6: Exclusion filter 1 result……………………………………………………………..27
Table 7: List of selected articles for SLR……………………………………………………27
Table 8: Detailed quality score of selected articles…………………………………………..31
Table 9: Algorithms used in each optimization approach……………………………………33
Table 10: List of factors affecting the performance of the algorithms……………………….36
Table 12: Performance Comparison of algorithms………………………………………......37
6
Chapter1: Introduction
This chapter defines the background and introduction of important topics come in this
systematic literature review
1. Introduction
Software development is evolving in nature because requirements of customers change over
time. To maintain the quality in software, testing is very important. Software testing is an
investigation method conducted to provide stakeholders with information about the quality of
the product or service under test [1]. Testing plays a vital role in every stage of software
development life cycle (SDLC) and if not done properly, it can mess-up the software product.
In integration of software product components and in modified software, where client
suggests some modifications during maintenance period; regression testing is done to validate
the correct working and to maintain quality of software after modifications. For quality we
have to retest the whole software again with all test cases, as little change in code may affect
some other parts of code which is pre-verified. This is regression testing which is time
consuming to the extent that it may take several weeks and becomes costly.
Normally, in practice there is not enough time to re-test the software in entirety with all test
cases, for this purpose we apply test case optimization, to conduct major or critical testing
first, that if either time is short or if we have to quit the testing because of resources then
major objective of testing i.e. at least coverage of essential requirements or code should be
achieved. This increases testing effectiveness in terms of time and cost.
Test case optimization is a technique which minimizes the number of test cases to be
executed by set criteria, which provides the same coverage as original test suite. For
optimizing the test cases, two main approaches can be considered; generate the optimized test
cases by using algorithms or to apply optimization strategy on already generated pool of test
cases.
Example if we have only 15 test cases then 15! = 1,307,647,368,000 different orders can
develop. It is very difficult task to know the most efficient order (or subset) among this
7
solution space to choose for testing. In existing research studies there are many
techniques/algorithms for automatic optimization. Classic techniques and Search based
techniques are two approaches. Classical algorithms are straight forward deterministic
algorithms which are often incapable in real world softwares because the problems have
objectives that cannot be characterized by a set of liner equations because these problems are
very complex.
But in this research my main focus is on search based techniques. As by search based
techniques we can achieve better results at lower cost, and freeing the testers to work on
testing problems that require more imagination and creativity.
This Systematic Literature Review (SLR) is on Search Based Techniques used for Test case
optimization having empirical evidence in literature. The scope comprises of search based
techniques used for test case prioritization, test case reduction, test case selection and
optimized test case generation.
1.1.
Test Case Optimization
The Test case Optimization techniques are employed to decrease the number test cases to be
tested by utilizing two methods. To acquire optimized test cases are generated by two
different systems, either generate optimized test cases or apply optimization on already
generated test cases.
For already generated test suites optimization, the approaches used are test case selection,
reduction and prioritization. Test case selection is to select some test cases in a subset from
original test set according to some criteria. Test-suite reduction is to reduce test suite by
removal of test cases and test-suite prioritization is an ordering of the test cases in the test
suite according to the basis of coverage or fault detection rate but the original number of test
cases remain same. These approaches are discussed in detail as follow:
1.1.1. Test Case Prioritization
Test case prioritization typically sorts existing test cases according to pre-set performance
goals. Prioritization is done either by coverage criteria or fault exposing rate. Coverage based
prioritization techniques prioritize test cases according to the quantity of statements,
branches, paths, or functions executed by the test cases. And fault exposing prioritization
8
techniques reorder test cases based on the capability of the test case in terms of exposing the
faults. History-based test prioritization techniques sort the test cases in terms of the historical
value, calculated from the test case execution history, such as covered program entities.
In general, test case prioritization, given program P and test suite T, test cases in T are
prioritized with the intent of finding an ordering of test cases that will be useful over a
succession of subsequent modified version of P.
The Test Case Prioritization Problem:
Given: T a test suite; PT the set of permutations of T; f a function from PT to the real
numbers. [63]
Problem: Find T` ϵ PT
Here, PT represents the set of all possible prioritizations (orderings) of T and f is a function
that, applied to any such ordering, yields an award value for that ordering.
1.1.2. Test Case Selection
Test case selection selects few test cases from test suite in a subset by a defined technique.
This subset has same coverage as main large test suite.
The problem of test case selection can be formalized as: “Given a test suite TS that detects a
set of faults (F) in the system, our goal is to maximize FD(sn), where sn is a subset of TS of
size n and FD(sn) is the percentage of F which is detected by sn” [51].
1.1.3. Test Case Reduction
When more than one test covers the same requirement, the tests have an overlap in coverage
that suggests the possibility of removing some tests while maintaining test suite
completeness. Even though test suite reduction maintains 100% coverage of the
requirements, it does not guarantee the same fault detection capabilities as the original test
suite. The formal definition of test suite reduction problem is as follows:
Problem: Find minimal test suite T' from T which satisfies all ris covered by
original suite T.
9
Given: {t1, t2,…, tm} is test suite T from m test cases and {r1, r2,…, rn} is set of
test requirement that must be covered in order to provide desirable coverage of
the program entities and each subsets {T1, T2,…, Tn} from T are related to one
of ris such that each test case tj belonging to Ti satisfies ri. [39]
1.1.4. Test Case Generation
Test Cases are generated by automated tools e.g. JUnit, by using UML models as input,
Therefore at this level if a an optimized technique is used on UML model’s input, then
optimized set of test cases can be generated. Genetic algorithm is widely used for this
purpose.
There is a lot of work in literature related to optimization of test cases, as Zehng Li [4] et al
discussed and compared different search based algorithms for test case prioritization. And L.
Camila [12] performed comparison between NSGA-II, MoCell, SPEA2 and Random
algorithm. Here comparison is done on the bases of requirements coverage and execution
time. On the base of Requirements coverage; Random is worst whereas, SPEA2 seems to
have a better performance for smaller search spaces, while NSGA-II and MOCell have better
performance for larger search spaces and on time bases; Random algorithm had the best
lowest execution time. The second best execution time was obtained by MOCell, followed by
NSGA-II. SPEA2 was slowest algorithm.
1.2. Search Based Software Testing
Search based software engineering (SBSE) is a new system which was introduced by “Mark
Harman and Jones” in 2001 [2]. “SBSE is an approach in which search based optimization
algorithms are used to address problems in software engineering” [3]. This approach is most
practical and adaptive now-a-days because it offers automated and semi-automated solutions
for Software engineering problems having large and complex problems with multiple
competing and conflicting objectives [3]. In 2007 [5], Mark Harman in his article gave an
overview of current state of widely used search based optimization techniques and discussed
the open areas, challenges and gaps for future work in SBSE. Later on in 2009 [3], he assisted
by his companions conducted a complete systematic literature review on SBSE. In this SLR,
he conferred the importance of search based techniques in every field of software engineering
collectively and discussed trends and relationships between the techniques and the
applications to which they have been applied. Leo Rela [11] also performed a study on
10
evolutionary algorithms used to solve different search and optimization problems in the area
of software engineering.
Most commonly used Search based techniques are hill climbing, simulated annealing, greedy
algorithm, genetic algorithm, evolutionary algorithm, particle swarm optimization etc [8].
Previously used techniques were not providing good optimal and reliable solutions and were
not cost effective as well. On the other hand search based technique provides cost effective,
reliable and efficient solutions.
59% work of SBSE done on software testing [3][8]. Search based software testing is a unique
testing which is useful for very large software, like real time applications where it is almost
impossible to test all the numerous test cases in given period of time.
There is a lot of search work done in Software testing by Search based techniques. McMinne
[8] has surveyed the application of Meta heuristics search techniques to software test data
generation. Ali et al, [6][7] conducted systematic literature review (SLR) on Search based test
case generation (SB-TCG) in which he discussed the application of SB-TCG observed
through empirical evidence present in literature. W. Afzal performed systematic mapping [9]
and SLR [10] on search based software testing for non-functional system properties and pinpointed challenges in the application of search based techniques for testing the non-functional
properties of the system. Cagatay Catal performs SLR on the application of Genetic
Algorithm on TCP [14].
As mentioned, surveys and SLRs already had been done on search based software testing but
not specifically cover Search based test case optimization (SB-TCO). A bundle of work on
search based test case optimization techniques is present in literature in wide variation. But
not all together on one place, so there is need to do a systematic literature review on SBTCO, which provides state-of-the-art on SB-TCO for future research and help to identify the
gaps.
1.3. Aims and Objectives
A bundle of work on search based test case optimization techniques is present in literature in
wide variation. But not all together on one place, so there is need to do a systematic literature
11
review on SB-TCO, which provides state-of-the-art on SB-TCO for future research and help
to identify the gaps.
My aim is to identify the impact of search based test case optimization techniques on scope,
performance and cost of software testing. And in this SLR my objective is to answer the
following:

What search based techniques are being used for test case optimization?
o Group these techniques on the base of procedure they follow.
o A comparison will be conducted within each approach of optimization to measure
the performance of search based techniques on the following criteria: coverage
effectiveness, response time, defect detection rate.
o And find the factors that affect the performance of these techniques.
1.4. Expected outcome
Expected outputs from my SLR are:

List of grouped search based techniques used in TCO.

List of factors affecting the performance of these techniques.

Best search Based techniques for test case optimization approaches.
1.5. Research Methodology
To answer my question I will conduct Systematic Literature Review (SLR) as research
methodology. “A SLR is a means of evaluating and interpreting all available research
relevant to a particular research question, topic area, or phenomena of interest. Systematic
reviews aim to present a fair evaluation of a research topic by using a trustworthy, rigorous
and auditable methodology” [13].
Systematic review consists of three phases planning the review, conducting the review, and
reporting the review. In proposal document there was only the planning document called
systematic review protocol. For SLR I followed the guidelines of Kitchenham and my
Systematic review protocol consists of [13]:
• Background.
• The research questions that the review is intended to answer.
12
• The strategy that will be used to search for primary studies including search terms and
resources to be searched.
• Study selection criteria. Study selection criteria are used to determine which studies are
included in, or excluded from, a systematic review.
• Study selection procedures.
• Study quality assessment checklists and procedures.
• Data extraction strategy.
• Synthesis of the extracted data.
• Project timetable.
Detail of my systematic review protocol is given in Appendix A.
13
Chapter: 2. Protocol Definition
This chapter describes the procedure followed to conduct the Systematic Literature Review
on search based test case optimization.
2.
Protocol Definition
2.1.
Research Questions
One of the rationales for this SLR is to summarize the existing evidence concerning Search
Based Test Case Optimization (TCO). So my evidence based investigation is mainly focused
on this. To understand state-of-the-art empirical work in Search Based TCO, in terms of gaps,
commonalities in existing empirical results and the comparison between Search Based
techniques used for TCO, the following three research questions are being formulated. There
is vast amount of variations in this context, and until now no clear guidance for applying the
right evolutionary algorithm is in the right situation. The main driver for conducting the
systematic review was to identify gaps, commonalities in empirical research related to Search
Based Test Case Prioritization and sum up the existing practices to answer the questions:
who, where, what, why and how in use of Search techniques. The research questions are
formulated to form a baseline for state-of-the-art empirical work and hence the objective is
that the systematic review should form a milestone for both future research and for practical
use by practitioners. Thus my major concerns are as follows:
RQ.1: Determine the Search Based Techniques being used for test case prioritization and
group these techniques?
RQ.2: What are different factors that are affecting the performance of these techniques?
RQ.3: What is the performance comparison of selected Techniques?
2.2. Detail of Questions
RQ.1: Determine the Search Based Techniques being used for test case optimization
and group these techniques?
The aim is to understand the existing research directions of test case optimization with
the help of search based techniques and mainly the work is considered having some
empirical work attached with it. There are many search based techniques are used for
test case optimization such as Greedy algorithm, Genetic algorithm, particle swarm
optimization, etc., and there versions. So from systematic literature review I have list
down all techniques (algorithms) and group those according to their similarities.
Grouping of techniques have been based on two different criteria. The first criteria is
based on the versions of algorithm/heuristic being used i.e. different versions of Genetic
Algorithm (GA) are considered as one group. Second grouping is based on the software
14
testing life cycle phase where algorithm or the technique is being applied i.e. Test Case
generation, Test Case Reduction, Test Case Optimization etc. The solution to this
question gives us the state-of-the-art empirical work of search based techniques used for
test case optimization.
RQ.2: What are different factors that are affecting the performance of these
techniques?
From listed techniques for all optimization approaches, I select some (not all)
evolutionary algorithms. And from empirical work I have identify the factors that can
affect the performance of these techniques such as size of the code, size of test suite,
size of test cases, etc. The solution of this question consist of all factors on which the
performance of these techniques be measured.
RQ.3: What is the performance comparison of selected Techniques?
In empirically study on search based TCO there is much diversity, hence conducting the
performance comparison is difficult. In this SLR I have conduct the performance
comparison of techniques with in each optimization approach on the bases of: coverage
effectiveness, response time, defect detection rate. In the solution I’ll get performance
of selected search based optimization techniques in comparison with each other.
Structured Questions
RQ.1: Determine the Search Based Techniques being used for test case optimization and
group these techniques?
Population: Empirical Literature for Search based test case optimization.
Outcome: List of grouped search based techniques used in TCO.
Intervention: SB-TCO.
Comparison: On the bases of similarities.
RQ.2: What are different factors that are affecting the performance of these techniques?
Population: Data Regarding Performance of Search Algorithm for test case
optimization.
Outcome: List of factors affecting the performance of these techniques.
Intervention: Search Based Test Case optimization
Comparison: No.
RQ.3: What is the performance comparison of selected Techniques?
Population: selected search based techniques
Outcome: Best search Based techniques for test case optimization approaches.
Intervention: SB-TCO.
Comparison: On the bases of coverage effectiveness, response time, defect detection rate.
15
2.3. Search Strategy
2.3.1. Strategy for Search terms
In search strategy, the search process will be conducted according to the following decided
steps:
1. Derive major search strings from my SLR topic.
2. Identify alternative spellings and synonyms for major terms; also alternative terms used in
literature will be considered (e.g. prioritization, optimization)
3. When database allows, use the Boolean OR to incorporate alternative spellings and
synonyms; and when database allows, use the Boolean AND to link the major terms from
population, intervention and outcome.
4. Break down the strings so that it can be applied to different databases. Give unique ID to
every segmented search string. And I will customize them for given resources to be
applied for searches.
5. Tools used for automating the search process
Step1: Major Search Terms:
-
Search Based techniques
Test case
Optimization
Empirical
Step 2: alternative spellings and synonyms for major terms
-
-
Search Based optimization algorithm (Search Based Optimization Algorithm OR
Genetic algorithm OR GA OR Particle swarm optimization OR PSO OR Evolutionary
Algorithm OR metaheuristic)
Test case (Test case OR Test suite OR Test data)
Prioritization (Prioritization OR Optimization OR Reduction OR Selection OR
Minimization OR optimized Generation)
Empirical (Empirical study OR Case study OR Survey OR Technical Report OR
Experiment)
Step 3: Integrating Boolean <OR> and <AND>
(Search Based Optimization Algorithm OR Genetic algorithm OR GA OR Particle swarm
optimization OR PSO OR Evolutionary Algorithm OR metaheuristic) AND (Test Case OR
Test Suit) AND (Prioritization OR Optimization OR Reduction OR Selection OR
Minimization OR optimized Generation) AND (Empirical study OR Case study OR
Survey OR Technical Report OR Experiment)
Step 5: Tools used for automating the search process
16
To speed up the process of documenting my search results I will be using two tools which
will help me in automating the process. These tools are as follows:
-
End notes [for documenting bibliography from searches]; I will retrieve search results
in End notes and later, on the basis of titles and abstracts I will include or exclude
papers. It will help me in documenting my search results for both primary searches and
for primary studies.
MS Access [to track the references].
-
2.3.2. Resources to be searched
A range of data bases has been selected for rigorous search and to reduce the bias. Following
data bases will be searched for the retrieval of primary studies:
1.
2.
3.
4.
Springer link
IEEE Explore
ACM Digital library
Science Direct
2.3.3. Other sources to be searched
-
Various databases are being selected to reduce the bias of study including published
Technical Reports, Journal Papers and Conference Proceedings.
Then I will go for secondary studies mentioned in the references of the each primary
study by applying same procedures as applied for primary studies.
-
2.4. Study selection criteria







The initial selection is done on the basis of the TITLE and ABSTRACT of the paper.
All data obtained from the search process are archived in database according to the
journal from which it is retrieved.
From database the duplicates originating from various search engines are removed
after initial scan of results.
Inclusion and exclusion criteria are applied on the remaining articles to sort out the
accepted ones.
On accepted papers detailed inclusion criteria is applied which is a Quality
Instrument.
Full papers of all studies that are not clearly ineligible to fit in the criteria, is discussed
with supervisor for their final exclusion.
The excluded papers and reasons for exclusion are recorded in a file, and the included
papers and study type are recorded in another file.
17
2.4.1. Study of inclusion and Exclusion Criteria
Various criteria are formulated with the intension to identify those studies that provide direct
evidence for the research questions. Following are the inclusion and exclusion criteria for
my research questions:
Inclusion criteria:
In order to answer the stated research questions, I have been searching for research articles
by reading the abstracts that contain information about search based test case optimization
having empirical work done in it. So the papers including a case study, an experiment,
survey or industrial experience reports are focused. When it is confirmed after reading the
abstract that article is relevant to my research, I have study the whole paper. The objective
of the selection process was to identify the Articles relevant to the objectives of the
systematic review. The search strings, are quite broad and hence it is expected that not all
studies identified would find a place in the final phase of selection process. Only those
studies are included that are empirical based and where the main focus is test case
optimization with search based technique(s).
Exclusion criteria:
-
Those studies are excluded that are based on personal expert opinion.
One study can be reported in multiple papers so on the basis of validity score I included
highest ranked journals and exclude others.
Multiple studies can be reported in one paper; if any of them do not stand to our
inclusion criteria and differ variably then only the irrelevant study is excluded.
2.4.2. Quality Instrument for assessing validity
After initial selection of studies, a more detail criteria is required to judge the quality of study
to see whether it is worth considering as evidence to answer our research question or not.
Quality Instrument will be designed for assigning numerical values for factors in the checklist
to be evaluated for each study. My main focus is on the study design (of the paper).
The research questions and my inclusion/exclusion criteria will suggest me that I am going to
include only empirical studies like Case studies, Industrial Experience Reports etc.
If any paper is considered to be very poor in quality it will not be excluded at this stage and
its record will be maintained. Quality assessment checklists for empirical data e.g. survey,
case study, experiment and experience report etc will be used from guidelines of Kitchenham
(2007) [13].
A Quality Instrument including check list for assessing the quality of study and assigning
numerical values to the questions to rank the selected papers.
18
Table 1: Quality Instrument Checklist for Assessing the Articles
Sr.
Quality Assessment Questions
No.
General Paper Evaluation
Do the author(s) clearly mention the issues
1
or hypothesis for the paper?
Is the paper clearly describe/define the
2
design (procedure followed, treatments and
all controls) of empirical study used?
3
Are the findings or results clearly stated?
Does the case/data choose for analysis is
4
big enough to narrate with real life
problem?
Is the data used for analysis sufficiently
5
rigorous?
Is the article written in a balanced and
6
understandable manner?
7
Are threats to validity described?
According to Research Question Evaluation
Relevant to my research question?
8
(Score 1 for each research question)
Does the article include more than one
9
algorithm?
Do the search based algorithm(s) described
10
in detail?
Are the factors affecting the performance
11
stated?
Is the performance comparison between
12
algorithms clearly presented?
Total
Scale
Score
Y=1/Partly=0.5/N=0
Y=1/Partly=0.5/N=0
Y=1/Partly=0.5/N=0
Y=1/Partly=0.5/N=0
Y=1/Partly=0.5/N=0
Y=1/Partly=0.5/N=0
Y=1/Partly=0.5/N=0
Strongest=3Lowest=0
Y=1/N=0
Y=1/Partly=0.5/N=0
Y=1/Partly=0.5/N=0
Y=1/Partly=0.5/N=0
14
19
Chapter: 3 Protocol Executions
This chapter comprises of protocol execution, by which we get the result of search strings,
apply inclusion/exclusion criteria on the result and get the final list of articles to include in
SLR.
3.
Protocol Execution
3.1. Search String Results
Result of Search Strings on Digital library Databases:
Search String Result of ACM
As ACM does not allow that huge search string so it is divided into 7 small search strings and
with filter find in abstract. Their results are given in the table below.
Table 2: Search string result of ACM
ACM
(Abstract:"Particle swarm optimization" or Abstract: PSO or
Papers Found
26
Abstract: swarm) and (Abstract:"test case" or Abstract:"test
1
suite") and (Abstract: prioritization or Abstract: optimization
or Abstract: reduction or Abstract: selection or Abstract:
minimization or Abstract:"optimized generation")
(Abstract:"Genetic" or Abstract: GA) and (Abstract:"test
62
case" or Abstract:"test suite") and (Abstract: prioritization or
2
Abstract: optimization or Abstract: reduction or Abstract:
selection or Abstract: minimization or Abstract:"optimized
generation")
((Abstract:"ant colony optimization" or Abstract: ACO or
9
Abstract: colony) and (Abstract:"test case" or Abstract:"test
3
suite") and (Abstract: prioritization or Abstract: optimization
or Abstract: reduction or Abstract: selection or Abstract:
minimization or Abstract:"optimized generation"))
20
(Abstract:"Evolutionary Algorithm" or Abstract: Evolution
46
or Abstract: bacteria or Abstract: bacterial) and
4
(Abstract:"test case" or Abstract:"test suite") and (Abstract:
prioritization or Abstract: optimization or Abstract:
reduction or Abstract: selection or Abstract: minimization or
Abstract:"optimized generation")
(((Abstract:"Search Based" OR Abstract: "SBST" OR
Abstract: metaheuristic OR Abstract: heuristic) AND
5
(Abstract: "test case" OR Abstract: "test suite") AND
65
(Abstract: "prioritization" OR Abstract: "optimization" OR
Abstract: "reduction" OR Abstract: "selection" OR Abstract:
"minimization" OR Abstract: optimized generation)))
((((Abstract:"neural network") and (Abstract:"test case" or
Abstract:"test suite") and (Abstract: prioritization or
6
3
Abstract: optimization or Abstract: reduction or Abstract:
selection or Abstract: minimization or Abstract:"optimized
generation"))))
((((Abstract: greedy) and (Abstract:"test case" or
Abstract:"test suite") and (Abstract: prioritization or
7
8
Abstract: optimization or Abstract: reduction or Abstract:
selection or Abstract: minimization or Abstract:"optimized
generation"))))
TOTAL
219
After removing duplicates
167
Total of all search strings results are 219. And after removing duplicates remaining articles
are 167.
Search String Result of IEEE
As IEEE does not allow that huge search string so it is divided into 7 small search strings and
with filter find in abstract refined by Topic: Computing & Processing (Hardware/Software).
Their results are given in the table below.
Table 3: Search string result of IEEE
21
1
IEEE
Results
((((("Abstract":"particle swarm optimization") OR
36
"Abstract":"PSO") OR "Abstract":swarm) AND
(("Abstract":"test case") OR "Abstract":"test suite")) AND
(((((("Abstract":optimization) OR "Abstract":prioritization)
OR "Abstract":selection) OR "Abstract":optimized
generation) OR "Abstract":reduction) OR
"Abstract":minimization))
2
(((("Abstract":Genetic) OR "Abstract":GA) AND
44
((“Abstract”:"test case") OR "Abstract":"test suite")) AND
((((((“Abstract”:optimization) OR "Abstract":prioritization)
OR "Abstract":selection) OR "Abstract":optimized
generation) OR "Abstract":reduction) OR
"Abstract":minimization))
3
((((("Abstract":"ant colony optimization") OR
11
"Abstract":"ACO") OR "Abstract":colony) AND
(("Abstract":"test case") OR "Abstract":"test suite")) AND
(((((("Abstract":optimization) OR "Abstract":prioritization)
OR "Abstract":selection) OR "Abstract":optimized
generation) OR "Abstract":reduction) OR
"Abstract":minimization))
4
(((((("Abstract":"Evolutionary Algorithm") OR
49
"Abstract":Evolution) OR "Abstract":bacteria) OR
"Abstract":bacterial)AND (("Abstract":"test case") OR
"Abstract":"test suite")) AND (((((("Abstract":optimization)
OR "Abstract":prioritization) OR "Abstract":selection) OR
"Abstract":optimized generation) OR "Abstract":reduction)
OR "Abstract":minimization))
5
(((((("Abstract":"Search Based") OR "Abstract":SBST) OR
37
"Abstract":metaheuristic) OR "Abstract":heuristic)AND
(("Abstract":"test case") OR "Abstract":"test suite")) AND
(((((("Abstract":optimization) OR "Abstract":prioritization)
OR "Abstract":selection) OR "Abstract":optimized
22
generation) OR "Abstract":reduction) OR
"Abstract":minimization))
6
(((("Abstract":“neural network”) AND ((“Abstract”:"test
3
case") OR "Abstract":"test suite")) AND
((((((“Abstract”:optimization) OR "Abstract":prioritization)
OR "Abstract":selection) OR "Abstract":optimized
generation) OR "Abstract":reduction) OR
"Abstract":minimization))
7
((((("Abstract":greedy) AND (("Abstract":"test case") OR
0
"Abstract":"test suite")) AND (((((("Abstract":optimization)
OR "Abstract":prioritization) OR "Abstract":selection) OR
"Abstract":optimized generation) OR "Abstract":reduction)
OR "Abstract":minimization)))
TOTAL
180
After Remove Duplicates
129
Total of all search strings results are 180. And after removing duplicates remaining articles
are 129.
Search String Result of Science Direct
As Science Direct does not allow that huge search string so it is divided into 7 small search
strings and with filter find in title and abstract refined by Topic: All Sources (Computer
Science). Their results are given in the table below.
Table 4: Search string result of Science Direct
Science Direct
1
(TITLE-ABSTR-KEY("Particle swarm optimization") OR
21
TITLE-ABSTR-KEY(PSO) OR TITLE-ABSTR-KEY(swarm))
and (TITLE-ABSTR-KEY("test case") OR TITLE-ABSTRKEY("test suite")) and (TITLE-ABSTR-KEY(optimization) OR
TITLE-ABSTR-KEY(prioritization)OR TITLE-ABSTRKEY(reduction) OR TITLE-ABSTR-KEY(minimization) OR
TITLE-ABSTR-KEY(selection) OR TITLE-ABSTRKEY("optimized generation"))
23
2
(TITLE-ABSTR-KEY(Genetic) or TITLE-ABSTR-KEY(GA))
35
and (TITLE-ABSTR-KEY("test case") or TITLE-ABSTRKEY("test suite")) and (TITLE-ABSTR-KEY(optimization) or
TITLE-ABSTR-KEY(prioritization) or TITLE-ABSTRKEY(reduction) or TITLE-ABSTR-KEY(selection) or TITLEABSTR-KEY(minimization) or TITLE-ABSTR-KEY("optimized
generation"))
3
(TITLE-ABSTR-KEY("ant colony optimization") OR TITLE-
4
ABSTR-KEY(ACO) OR TITLE-ABSTR-KEY(colony)) and
(TITLE-ABSTR-KEY("test case") OR TITLE-ABSTRKEY("test suite")) and (TITLE-ABSTR-KEY(optimization) OR
TITLE-ABSTR-KEY(prioritization)OR TITLE-ABSTRKEY(reduction) OR TITLE-ABSTR-KEY(minimization) OR
TITLE-ABSTR-KEY(selection) OR TITLE-ABSTRKEY("optimized generation"))
4
(TITLE-ABSTR-KEY("Search Based") OR TITLE-ABSTR-
37
KEY(SBST) OR TITLE-ABSTR-KEY(metaheuristic) OR
TITLE-ABSTR-KEY(heuristic)) and (TITLE-ABSTR-KEY("test
case") OR TITLE-ABSTR-KEY("test suite")) and (TITLEABSTR-KEY(optimization) OR TITLE-ABSTRKEY(prioritization)OR TITLE-ABSTR-KEY(reduction) OR
TITLE-ABSTR-KEY(minimization) OR TITLE-ABSTRKEY(selection) OR TITLE-ABSTR-KEY("optimized
generation"))
5
(TITLE-ABSTR-KEY("Evolution") OR TITLE-ABSTR-
22
KEY("Evolutionary Algorithm") OR TITLE-ABSTRKEY(Bacteria) OR TITLE-ABSTR-KEY(bacterial)) and (TITLEABSTR-KEY("test case") OR TITLE-ABSTR-KEY("test suite"))
and (TITLE-ABSTR-KEY(optimization) OR TITLE-ABSTRKEY(prioritization)OR TITLE-ABSTR-KEY(reduction) OR
TITLE-ABSTR-KEY(minimization) OR TITLE-ABSTRKEY(selection) OR TITLE-ABSTR-KEY("optimized
generation"))
24
6
(TITLE-ABSTR-KEY("neural network")) and (TITLE-ABSTR-
12
KEY("test case") or TITLE-ABSTR-KEY("test suite")) and
(TITLE-ABSTR-KEY(optimization) or TITLE-ABSTRKEY(prioritization) or TITLE-ABSTR-KEY(reduction) or
TITLE-ABSTR-KEY(selection) or TITLE-ABSTRKEY(minimization) or TITLE-ABSTR-KEY("optimized
generation"))
7
(TITLE-ABSTR-KEY(greedy)) and (TITLE-ABSTR-KEY("test
4
case") or TITLE-ABSTR-KEY("test suite")) and (TITLEABSTR-KEY(optimization) or TITLE-ABSTRKEY(prioritization) or TITLE-ABSTR-KEY(reduction) or
TITLE-ABSTR-KEY(selection) or TITLE-ABSTRKEY(minimization) or TITLE-ABSTR-KEY("optimized
generation"))[All Sources(Computer Science)]
TOTAL
135
After Remove Duplicates
93
Total of all search strings results are 135. And after removing duplicates remaining articles
are 93.
Search String Result of Springerlink
As Springerlink does not allow that huge search string so it is divided into 13 small search
strings and with filter find in abstract and with Topic Computer Science. Their results are
given in the table below.
Table 5: Search string result of Springerlink
1
Springerlink
Results
ab:(("Particle swarm optimization" or PSO or swarm) and ("test
9
case" or "test
suite") and (optimization or prioritization or minimizationor selec
tion))
ab:(("Particle swarm optimization" or PSO or swarm) and ("test
3
case" or "test suite") and (reduction or generation))
25
2
ab:((genetic or GA) and ("test case" or "test
17
suite")and (optimization or prioritization or minimization or selec
tion or reduction or generation))
3
ab:(("ant colony optimization" or colony or ACO)and ("test
4
case" or "test
suite") and (optimization or prioritization or minimization or
selection or reduction))
ab:(("ant colony optimization" or colony or ACO)and ("test
2
case" or "test suite") and (generation))
4
ab:(("search based" or SBST or metaheuristic or
5
heuristic) and ("test case" or "test
suite") and (optimization or minimization or prioritization))
ab:(("search based" or SBST or metaheuristic or
16
heuristic) and ("test case" or "test
suite") and (selection or reduction or generation))
5
ab:((evolution or "evolutionary algorithm" or
10
bacteria or bacterial) and ("test case" or "test
suite") and (optimization or minimization or prioritization))
ab:((evolution or "evolutionary algorithm" or
5
bacteria or bacterial) and ("test case" or "test
suite") and (selection or reduction or generation))
6
'ab:(("neural network") and ("test case" or "test
3
suite") and (optimization or prioritization or minimization or sele
ction or reduction orgeneration))' with no filters
7
ab:((greedy) and ("test case" or "test
2
suite") and(optimization or prioritization or minimization or selec
tion or reduction or generation))
TOTAL
76
After duplicates remove
54
Total of all search strings results are 76. And after removing duplicates remaining articles are
54.
26
3.2. After Exclusion Filter 1
In filter 1, papers are discarded after reading abstract. Discarding criteria was:

Paper not related to software testing.

Does not include in empirical study.
Filter 1 results are in table given below
Table 6: Exclusion filter 1 result
1st filter
IEEE
60
Science Direct
23
Springerlink
30
ACM
84
Total 197
After removing duplicates 152
3.3. After Exclusion Filter 2
Inclusion/exclusion criteria described in protocol definition is applied as filter 2. And
remaining articles are 60. And two articles are selected from secondary search.
After filter 2, as a result, 62 articles selected are listed below:
Table 7: List of selected articles for SLR
Study
Included Articles
Reference
s-1
Test Case Prioritization Technique based on Genetic Algorithm
[63]
s-2
A history-based cost-cognizant test case prioritization technique in [74]
#
regression testing
s-3
Automated Test Case Prioritization with Reactive GRASP
[59]
s-4
Analyzing Test Case Selection & Prioritization using ACO
[71]
s-5
Applying Particle Swarm Optimization to Prioritizing Test Cases for [32]
Embedded Real Time Software Retesting
s-6
Building Prioritized Pairwise Interaction Test Suites with Ant Colony [41]
27
Optimization
s-7
Design and Analysis of Cost-Cognizant Test Case Prioritization Using [57]
Genetic Algorithm with Test History
s-8
Test Case Filtering and Prioritization Based on Coverage of [35]
Combinations of Program Elements
s-9
Empirically Studying the Role of Selection Operators During Search- [55]
Based Test Suite Prioritization
s-10
Using Synthetic Test Suites to Empirically Compare Search-Based and [44]
Greedy Prioritizers
s-11
Time-Aware
Test-Case
Prioritization
using
Integer
Linear [34]
Programming
s-12
Time-Aware Test Suite Prioritization
[22]
s-13
Automated Test Case Generation and Its Optimization for Path Testing [43]
Using Genetic Algorithm and Sampling
s-14
Automated Generation of Test Cases from Output Domain of an [70]
Embedded System using Genetic Algorithms
s-15
Java based Test case Generation and Optimization Using Evolutionary [26]
Testing
s-16
A Multi-Objective Particle Swarm Optimization for Test Case [65]
Selection Based on Functional Requirements Coverage and Execution
Effort
s-17
An Enhanced Test Case Selection Approach for Model-Based [54]
Testing: An Industrial Case Study
s-18
An Industrial Investigation of Similarity Measures for Model-Based [51]
Test Case Selection
s-19
The Research of Test-Suite Reduction Technique
[64]
s-20
A Novel Chaos Discrete Particle Swarm Optimization Algorithm for [50]
Test Suite Reduction
s-21
A Technique to Reduce the Test Case Suites for Regression Testing [23]
Based on a Self-Organizing Neural Network Architecture
s-22
Test Case Reduction Based on Program Invariant and Genetic [45]
Algorithm
s-23
An Evolutionary Algorithm for Regression Test Suite Reduction
[53]
28
s-24
Automated software test optimization framework – an artificial bee [60]
colony optimization-based approach
s-25
Evolutionary Testing of Object-Oriented Software
[52]
s-26
An Optimization Strategy for Evolutionary Testing Based on [49]
Cataclysm
s-27
Test Case Evaluation and Input Domain Reduction strategies for the [36]
Evolutionary Testing of Object-Oriented software
s-28
An experimental study of four typical test suite reduction techniques
[29]
s-29
Strongly-Typed Genetic Programming and Purity Analysis: Input [27]
Domain Reduction for Evolutionary Testing Problems
s-30
Search-Based Test Case Generation for Object-Oriented Java Software [28]
Using Strongly-Typed Genetic Programming
s-31
An Empirical Evaluation of Test Suite Reduction for Boolean [30]
Specification-based Testing
s-32
Test-Suite Reduction Using Genetic Algorithm
[19]
s-33
Black-box System Testing of Real-Time Embedded Systems Using [58]
Random and Search-based Testing
s-34
Multi-Objective Test Suite Minimization Using Quantum-Inspired [73]
Multi-Objective Differential Evolution Algorithm
s-35
A new heuristic for test suite reduction
[15]
s-36
Search Algorithms for Regression Test Case Prioritization
[4]
s-37
A Genetic Algorithm for Test-Suite Reduction
[20]
s-38
Automatic test case optimization using a bacteriological adaptation [18]
model: application to .NET components
s-39
A Case Study on the Use of Genetic Algorithms to Generate Test [69]
Cases for Temporal Systems
s-40
A non-pheromone based intelligent swarm optimization technique in [38]
software test suite optimization
s-41
A concept analysis inspired greedy algorithm for test suite [21]
minimization
s-42
An Empirical Study of Incorporating Cost into Test Suite Reduction [40]
and Prioritization
s-43
A New Algorithm to Test Suite Reduction Based on Cluster Analysis
[39]
29
s-44
Study of Test Suite Reduction Based on Quantum Evolutionary [46]
Algorithm
s-45
Automated GUI Test Coverage Analysis using GA
[61]
s-46
An Improved Quantum Genetic Algorithm for Test Suite Reduction
[66]
s-47
Designing Better Fitness Functions for Automated Program Repair
[56]
s-48
Variable Strength Interaction Testing with an Ant Colony System [33]
Approach
s-49
Comparison of nearest point algorithms by genetic algorithms
[68]
s-50
A Simulation Study on Some Search Algorithms for Regression Test [47]
Case Prioritization
s-51
On the Value of Combining Feature Subset Selection with Genetic [37]
Algorithms: Faster Learning of Coverage Models
s-52
Search based constrained test case selection using execution effort
[75]
s-53
Genes and Bacteria for Automatic Test Cases Optimization in the [17]
.NET Environment
s-54
Automated test data generation using a scatter search approach
[42]
s-55
Quality Improvement and Optimization of Test Cases – A Hybrid [48]
Genetic Algorithm Based Approach
s-56
A variable strength interaction test suites generation strategy using [62]
Particle Swarm Optimization
s-57
Software Reliability Testing Covering Subsystem Interactions
[72]
s-58
Testing an Intentional Naming Scheme Using Genetic Algorithms
[16]
s-59
Automatic, evolutionary test data generation for dynamic software [31]
testing
s-60
Evolutionary mutation testing
[67]
s-61
Iterative automatic test generation method for telecommunication [25]
protocols
s-62
Scatter Search in software testing, comparison and collaboration with [24]
Estimation of Distribution Algorithms
There Quality Score detail is detailed in the table below
30
Table 8: Detailed quality score of selected articles
Ref#\Q General Paper Evaluation
1
2
3
4
5
6
7
According to Research
Questions
8
9
10
11
Total
1
0
1
0.5
0.5
0.5
0.5
0
1
1
1
1
1
8
2
1
1
1
1
1
1
0
3
1
1
1
1
13
3
1
1
1
1
1
1
0.5
2
1
0.5
0
1
11
4
1
0.5
1
1
1
1
0
2
0
0.5
1
0
9
5
1
1
1
0.5
0.5
1
0
1
1
1
0
1
9
6
1
1
1
0.5
0.5
1
0
3
1
1
0
1
11
7
1
1
1
1
1
1
0
3
1
1
1
1
13
8
1
1
1
1
1
1
0
2
1
1
0
0
10
9
1
1
1
0.5
1
1
1
3
1
1
1
1
13.5
10
0.5
0.5
0.5
0
1
0.5
0
1
1
0
0.5
0.5
6
11
1
1
1
0.5
1
1
1
0.5
1
0
0
0.5
8.5
12
0.5
1
1
0.5
0.5
1
1
2
0
1
1
1
11
13
0.5
0.5
0.5
0.5
0.5
0.5
0
1
0
0.5
0
0
4.5
14
0.5
1
1
0.5
0.5
1
0
1
0
1
0
0
6.5
15
0
1
0.5
0.5
0
1
0
1
0
1
0
0
5
16
1
1
1
1
1
1
0.5
2
1
1
0
1
11.5
17
1
1
1
1
1
1
1
1
1
0
0.5
0.5
10
18
1
1
1
0.5
1
1
1
1.5
1
0.5
1
1
11.5
19
1
0.5
0.5
0.5
0.5
1
0
1.5
1
1
0
0.5
8
20
1
1
1
0.5
0.5
1
0
2
1
1
1
1
11
21
1
0.5
1
1
0.5
1
0
1.5
1
0.5
0
0.5
8.5
22
0.5
1
1
0.5
0.5
1
0
1.5
1
1
0
0.5
8.5
23
0.5
0.5
1
1
1
1
0
1
0
0.5
0
0
6.5
24
1
1
1
1
1
1
0
3
1
1
1
1
13
25
1
1
1
1
1
0.5
0
2
1
0
1
0.5
10
26
1
0.5
0.5
0.5
1
0.5
0
2
1
0
1
0.5
8.5
27
1
0.5
0.5
1
0.5
1
0
0.5
0
0
0.5
0
5.5
28
1
0.5
1
0.5
1
1
1
2.5
1
0.5
1
1
12
29
1
0.5
0.5
0.5
0.5
0.5
0
0.5
0
0
0
0
4
12
Score
31
30
1
0.5
0.5
0.5
0
0.5
0
1
0
0.5
1
0
5.5
31
1
0.5
1
1
1
1
0
3
1
0
1
1
11.5
32
1
1
1
1
1
1
0
3
1
1
1
1
13
33
1
0.5
0.5
1
1
1
1
2.5
1
0
1
1
12
34
1
1
1
1
1
1
0
2
1
1
0
1
11
35
1
1
0.5
0.5
0.5
1
0
2
1
0.5
0
0.5
9.5
36
1
1
1
1
1
1
1
3
1
1
1
1
14
37
1
1
0.5
1
1
1
0
2
1
1
0
1
10.5
38
1
1
0.5
0.5
0.5
0.5
0
2.5
1
1
1
1
11
39
1
0.5
0.5
0.5
0.5
0.5
0
2
1
0.5
1
0.5
8.5
40
1
1
1
1
1
1
0
3
1
1
1
1
13
41
1
1
0.5
1
0.5
0.5
0
2
1
1
0
0.5
9
42
1
0.5
1
1
1
0.5
0
2
1
0.5
0.5
1
10
43
1
1
1
0.5
1
1
1
2
1
1
0
1
11.5
44
1
0.5
1
0.5
0.5
1
0
2.5
1
1
0.5
1
10.5
45
1
1
1
1
1
1
0
2
0
1
1
0
10
46
1
0.5
1
0.5
0.5
1
0
3
1
1
1
1
11.5
47
1
1
1
1
1
0.5
0
3
1
0
1
1
11.5
48
1
0.5
0.5
0.5
0
0.5
0
2.5
1
1
1
0.5
9
49
1
1
1
1
1
1
0
3
1
1
1
1
13
50
1
1
1
0
1
1
1
2
1
1
0
1
11
51
1
1
1
0.5
1
1
0
0.5
0
0.5
0
0
6.5
52
1
1
1
1
1
1
0
3
1
1
1
1
13
53
1
0.5
1
1
0.5
1
0
3
1
1
1
1
12
54
1
0.5
1
0.5
0.5
1
0
2
1
1
0
1
9.5
55
1
1
1
1
1
1
0
3
1
1
1
1
13
56
1
0.5
0.5
0.5
0.5
0.5
0
1
1
1
0
0
6.5
57
1
1
0.5
0.5
0.5
1
0
0.5
0
0.5
0
0
5.5
58
0.5
0.5
0.5
1
1
1
0
1
0
1
0
0
6.5
59
1
0.5
1
1
1
1
0
1.5
1
1
0
0.5
9.5
60
1
1
1
1
1
1
1
3
1
1
1
1
14
61
1
0.5
0.5
0.5
1
1
0
2.5
1
1
0.5
1
10.5
62
1
1
1
1
1
1
0.5
3
1
1
1
1
13.5
32
Chapter 4: Results
This is the Chapter describing results of all research questions of this SLR and these results
are based on selected articles which have been included in this SLR. Results are taken from
general extraction derived from Articles attached in Appendix B. There are three tables as
under to address research questions respectively.
4. Result for Research Questions
4.1. Research Question 1
Determine the Search Based Techniques being used for test case optimization and group
these techniques?
The solution to this question lists down all techniques (algorithms) used in literature for each
approach.
Table 9: Algorithms used in each optimization approach.
Optimization
Approach
Prioritization
Algorithms
Random prioritization[s-2][s-5][ s-9][ s-11][ s-49]
Genetic Algorithm[s-1][ s-3][ s-8][ s-9][ s-12][ s-36][ s-49][ s-50]
 GA-hist[s-2][ s-7]
 time-aware statement coverage prioritization via GA[s-11]
 time-aware method coverage prioritization via GA[s-11]
 Time-aware test suite prioritization (GA-fun)[ s-2][ s-7]
 Improved Genetic Algorithm[s-1]
Greedy algorithm[s-3][ s-8][ s-9][ s-10][ s-36][ s-42]
 Additional Greedy algorithm[s-3][ s-36][ s-50]
 Total Greedy Algorithm [s-50]
 delayed greedy[s-42]
 2-Optimal Greedy Algorithm[s-36][ s-42][ s-50]
 Reactive GRASP[s-3]
Simulated annealing[s-3]
Ant Colony Optimization (ACO)[ s-4]
 Ant System (AS)[s-6]
 Ant System with Elitist (ASelite)[ s-6]
 Ant Colony System (ACS)[s-6]
 Max-Min Ant System(MMAS)[ s-6]
Particle Swarm Optimization[s-5]
Nearest point algorithm[s-49]
Hill Climbing[s-9][ s-10][ s-36][ s-50]
Heuristics
 Harrold Gupta Soffa (HGS)[ s-42]
 hist-value[s-7]
33


Reduction
Selection
Optimal prioritization (optimal)[ s-2]
Historical fault detection effectiveness prioritization (histfaultdetected)[ s-2][ s-7]
 Total function coverage prioritization (total-fun)[ s-2][ s-7]
 Cost-cognizant total function coverage prioritization (cctotal-fun)
[s-2][ s-7]
 traditional total statement coverage prioritization[s-11]
 traditional additional statement coverage prioritization[s-11]
 traditional total method coverage prioritization[s-11]
 traditional additional method coverage prioritization[s-11]
 time-aware total statement coverage prioritization via ILP[s-11]
 time-aware additional statement coverage prioritization via ILP[s11]
 time-aware total method coverage prioritization via ILP[s-11]
 time-aware additional method coverage prioritization via ILP[s-11]
Ant colony algorithm[s-19][ s-40]
Artificial Bee Colony Optimization (ABC)[ s-40]
Greedy algorithm, [s-19][ s-31][ s-41][ s-46]
 G`[ s-31]
 Greedy (MgrA)[ s-37]
 Greedy Algorithm (ADD-DELTA)[ s-32][ s-42]
 Delayed-Greedy heuristic (DelGreedy)[ s-41][ s-42]
 GRE algorithm[s-19][ s-20][ s-28][ s-31][ s-35][ s-44][ s-46]
 GRE`[ s-31]
 GE[s-20][ s-31][ s-35][ s-44]
 GE`[ s-31]
 2-optimal greedy (2OPT)[ s-42]
Heuristic algorithm,[ s-19][ s-20][ s-28][ s-31][ s-35][ s-43]
 HGS (Harrold, Gupta and Soffa)[ s-41][ s-42][ s-44][ s-46]
 SMSP[s-41]
 H`[ s-31]
 Coverage-based and distribution-based heuristic algorithm
(clustering) [s-43]
 ART (Adaptive Resonance Theory) neural network with Clustering
[s-21]
 ILP[s-28]
Particle swarm optimization
 chaos discrete particle swarm optimization algorithm(CDPSO)[ s20]
 discrete particle swarm optimization (DPSO)[ s-20]
Random Algorithm[s-21][ s-22][s-60]
Genetic Algorithm [s-22][ s-23][ s-28][ s-32][ s-37][ s-44][ s-46][ s51][ s-53][ s-55][ s-60]
 Hybrid Genetic Algorithm (HGA)[ s-55]
 Quantum evolutionary Algorithm[s-44][ s-46]
 IQGA[s-46]
Bacteriological algorithms[s-53][ s-55]
Ant Colony Optimization (ACO)[ s-4]
Genetic Algorithm[s-8][ s-17][ s-33][ s-47]
34



Generation
NSGA-II[s-34]
coverage-based GA selection (CovGA), [s-18]
Quantum-inspired Multi-objective differential Evolution
Algorithm[s-34]
Greedy Algorithm[s-8][ s-17]
 coverage-based Greedy selection (CovGr),[ s-18]
Random selection (Rnd),[ s-18][ s-33][ s-47][ s-52]
 Adaptive Random Testing (ART)[ s-33]
HC (Hill-Climbing)[ s-52]
Heuristics
 the state of the art in STCS (TransGr).[ s-18]
 FS (Forward Selection)[ s-52]
 BE (Backward Elimination)[ s-52]
PSO
 BMOPSO[s-16]
 BMOPSO-CDR[s-16]
 BCPSO (Binary Constrained PSO)[ s-52]
 BCPSO-FS (hybrid algorithm)[ s-52]
 BCPSO-HC (hybrid algorithm)[ s-52]
Ant Colony Optimization (ACO)[ s-40]
 Ant System (AS), [s-6]
 Ant System with Elitist (ASelite),[ s-6]
 Ant Colony System (ACS) and[s-6][ s-48]
 Max-Min Ant System(MMAS)[ s-6]
Simulated annealing[s-48]
Greedy algorithms [s-39]
 Density [s-48]
 ParaOrder[s-48]
 gradient descent[s-59]
Genetic Algorithm[s-13][ s-14][ s-15][ s-24][ s-25][ s-33][ s-38][ s-45][
s-57][ s-58][ s-59]
 GA with Close-Up (CU) algorithm and Batch-Optimistic (BO)
algorithm[s-59]
 Genetic Single point crossover/mutation[s-39]
 Genetic Multi point crossover/mutation[s-39]
 differential genetic (diffGA)[ s-59]
 Strongly-Typed Genetic Programming[s-27][ s-29][ s-30]
Artificial Bee Colony Optimization (ABC)[ s-40]
 Sequential ABC[s-24]
 Parallel ABC[s-24]
Random Search[s-24][ s-25][ s-33][ s-59]
 random trace test derivation algorithm[s-61]
 random event sequences[s-61]
 Adaptive Random Testing (ART)[ s-33]
The proposed iterative test generation method[s-61]
 its enhancement with the immediate test selection[s-61]
 its extension with memory[s-61]
Evolutionary Algorithm with Optimization Strategy Based on
Cataclysm (OSBC) [s-26]
35
Evolutionary Algorithm with Dynamic Optimization of the possibility
(DOMP) [s-26]
Bacteriological[s-38]
Particle Swarm Optimization[s-56]
Estimation of Distribution Algorithms (EDA)[ s-62]
Scatter search[s-62]
 TCSS (test coverage scatter search)[ s-54]
 TCSS-LS (test coverage scatter search-local search)[ s-54]
 Combination of Scatter Search and EDA (SS-EDA)[ s-62]
4.2. Research Question 2
What are different factors that are affecting the performance of these techniques?
The solution to this research question lists down all the factors affecting the performance of
algorithms from literature.
Table 10: List of factors affecting the performance of the algorithms
Algorithms
Genetic
Strongly-Typed
Genetic Programming
Quantum genetic
Ant Colony
Optimization
Search Based
Factors
Local optima position
No. of generations
Fault severities
Population size
Cost of test cases vary
Parameter values of GA
Parallelization
Can be interrupted during execution
construct diverse test orderings
time budget
non-memorization
prematurity of population
time consuming
Computational overhead
fitness function
non linear suboptimal solution
Configuration of parameters
Computational overhead
maintain the population diversity
full cross -interference
Number of iterations
Time Constraint(TC) value
Sequential behaviour
Solution selection is done at the end
Computational overhead
Memory limit problem
Parameter setting
inherited randomness in ACS
Lack of uniformity in execution time
In Studies
1,2,24,55
2,7,9,45,53,55
2,7
2,7
2,7
9,17,32,39,49,53,55,60
9
9,33
9
12,18,25,28
24,38,53
26
33
47
49
55
29
47
44,46
44,46
4
4,40
40
40
40
40
48
48
10
36
PSO
Nearest point
algorithm
Greedy
Effort Limits (time)
fitness function
genetic operators
Initial set
Time budget
Time overheads
Time overheads
Time budget
memorization of best test case
No. of Test cases
linear and optimal solutions
Parallel behaviour
Optimization is done at each iteration
Computational overhead
Memory limit issue
Complexity of test cases
Parameter value
2--optimal greedy
DelGreedy
Heuristic
Bacteriological
Artificial bee colony
Iterative algorithms
Scatter Search
52
49
49
20
31
42
42
31
38,53
53
55
40
40
40
40
61
62
4.3. Research Question 3
What is the performance comparison of selected Techniques?
In solution to this research question a performance comparison of algorithms in each
optimization approach is conducted. In empirically study on search based TCO there is much
diversity, hence conducting the performance comparison is difficult. Following is the table
of performance comparison of algorithms, extracting from appendix B.
Table 12: Performance Comparison of algorithms.
Comparison Base
Algorithms
Testing Cost
GA
IGA
BMOPSO-CDR
BMOPSO
Coverage/Error
finding Rate
GA
ADD-DELTA
Modified GA
Simple GA
Modified Greedy
IGA
Greedy algorithm,
genetic algorithm,
simulated annealing
Reactive GRASP
GA
Additional Greedy
Hybrid Genetic
Algorithm (HGA)
QMDEA
Genetic Algorithm
(GA)
Bacteriologic
Algorithm (BA)
NSGA-II
Comparison
IGA’s low[1]
BMOPSO-CDR
outperform BMOPSO[16]
GA is better than ADDDELTA[32]
Modified GA has smallest
cost[37]
IGA increases the rate[1]
Additional greedy
performed significantly
better both for the
four small and large
programs[3]
HGA out performs the
other two in faults
revealing and coverage
[55]
QMDEA out
performs[34]
37
DelGreedy
GA-hist
HGS
Greedy
2-opt
GA-fun
Clustering Heuristic
H
PSO
AS
Random
MMAS
ACS
ASelite
GA
Greedy
Hill Climbing
Nearest point
algorithm
Iterative algorithm
Random search
Genetic algorithms
Random
ILP
GA
GA
Random
BMOPSO-CDR
BMOPSO
Combination of
Scatter Search and
EDA (SS-EDA)
Estimation of
Distribution
Algorithms (EDA)
Scatter Search (SS)
BCPSO (Binary
Constrained PSO)
HC (Hill-Climbing)
FS (Forward
Selection)
BE (Backward
Elimination)
Random
Greedy
Genetic Multi point
crossover/mutation
Random
Greedy
Coverage-based GA
BCPSO-FS (hybrid
algorithm)
BCPSO-HC (hybrid
algorithm)
Genetic Single point
crossover/mutation
GA with Jac Index
Greedy and 2-opt has
highest coverage rate[42]
GA-hist is more efficient
in error finding[2][7]
Clustering Heuristic has
better percentage of fault
detection[43]
PSO is effective[5]
ACS and MMAS has
fastest coverage, AS hass
lowest rate and ASelite is
between them[6]
GA is more effective[9]
In some cases GA is good
otherwise it worse[36]
GA is worse[50]
Nearest point has better
performance[49]
Iterative provides similar
or slightly better results
for fault coverage [61]
ILP is more faster[11][28]
GA perform
better[12][22][25][47][60]
Non is
dominating[33][47]
GA is worse because of
computational
overhead[47][49]
BMOPSO-CDR
outperform BMOPSO[16]
SS-EDA is best whereas
other two are also good in
coverage[62]
Both BCPSO-FS and
BCPSO-HC were better
than BCPSO, FS and
HC[52]
Genetic Single point
crossover/mutation
performs well[39]
GA with Jac Index is
better[18]
38
Ant Colony
Algorithm
Greedy algorithm,
Heuristic algorithm,
GRE algorithm
DPSO
H
GRE
GE
Random
AC is better[19]
EA with OSBC
EA with DOMP
Greedy
G’
GE, GE’
GRE, GRE’
H, H’
Additional greedy
2-optimal greedy
ACO
EA with OSBC works
better than EA with
DOMP, but not work well
for Complicated Branch
Conditions.[26]
GRE’ is better and H’ is
best[31]
CDPSO
ART (Adaptive
Resonance Theory)
neural network with
Clustering
Greedy
ABC
CDPSO is fastest[20]
ART is better [21]
Greedy is worse, 2optimal is best [36]
ABC is more
effective[40]
TCSS (test coverage
scatter search)
Genetic algorithms
with Close-Up (CU)
algorithm and
Batch-Optimistic
(BO) algorithm
No. of Generations GA-hist
random
gradient descent
standard genetic and
differential genetic
Proposed technique has
best coverage[59]
GA-fun
Population size
GA-hist
GA-fun
Execution time
GA-hist
GA-fun
Radom
Optimal
histfault-detected
total-fun
genetic algorithm,
simulated annealing
Reactive GRASP
Additional greedy
GE algorithm
When no. of generation is
greater than population
size GA-hist provides
higher fault detection
rate[2][7]
When size is too low GAhist’s fault detection rate
is low[2][7]
GA-hist has low
execution time and more
effective[2][7]
Greedy
quantum genetic
Greedy algorithm
performed more
efficiently than all other
algorithms.[3]
Quantum genetic has
39
algorithm
IQGA
PSO
GA
AS
Greedy
Hill Climbing
ILP
GA
ILP
H
GRE
Greedy
Greedy
Genetic Single point
crossover/mutation
Genetic
Genetic multi point
crossover/mutation
Parallel-ABC
Sequential-ABC
Genetic
Random
EA with OSBC
EA with DOMP
Genetic
Random
ART
NSGA-II
Genetic
QMDEA
Bacteriological
ABC
DelGreedy
Historical
GRE algorithm,
HGS algorithm,
genetic algorithm
greedy algorithm
HGS algorithm
GRE algorithm
Genetic algorithm
and
QGA separately
Greedy
GRE
ILP
H
MMAS
ACS
ASelite
GA-hist
ACO
HGS
Greedy
2-opt
GA-fun
smallest execution
time[44]
IQGA and QGA has
smallest[46]
PSO is efficient[5]
GA is worst[28]
ILP is Best[28]
AS has shortest, ASelite
has longest and, MMAS
and ACS is between
them[6]
HC lack uniformity in
execution time, and
greedy is much faster[10]
GA is less effective than
ILP[11][28]
ILP
GA is better[17]
Genetic Multi point
crossover/mutation is
faster[39]
Parallel-ABC takes least
time
GA has greatest time
because of stuck in local
optima and nonmemorization of best
individual in each
iteration[24]
EA with OSBC’s testing
cost is very high[26]
GA more time
consuming[33]
QMDEA is faster[34]
Bacteriological is far
speedy[38][53]
ABC is faster[40]
Greedy and HGS are
efficient [42]
GA-hist provides high
40
information
Bacteriological
Genetic
Test case cost
GA-hist
Fault severity
GA-hist
Initial Set
DPSO
GA-fun
Radom
Optimal
histfault-detected
total-fun
GA-fun
Radom
Optimal
histfault-detected
total-fun
H
GE
GRE
Size of test suite
GA
Greedy Algorithm
(ADD-DELTA)
GRE’
G (greedy) and G’
GE and GE’
GRE
H and H’
GE
GRE
GRE
H
GA
Sequential ABC
Random testing
Genetic Algorithm
Simple GA
Modified Greedy
H
ILP
Parallel ABC
Modified GA
DelGreedy
DelGreedy
Greedy
HGS
SMSP
HGS
Greedy
2-opt
Clustering Heuristic
H
quantum genetic
algorithm
GE algorithm
GRE algorithm,
HGS algorithm,
effectiveness [2][7]
Bacteriological has
memorization of best test
case, so more effective
[38][53]
GA-hist is more effective
in both cases when cost
vary or equal[2][7]
GA-hist is more effective
in both cases when fault
severity vary or
equal[2][7]
DPSO is stable as it has
nothing to do with initial
values, where as others
have [20]
Size of test suite achieved
by GA is smaller than
ADD-DELTA[32]
GRE’ has minimum[31]
H has largest[35]
ILP has smallest[28]
Parallel ABC has
smallest[24]
Modified GA’s sizes are
significantly smaller than
simple GA and modified
greedy [37]
DelGreedy has smallest
[41]
DelGreedy and HGS are
best methods for
minimizing the test suite
size [42]
Clustering Heuristic has
less size [43]
Genetic and quantum
genetic algorithm has
better minimized suite
41
genetic algorithm
IQGA
ACS
greedy algorithm
HGS algorithm
GRE algorithm
Genetic algorithm
and
QGA separately
simulated annealing
greedy algorithms
Density and
ParaOrder
specially when original
test suite is larger in size
[44]
IQGA has smallest[46]
ACS has smaller
But in some cases equal
to simulated annealing
[48]
42
Chapter 5: Analysis
This Chapter gives analysis of research questions which describes algorithms used for each
approach of test case optimisation, factors affecting the performance of these algorithms and
the their performance based comparison.
5. Analysis for Questions.
5.1. Research Question 1
RQ.1: Determine the Search Based Techniques being used for test case optimization
and group these techniques?
The aim is to understand the existing research directions of test case optimization with the
help of search based techniques and mainly the work is considered having some empirical
work attached with it. This systematic literature review lists down all techniques (algorithms)
and group those according to their similarities.
Number of Articles
After exclusion filter 2 finalized articles left are 62 and each optimization approach contains:
in prioritization 16, in selection 9, in generation 23 and in reduction 20.
25
20
15
10
5
0
Prioritization
Selection
Reduction
Optimized
Generation
Figure 1: Number of Algorithm used in each Approach
It means there is more empirical work in test case optimized generation and least in test case
selection.
Algorithms used in each optimization approach
Test Case Prioritization:
43
In prioritization, worked algorithms are 8 along their versions. Random algorithm is stated in
4 studies, Genetic Algorithm (GA) in 11 studies along with versions (GA-hist, time-aware
statement coverage prioritization via GA, time-aware method coverage prioritization via GA,
Time-aware test suite prioritization (GA-fun), Improved Genetic Algorithm(IGA)),
Greedy(G) in 7 studies along with versions (Additional Greedy algorithm, Total Greedy
Algorithm, delayed greedy, 2-Optimal Greedy Algorithm, Reactive GRASP), Simulated
Annealing(SA) in 1 study, Ant Colony Optimization(ACO) in 2 studies along with versions
(Ant System (AS), Ant System with Elitist (ASelite), Ant Colony System (ACS), Max-Min
Ant System(MMAS)) , Particle Swarm Optimization (PSO) in 2 studies with version (Nearest
point algorithm), Hill Climbing(HC) in 4 studies and Heuristics(H) in 4 studies with many
versions.
In this technique Genetic Algorithm is used most.
12
10
8
6
4
2
0
Random
GA
G
SA
ACO
PSO
HC
H
Figure 2: Algorithms used in Prioritization
Test Case Selection
In test case selection, worked algorithms are 7 with their versions. Random is used in 4
studies with its one adaptive version, GA in 6 studies with versions (NSGA-II, coveragebased GA selection (CovGA), Quantum-inspired Multi-objective differential Evolution
Algorithm), G in 3 studies with version (coverage-based Greedy selection (CovGr)), ACO in
1 study, PSO in 2 studies with versions (BMOPSO, BMOPSO-CDR, BCPSO (Binary
Constrained PSO), BCPSO-FS (hybrid algorithm), BCPSO-HC (hybrid algorithm)), HC in 1
study, H in 2 studies with versions.
In this technique Genetic Algorithm is used most.
44
7
6
5
4
3
2
1
0
Random
GA
G
ACO
PSO
HC
H
Figure 3: Algorithms used in Selection
Test Case Reduction
In test case Reduction, 8 algorithms have been applied. ACO is used in 2 studies, Artificial
Bee Colony (ABC) in 1 study, G in 11 studies with versions (G`, Greedy (MgrA), ADDDELTA, Delayed-Greedy heuristic (DelGreedy), GRE algorithm, GRE`, GE, GE`, 2-optimal
greedy (2OPT)), GA in 11 studies with versions (Hybrid Genetic Algorithm (HGA),
Quantum evolutionary Algorithm(QGA), IQGA), Bacteriological algorithm(BA) in 2 studies,
PSO in 1 study with two versions (DPSO,CDPSO), Random in 3 studies and H in 11 studies
with versions.
In this technique Genetic Algorithm, greedy algorithm and heuristics are used most.
12
10
8
6
4
2
0
Random
GA
G
ACO
PSO
ABC
H
BA
Figure 4: Algorithms used in Reduction
Test Case Optimized Generation
In optimized test case generation, total number of algorithms have been worked are 13. ACO
is used in 3 studies with versions (AS, ASelite, ACS, MMAS), SA in 1 study, G in 3 studies
with versions (Density, ParaOrder, gradient descent), GA in 15 studies with versions (GA
with Close-Up (CU) algorithm and Batch-Optimistic (BO) algorithm, Genetic Single point
crossover/mutation, Genetic Multi point crossover/mutation, differential genetic (diffGA),
45
Strongly-Typed Genetic Programming), ABC in 2 studies with versions (Sequential and
parallel), Random in 5 with versions, G. Kovacs proposed iterative test generation method in
1 study with 2 variations, Evolutionary Algorithm with Optimization Strategy Based on
Cataclysm (OSBC) in 1 study, Evolutionary Algorithm with Dynamic Optimization of the
possibility (DOMP) in 1 study, BA in 1, PSO in 1 study, Estimation of Distribution
Algorithms (EDA) in 1 study, and Scatter search (SS) in 2 with versions (TCSS (test
coverage scatter search), TCSS-LS (test coverage scatter search-local search), Combination
of Scatter Search and EDA (SS-EDA)).
In this technique Genetic Algorithm is used most.
16
14
12
10
8
6
4
2
0
Figure 5: Algorithms used in Optimized Generation
Grouping of Algorithms
Random algorithm and heuristic algorithms are not part of this SLR, but as in many studies
Search Based algorithms are compared with them. So they are discussed but not in detail.
Algorithms are grouped according to their similarities and versions.
Algorithms and their versions
Random prioritization
Genetic Algorithm
 GA-hist
 time-aware statement coverage prioritization via GA
 time-aware method coverage prioritization via GA
 Time-aware test suite prioritization (GA-fun)
 Improved Genetic Algorithm
 NSGA-II
 coverage-based GA selection (CovGA),
 Hybrid Genetic Algorithm (HGA)
 Quantum evolutionary Algorithm
 IQGA
 Quantum-inspired Multi-objective differential Evolution Algorithm
46





GA with Close-Up (CU) algorithm and Batch-Optimistic (BO) algorithm
Genetic Single point crossover/mutation
Genetic Multi point crossover/mutation
differential genetic (diffGA)
Strongly-Typed Genetic Programming
Greedy algorithm
 Total Greedy Algorithm
 Additional Greedy algorithm
 delayed greedy
 2-Optimal Greedy Algorithm
 Reactive GRASP
 G`
 Greedy (MgrA)
 Greedy Algorithm (ADD-DELTA)
 GRE algorithm and GRE`
 GE and GE`
 coverage-based Greedy selection (CovGr),
 Density
 ParaOrder
 gradient descent
Simulated annealing
Ant Colony Optimization (ACO)
 Ant System (AS)
 Ant System with Elitist (ASelite)
 Ant Colony System (ACS)
 Max-Min Ant System(MMAS)
Particle Swarm Optimization (PSO)
 BMOPSO
 BMOPSO-CDR
 BCPSO (Binary Constrained PSO)
 BCPSO-FS (hybrid algorithm)
 BCPSO-HC (hybrid algorithm)
 Nearest point algorithm
 chaos discrete particle swarm optimization algorithm(CDPSO)
 discrete particle swarm optimization (DPSO)
Bacteriological algorithms
Artificial Bee Colony Optimization (ABC)
 Sequential ABC
 Parallel ABC
Hill Climbing
Scatter search
 TCSS (test coverage scatter search)
 TCSS-LS (test coverage scatter search-local search)
 Combination of Scatter Search and EDA (SS-EDA)
Heuristics
Grouping depend on Similarities
47
Search based algorithms are grouped into two main sub groups Genetic and Swarm
intelligence. Genetic also include bacteriological Algorithm as they follow same steps and
Swarm intelligence include Particle swarm optimization, Ant Colony optimization, Artificial
Bee Colony Algorithm, they all belong to same group as they have some parameters to help
in optimization.
5.2. Research Question 2
RQ.2: What are different factors that are affecting the performance of these
techniques?
The solution to this research question lists down all the factors which affects the performance
of the algorithms. Following are the factors showing algorithms affected by them.
Parameter setting
Most important factor is parameter setting that affect about all techniques, setting of
parameters directly affect the performance of algorithms. Techniques that affects by this are
Genetic Algorithm [s-9][s-17][s-32][s-39][s-49][s-53][s-55][s-60], Strongly-Typed Genetic
Programming [s-29], Ant Colony Optimization[s-48], Nearest point algorithm[s-49], Scatter
Search[s-62].
Number of iterations
Number of iterations are specified as a parameter in start, when set less genetic algorithm
does not perform well[s-2][s-7][s-9][s-45][s-53][s-55], Ant Colony Optimization also
perform good in large number of iterations[s-4].
Local optima position
Genetic algorithm trapped in local optima positions so it is time consuming[s-1][s-2][s-24][s55], by giving similar solutions. Whereas IGA, has overcome this problem by adding baseline
testing[s-1], GA-hist overcome it by apply diversity in mutation, ABC works in both local
and global optima[s-24], BA also overcome this problem and provide rapid and optimal
convergence[s-55], and HGA provides near global optimal solutions[s-55].
Fault severities
Fault severity means how much critical the fault in some part of the program i.e., if some part
of the program is are very crucial than the fault found in this part is payable. GA-hist
performs good with both vary and equal fault severity or can say all algorithms with
historical data performs good with vary fault severity[s-2][s-7].
Population size
The size of the population defined affects the iterations to get the optimal results, when
population size is greater less iterations are needed to get the optimal results by GA[s-2][s-7].
48
Time budget
Time budget is measured in terms of time taken by test cases to execute. As testing is major
part in SDLC and it is very time consuming. Search based algorithms have lack of uniformity
in execution time[s-10]. When time budget is allocated longer GA provides best results[s18][s-25], when execution time of test cases is known GA can order test cases in better way
[s-12], whereas according to some studies GA is more time consuming to produce
representative set [s-28][s-33], GA-Hist is more efficient as it calculates fitness of each
chromosome to consume the time overhead [s-2][s-7]. GRE produce minimum size of test
suites then greedy and heuristic; when time resources are enough[s-31]. 2-optimal greedy and
delayed greedy are more time consuming then Harrold Gupta Soffa (HGS) and traditional
greedy[s-42], ACO also performs well with greater time to run[s-4], BCPSO-FS also has
good results[s-52].
Memorization
In iterative algorithms, at each iteration algorithm has to calculate the fitness value of
selected test cases which increases the computational overhead as in Genetic algorithm [s24][s-38][s-53]. GA-hist has partially overcome this problem by adding historical value of
test cases(execution time, faults detected and detected fault’s severity) in historical
information repository, it helps in optimizing test cases[s-2][s-7]. Bacteriological algorithm
memorizes the best test cases and produces best results [s-38][s-53]
Parallel/Sequential Behavior
Parallel behavior i.e., multithreading helps algorithms work fast. ABC has parallel behavior
that makes algorithm work fast to reach the near global optimal solution[s-40]. GA can also
enable parallel behavior[s-9]. ACO works in sequential behavior which makes it time
consuming due to computational overhead and can cause memory limit issue[s-40]
Memory Limit
Memory limit problem comes under available resources; due to much computation memory
limit issue may occur as in ACO[s-40], which is handled in ABC by applying parallel
behavior[s-40].
Computational Overhead
Algorithms have lots of calculations to choose the best to fit test case, if this calculation is
done repeatedly called computational overhead. ACO is a sequential behavior algorithm, its
solution selection is done at which causes computational overhead[s-40]. This issue is
lessened by ABC a parallel behavior algorithm which selects solution at each iteration [s-40].
GA has also much overhead because of its sample selection at each iteration behavior [s-47].
Population Diversity
49
It is better to use different test orderings, to cover the same requirements that it can ensure
product reliability; may be one same test ordering can miss the fault. Greedy with same time
and CE value produces same test orderings whereas GA produces diverse test orderings [s-9].
QEA[s-44] and QGA[s-46] also produce population diversity. ACS achieves this by pseudorandom proportional rule[s-48].
Population Prematurity
GA may get stuck into a local optimal solution, which shows the population of GA is
premature[s-26].
Interrupted In Execution
When resources reach the limit in ART and SBT (GA) test case execution can be stopped
without any side effect[s-33][s-9].
Fitness Function
Fitness functions are used to evaluate the relative superiority of different solution candidates
and to determine the direction of the search, explicitly or implicitly[s-49]. Fitness function
gives criteria to algorithm to judge whether the algorithm gives desired results. When fitness
function’s criteria is fulfilled, means desired outcome has been achieved and iterations should
be stop now. Different fitness functions are proposed to compare the GA and Nearest point
algorithm[s-49].
Linear/Non Linear Solutions
Generating test cases is a linear process, but automatically improving their efficiency from
time to time is a non linear process. GA produces non linear suboptimal solutions as it could
strikes up in local optima. Whereas HGA and BA have linear optimal solutions.[s-55]
Initial Set
Initial Set is the first sub-population chosen by the Algorithm to start for optimization.
Greedy and heuristics may produce different results with different initial set [s-20].
Full Cross–Interference
In full-interference crossover operation, all chromosomes in the population are involved in
crossover; it takes the advantage of all chromosomes information as much as possible and
brings a new evolutionary force even if premature population. By using full cross interference
in QGA, it is easy to maintain the population diversity with a good convergence speed [s44][s-46].
Solution Selection Stage
50
Solution selection in ACO is done at the end of all iterations because of sequential behavior
of algorithm, whereas in ABC solution selection is done at each iteration as its behavior is
parallel [s-40]
Complexity of Test Cases
Complexity of test cases depends on the number of input/output variables and number of
decision branches in code. Iterative algorithm takes more time with simple test cases then
random algorithm, where as best with complex programs [s-61].
5.3. Research Question 3
RQ.3: What is the performance comparison of selected Techniques?
In solution to this research question a performance comparison of algorithms in each
optimization approach is conducted. In empirically study on search based TCO there is much
diversity, hence conducting the performance comparison is difficult. Following are the test
case optimization approaches having algorithms and their comparison on the bases of
coverage, execution time, error finding rate, stability of results, and minimum resulted test
suite produce.
Test Case Selection
Genetic Algorithm

Coverage
GA compared to greedy has better results [s-17].

Execution time
GA compared to greedy has better results [s-17]. GA compared to greedy and
random, give best results when execution time is more [s-18]. In [s-33][s-47] none is
dominating the other; in GA and random on the base of fault detection rate where is
GA is more time consuming because of computational overhead. Quantum-inspired
Multi-objective Differential Evolution Algorithm (QMDEA) compared to NSGA II,
has faster convergence speed[s-34].
Particle Swarm Optimization

Requirement coverage and execution effort
Binary Multi-Objective PSO (BMOPSO), Binary Multi-Objective PSO using
Crowding Distance and Roulette Wheel (BMOPSO-CDR) and Random search are
compared, BMOPSO-CDR outperforms the both [s-16]. BCPSO compared with two
hybrid algorithms integrated in BCPSO (BCPSO-HC and BCPSO-FS), Hill Climbing
and random Search, both Hybrid techniques give promising results [s-52].
51
Test Case prioritization
Genetic Algorithm

Execution Time
Improved GA is more rapid than GA[s-1]. GA-hist is faster than GA-fun[s-2][s-7].
Additional Greedy is more efficient than greedy, genetic, simulated annealing and
Reactive GRASP [s-3]. ILP requires less analysis time than GA but greater than
traditional techniques [s-11].

Coverage
There is no significant difference in coverage performance between Additional
Greedy, greedy, genetic, simulated annealing and Reactive GRASP [s-3]. GA is more
effective than greedy and Hill climbing [s-9], in some cases GA may not perform well
[s-36]. When time budget is not concern traditional prioritization techniques
outperform the ILP and GA [s-11]. Nearest point algorithm has better performance
then greedy and GA, whereas GA has worse because of computational overhead [s49]. Additional Greedy Algorithm and 2-Optimal Greedy Algorithm perform better,
Greedy and hill climbing performs weaker, and GA is worse [s-50].

Error finding rate
GA-hist has best fault detection rate, in comparison to random, hist-fault-detect, histvalue, GA-fun, total-fun, cc-total-fun[s-2][s-7]. Fault detection rate of ILP is much
better than GA and traditional prioritization techniques [s-11]. GA has better APFD
than random search [s-12].

Stable Results
Greedy, additional greedy and 2-optimal show same results when performs
repeatedly, whereas hill climbing and genetic have vary results with each experiment
[s-36].
Ant Colony Optimization

Coverage
ACS and MMAS has fastest coverage rate, AS has lowest rate and ASelite is between
them [s-6].
Particle Swarm Optimization

Execution Time
52
PSO is more efficient than random search [s-5].

Coverage
PSO is more effective then random search [s-5].
Greedy

Execution Time
Greedy algorithm is faster than hill climbing [s-10]. Greedy and HGS are efficient
than Delayed greedy and 2-optimal [s-42].

Coverage
In comparison of greedy, additional greedy and 2-optimal greedy; greedy is worse and
2-optimal greedy is best [s-36]. Greedy and 2-opt has highest coverage rate and the
Delayed greedy does better than HGS [s-42].
Test Case Generation
Genetic

Coverage
Evolutionary Algorithm with Optimization Strategy Based on Cataclysm (OSBC) is
better than Evolutionary Algorithm with Dynamic Optimization of the possibility
(DOMP) [s-26]. Genetic Single point crossover/mutation performs well than Greedy
and Genetic Multi point crossover/mutation [s-39]. Genetic algorithm with Close-Up
(CU) algorithm and Batch-Optimistic (BO) algorithm has best coverage than random,
gradient descent, standard genetic and differential genetic [s-59].

Execution Time
In GA and random none is dominating, but GA is time consuming [s-33]. GA is
speedy than Bacteriological algorithm in convergence [s-38]. Genetic Multi point
crossover/mutation is faster than Greedy and Genetic Single point crossover/mutation
[s-39]. Evolutionary Algorithm with Optimization Strategy Based on Cataclysm
(OSBC) testing cost is high than Evolutionary Algorithm with Dynamic Optimization
of the possibility (DOMP) [s-26]. Iterative algorithms decreases the total test case
generation time [s-61].

Fault Detection
GA has better fault detection rate than random [s-25]. Iterative algorithms provide
similar or better results than random [s-61].
53

Minimized test Suite
Iterative algorithm has smaller number of final test cases than random [s-61].
Artificial Bee Colony and Ant Colony Optimization

Execution Time
AS has shortest execution time, ASelite has longest and, MMAS and ACS are
between them [s-6]. Parallel-ABC takes least time, GA has greatest time and
Sequential-ABC and Random lie between them [s-24]. ABC is fast and attains
coverage 100% in few runs [s-40].

Coverage
ABC has better coverage than ACO [s-40].

Minimized Test Suite
Parallel-ABC has smallest generated test suite than sequential-ABC, GA and Random
[s-24]. ACS generates smaller size of test suites than simulated annealing and greedy,
but in some cases similar to simulated annealing [s-48].
Scatter Search

Coverage
Combination of Scatter Search and EDA (SS-EDA) is best, whereas Estimation of
Distribution Algorithms (EDA) and Scatter Search (SS) are also good in coverage [s62]. TCSS-LS (test coverage scatter search-local search) outperforms than TCSS (test
coverage scatter search) [s-54].
Test Case Reduction
Genetic

Coverage
GA is better than random search [s-22][s-60].

Execution Time
Genetic performs worst among H, ILP and GRE, whereas H is fastest [s-28]. GA has
less execution time than greedy (ADD-DELTA) [s-32]. Modified GA is faster than
simple GA and modified greedy [s-37]. Quantum genetic algorithm has least
execution time than genetic algorithm, GE, GRE and HGS [s-44]. Improved QGA and
54
QGA have small execution time than GA [s-46]. Bacteriological Algorithm is speedy
than GA [s-53].

Minimized Test Suite
ILP has smallest representative set in Genetic, H and GRE, and Genetic not perform
well [s-28]. GA has smaller reduced test suite than greedy (ADD-DELTA) [s-32].
Modified GA has smallest size than simple GA and Modified greedy [s-37]. Quantum
Genetic algorithm and GA have better minimized test suite than GE, GRE and HGS
[s-44]. IQGA has smallest than QGA, GA, HGS, GRE and greedy [s-46].

Fault detection
ILP considers the fault detection capability [s-28]. Hybrid GA has better fault
revealing percentage than GA and BA [s-55].
Ant Colony Optimization

Coverage
Ant colony algorithm is better than greedy, heuristic and GRE [s-19].
Particle Swarm Optimization

Coverage
CDPSO is best than DPSO, GE, H and GRE [s-20].

Execution time
Convergence speed of CDPSO is faster as compared to DPSO, GE, H and GRE [s20].

Stable Results
In different initial conditions GE, H and GRE may produce different results. Whereas
CDPSO and DPSO are stable as they have nothing to do with initial values [s-20].
Neural Network

Coverage
ART (Adaptive Resonance Theory) neural network with Clustering is better than
Random Algorithm [s-21].
Greedy and Heuristic
55

Coverage
Advance GRE and advance H are best than GRE, H, G, GE, G’ and GE’ [s-31].

Fault Detection
Clustering Heuristic has better fault detection rate than simple heuristic[s-43].

Minimized Test Suite
Advance GRE has minimum reduced test Suite [s-31]. Heuristic has larger
representative set than GRE and GE [s-35]. DelGreedy has smallest test suite than
greedy, HGS and SMPS [s-41]. DelGreedy and HGS are best for minimizing the test
suite than 2-optimal and greedy [s-42]. Clustering Heuristic has smaller reduced test
suite than Heuristic [s-43].
Artificial Bee Colony

Execution Time
ABC is fast and attains coverage 100% in few runs [s-40].

Coverage
ABC has better coverage than ACO [s-40].

Consistency in Result
ABC provides Consistent Results.
56
Chapter 6: Conclusion
In order to decrease the cost of regression testing, researchers have emphasized and shown
importance to test case optimization (TCO) techniques, which identifies a subset of test cases
that provide same coverage of software as the original test suite. There are many techniques
for test case optimization but their main focus is on search based techniques as they find the
optimal solutions for the problems. For optimizing the test cases, two main approaches can be
considered; either generate the optimized test cases by using algorithms or to apply
optimization approaches on already generated pool of test cases. There are three approaches
for optimizing the already generated pool of test cases; which are prioritization, selection and
reduction.
There are many techniques for optimizing the test suite but it is focused on search based
algorithms in this SLR. By search based techniques one can achieve better results at lower
cost, and freeing the testers to work on testing problems that require more imagination and
creativity.
The aim of this SLR is to find the State-of-the-Art in search-based test case optimization and
find which search-based algorithms are used for each optimization approach, factors affecting
the performance of these algorithms and their performance comparison within each
optimization approach. For this SLR research questions (RQ) are:RQ.1: Determine the Search Based Techniques being used for test case optimization and
group these techniques?
RQ.2: What are different factors that are affecting the performance of these techniques?
RQ.3: What is the performance comparison of selected Techniques?
The string results by all data bases are 443 articles. After exclusion filter 1 and removing the
duplicates; 152 articles are selected. And after applying filter 2 final selected articles are 60
and 2 articles are added from reference scanning. It means 62 papers are finalized. The papers
are distributed in optimized approaches as: prioritization (16 papers), selection (9 papers),
reduction (20 papers) and optimized generation (23 papers).
From table 1 number of algorithms in each approach are:- prioritization: 8 and their versions,
selection: 7 with their versions, Reduction: 8 with their versions, and in optimized test case
57
generation: 13 with their versions. The used algorithms are genetic algorithm, greedy
algorithm, ant colony optimization, Bacteriological algorithm, artificial bee colony, particle
swarm optimization, heuristics, simulated annealing, hill climbing and scatter search
algorithm.
Factors which affect the performance of algorithms are Parameter setting, Number of
iterations, Local optima position, Fault severities, Population size, Time budget,
Memorization, Parallel/Sequential Behavior, Memory Limit, Computational Overhead,
Population Diversity, Population Prematurity, Interrupted In Execution, Fitness Function,
Linear/Non Linear Solutions, Initial Set, Full Cross–Interference, Solution Selection Stage,
Complexity of Test Cases. These have been extracted from literature.
And the performance of the Algorithms are compared on the bases of coverage, execution
time, error finding rate, stability of results, and minimum resulted test suite produce, within
each approach.
By this SLR, researchers will find the gaps for their research. Researchers mostly haven’t
applied real world data for validating their research, so it should be applied more for gaining
more confidence on the algorithms. There should be more work on the performance
comparisons of algorithms to find best algorithm for the purpose.
58
References
1. http://en.wikipedia.org/wiki/Software_testing 13 May, 2011.
2. Mark Harman, Bryan F. Jones, “Search-Based Software Engineering”, Information and
Software Technology, vol-43, pg. 833-839, 2001.
3. M. Harman, A. Mansouri, and Y. Zhang, “Search based software engineering: A
comprehensive analysis and review of trends techniques and applications”, Technical
Report TR-09-03, Department of Computer Science, King's College London, April 2009.
4. Zheng Li, Mark Harman, and Robert M. Hierons, “Search Algorithms for Regression Test
Case Prioritization”, IEEE Transactions on Software Engineering, Vol. 33, No. 4, p. 225237, April 2007.
5. Mark Harman, “The Current State and Future of Search Based Software Engineering”,
Future of Software Engineering, 2007.
6. S. Ali, L.C. Briand, H. Hemmati, and R.K. Panesar-Walawege, “A Systematic Review of
the Application and Empirical Investigation of Evolutionary Testing”, Technical Report
Simula.SE.293, Simula Research Laboratory, 2008.
7. S. Ali, L.C. Briand, H. Hemmati, and R.K. Panesar-Walawege, “A Systematic Review of
the Application and Empirical Investigation of Search based test case generation”, IEEE
Transactions on Software Engineering, vol 36, No. 6, 2010.
8. Phil McMinn, “Search based software test data generation: A survey”, Software Testing,
Verification and Reliability, 14(2), pp. 105-156, 2004.
9. W. Afzal, R. Torkar, R. Feldt, “A systematic mapping study on non-functional search-
based software testing”, 20th International Conference on Software Engineering and
Knowledge Engineering (SEKE), 2008.
10. W. Afzal, R. Torkar, R. Feldt, “A systematic review of search based software testing for
non-functional system properties”, Information and software technology, 2009.
11. Leo Rela, “MS thesis Evolutionary computing in search based software engineering”,
Lappeenranta university of technology, department of Information Technology, 2003.
12. Camila Loiola Brito Maia, et al, “Applying Search-Based Techniques for Requirements-
Based Test Case Prioritization”, Proceedings of the Brazilian Workshop on Optimization
in Software Engineering (WOES '10), 2010.
13. B. Kitchenham, “Guidelines for performing Systematic Literature Reviews in Software
Engineering”, Version 2.3, EBSE Technical Report EBSE-2007-01, Keele University and
University of Durham, 2007.
59
14. Cagatay Catal, “On the Application of Genetic Algorithms for Test Case Prioritization: A
Systematic Literature Review”, EAST’12, Lund, Sweden, ACM, 2012.
15. T.Y. Chen, and M.F. Lau, “A new heuristic for test suite reduction”. Information and
Software Technology, 40: p. 347-354, 1998.
16. S. Khurshid, T. Margaria, and W. Yi, “Testing an Intentional Naming Scheme Using
Genetic Algorithms”, Tools and Algorithms for the Construction and Analysis of
Systems, Springer Berlin / Heidelberg, p. 358-372, 2001.
17. B. Baudry, et al., “Genes and bacteria for automatic test cases optimization in the .NET
environment”, 13th International Symposium on Software Reliability Engineering, ISSRE
2003.
18. B. Baudry, et al., “Automatic test case optimization using a bacteriological adaptation
model: application to .NET components”, 17th IEEE International Conference on
Automated Software Engineering, ASE 2002.
19. M. Xue-ying, S. Bin-kui, and Y. Cheng-qing, “Test-Suite reduction using genetic
algorithm”, in Proceedings of the 6th international conference on Advanced Parallel
Processing Technologies, Springer-Verlag: Hong Kong, China, 2005.
20. M.A. Xue-ying, et al., “A genetic algorithm for test-suite reduction”. International
Conference on Systems, Man and Cybernetics, IEEE, 2005.
21. T. Sriraman and G. Neelam, “A concept analysis inspired greedy algorithm for test suite
minimization”, Proceedings of the 6th ACM SIGPLAN-SIGSOFT workshop on Program
analysis for software tools and engineering, ACM: Lisbon, Portugal, 2005.
22. R.W. Kristen, et al., “Time Aware test suite prioritization”, Proceedings of the 2006
international symposium on Software testing and analysis, ACM: Portland, Maine, USA,
2006.
23. Adenilso da Silva, S., M. Rodrigo Fernandes de, and S. Luciano Jose, “A Technique to
Reduce the Test Case Suites for Regression Testing Based on a Self-Organizing Neural
Network Architecture”, Proceedings of the 30th Annual International Computer Software
and Applications Conference, IEEE Computer Society, Volume 02. 2006.
24. R.N. Sagarna, and J.A. Lozano, “Scatter Search in software testing, comparison and
collaboration with Estimation of Distribution Algorithms”, European Journal of
Operational Research, 169(2): p. 392-412, 2006.
25. G. KovÃcs, et al., “Iterative automatic test generation method for telecommunication
protocols”, Computer Standards &amp; Interfaces, 28(4): p. 412-427, 2006.
60
26. C.S.S. Dharsana, et al., “Java Based Test Case Generation and Optimization Using
Evolutionary Testing”, Proceedings of the International Conference on Computational
Intelligence and Multimedia Applications (ICCIMA 2007) - Volume 01, IEEE Computer
Society, 2007.
27. Jos, et al., “Strongly-typed genetic programming and purity analysis: input domain
reduction for evolutionary testing problems”, Proceedings of the 10th annual conference
on Genetic and evolutionary computation, ACM: Atlanta, GA, USA, 2008.
28. Jos and R. Carlos Bregieiro, “Search-based test case generation for object-oriented java
software using strongly-typed genetic programming”, Proceedings of the 2008 GECCO
conference companion on Genetic and evolutionary computation, ACM: Atlanta, GA,
USA, 2008.
29. H. Zhong, L. Zhang, and H. Mei, “An experimental study of four typical test suite
reduction techniques”, Information and Software Technology, 50(6): p. 534-546, 2008.
30. Z. Xiaofang, et al., “An Empirical Evaluation of Test Suite Reduction for Boolean
Specification-Based Testing”, Proceedings of the 2008 The Eighth International
Conference on Quality Software, IEEE Computer Society, 2008.
31. A.A. Sofokleous, and A.S. Andreou, “Automatic, evolutionary test data generation for
dynamic software testing”, Journal of Systems and Software, 81(11): p. 1883-1898, 2008.
32. K.H.S. Hla, C. YoungSik, and P. Jong Sou. “Applying Particle Swarm Optimization to
Prioritizing Test Cases for Embedded Real Time Software Retesting”, IEEE 8th
International Conference on Computer and Information Technology Workshops, CIT
Workshops, 2008.
33. C. Xiang, et al., “Variable Strength Interaction Testing with an Ant Colony System
Approach”, Proceedings of the 2009 16th Asia-Pacific Software Engineering Conference,
IEEE Computer Society, 2009.
34. Z. Lu, et al., “Time-aware test-case prioritization using integer linear programming”,
Proceedings of the eighteenth international symposium on Software testing and analysis,
ACM: Chicago, IL, USA, 2009.
35. M. Wes, and E.G. Marwa, “Test case filtering and prioritization based on coverage of
combinations of program elements”, Proceedings of the Seventh International Workshop
on Dynamic Analysis, ACM: Chicago, Illinois, 2009.
36. J.C.B. Ribeiro, et al., “Test Case Evaluation and Input Domain Reduction strategies for
the Evolutionary Testing of Object-Oriented software”, Information and Software
Technology, 51(11): p. 1534-1548, 2009.
61
37. H.A. James, and M. Tim, “On the value of combining feature subset selection with
genetic algorithms: faster learning of coverage models”, Proceedings of the 5th
International Conference on Predictor Models in Software Engineering, ACM:
Vancouver, British Columbia, Canada, 2009.
38. D.J. Mala, et al., “A non-pheromone based intelligent swarm optimization technique in
software test suite optimization”, International Conference on Intelligent Agent & MultiAgent Systems, IAMA 2009.
39. S. Parsa, A. Khalilian, and Y. Fazlalizadeh, “A new algorithm to Test Suite Reduction
based on cluster analysis”, 2nd IEEE International Conference on Computer Science and
Information Technology, ICCSIT 2009.
40. M.S. Adam, and M.K. Gregory, “An empirical study of incorporating cost into test suite
reduction and prioritization”, Proceedings of the 2009 ACM symposium on Applied
Computing, ACM: Honolulu, Hawaii, 2009.
41. C. Xiang, et al., “Building Prioritized Pairwise Interaction Test Suites with Ant Colony
Optimization”, 9th International Conference on Quality Software, QSIC '09, 2009.
42. R. Blanco, J. Tuya, and B. Adenso-Díaz, “Automated test data generation using a
scatter search approach”, Information and Software Technology, 51(4): p. 708-720, 2009.
43. D. Mohapatra, P. Bhuyan, and D.P. Mohapatra, “Automated Test Case Generation and Its
Optimization for Path Testing Using Genetic Algorithm and Sampling”, WASE
International Conference on Information Engineering, ICIE '09, 2009.
44. D.W. Zachary, and M.K. Gregory, “Using synthetic test suites to empirically compare
search-based and greedy prioritizers”, Proceedings of the 12th annual conference
companion on Genetic and evolutionary computation, ACM: Portland, Oregon, USA,
2010.
45. P. Nenggang, Z. FanPing, and H. Yu-Han, “Test Case Reduction Based on Program
Invariant and Genetic Algorithm”, 6th International Conference on Wireless
Communications Networking and Mobile Computing (WiCOM), 2010.
46. Z. Yi-kun, et al., “Study of Test Suite Reduction Based on Quantum Evolutionary
Algorithm”, International Conference on Artificial Intelligence and Computational
Intelligence (AICI), 2010.
47. L. Sihan, et al., “A Simulation Study on Some Search Algorithms for Regression Test
Case Prioritization”, Proceedings of the 2010 10th International Conference on Quality
Software, IEEE Computer Society, 2010.
62
48. D.J. Mala, and V. Mohan, “Quality improvement and optimization of test cases: a hybrid
genetic algorithm based approach”, SIGSOFT Softw. Eng. Notes, 35(3): p. 1-14, 2010.
49. W. Meng, et al., “An Optimization Strategy for Evolutionary Testing Based on
Cataclysm”, Proceedings of the 2010 IEEE 34th Annual Computer Software and
Applications Conference Workshops, IEEE Computer Society, 2010.
50. S. Jia-ze, and W. Shu-yan, “A novel chaos discrete particle swarm optimization algorithm
for test suite reduction”, 2nd International Conference on Information Science and
Engineering (ICISE), 2010.
51. H. Hemmati, and L. Briand, “An Industrial Investigation of Similarity Measures for
Model-Based Test Case Selection”, IEEE 21st International Symposium on Software
Reliability Engineering (ISSRE), 2010.
52. S. Lucas Serpa, and S. Maarten van, “Evolutionary testing of object-oriented software”,
Proceedings of the 2010 ACM Symposium on Applied Computing, ACM: Sierre,
Switzerland, 2010.
53. S. Nachiyappan, A. Vimaladevi, and C.B. SelvaLakshmi, “An evolutionary algorithm for
regression test suite reduction”, International Conference on Communication and
Computational Intelligence (INCOCCI), 2010.
54. H. Hadi, et al., “An enhanced test case selection approach for model-based testing: an
industrial case study”, Proceedings of the eighteenth ACM SIGSOFT international
symposium on Foundations of software engineering, ACM: Santa Fe, New Mexico, USA,
2010.
55. P.C. Alexander, S.R. Robert, and M.K. Gregory, “Empirically studying the role of
selection operators during search-based test suite prioritization”, Proceedings of the 12th
annual conference on Genetic and evolutionary computation, ACM: Portland, Oregon,
USA, 2010.
56. F. Ethan, et al., “Designing better fitness functions for automated program repair”,
Proceedings of the 12th annual conference on Genetic and evolutionary computation,
ACM: Portland, Oregon, USA, 2010.
57. H. Yu-Chi, et al., “Design and Analysis of Cost-Cognizant Test Case Prioritization Using
Genetic Algorithm with Test History”, Computer Software and Applications Conference
(COMPSAC), IEEE 34th Annual, 2010.
58. A. Arcuri, et al., “Black-Box System Testing of Real-Time Embedded Systems Using
Random and Search-Based Testing”, Testing Software and Systems, Springer Berlin /
Heidelberg. p. 95-110, 2010.
63
59. M. Camila Loiola Brito, et al., “Automated test case prioritization with reactive GRASP”,
Adv. Soft. Eng., p. 1-13, 2010.
60. D. Jeya Mala, V. Mohan, and M. Kamalapriya, “Automated software test optimisation
framework - an artificial bee colony optimization-based approach”, Software, IET, 4(5):
p. 334-348, 2010.
61. R. Abdul, et al., “Automated GUI Test Coverage Analysis Using GA”, Proceedings of the
2010 Seventh International Conference on Information Technology: New Generations,
IEEE Computer Society, 2010.
62. B.S. Ahmed, and K.Z. Zamli, “A variable strength interaction test suites generation
strategy using Particle Swarm Optimization”, Journal of Systems and Software, 84(12): p.
2171-2185, 2011.
63. J. Wang, Y. Zhuang, and C. Jianyun, “Test Case Prioritization Technique Based on
Genetic Algorithm”, Proceedings of the 2011 International Conference on Internet
Computing and Information Services, IEEE Computer Society, 2011.
64. D. Cui, and W. Yin, “The research of test-suite reduction technique”, International
Conference on Consumer Electronics, Communications and Networks (CECNet), 2011.
65. S.d.S. Luciano, et al., “A Multi-objective Particle Swarm Optimization for Test Case
Selection Based on Functional Requirements Coverage and Execution Effort”,
Proceedings of the 2011 IEEE 23rd International Conference on Tools with Artificial
Intelligence, IEEE Computer Society, 2011.
66. Z. Yi-kun, et al., “An improved quantum genetic algorithm for test suite reduction” IEEE
International Conference on Computer Science and Automation Engineering (CSAE),
2011.
67. J.J. Dom, et al., “Evolutionary mutation testing”, Information and Software Technology,
53(10): p. 1108-1123, 2011.
68. J. Koljonen, “Comparison of nearest point algorithms by genetic algorithms”, Expert
Systems with Applications, 38(8): p. 10303-10311, 2011.
69. K. Derderian, et al., “A Case Study on the Use of Genetic Algorithms to Generate Test
Cases for Temporal Systems”, Advances in Computational Intelligence, Springer Berlin /
Heidelberg, p. 396-403, 2011.
70. C.P. Vudatha, et al., “Automated generation of test cases from output domain of an
embedded system using Genetic algorithms”, 3rd International Conference on Electronics
Computer Technology (ICECT), 2011.
64
71. S. Bharti, and S. Shweta, “Analyzing test case selection & prioritization using ACO”,
SIGSOFT Softw. Eng. Notes, 36(6): p. 1-5, 2011.
72. M. Matthias, and S. Francesca, “Software reliability testing covering subsystem
interactions”, Proceedings of the 16th international GI/ITG conference on Measurement,
Modeling, and Evaluation of Computing Systems and Dependability and Fault Tolerance,
Springer-Verlag: Kaiserslautern, Germany, 2012.
73. A.C. Kumari, K. Srinivas, and M.P. Gupta, “Multi-objective test suite minimization using
Quantum-inspired Multi-objective Differential Evolution Algorithm”, IEEE International
Conference on Computational Intelligence & Computing Research (ICCIC), 2012.
74. H. Yu-Chi, P. Kuan-Li, and H. Chin-Yu, “A history-based cost-cognizant test case
prioritization technique in regression testing”, J. Syst. Softw., 85(3): p. 626-637, 2012.
75. L.S. de Souza, et al., “Search based constrained test case selection using execution
effort”, Expert Systems with Applications, 40(12): p. 4887-4896, 2013.
65
Appendix
66
Appendix A: Protocol
P ROTOCOL
Protocol of Systematic Review of
Empirical Literature in Search Based Test
Case Prioritization
Beenish Fatima
293-FBAS/MSSE-F09
Protocol Version 1.2
May, 2011.
Department of Software Engineering
Faculty of Basic and Applied Sciences
International Islamic University Islamabad
http://iiu.edu.pk
67
Protocol of Systematic Review of Empirical Literature in
Search Based Test Case Prioritization
Preamble
Software is evolving in nature because requirements of customers change over time. To maintain
the quality in software, testing is very important. For testing purpose, test cases are designed for
initial version of software, but by modifications in software more test cases are added to test suit
which make test suit large. But for quality we have to retest the whole software again with all test
cases, as little change in code may affect some other part of code which is pre-verified, this is
regression testing which is time consuming and may take several weeks. Normally, in practice
there is not enough time to retest the software with all test suits, for this we apply test case
optimization, to do major or critical testing first, that if due to time shortage, and if we have to quit
the testing for saving the time then major objective of testing should be achieved.
Test case optimization is minimizing the number of test cases to be executed by some criteria,
which provide the same coverage as original test suite and to save the cost of test case execution.
For optimizing the test cases two main approaches can be considered; generate the optimized test
cases or to apply optimization strategy on already generated pool of test cases. For already
generated test suites optimization the approaches used are test case selection, reduction and
prioritization. Test case selection is to select some test cases in a subset from original test set
according to some criteria. Test-suite reduction is to reduced test suite by removal of test cases
and test-suite prioritization is an ordering of the test cases in the test suite according to some
criteria or goals. In existing research studies there are many techniques/algorithms for each
approach of optimization. But in this research my main focus is on search based techniques.
Search based software engineering (SBSE) is a new terminology which was introduced by Mark
Harman and Jones in 2001 [1]. “SBSE is an approach to software engineering in which search based
optimization algorithms are used to address problems in software engineering” [2]. This approach
is most adaptive now-a-days because it offers automated and semi automated solutions for
Software engineering problems having large and complex problems spaces with multiple
competing and conflicting objectives [2].
My focus here is to systematically review those empirical studies on the search based techniques
for test case prioritization, as a number of empirical studies for this topic exist in literature. The
protocol for my SLR is developed based on the guidelines of Kitchenham (2007) [21].
Background
Testing is the most important and critical part of software life cycle development. But this is the
most expensive phase to apply, so it is not recommendable to test rigorously due to short of
resources such as time and cost. For example, “one of our industrial partner reports that for one of
68
its products of about 20,000 lines of code, for this test suite requires seven weeks to run” [3]. For
this purpose test case optimization techniques selection, prioritization and reduction are being
used to enhance the efficiency of testing [3][4]. In test case reduction there is possibility to skip
major test case, which tests the important portion of the system under test, if the technique used
is not good. Whereas in test case prioritization; test cases are arranged by some priority criteria
which are of importance to the testers. Prioritization is majorly done on the criteria of coverage
and cost. Coverage consist of requirement’s coverage, statement coverage, branch coverage, block
coverage, etc. and cost consist of execution time [5][6] and financial cost associated with the
running of test cases [3].
A test suite is typically developed for the initial version of evolving software, later on it is reused
and embed to test each subsequent version. With the integration of additional functionality in
software increases the test suit, and it is not feasible for regression testing because of big test suit,
and this is not efficient. To increase the efficiency of testing performance test suit should be
optimized.
There are many techniques for test case prioritization and test case reduction [2][3][4][7]. But in
this research my main focus is on search based techniques. Search Based Software Engineering is
approach of software engineering in which search based techniques are used to address the
problems in Software Engineering [7]. These algorithms are frequently in use since after 2001, in
different fields of software engineering. The most commonly used techniques are local search,
simulated annealing, genetic algorithms [2][6][7][8][9], and genetic programming. Other
techniques are also in use for example greedy algorithm [10], random algorithm [11], particle
swarm optimization [12], ant colony, etc and there different versions.
Genetic algorithm is commonly in practice and more prefer on other algorithms [13][14]. But in
some scenarios, Additional greedy algorithm [14] and particle swarm optimization [12] are
preferred. The major flaw of genetic algorithm is big list of parameters that are difficult to handle
[12]. These techniques use artificial intelligence which is human competitive i.e. these algorithm
can produce solutions without involving humans and these test suit are more efficient [15].
Several authors have provided comparative studies on the performance of several Search based
techniques regarding test case optimization approaches [6, 12, 13, 14, 15, 16]. In [14], there is
comparison between NSGA-II, MoCell, SPEA2 and Random algorithm. Here comparison is done on
the bases of requirements coverage and execution time. On the base of Requirements coverage;
Random is worst whereas, SPEA2 seems to have a better performance for smaller search spaces,
while NSGA-II and MOCell have better performance for larger search spaces and on time bases;
Random algorithm had the best lowest execution time. The second best execution time was
obtained by MOCell, followed by NSGA-II. SPEA2 was slowest algorithm.
59% work of SBSE done on software testing [7][17]. Hence, there is a lot work in Software testing.
McMinne [17] has surveyed the application of Meta heuristics search techniques to software test
data generation. Ali et al, [18] conducted systematic literature review (SLR) on Search based test
case generation (SB-TCG) in which he discussed the application of SB-TCG through empirical
evidence present in literature. W. Afzal performed systematic mapping [19] and SLR [20] on search
69
based software testing for non functional system properties and pin-point challenges in the
application of search based techniques for testing the non-functional properties of the system.
As mentioned, surveys and SLRs already had been done on search based software testing but not
specifically cover Search based test case optimization (SB-TCO). A bundle of work on search based
test case optimization techniques is present in literature in wide variation. But not all together on
one place, so there is need to do a systematic literature review on SB-TCO, which provides stateof-the-art on SB-TCO for future research and help to identify the gaps.
Research Questions
One of the rationales for this SLR is to summarize the existing evidence concerning Search Based
Test Case Optimization (TCO). So my evidence based investigation is mainly focused on this. To
understand state-of-the-art empirical work in Search Based TCO, in terms of gaps, commonalities
in existing empirical results and the comparison between Search Based techniques used for TCO
approaches, the following three research questions are being formulated. There is vast amount of
variations in this context, and until now no clear guidance for applying the right evolutionary
algorithm is in the right situation. The main driver for conducting the systematic review was to
identify gaps, commonalities in empirical research related to Search Based Test Case Prioritization
and sum up the existing practices to answer the questions: who, where, what, why and how in use
of Search techniques. The research questions are formulated to form a baseline for state-of-theart empirical work and hence the objective is that the systematic review should form a milestone
for both future research and for practical use by practitioners. Thus my major concerns are as
follows:
RQ.1: Determine the Search Based Techniques being used for test case prioritization and group
these techniques?
RQ.2: What are different factors that are affecting the performance of these techniques?
RQ.3: What is the performance comparison of selected Techniques?
Detail of Questions
RQ.1: Determine the Search Based Techniques being used for test case
optimization and group these techniques?
The aim is to understand the existing research directions of test case
optimization with the help of search based techniques and mainly the work is
considered having some empirical work attached with it. There are many
search based techniques are used for test case optimization such as Greedy
algorithm, Genetic algorithm, particle swarm optimization, etc., and there
versions. So from systematic literature review I will list down all techniques
(algorithms) and group those according to their similarities i.e. versions of any
algorithm such as greedy algorithm and additional greedy algorithm, or
genetic algorithm and NSGA etc. The solution to this question gives us the
70
state-of-the-art empirical work of search based techniques used for test case
optimization.
RQ.2: What are different factors that are affecting the performance of these
techniques?
From listed techniques for all optimization approaches, I select some (not all)
evolutionary algorithms. And from empirical work I will identify the factors
that can affect the performance of these techniques such as size of the code,
size of test suite, size of test cases, etc. The solution of this question consist of
all factors on which the performance of these techniques be measured.
RQ.3: What is the performance comparison of selected Techniques?
In empirically study on search based TCO there is much diversity, hence
conducting the performance comparison is difficult. In this SLR I will conduct
the performance comparison of techniques with in each optimization approach
on the bases of: coverage effectiveness, response time, defect detection rate. In
the solution I’ll get performance of selected search based optimization
techniques in comparison with each other.
Structured Questions
RQ.1: Determine the Search Based Techniques being used for test case optimization
and group these techniques?
Population: Empirical Literature for Search based test case optimization.
Outcome: List of grouped search based techniques used in TCO.
Intervention: SB-TCO.
Comparison: On the bases of similarities.
RQ.2: What are different factors that are affecting the performance of these
techniques?
Population: Data Regarding Performance of Search Algorithm for test case
optimization.
Outcome: List of factors affecting the performance of these techniques.
Intervention: Search Based Test Case optimization
Comparison: No.
RQ.3: What is the performance comparison of selected Techniques?
Population: selected search based techniques
Outcome: Best search Based techniques for test case optimization approaches.
Intervention: SB-TCO.
Comparison: On the bases of coverage effectiveness, response time, defect detection
rate.
71
Search Strategy
Strategy for Search terms
In search strategy, the search process will be conducted according to the following decided
steps:
6. I will derive major search strings from my SLR topic.
7. Identify alternative spellings and synonyms for major terms; also alternative terms used
in literature will be considered (e.g. prioritization, optimization)
8. When database allows, use the Boolean OR to incorporate alternative spellings and
synonyms; and when database allows, use the Boolean AND to link the major terms
from population, intervention and outcome.
9. Break down the strings so that it can be applied to different databases. Give unique ID to
every segmented search string. And I will customize them for given resources to be
applied for searches.
10. Tools used for automating the search process
Step1: Major Search Terms:
-
Search Based techniques
Test case
Optimization
Empirical
Step 2: alternative spellings and synonyms for major terms
-
-
Search Based optimization algorithm (Search Based Optimization Algorithm
OR Genetic algorithm OR GA OR Particle swarm optimization OR PSO OR
Evolutionary Algorithm OR metaheuristic)
Test case (Test case OR Test suite OR Test data)
Prioritization (Prioritization OR Optimization OR Reduction OR Selection OR
Minimization OR optimized Generation)
Empirical (Empirical study OR Case study OR Survey OR Technical Report
OR Experiment)
Step 3: Integrating Boolean <OR> and <AND>
(Search Based Optimization Algorithm OR Genetic algorithm OR GA OR Particle swarm
optimization OR PSO OR Evolutionary Algorithm OR metaheuristic) AND (Test Case OR Test
Suit) AND (Prioritization OR Optimization OR Reduction OR Selection OR Minimization OR
optimized Generation) AND (Empirical study OR Case study OR Survey OR Technical Report
OR Experiment)
Step 5: Tools used for automating the search process
72
To speed up the process of documenting my search results I will be using two tools which
will help me in automating the process. These tools are as follows:
-
-
End notes [for documenting bibliography from searches]; I will retrieve search results
in End notes and later, on the basis of titles and abstracts I will include or exclude
papers. It will help me in documenting my search results for both primary searches
and for primary studies.
MS Access [to track the references].
Resources to be searched
A range of data bases has been selected for rigorous search and to reduce the bias.
Following data bases will be searched for the retrieval of primary studies:




Springer link
IEEE Explore
ACM Digital library
Science Direct
Other sources to be searched
-
Various databases are being selected to reduce the bias of study including published
Technical Reports, Journal Papers and Conference Proceedings.
I will also contact key authors directly to check for any new works that have been
accepted for publication, but these are still in printing process.
Then I will go for secondary studies mentioned in the references of the each primary
study by applying same procedures as applied for primary studies.
Study selection criteria







The initial selection will be on the basis of the TITLE and ABSTRACT of the paper.
All data obtained from the search process will be archived in database according to the
journal from which it is retrieved.
From database the duplicates originating from various search engines will be removed after
initial scan of results.
Inclusion and exclusion criteria will be applied on the remaining articles to sort out the
accepted ones.
On accepted papers detailed inclusion criteria will be applied which is a Quality Instrument
for studies to see whether these qualify to be included or not.
Full papers of all studies that are not clearly ineligible to fit in the criteria, will then be
discussed with supervisor for their final exclusion.
The excluded papers and reasons for exclusion will be recorded in a file, and the included
papers and study type will be recorded in another file.
73
Study of inclusion and Exclusion Criteria
Various criteria are formulated with the intension to identify those studies that provide
direct evidence for the research questions. Following are the inclusion and exclusion criteria
for my research questions:
Inclusion criteria:
In order to answer the stated research questions, I have been searching for research
articles by reading the abstracts that contain information about search based test case
optimization with some empirical work done in it. So the papers including a case study, an
experiment, survey or industrial experience reports are focused. When it will be confirmed
after reading the abstract that article is relevant to my research, I will study the whole
paper. The objective of the selection process was to identify the Articles relevant to the
objectives of the systematic review. The search strings, are quite broad and hence it is
expected that not all studies identified would find a place in the final phase of selection
process. Only those studies will be included that are empirical based and where the main
focus is test case optimization with search based technique(s).
Exclusion criteria:
-
Those studies will be excluded that are based on personal expert opinion.
One study can be reported in multiple papers so on the basis of validity score I will
include highest ranked journals and exclude others.
Multiple studies can be reported in one paper; if any of them do not stand to our
inclusion criteria and differ variably then only the irrelevant study will be excluded.
Quality Instrument for assessing validity
After initial selection of studies, a more detail criteria is required to judge the quality of
study to see whether it is worth considering as evidence to answer our research question or
not.
Quality Instrument will be designed for assigning numerical values for factors in the checklist
to be evaluated for each study. My main focus is on the study design (of the paper).
The research questions and my inclusion/exclusion criteria will suggest me that I am going to
include only empirical studies like Case studies, Industrial Experience Reports etc.
If any paper is considered to be very poor in quality it will not be excluded at this stage and
its record will be maintained. Quality assessment checklists for empirical data e.g. survey,
case study, experiment and experience report etc will be used from guidelines of
Kitchenham (2007) [21].
A Quality Instrument including check list for assessing the quality of study and assigning
numerical values to the questions to rank the selected papers.
74
Quality Instrument Checklist for Assessing the Articles
Sr. No.
Quality Assessment Questions
General Paper Evaluation
Do the author(s) clearly mention the issues or
1
hypothesis for the paper?
Is the paper clearly describe/define the design
2
(procedure followed, treatments and all
controls) of empirical study used?
3
Are the findings or results clearly stated?
Does the case/data choose for analysis is big
4
enough to narrate with real life problem?
Is the data used for analysis sufficiently
5
rigorous?
Is the article written in a balanced and
6
understandable manner?
7
Are threats to validity described?
According to Research Question Evaluation
Relevant to my research question?
8
(Score 1 for each research question)
Does the article include more than one
9
algorithm?
Do the search based algorithm(s) described in
10
detail?
Are the factors affecting the performance
11
stated?
Is the performance comparison between
12
algorithms clearly presented?
Total
Scale
Score
Y=1/Partly=0.5/N=0
Y=1/Partly=0.5/N=0
Y=1/Partly=0.5/N=0
Y=1/Partly=0.5/N=0
Y=1/Partly=0.5/N=0
Y=1/Partly=0.5/N=0
Y=1/Partly=0.5/N=0
Strongest=3-Lowest=0
Y=1/N=0
Y=1/Partly=0.5/N=0
Y=1/Partly=0.5/N=0
Y=1/Partly=0.5/N=0
14
Search Process Documentation
Primary Search Documentation
The customized search strings will be applied to the databases according to decided
strategy. As the process will go on, the results will be saved by the following decided
strategy.
i.
ii.
iii.
iv.
v.
Main folder by name of RQ.1 will be created.
Within folder further sub folders by the name of specified database or journal will be
created.
Within each folder for different search string terms different folders will be created by
the name of their IDs.
All records will be maintained for one search string in library of reference manager
software (endnotes).
Results of that specific search strings will be placed in that folder created by the name
of that search string.
75
vi.
vii.
viii.
ix.
x.
xi.
Duplicates (in papers and studies in papers) will be removed from data base after
scanning the records.
After applying inclusion/exclusion criteria within the folder by the name of journal, I
will create two folders for included and excluded papers. This will also give indication
that which journal gave more evidence then others.
Data base will be updated for included and excluded papers.
Reasons for exclusion will be recorded in a file.
All included papers will be moved to one folder.
Conflicts for papers where inclusion or exclusion is ambiguous will be consulted
according to the decided rules, another file will be created to record these activities,
the decisions will be recorded accordingly and papers will either be accepted or
rejected.
Secondary Search Documentation
From accepted primary studies, secondary searches will be made and same procedure will
be followed as followed for the documentation of primary searches.
Data Extraction








Data-Extraction form will be applied it to all the accepted papers and differences between
results will be resolved by returning to the relevant literature, discussion, and when
necessary consultation with supervisors.
The data-extraction forms will be entered into the Data base for results.
Duplicate publication will be identified by cross-checking the study population and location
for all studies reporting same results.
When duplicates are identified, the paper that reports the most recent results of the study
will be prefer.
Data extraction will be performed only for that decided paper.
For each paper I will give an ID to the study.
For one paper having multiple studies I will give separate Study ID to each study.
The form will first obtain general information about the paper and then data extraction
procedure will be applied related to the research questions.
General Information required for a single Paper
Reference No.
Title
Authors
Published
In
Year
Abstract
Algorithms
Optimization Approach
Selection/Reduction/Prioritization/Generation
Issues
Type of research
Experiment/Survey/Case Study/Industrial Report
76
Experiment
Research
Methodology
Quality
Score
Optimization
Approach
Prioritization
Factors
limiting the
performance
2 instances, A
30 test case
and B 90 test
cases.
Requirement
coverage
Optimization
Based on
criteria
NSGA-II,
MoCell,
SPEA2
Algorithms
Based on
requirement
coverage and
execution
time
Comparison
Any Comparison?
Procedure followed
Factors effecting the
performance
Results
Remarks
Yes/No
(if yes)
Extraction Data Table
77
s-3
s-2
[14]
s-1
.
.
.
Paper
Reference
Study ID
Validation of Review Process
The final version of the protocol will be formed after performing two steps; Protocol Evaluation
and Pilot testing.
Protocol Evaluation
Following strategy will be applied for evaluation;
o
o
o
The protocol will be initially given for peer review.
Then will be evaluated from supervisor.
It would then be sent for external evaluation to selected panel of independent
reviewers.
Protocol will be updated into final version after comments from external reviewers and pilot
testing.
Pilot testing
For pilot testing, I run the atomic search strings on IEEE search engine only and
results are:
Sr. No.
Search Engine: IEEE Explorer
Search String
Refined By
St-1.1
(("test case optimization") AND
genetic)
St-1.2
(("test case prioritization") AND
genetic)
St-1.3
(("test suite prioritization") AND
genetic)
St-1.4
(("test suite optimization") AND
genetic)
St-1.5
(("test suite reduction") AND
genetic)
St-1.6
(("test case reduction") AND
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Date
Results
02-05-11
4
02-05-11
5
02-05-11
0
02-05-11
1
02-05-11
2
02-05-11
1
78
genetic)
St-1.7
(("test suite generation") AND
genetic)
St-1.8
(("test case generation") AND
genetic)
St-1.9
(("test suite generation") AND
evolution)
St-1.10
(("test case generation") AND
evolution)
St-1.11
(("test case optimization") AND
evolution)
St-1.12
(("test suite optimization") AND
evolution)
St-1.13
(("test suite prioritization") AND
evolution)
St-1.14
(("test case prioritization") AND
evolution)
St-1.15
(("test case reduction") AND
evolution)
St-1.16
(("test suite reduction") AND
evolution)
St-1.17
((("particle swarm
optimization") OR "PSO") AND
"test suite reduction")
((("particle swarm
optimization") OR "PSO") AND
"test case reduction")
((("particle swarm
optimization") OR "PSO") AND
"test case prioritization")
((("particle swarm
optimization") OR "PSO") AND
"test suite prioritization")
((("particle swarm
optimization") OR "PSO") AND
"test suite optimization")
((("particle swarm
optimization") OR "PSO") AND
"test case optimization")
((("particle swarm
St-1.18
St-1.19
St-1.20
St-1.21
St-1.22
St-1.23
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
02-05-11
2
02-05-11
49
02-05-11
0
02-05-11
9
02-05-11
0
02-05-11
1
02-05-11
0
02-05-11
3
02-05-11
0
02-05-11
2
02-05-11
1
02-05-11
0
02-05-11
1
02-05-11
0
02-05-11
1
02-05-11
0
02-05-11
0
79
St-1.24
St-1.25
optimization") OR "PSO") AND
"test case generation")
((("particle swarm
optimization") OR "PSO") AND
"test suite generation")
((("ant colony") OR "ATO") AND
"test suite generation")
St-1.26
((("ant colony") OR "ATO") AND
"test case generation")
St-1.27
((("ant colony") OR "ATO") AND
"test case prioritization")
St-1.28
((("ant colony") OR "ATO") AND
"test suite prioritization")
St-1.29
((("ant colony") OR "ATO") AND
"test suite optimization")
St-1.30
((("ant colony") OR "ATO") AND
"test case optimization")
St-1.31
((("ant colony") OR "ATO") AND
"test case reduction")
St-1.32
((("ant colony") OR "ATO") AND
"test suite reduction")
St-1.33
Search based") AND "test suit
reduction")
St-1.34
(("Search based") AND "test
case reduction")
St-1.35
(("Search based") AND "test
case optimization")
St-1.36
(("Search based") AND "test
suite optimization")
St-1.37
(("Search based") AND "test
suite prioritization")
St-1.38
(("Search based") AND "test
case prioritization")
St-1.39
(("Search based") AND "test
case generation")
St-1.40
(("Search based") AND "test
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
02-05-11
0
02-05-11
0
02-05-11
0
02-05-11
0
02-05-11
0
02-05-11
1
02-05-11
0
02-05-11
0
02-05-11
0
03-05-11
0
03-05-11
0
03-05-11
0
03-05-11
0
03-05-11
0
03-05-11
0
03-05-11
8
03-05-11
0
80
suite generation")
St-1.41
((Bacteriological) AND "test
suite generation")
St-1.42
((Bacteriological) AND "test case
generation")
St-1.43
((Bacteriological) AND "test case
optimization")
St-1.44
((Bacteriological) AND "test
suite optimization")
St-1.45
((Bacteriological) AND "test
suite prioritization")
St-1.46
((Bacteriological) AND "test case
prioritization")
St-1.47
((Bacteriological) AND "test case
reduction")
St-1.48
((Bacteriological) AND "test
suite reduction")
St-1.49
((Bacteriological) AND "test
suite selection")
St-1.50
((Bacteriological) AND "test case
selection")
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
Subject: Computing &
Processing
(Hardware/Software)
03-05-11
0
03-05-11
0
03-05-11
2
03-05-11
0
03-05-11
0
03-05-11
0
03-05-11
0
03-05-11
0
03-05-11
13
03-05-11
75
And the results of integrated queries are:
Sr. No.
St-IEEE1
St-IEEE2
Search Engine: IEEE Explorer
Search String
Refined By
((((Abstract:Genetic) OR
Abstract:GA) AND (((Abstract:"test
case") OR Abstract:“test suite”) OR
Abstract:“test data”)) AND
((((((Abstract:optimization) OR
Abstract:prioritization) OR
Abstract:selection) OR
Abstract:optimized generation) OR
Abstract:reduction) OR
Abstract:minimization))
(((((Abstract:"Particle swarm
optimization") OR Abstract: "PSO")
OR Abstract: swarm) AND
Date
Results
Subject: Computing &
Processing
(Hardware/Software)
08-07-11
318
Subject: Computing &
Processing
(Hardware/Software)
08-07-11
175
81
St-IEEE3
St-IEEE4
St-IEEE5
(((Abstract:"test case") OR
Abstract:“test suite”) OR
Abstract:“test data”)) AND
((((((Abstract:optimization) OR
Abstract:prioritization) OR
Abstract:selection) OR
Abstract:optimized generation) OR
Abstract:reduction) OR
Abstract:minimization))
(((((Abstract:"ant colony
optimization") OR Abstract:ACO)
OR Abstract:"ant colony") AND
(((Abstract:"test case") OR
Abstract:“test suite”) OR
Abstract:“test data”)) AND
((((((Abstract:optimization) OR
Abstract:prioritization) OR
Abstract:selection) OR
Abstract:optimized generation) OR
Abstract:reduction) OR
Abstract:minimization))
((((((Abstract:"Search Based") OR
Abstract:SBST) OR
Abstract:"metaheuristic") OR
Abstract:heuristic) AND
(((Abstract:"test case") OR
Abstract:“test suite”) OR
Abstract:“test data”)) AND
((((((Abstract:optimization) OR
Abstract:prioritization) OR
Abstract:selection) OR
Abstract:optimized generation) OR
Abstract:reduction) OR
Abstract:minimization))
((((Abstract:"Evolutionary
Algorithm") OR Abstract:Evolution)
AND (((Abstract:"test case") OR
Abstract:“test suite”) OR
Abstract:“test data”)) AND
((((((Abstract:optimization) OR
Abstract:prioritization) OR
Abstract:selection) OR
Abstract:optimized generation) OR
Abstract:reduction) OR
Abstract:minimization))
Subject: Computing &
Processing
(Hardware/Software)
08-07-11
39
Subject: Computing &
Processing
(Hardware/Software)
08-07-11
149
Subject: Computing &
Processing
(Hardware/Software)
08-07-11
139
12-07-11
30
Search Engine: ACM
StACM-1
((Abstract:"Particle swarm
optimization" OR Abstract: "PSO"
OR Abstract: swarm) AND
(Abstract: "test case" OR Abstract:
"test suite" OR Abstract: "test
data") AND (Abstract:
The ACM Guide to
Computing Literature
82
StACM-2
StACM-3
StACM-4
StACM-5
"prioritization" OR Abstract:
"optimization"OR Abstract:
"reduction" OR Abstract:
"selection" OR Abstract:
"minimization" OR Abstract:
optimized generation))
((Abstract:"Genetic" OR Abstract:
"GA") AND (Abstract: "test case"
OR Abstract: "test suite" OR
Abstract: "test data") AND
(Abstract: "prioritization" OR
Abstract: "optimization" OR
Abstract: "reduction" OR Abstract:
"selection" OR Abstract:
"minimization" OR Abstract:
optimized generation))
((Abstract:"ant colony
optimization" OR Abstract: "ACO"
OR Abstract: "ant colony") AND
(Abstract: "test case" OR Abstract:
"test suite" OR Abstract: "test
data") AND (Abstract:
"prioritization" OR Abstract:
"optimization" OR Abstract:
"reduction" OR Abstract:
"selection" OR Abstract:
"minimization" OR Abstract:
optimized generation))
((Abstract:"Search Based" OR
Abstract: "SBST" OR Abstract:
metaheuristic OR Abstract:
heuristic) AND (Abstract: "test
case" OR Abstract: "test suite" OR
Abstract: "test data") AND
(Abstract: "prioritization" OR
Abstract: "optimization" OR
Abstract: "reduction" OR Abstract:
"selection" OR Abstract:
"minimization" OR Abstract:
optimized generation))
((Abstract:"Evolution" OR Abstract:
"Evolutionary Algorithm") AND
(Abstract: "test case" OR Abstract:
"test suite" OR Abstract: "test
data") AND (Abstract:
"prioritization" OR Abstract:
"optimization" OR Abstract:
"reduction" OR Abstract:
"selection" OR Abstract:
"minimization" OR Abstract:
optimized generation))
The ACM Guide to
Computing Literature
12-07-11
37
The ACM Guide to
Computing Literature
14-07-11
2
The ACM Guide to
Computing Literature
14-07-11
44
The ACM Guide to
Computing Literature
14-07-11
18
Search Engine: Science Direct
83
St-SD-1
St-SD-2
St-SD-3
St-SD-4
St-SD-5
(TITLE-ABSTR-KEY(genetic) OR
TITLE-ABSTR-KEY(GA)) and (TITLEABSTR-KEY("test case") OR TITLEABSTR-KEY("test suite") OR TITLEABSTR-KEY("test data")) and (TITLEABSTR-KEY(optimization) OR TITLEABSTR-KEY(prioritization)OR TITLEABSTR-KEY(reduction) OR TITLEABSTR-KEY(minimization) OR TITLEABSTR-KEY(selection) OR TITLEABSTR-KEY(optimized generation))
(TITLE-ABSTR-KEY(“Particle swarm
optimization”) OR TITLE-ABSTRKEY(PSO) OR TITLE-ABSTRKEY(swarm)) and (TITLE-ABSTRKEY("test case") OR TITLE-ABSTRKEY("test suite") OR TITLE-ABSTRKEY("test data")) and (TITLE-ABSTRKEY(optimization) OR TITLE-ABSTRKEY(prioritization)OR TITLE-ABSTRKEY(reduction) OR TITLE-ABSTRKEY(minimization) OR TITLE-ABSTRKEY(selection) OR TITLE-ABSTRKEY(optimized generation))
(TITLE-ABSTR-KEY("ant colony
optimization") OR TITLE-ABSTRKEY(ACO) OR TITLE-ABSTR-KEY("ant
colony")) and (TITLE-ABSTRKEY("test case") OR TITLE-ABSTRKEY("test suite") OR TITLE-ABSTRKEY("test data")) and (TITLE-ABSTRKEY(optimization) OR TITLE-ABSTRKEY(prioritization)OR TITLE-ABSTRKEY(reduction) OR TITLE-ABSTRKEY(minimization) OR TITLE-ABSTRKEY(selection) OR TITLE-ABSTRKEY(optimized generation))
(TITLE-ABSTR-KEY("Search Based")
OR TITLE-ABSTR-KEY(SBST) OR TITLEABSTR-KEY(metaheuristic) OR TITLEABSTR-KEY(heuristic)) and (TITLEABSTR-KEY("test case") OR TITLEABSTR-KEY("test suite") OR TITLEABSTR-KEY("test data")) and (TITLEABSTR-KEY(optimization) OR TITLEABSTR-KEY(prioritization)OR TITLEABSTR-KEY(reduction) OR TITLEABSTR-KEY(minimization) OR TITLEABSTR-KEY(selection) OR TITLEABSTR-KEY(optimized generation))
(TITLE-ABSTR-KEY("Evolutionary
Algorithm") OR TITLE-ABSTR-
All Sources(Computer Science)
14-07-11
43
All Sources(Computer
Science)
25-07-11
18
All Sources(Computer
Science)
25-07-11
3
All Sources(Computer
Science)
25-07-11
45
All Sources(Computer
Science)
25-07-11
35
84
KEY(Evolution)) and (TITLE-ABSTRKEY("test case") OR TITLE-ABSTRKEY("test suite") OR TITLE-ABSTRKEY("test data")) and (TITLE-ABSTRKEY(optimization) OR TITLE-ABSTRKEY(prioritization)OR TITLE-ABSTRKEY(reduction) OR TITLE-ABSTRKEY(minimization) OR TITLE-ABSTRKEY(selection) OR TITLE-ABSTRKEY(optimized generation))
Schedule of Activities
Activity
Starting Date
Completion Date
4th March 2011
13th April 2011
15th April 2011
25th April 2011
26th April 2011
20th May 2011
Pilot Testing
21st May 2011
15th August 2011
Analysis
16th August 2011
15th September 2011
Documentation of
Results
15th September 2011
25th October 2011
Comments
Protocol development
Version V1.1
Protocol Evaluation
Protocol
Documentation
version 1.2
References
76. Mark Harman, Bryan F. Jones, “Search-Based Software Engineering”, Information and
Software Technology, vol-43, pg. 833-839, 2001.
85
2. Zheng Li, Mark Harman, and Robert M. Hierons, “Search Algorithms for Regression
Test Case Prioritization”, IEEE Transactions On Software Engineering, Vol. 33, No.
4, April 2007.
3. James A. Jones, Mary Jean Harrold, “Test-Suite Reduction and Prioritization for Modified
Condition/Decision Coverage”, Software Engineering, IEEE Transactions, 2003.
4. Saif ur Rehman Khan, et al, “The Impact of Test Case Reduction and Prioritization on
Software Testing Effectiveness”, International Conference on Emerging
Technologies, IEEE, 2009.
5. Lu Zhang, et al, “Time-Aware Test-Case Prioritization using Integer Linear
Programming”, ISSTA’09, Chicago, Illinois, USA, 2009.
6. Kristen R. Walcott, Gregory M. Kapfhammer, “Time Aware Test Suite
Prioritization”, ISSTA’06, July 17–20, 2006.
7. M. Harman, A. Mansouri, and Y. Zhang, “Search based software engineering: A
comprehensive analysis and review of trends techniques and applications”, Technical Report
TR-09-03, Department of Computer Science, King's College London, April 2009.
8. Mark Harman, Afshin Mansouri, “Search Based Software Engineering: Introduction
to the Special Issue of the IEEE Transactions on Software Engineering”, IEEE
Transactions on Software Engineering, Vol. 36, No. 6, November/December 2010.
9. Alexander P. Conrad, Robert S. Roos, “Empirically Studying the Role of Selection
Operators during Search-Based Test Suite Prioritization”, GECCO’10, July 7.11,
2010.
10. Camila Loiola Brito Maia, et al, “Automated Test Case Prioritization with Reactive
GRASP”, Hindawi Publishing Corporatio, Advances in Software Engineering,
Volume 2010.
11. Bo Jiang, et al, “Adaptive Random Test Case Prioritization”, 24th IEEE/ACM
International Conference on Automated Software Engineering, 2009.
12. K. H.Saw Hla; Y. Choi; Jong Sou Park;, “Applying Particle Swarm Optimization to
Prioritizing Test Cases for Embedded Real Time Software Retesting”, IEEE 8th
International Conference on Computer and Information Technology Workshops,
2008.
13. S. Praveen Ranjan, Tai-hoon Kim, “Application of Genetic Algorithm in Software
Testing”, International Journal of Software Engineering and Its Applications, Vol. 3,
No.4, October 2009.
14. Camila Loiola Brito Maia, et al, “Applying Search-Based Techniques for
Requirements-Based Test Case Prioritization”, Proceedings of the Brazilian
Workshop on Optimization in Software Engineering (WOES '10), 2010.
15. Jerffeson Teixeira de Souza, et al, “The Human Competitiveness of Search Based
Software Engineering”, 2nd International Symposium on Search Based Software
Engineering, IEEE 2010.
16. Yu-Chi Huang, et al, “Design and Analysis of Cost-Cognizant Test Case
Prioritization Using Genetic Algorithm with Test History”, IEEE 34th Annual
Computer Software and Applications Conference, 2010.
17. Phil McMinn, “Search based software test data generation: A survey”, Software Testing,
Verification and Reliability, 14(2), pp. 105-156, 2004.
86
18. S. Ali, L.C. Briand, H. Hemmati, and R.K. Panesar-Walawege, “A Systematic Review of the
Application and Empirical Investigation of Search based test case generation”, IEEE
Transactions on Software Engineering, vol 36, No. 6, 2010.
19. W. Afzal, R. Torkar, R. Feldt, “A systematic mapping study on non-functional search-based
software testing”, 20th International Conference on Software Engineering and Knowledge
Engineering (SEKE), 2008.
20. W. Afzal, R. Torkar, R. Feldt, “A systematic review of search based software testing for nonfunctional system properties”, Information and software technology, 2009.
21. B. Kitchenham, “Guidelines for performing Systematic Literature Reviews in Software
Engineering”, Version 2.3, EBSE Technical Report EBSE-2007-01, Keele University and
University of Durham, 2007.
22. M. Höst, P. Runeson, “Checklists for Software Engineering Case Study Research”, In
Proceedings First International Symposium on Empirical Software Engineering and
Measurement, pp 479–481, 2007.
87
Appendix B: General Extraction
Reference No.
1
Title
Test Case Prioritization Technique based on Genetic Algorithm
Authors
Wang Jun, Zhuang Yan, Jianyun Chen
Published
In
2011 International Conference on Internet Computing and Information
Services IEEE
Year
2011
Abstract
With the rapid development of information technology, software testing,
as a software quality assurance, is becoming more and more important. In
the software life cycle, each time the code has changed need to be
regression testing. The huge test case library makes running a full test case
library being challenged. To this end, we designed a genetic algorithmbased test case prioritization algorithm and improved the genetic
algorithm proposed software test case prioritization algorithm.
Algorithms
Optimization Approach
 Genetic Algorithm
 Improved Genetic Algorithm
Prioritization
Issues
--
Type of research
Experiment
Any Comparison?
Yes
The improved algorithm based on baseline testing is more rapid.
It can increase the finding rate of test sequence errors, and more
quickly to identify the system’s reliability, reduce software testing
costs.
Fitness metrics
Block coverage
Factors
effecting
performance
the
Trapping in Local optima position
Quality Instrument score
8
Reference No.
2
Title
A history-based cost-cognizant test case prioritization technique in
regression testing
Authors
Yu-Chi Huang, Kuan-Li Peng, Chin-Yu Huang
Published
In
The Journal of Systems and Software 85 (2012) 626– 637
88
Year
2012
Abstract
Software testing is typically used to verify whether the developed software
product meets its requirements. From the result of software testing,
developers can make an assessment about the quality or the acceptability
of developed software. It is noted that during testing, the test case is a
pair of input and expected output, and a number of test cases will be
executed either sequentially or randomly. The techniques of test case
prioritization usually schedule test cases for regression testing in an order
that attempts to increase the effectiveness. However, the cost of test
cases and the severity of faults are usually varied. In this paper, we
propose a method of cost-cognizant test case prioritization based on the
use of historical records. We gather the historical records from the latest
regression testing and then propose a genetic algorithm to determine the
most effective order. Some controlled experiments are performed to
evaluate the effectiveness of our proposed method. Evaluation results
indicate that our proposed method has improved the fault detection
effectiveness. It can also been found that prioritizing test cases based on
their historical information can provide high test effectiveness during
testing.
Algorithms
• Optimal prioritization (optimal)
• Random prioritization
• Historical fault detection effectiveness prioritization (histfault-detected)
• Time-aware test suite prioritization (GA-fun)
• GA-hist
• Total function coverage prioritization (total-fun)
• Cost-cognizant total function coverage prioritization (cctotal-fun)
Prioritization
Optimization Approach
Issues
The cost of test cases and the severity of faults are usually varied. In this
paper, we propose a method of cost-cognizant test case prioritization
based on the use of historical records
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
Average percentage of faults detected per cost (APFDc)
Factors
effecting
performance
the
Results have shown that on average, the proposed technique can
significantly improve the effectiveness in comparison with three
coverage-based techniques and two other history-based
techniques. It also performed well when test case costs and fault
severities were equal. In contrast to the other GA-produced
prioritization technique used in the experiment, the proposed
technique provided higher efficiency because the number of faults
was far fewer than the number of functions. According to the
experimental results, we verify that the proposed technique is
useful. We also confirm that prioritizing test cases based on their
historical information can provide high effectiveness.
Test case costs(time) and faults severities vary
89
GA stuck in local optima
Impact of generations and population size
Quality Instrument score
13
Reference No.
3
Title
Automated Test Case Prioritization with Reactive GRASP
Authors
Camila Loiola Brito Maia, Rafael Augusto Ferreira do Carmo, Fabr´ıcio
Gomes de Freitas, Gustavo Augusto Lima de Campos, and Jerffeson
Teixeira de Souza
Published
In
Hindawi Publishing Corporation Advances in Software Engineering Volume
2010, Article ID 428521, 18 pages
Year
2010
Abstract
Modifications in software can affect some functionality that had been
working until that point. In order to detect such a problem, the ideal
solution would be testing the whole system once again, but there may be
insufficient time or resources for this approach. An alternative solution is
to order the test cases so that the most beneficial tests are executed first,
in such a way only a subset of the test cases can be executed with little
lost of effectiveness. Such a technique is known as regression test case
prioritization. In this paper, we propose the use of the Reactive GRASP
metaheuristic to prioritize test cases. We also compare this metaheuristic
with other search-based algorithms previously described in literature. Five
programs were used in the experiments. The experimental results
demonstrated good coverage performance with sometime overhead for
the proposed technique. It also demonstrated a high stability of the results
generated by the proposed approach.
Algorithms
 Greedy algorithm
 Additional Greedy algorithm
 Genetic algorithm
 Simulated annealing
 Reactive GRASP
Prioritization
Optimization Approach
Issues
In this paper, we propose the use of the Reactive GRASP metaheuristic to
prioritize test cases. We also compare this metaheuristic with other
search-based algorithms previously described in literature.
Type of research
Experiment
Any Comparison?
Yes
 The Greedy algorithm performed the worst
 The Reactive GRASP algorithm had the second best
performance.
 Additional Greedy algorithm had the best performance in
effectiveness of all tests.
90
 Genetic algorithm approach performed the fourth best in our
evaluation.
 Simulated Annealing is the 3rd best
 Reactive GRASP had the minor difference in performance
between the best and the worst coverage criterion, which
demonstrates an interesting characteristic of this algorithm: its
stability.
APBC, APDC, and APSBC
Fitness metrics
Factors
effecting
performance
the
--
Quality Instrument score
11
Reference No.
4
Title
Analyzing Test Case Selection & Prioritization using ACO
Authors
Bharti Suri, Shweta Singhal
Published
In
ACM SIGSOFT Software Engineering Notes, November 2011 Volume 36
Number 6
Year
2011
Abstract
Regression testing is primarily a maintenance activity that is performed
frequently to ensure the validity of the modified software. In such cases,
due to time and cost constraints, the entire test suite cannot be run. Thus,
it becomes essential to select or prioritize the tests in order to cover
maximum faults in minimum time. Recently, Ant Colony Optimization
(ACO), which is a new way to solve time constraint prioritization problem,
has been utilized. This paper presents the analysis of the regression test
prioritization technique to reorder test suites in time constraint
environment along with the sample runs on various programs. Our
analysis concluded that the ACO finds better orderings at higher values of
the time constraint (TC). The correctness of the technique has also been
recorded to be near optimal at an average.
Algorithms
Ant Colony Optimization (ACO)
Optimization Approach
Selection
Prioritization
Issues
Time and cost analysis for TCP by ACO
Type of research
Experiment
Any Comparison?
No
It can now be summarized that higher the value of the TC, more
is the number of iterations and hence, more correct is the final
outcome. ACO is a strong & robust heuristic approach involving
positive feedback and parallel computations and hence, leads to
91
the best or near best solutions in optimum time.
Fitness metrics
Optimal path found
Factors
effecting
performance
the
Number of iterations
Time Constraint(TC) value
Quality Instrument score
9
Reference No.
5
Title
Applying Particle Swarm Optimization to Prioritizing Test Cases for
Embedded Real Time Software Retesting
Authors
Khin Haymar Saw Hla, YoungSik Choi, Jong Sou Park
Published
In
IEEE 8th International Conference on Computer and Information
Technology Workshops
Year
2008
Abstract
In recent years, complex embedded systems are used in every device that
is infiltrating our daily lives. Since most of the embedded systems are
multi-tasking real time systems, the task interleaving issues, deadlines and
other factors needs software units retesting to follow the subsequence
changes. Regression testing is used for the software maintenance that
revalidates the old functionality of the software unit. Testing is one of the
most complex and time-consuming activities, in which running of all
combination of test cases in test suite may require a large amount of
efforts. Test case prioritization techniques can take advantage that orders
test cases, which attempts to increase effectiveness in regression testing.
This paper proposes to use particle swarm optimization (PSO) algorithm to
prioritize the test cases automatically based on the modified software
units. Regarding to the recent investigations, PSO is a multi-object
optimization technique that can find out the best positions of the objects.
The goal is to prioritize the test cases to the new best order, based on
modified software components, so that test cases, which have new higher
priority, can be selected in the regression testing process. The empirical
results show that by using the PSO algorithm, the test cases can be
prioritized in the test suites with their new best positions effectively and
efficiently.
Algorithms
Particle Swarm Optimization
Greedy
random
Prioritization
Optimization Approach
Issues
The goal is to prioritize the test cases to the new best order, based on
modified software components, so that test cases, which have new higher
92
priority, can be selected in the regression testing process.
Type of research
Experiment
Any Comparison?
Yes
By using the PSO algorithm, the test cases can be prioritized in
the test suites with their new best positions effectively and
efficiently.
PSO is better in coverage criteria and time execution
Fitness metrics
Nil –100 iterations to stop
Factors
effecting
performance
the
---
Quality Instrument score
9
Reference No.
6
Title
Building Prioritized Pairwise Interaction Test Suites with Ant Colony
Optimization
Authors
Xiang Chen, Qing Gu, Xin Zhang, Daoxu Chen
Published
In
Ninth International Conference on Quality Software
Year
2009
Abstract
Interaction testing offers a stable cost-benefit ratio in identifying faults.
But in many testing scenarios, the entire test suite cannot be fully
executed due to limited time or cost. In these situations, it is essential to
take the importance of interactions into account and prioritize these tests.
To tackle this issue, the biased covering array is proposed and the
Weighted Density Algorithm (WDA) is developed. To find a better solution,
in this paper we adopt ant colony optimization (ACO) to build this
prioritized pairwise interaction test suite (PITS). In our research, we
propose four concrete test generation algorithms based on Ant System,
Ant System with Elitist, Ant Colony System and Max-Min Ant System
respectively. We also implement these algorithms and apply them to two
typical inputs and report experimental results. The results show the
effectiveness of these algorithms.
Algorithms
Ant colony optimization
 Ant System (AS),
 Ant System with Elitist (ASelite),
 Ant Colony System (ACS) and
 Max-Min Ant System(MMAS)
 Prioritization
 Generation
(1) The importance of interactions is taken into account and these tests
are prioritized. (2) The strength of different factors may not be fixed, but a
variable. (3) In many situations, there are constraints between specific
Optimization Approach
Issues

93
factors that render certain interactions invalid.
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
--
Factors
effecting
performance
the
 Randomness inherent in this algorithm.
 ACS and MMAS generally produce the fastest rate of coverage of
first 10 tests, AS has the slowest rate of coverage, and ASelite is
between these strategies.
 The generation time is longer than WDA, and the generation
time of ASelite is longest while the generation time of AS is
shortest, MMAS and ACS is between these two.
--
Quality Instrument score
11
Reference No.
7
Title
Design and Analysis of Cost-Cognizant Test Case Prioritization Using
Genetic Algorithm with Test History
Authors
Yu-Chi Huang, Chin-Yu Huang, Jun-Ru Chang and Tsan-Yuan Chen
Published
In
IEEE 34th Annual Computer Software and Applications Conference
Year
2010
Abstract
During software development, regression testing is usually used to assure
the quality of modified software. The techniques of test case prioritization
schedule the test cases for regression testing in an order that attempts to
increase the effectiveness in accordance with some performance goal. The
most general goal is the rate of fault detection. It assumes all test case
costs and fault severities are uniform. However, those factors usually vary.
In order to produce a more satisfactory order, the cost-cognizant metric
that incorporates varying test case costs and fault severities is proposed.
In this paper, we propose a cost-cognizant test case prioritization
technique based on the use of historical records and a genetic algorithm.
We run a controlled experiment to evaluate the proposed technique’s
effectiveness. Experimental results indicate that our proposed technique
frequently yields a higher Average Percentage of Faults Detected per Cost
(APFDc). The results also show that our proposed technique is also useful
in terms of APFDc when all test case costs and fault severities are uniform.
Algorithms






GA-hist,
hist-value
hist-fault-detected (historical fault detection effectiveness
prioritization)
cc-total-fun (cost-cognizant total function coverage prioritization),
total-fun (total function coverage prioritization)
GA-fun
94
Optimization Approach
Prioritization
Issues
Which techniques frequently yields a higher Average Percentage of Faults
Detected per Cost (APFDc).
The proposed technique is to search for an order of the existing test suite
which has the greatest effectiveness in terms of cost-cognizant test case
prioritization in the preceding regression testing.
Type of research
Experiment
Any Comparison?
Yes
Condition 1: When variety of test case costs and fault
severities.
GA-hist has a greater mean APFDc value than the other
techniques. Comparing GA-hist with random, GA-hist is a
significant improvement on mean APFDc. GA-hist also performs
well in comparison to history-based prioritization techniques
such as hist-fault-detect and hist-value. In contrast with
coverage-based prioritization techniques such as GA-fun, totalfun, and cc-total-fun, on average, GA-hist improves 2% to 4%,
but is a significant improvement in comparison to total-fun.
Condition 2: the cost and total detected fault severity of the
test cases are uniform.
GA-hist also provides higher effectiveness than the other
techniques.
The proposed technique prioritizes test cases according to their
various test costs and fault severities without analyzing the
source code.
Fitness metrics
APFDc
Quality Instrument score




13
Reference No.
8
Title
Test Case Filtering and Prioritization Based on Coverage of Combinations
of Program Elements
Authors
Wes Masri, Marwa El-Ghali
Factors
effecting
performance
Published
the
Number of generations
Population size
Fault severities
Cost of test cases vary
In
WODA ’09, July 20, 2009, Chicago, Illinois, USA. ACM
Year
2009
95
Abstract
Test case filtering is concerned with selecting from a test suite T a subset
T’ that is capable of revealing most of the defects revealed by T. A smaller
T’ is desirable since it translates to fewer test runs to be audited manually.
Test case prioritization, a related technique, aims at scheduling the tests in
T so that the defects are revealed as early as possible when T gets
executed. We propose techniques that are based on coverage of
combinations of program elements of different types. Clearly, exploring all
possible combinations induced at runtime is infeasible, which calls for the
use of an approximation algorithm. In this paper we investigate the use of
a genetic algorithm to select a number of suitable combinations of
program elements to be covered. We compared our technique to other
coverage based techniques that consider program elements of the same
type and that do not take into account their combinations; our preliminary
results were promising. For example, after filtering the original test suite T
for JTidy, the resulting T’ revealed all the defects in T and was only 14.1%
its size.
Algorithms
Genetic Algorithm
Greedy Algorithm
 Selection
 Prioritization
Existing coverage-based test case filtering and prioritization techniques
seek to cover program elements of varying granularity, e.g., methods,
statements, branches, def-use pairs, slice pairs and information flow pairs.
Combinations of these program elements are more likely to characterize
complex failures, i.e., failures induced by defects involving multiple
program interactions.


Optimization Approach
Issues
Type of research
Experiment
Any Comparison?
No
Fitness metrics
Coverage of faults
Factors
effecting
performance
the
--
Quality Instrument score
10
Reference No.
9
Title
Empirically Studying the Role of Selection Operators During Search-Based
Test Suite Prioritization
Authors
Alexander P. Conrad, Robert S. Roos
Published
Abstract
In
GECCO’10, July 7.11, 2010, Portland, Oregon, USA. ACM
Year
2010
Regression test suite prioritization techniques reorder test cases so that,
96
on average, more faults will be revealed earlier in the test suite's
execution than would otherwise be possible. This paper presents a genetic
algorithm-based test prioritization method that employs a wide variety of
mutation, crossover, selection, and transformation operators to reorder a
test suite. Leveraging statistical analysis techniques, such as tree model
construction through binary recursive partitioning and kernel density
estimation, the paper's empirical results highlight the unique role that the
selection operators play in identifying an effective ordering of a test suite.
The study also reveals that, while truncation selection consistently
outperformed the tournament and roulette operators in terms of test
suite effectiveness, increasing selection pressure consistently produces the
best results within each class of operator. After further explicating the
relationship between selection intensity, termination condition, fitness
landscape, and the quality of the resulting test suite, this paper
demonstrates that the genetic algorithm-based prioritizer is superior to
random search and hill climbing and thus suitable for many regression
testing environments.
Algorithms
Random Search
Genetic Algorithm
Greedy Algorithm
Hill Climbing
Prioritization
Optimization Approach
Issues
To highlight the unique role that the selection operators (mutation,
crossover, and selection operators) in identifying an effective ordering of a
test suite.
Type of research
Experiment
Any Comparison?
Yes
GA has three additional advantages over greedy algorithm
1. Parallelization
2. Can be interrupted during execution
3. Can construct diverse test orderings that achieve
equivalent coverage effectiveness scores
 GA produce greater amount of variable and effective test
orderings than greedy and hill climbing in given execution
time.
 High selection intensity is important for producing good test
orderings.
Coverage Effectiveness
Fitness metrics
Factors
effecting
performance
the

Execution time
Number of generations
Selection operator values of GA
Quality Instrument score
13.5
Reference No.
10
97
Title
Using Synthetic Test Suites to Empirically Compare Search-Based and
Greedy Prioritizers
Authors
Zachary D. Williams, Gregory M. Kapfhammer
Published
In
GECCO'10, July 7.11, 2010, Portland, Oregon, USA. ACM
Year
2010
Abstract
The increase in the complexity of modern software has led to the
commensurate growth in the size and execution time of the test suites for
these programs. In order to address this alarming trend, developers use
test suite prioritization to reorder the test cases so that faults can be
detected at an early stage of testing. Yet, the implementation and
evaluation of greedy and search-based test prioritizers requires access to
case study applications and their associated test suites, which are often
difficult to find, configure, and use in an empirical study. This paper
presents two types of synthetically generated test suites that support this
process of experimentally evaluating prioritization methods. Using
synthetic test suites affords greater control over test case characteristics
and supports the identification of empirical trends that contradict the
established wisdom about search-based and greedy prioritization. For
instance, we find that the hill climbing algorithm often exhibits a lower
time overhead than the greedy test suite prioritizer while often producing
test orderings with comparable effectiveness scores.
Algorithms
Optimization Approach
 Greedy
 Hill-climbing
Prioritization
Issues
Emperically compare serach-based and greedy algorithm
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
Cost and coverage effectiveness
Factors
effecting
performance
the
greedy algorithms are faster than search-based methods by
showing that HC's arithmetic means for prioritization time are
lower than those of GRD.
searched-based methods may lack uniformity in their execution times and
construct prioritizations with modest effectiveness ratings
Quality Instrument score
6
Reference No.
11
Title
Time-Aware Test-Case Prioritization using Integer Linear Programming
Authors
Lu Zhang, Shan-Shan Hou, Chao Guo, Tao Xie, Hong Mei
98
Published
In
ISSTA’09, July 19–23, 2009, Chicago, Illinois, USA.
Year
2009
Abstract
Techniques for test-case prioritization re-order test cases to increase their
rate of fault detection. When there is a fixed time budget that does not
allow the execution of all the test cases, time-aware techniques for testcase prioritization may achieve a better rate of fault detection than
traditional techniques for test-case prioritization. In this paper, we
propose a novel approach to time-aware test-case prioritization using
integer linear programming. To evaluate our approach, we performed
experiments on two subject programs involving four techniques for our
approach, two techniques for an approach to time-aware test-case
prioritization based on genetic algorithms, and four traditional techniques
for test-case prioritization. The empirical results indicate that two of our
techniques outperform all the other techniques for the two subjects under
the scenarios of both general and version-specific prioritization. The
empirical results also indicate that some traditional techniques with lower
analysis time cost for test-case prioritization may still perform
competitively when the time budget is not quite tight.
Algorithms
Optimization Approach
 random prioritization
 traditional total statement coverage prioritization
 traditional additional statement coverage prioritization
 traditional total method coverage prioritization
 traditional additional method coverage prioritization
 time-aware statement coverage prioritization via GA
 time-aware method coverage prioritization via GA
 time-aware total statement coverage prioritization via ILP
 time-aware additional statement coverage prioritization via ILP
 time-aware total method coverage prioritization via ILP
 time-aware additional method coverage prioritization via ILP
Prioritization
Issues
Mostly techniques does not consider time budget
Type of research
Experiment
Any Comparison?
Yes
-uFitness metrics
APFD
Factors
effecting
performance
the
Quality Instrument score
 GA techniques are less effective then proposed ILP
techniques in terms of time budget
 Fault detection rate of ILP is much better than GA and
traditional prioritization techniques
 ILP requires less analysis time than GA but greater traditional
techniques
---
8.5
99
Reference No.
12
Title
Time-Aware Test Suite Prioritization
Authors
Kristen R. Walcott, Mary Lou Soffa, Gregory M. Kapfhammer, Robert S.
Roos
Published
In
ISSTA’06, July 17–20, 2006, Portland, Maine, USA. ACM
Year
2006
Abstract
Regression test prioritization is often performed in a time constrained
execution environment in which testing only occurs for a fixed time
period. For example, many organizations rely upon nightly building and
regression testing of their applications every time source code changes are
committed to a version control repository. This paper presents a
regression test prioritization technique that uses a genetic algorithm to
reorder test suites in light of testing time constraints. Experiment results
indicate that our prioritization approach frequently yields higher average
percentage of faults detected (APFD) values, for two case study
applications, when basic block level coverage is used instead of method
level coverage. The experiments also reveal fundamental trade-offs in the
performance of time-aware prioritization. This paper shows that our
prioritization technique is appropriate for many regression testing
environments and explains how the baseline approach can be extended to
operate in additional time constrained testing circumstances.
Algorithms
Random
Genetic Algorithm
Optimization Approach
Prioritization
Issues
This paper shows that if the maximum time allotted for execution of the
test cases is known in advance, a more effective prioritization can be
produced.
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
APFD
Factors
effecting
performance
the
GA prioritization perform better then random prioritization
Time budget
Quality Instrument score
10.5
Reference No.
13
Title
Automated Test Case Generation and Its Optimization for Path Testing
100
Using Genetic Algorithm and Sampling
Authors
Published
Debasis Mohapatra, Prachet Bhuyan, Durga P. Mohapatra
In
2009 WASE International Conference on Information Engineering
Year
2009
Abstract
Software testing is an important activity of the software development
process, and automated test case generation contributes to reduce cost
and time efforts. In this paper, we have used genetic algorithm to optimize
the test cases that are generated using the category-partition and test
harness patterns. In order to investigate the effectiveness of the approach,
a graph is derived using category-partition and genetic algorithm is used to
optimize the test cases to cover the entire possible path. The optimal test
suites are devised by the method of sampling statistics.
Algorithms
Genetic Algorithm
Optimization Approach
Generation
Issues
We have used genetic algorithm to optimize the test cases that are
generated using the category-partition and test harness patterns. In order
to investigate the effectiveness of the approach, a graph is derived using
category-partition and genetic algorithm is used to optimize the test cases
to cover the entire possible path.
Type of research
Experiment
Any Comparison?
No
Fitness metrics
Path coverage
Factors
effecting
performance
the
--
--
Quality Instrument score
4.5
Reference No.
14
Title
Automated Generation of Test Cases from Output Domain of an
Embedded System using Genetic Algorithms
Authors
Chandra Prakash Vudatha, Dr. Sastry KR Jammalamadaka, Sateesh
Nalliboena, Bala Krishna Kamesh Duvvuri
Published
Abstract
In
3rd International Conference on Electronics Computer Technology (ICECT)
Year
2011
A primary issue in black-box testing is how to generate adequate test cases
from input domain of the system under test on the basis of user’s
requirement specification. However, for some types of systems including
101
embedded systems, developing test cases from output domain is more
suitable than developing from input domain, especially, when the output
domain is smaller. This approach ensures better reliability of the system
under test.
In this paper, the authors present a new approach to automate the
generation of test cases from output domain of a pilot project
‘Temperature Monitoring and Controlling of Nuclear Reactor System”
(TMCNRS) which is an embedded system developed using modified
Cleanroom Software Engineering methodology.
An Automated Test Case Generator (ATCG) that uses Genetic algorithms
(GAs) extensively and generates test cases from output domain is
proposed. The ATCG generates test cases which are useful to conduct
pseudo- exhaustive testing to detect single, double and several multimode
faults in the system. The generator considers most of the combinations of
outputs, and finds the corresponding inputs while optimizing the number
of test cases generated.
In order to investigate the effectiveness of this approach, test cases were
generated by ATCG and the tests were conducted on the target embedded
system at a minimum cost and time. Experimental results show that this
approach is very promising.
Algorithms
Genetic Algorithm
Optimization Approach
Generation
Issues
The authors present a new approach to automate the generation of test
cases from output domain of a pilot project ‘Temperature Monitoring and
Controlling of Nuclear Reactor System” (TMCNRS) which is an embedded
system developed using modified Cleanroom Software Engineering
methodology.
Type of research
Experiment
Any Comparison?
No
Fitness metrics
Requirement coverage
Factors
effecting
performance
the
--
Quality Instrument score
6.5
Reference No.
15
Title
Java based Test case Generation and Optimization Using Evolutionary
Testing
Authors
C.S.Siva Dharsana, Ms.A.Askarunisha, D.Nithila Jennifer, Dr.N.Ramaraj
102
Publish ed
In
International Conference on Computational Intelligence and Multimedia
Applications
Year
2007
Abstract
Testing is an iterative process of both validating functionality and even
more important attempting to break the software Test case is used for
testing which is a set of test data and test programs (test scripts) and their
expected results. This paper details on the tool TCGOJ, Test Case
Generation and Optimization for Java Programs, that generates test cases
for any Java programs and produces an optimized result of test cases.
Genetic Algorithm is used to create and optimize chromosomes which
describe the test cases. Mutation and crossover are done for gaining
maximum coverage.
Algorithms
Genetic Algorithm
Optimization Approach
Generation
Issues
--
Type of research
Experiment
Any Comparison?
No
Fitness metrics
Path coverage
Factors
effecting
performance
the
--
Quality Instrument score
5
Reference No.
16
Title
A Multi-Objective Particle Swarm Optimization for Test Case Selection
Based on Functional Requirements Coverage and Execution Effort
Authors
Luciano S. de Souza, Pericles B. C. de Miranda, Ricardo B. C. Prudencio,
Flavia de A. Barros
Published
Abstract
In
2011 23rd IEEE International Conference on Tools with Artificial
Intelligence
Year
2011
Although software testing is a central task in the software lifecycle, it is
sometimes neglected due to its high costs. Tools to automate the testing
process minor its costs, however they generate large test suites with
redundant Test Cases (TC). Automatic TC Selection aims to reduce a test
suite based on some selection criterion. This process can be treated as an
optimization problem, aiming to find a subset of TCs which optimizes one
or more objective functions (i.e., selection criteria). The majority of search-
103
based works focus on single-objective selection. In this light, we developed
a mechanism for functional TC selection which considers two objectives
simultaneously: maximize requirements’ coverage while minimizing cost in
terms of TC execution effort. This mechanism was implemented as a multiobjective optimization process based on Particle Swarm Optimization
(PSO). We implemented two multi-objective versions of PSO (BMOPSO
and BMOPSO-CDR). The experiments were performed on two real test
suites, revealing very satisfactory results (attesting the feasibility of the
proposed approach). We highlight that execution effort is an important
aspect in the testing process, and it has not been used in a multi-objective
way together with requirements coverage for functional TC selection.
Algorithms
BMOPSO
BMOPSO-CDR
Optimization Approach
Selection
Issues
Search based techniques widely focus on single-objective optimization and
rarely focus on cost of test cases execution
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
Maximum Requirement coverage
BMOPSO-CDR outperform BMOPSO as it covered a
wider region of the search space
Minimum test case running cost
Factors
effecting
performance
the
--
Quality Instrument score
11.5
Reference No.
17
Title
An Enhanced Test Case Selection Approach for Model-Based Testing: An
Industrial Case Study.
Authors
Hadi Hemmati, Lionel Briand, Andrea Arcuri, Shaukat Ali
Published
Abstract
In
18th ACM International Symposium on Foundations of Software
Engineering (FSE)
Year
2010
In recent years, Model-Based Testing (MBT) has attracted an increasingly
wide interest from industry and academia. MBT allows automatic
generation of a large and comprehensive set of test cases from system
models (e.g., state machines), which leads to the systematic testing of the
system. However, even when using simple test strategies, applying MBT in
104
large industrial systems often leads to generating large sets of test cases
that cannot possibly be executed within time and cost constraints. In this
situation, test case selection techniques are employed to select a subset
from the entire test suite such that the selected subset conforms to
available resources while maximizing fault detection. In this paper, we
propose a new similarity-based selection technique for state machinebased test case selection, which includes a new similarity function using
triggers and guards on transitions of state machines and a Genetic
algorithm-based selection algorithm. Applying this technique on an
industrial case study, we show that our proposed approach is more
effective in detecting real faults than existing alternatives. We also assess
the overall benefits of model-based test case selection in our case study by
comparing the fault detection rate of the selected subset with the
maximum possible fault detection rate of the original test suite.
Algorithms
Optimization Approach
 Greedy
 Genetic
Selection
Issues
--
Type of research
Case Study
Any Comparison?
Yes
Fitness metrics
Execution time
Factors
effecting
performance
the
GA is better
Parameter tuning of GA can produce better results
Quality Instrument score
10
Reference No.
18
Title
An Industrial Investigation of Similarity Measures for Model-Based Test
Case Selection
Authors
Hadi Hemmati and Lionel Briand
Published
Abstract
In
IEEE 21st International Symposium on Software Reliability Engineering
Year
2010
Applying model-based testing (MBT) in practice requires practical solutions
for scaling up to large industrial systems. One challenge that we have
faced while applying MBT was the generation of test suites that were too
large to be practical, even for simple coverage criteria. The goal of test
case selection techniques is to select a subset of the generated test suite
that satisfies resource constraints while yielding a maximum fault
detection rate. One interesting heuristic is to choose the most diverse test
cases based on a pre-defined similarity measure. In this paper, we
105
investigate and compare possible similarity functions to support similaritybased test selection in the context of state machine testing, which is the
most common form of MBT. We apply the proposed similarity measures
and a selection strategy based on genetic algorithms to an industrial
software system. We compare their fault detection rate based on actual
faults. The results show that applying Jaccard Index on test cases
represented as a set of trigger-guards is the most cost-effective similarity
measure. We also discuss the overall benefits of our test selection
approach in terms of test execution savings.
 random selection (Rnd),
 coverage-based Greedy selection (CovGr),
 coverage-based GA selection (CovGA),
 the state of the art in STCS (TransGr).
Selection
Algorithms
Optimization Approach
Issues
One challenge that we have faced while applying MBT was the generation
of test suites that were too large to be practical, even for simple coverage
criteria.
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
Average fault detection rate (AFDR)
Factors
effecting
performance
the
GA with jac index perform best
GA can perform best in coverage when longer stopping time (1 sec instead
of 175ms)
Quality Instrument score
11.5
Reference No.
19
Title
The Research of Test-Suite Reduction Technique
Authors
Cui Donghua, Yin Wenjie
Published
Abstract
In
IEEE
Year
2011
Ant colony algorithm is a bionic optimization algorithm, it can solve
combinatorial problems effectively. For the problem of the test suite
reduction, this algorithm could find the balance point between the speed
and the accuracy of solution. Unlike other existing algorithms, this
algorithm used test cost criteria, as well as the test coverage criteria.
Finally, the paper presented the results, the results is given by the others
classical algorithms compared with this algorithms. The results show that
this algorithm can significantly reduce the size and the cost of the testsuite, and achieved higher effectiveness of test-suite minimization.
106
Algorithms
 Ant colony algorithm
 Greedy algorithm,
 Heuristic algorithm,
 GRE algorithm
Reduction
Optimization Approach
Issues
This article will research how to generate the lowest cost of test suite and
can meet all needs.
Type of research
Experiment
Any Comparison?
Ant colony algorithm can get optimal solution.
Fitness metrics
Requirement coverage
Factors
effecting
performance
the
--
Quality Instrument score
8
Reference No.
20
Title
A Novel Chaos Discrete Particle Swarm Optimization Algorithm for Test
Suite Reduction
Authors
Jia-ze Sun, Shu-yan Wang
Published
In
IEEE
Year
2010
Abstract
Aiming at providing the more measure of coverage with the least number
of test cases for software regression testing, this paper presents a novel
chaos discrete particle swarm optimization algorithm(CDPSO) for test suite
reduction, which combines discrete particle swarm optimization (DPSO)
with a chaos searching strategy. In the algorithm, particle swarm is
initialized by chaotic series, and the position of the particle is produced by
stochastic algorithm. Moreover, it introduces chaos to DPSO, making every
particle select a suitable search direction from PSO search mechanism and
chaos search mechanism, to avoid PSO getting into local best and
appearing premature convergence. Finally, the classic example is used to
illustrate the performance of the proposed algorithm. The experimental
results indicate that the CDPSO algorithm can achieve higher performance,
faster speed than PSO, GE, H and GRE, and has nothing to do with the
initial value.
Algorithms
 chaos discrete particle swarm optimization algorithm(CDPSO)
 discrete particle swarm optimization (DPSO)
 GE
 H
 GRE
Reduction
Optimization Approach
107
Issues
A novel chaotic discrete particle swarm optimization algorithm (CDPSO) is
proposed, which can avoid slow search speed and premature convergence.
Type of research
Experiment
Any Comparison?
Yes
 The convergence speed of CDPSO is faster as attain full
coverage and the lowest redundancy, so it is stable and best.
 GE, H, GRE algorithms; in different initial conditions may
produce different results. Whereas DPSO has nothing to do
with initial values, so it is stable.
Requirement Coverage
Fitness metrics
Factors
effecting
performance
the
Initial set
Quality Instrument score
11
Reference No.
21
Title
A Technique to Reduce the Test Case Suites for Regression Testing Based
on a Self-Organizing Neural Network Architecture
Authors
Adenilso da Silva Simao, Rodrigo Fernandes de Mello, Luciano Jose Senger
Published
In
Proceedings of the 30th Annual International Computer Software and
Applications Conference (COMPSAC'06)
Year
2006
Abstract
This paper presents a technique to select subsets of the test cases,
reducing the time consumed during the evaluation of a new software
version and maintaining the ability to detect defects introduced. Our
technique is based on a model to classify test case suites by using an ART2. Self-organizing neural network architecture. Each test case is
summarized in a feature vector, which contains all the relevant
information about the software behavior. The neural network classifies
feature vectors into clusters, which are labeled according to software
behavior. The source code of a new software version is analyzed to
determine the most adequate clusters from which the test case subset will
be selected. Experiments compared feature vectors obtained from all-uses
code coverage information to a random selection approach. Results
confirm the new technique has improved the precision and recall metrics
adopted.
Algorithms
 ART (Adaptive Resonance Theory) neural network with Clustering
 Random Algorithm
Reduction
Optimization Approach
Issues
The mean precision of the new model is higher than random, what implies
in more quality to select test cases.
Type of research
Experiment
108
Any Comparison?
Yes
Fitness metrics
all-uses coverage
Factors
effecting
performance
the
Proposed method is better than random
--
Quality Instrument score
8.5
Reference No.
22
Title
Test Case Reduction Based on Program Invariant and Genetic Algorithm
Authors
Nenggang Pan, FanPing Zeng, Yu-Han Huang
Published
In
IEEE
Year
2010
Abstract
Test case reduction is the focus of the field of software testing. In this
paper, we integrate random testing methods, invariant technology and
genetic algorithms for test case reduction. Program invariant here refers
to the properties at some points of program, it can reveal the extent of
program data coverage and other valuable information. We evaluate the
parent program invariants to filter corresponding set of test cases, and
keep the better ones on the offspring. Generation by generation, we can
get the final reduced set of test cases. This method is easy to automate.
Experimental results and analysis show that compared to similar methods,
at the same coverage conditions, the method proposed in this paper have
higher efficiency and can get better set of test cases.
Algorithms
Optimization Approach
 Random
 Genetic
Reduction
Issues
Random test has some drawbacks
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
Branch coverage
Factors
effecting
performance
the
Proposed optimization is more affective and efficient
--
Quality Instrument score
8.5
Reference No.
23
109
Title
An Evolutionary Algorithm for Regression Test Suite Reduction
Authors
S.Nachiyappan, A.Vimaladevi, C.B.SelvaLakshmi
Published
In
Proceedings of the International Conference on Communication and
Computational Intelligence – 2010
Year
2010
Abstract
As the software is modified and new test cases are added to the test-suite,
the size of the test-suite grows and the cost of regression testing
increases. In order to decrease the cost of regression testing, researchers
have focused on the use of test-suite reduction techniques, which identify
a subset of test cases that provides the same coverage of the software,
according to some criterion, as the original test-suite. This paper
investigates the use of an evolutionary approach, called genetic
algorithms, for test-suite reduction. The proposed model builds the initial
population based on test history, calculates the fitness value using
coverage and run time of test case, and then selectively breeds the
successive generations using genetic operations and allows only the fit
tests to the reduced suite. This generational process is repeated until an
optimized test-suite is found.
Algorithms
Genetic Algorithm
Optimization Approach
Reduction
Issues
A mathematical model to reduce the cost of regression testing by
minimizing the size of the test-suite using an evolutionary algorithm called
Genetic Algorithm. Generally a limited time is allotted for testing.
Type of research
Experiment
Any Comparison?
No
Fitness metrics
Time and block coverage
Factors
effecting
performance
the
Quality Instrument score
--
6.5
Reference No.
24
Title
Automated software test optimization framework – an artificial bee colony
optimization-based approach
Authors
D. Jeya Mala, V. Mohan, M. Kamalapriya
Published
In
IET Softw., 2010, Vol. 4, Iss. 5, pp. 334–348
Year
2010
110
Abstract
Software test suite optimization is one of the most important problems in
software engineering research. To achieve this optimization, a novel approach
based on artificial bee colony (ABC) optimization is proposed here. The work
applied in this approach is motivated by the intelligent behavior of honey
bees. Since the ABC system combines local search methods carried out by
employed and onlooker bees with global search methods managed by scouts,
the approach attains global or near-global optima. Here, the parallel behavior
of the three bees is used to reach the solution generation faster. The
performance of the proposed approach is investigated based on coveragebased test adequacy criteria by comparing it with sequential ABC, random
testing and genetic algorithm-based approaches. Based on the experimental
results, it has been proved that the proposed parallel ABC approach
outperforms the other approaches in test suite optimization.
Algorithms
 Sequential ABC
 Parallel ABC
 Random testing
 Genetic Algorithm
Generation
Optimization Approach
Issues
The objective of the proposed approach is to generate an efficient test suite
that can cover the SUT with less time and cost by applying intelligent search
through the SUT using the parallel behavior of a group of three bees.
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
Factors effecting the
performance
Parallel ABC is higher when compared to the other three
approaches
 In some cases, random testing performs better than GA.
 In some other cases, GA performs better than sequential ABC
and random testing.
 The number of test cases needed to test the SUT with 100%
coverage is very less in parallel ABC.
 In case of GA, for some problems, even after the number of
generations is increased, the path coverage% reaches only 50%
due to strike up at local optimal solution.
 Even after increasing the number of runs/generations, random
testing and GA-based approaches achieved only 87.5 and
57.14% of coverage, respectively.
 In Parallel ABC: efficiency of the test cases in terms of their path
coverage measure is steadily improving
 Whereas in GA, the test cases improvement is non-linear and
usually strikes up at local optima.
 Parallel ABC-based approach takes less number of cycles to
achieve optimal or near-optimal solutions due to the parallel
behavior of the three agents.
 Parallel ABC takes least time in finishing the optimization task,
where as GA takes even more time than random due to nonmemorization of best individual in each iteration
Path Coverage

Parallel working and sequential working in local and global optimization.
GA strikes up at local optima. GA has non-memorization of best individuals in
111
each iteration.
Quality
score
Instrument
13
Reference No.
25
Title
Evolutionary Testing of Object-Oriented Software
Authors
Lucas Serpa Silva, Maarten van Someren
Published
In
SAC’10, March 22-26, 2010, Sierre, Switzerland
Year
2010
Abstract
It is estimated that 80% of software development cost is spent on detecting
and fixing defects. To tackle this issue, a number of tools and testing
techniques have been developed to improve the existing testing framework.
Although techniques such as static analysis, random testing and evolutionary
testing have been used to automate the testing process, it is not clear what is
the best approach. Previous research on evolutionary testing has mainly
focused on procedural programming languages with code coverage as the
main optimization parameter. In this work, we present an approach that
combines a genetic algorithm with static analysis to automatically test Eiffel
classes using the number of faults found as the optimization parameter. A
total of 115 experiments on 22 Eiffel classes were executed to evaluate the
effectiveness of evolutionary testing compared to the results obtained by
running a random test case generator for 15, 30 and 60 minutes. The results
show a genetic algorithm combined with static analysis can considerably
increase the number of faults found compared to a random testing approach.
In some cases, evolutionary testing found more faults in 15 minutes than a
random testing strategy found in 60 minutes.
Algorithms
Optimization Approach
 Random
 Random with Static Analysis
 Genetic Algorithm
Generation
Issues
Evolutionary testing finds more faults and in less time than random testing
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
Fault detection rate
Factors effecting the
performance
In GA time is an important element in effecting the performance
Quality
10
Instrument
Random perform worst, where as evolutionary algorithm (GA)
performs 25% better results
112
score
Reference No.
26
Title
An Optimization Strategy for Evolutionary Testing Based on Cataclysm
Authors
Meng Wang, Bixin Li, Zhengshan Wang, Xxiaoyuan Xie
Published
th
In
34 Annual IEEE Computer Software and Applications Conference Workshops
Year
2010
Abstract
Evolutionary Testing (ET) is an effective test case generation technique which
uses some meta-heuristic search algorithm, especially genetic algorithm, to
generate test cases automatically. However, the prematurity of the population
may decrease the performance of ET. To solve this problem, this paper
presents a novel optimization strategy based on cataclysm. It monitors the
diversity of population during the evolution process of ET. Once the
prematurity is detected, it will use the operator, cataclysm, to recover the
diversity of the population. The experimental results show that the proposed
strategy can improve the performance of ET evidently.
Algorithms
Optimization Approach
Evolutionary Algorithm with Optimization Strategy Based on
Cataclysm(OSBC)
 Evolutionary Algorithm with Dynamic Optimization of the
possibility(DOMP)
Generation
Issues
The prematurity of the population may decrease the performance of ET.
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
Branch Coverage
Factors effecting the
performance
Prematurity of Population
Quality
score
8.5

Instrument
OSBC works better than DOMP, but not work well for Complicated
Branch Conditions.
Reference No.
27
Title
Test Case Evaluation and Input Domain Reduction strategies for the
Evolutionary Testing of Object-Oriented software
Authors
José Carlos Bregieiro Ribeiro, Mário Alberto Zenha-Rela, Francisco Fernández
de Vega
113
Published
In
Information and Software Technology 51 (2009) 1534–1548
Year
2009
Abstract
In Evolutionary Testing, meta-heuristic search techniques are used for
generating test data. The focus of our research is on employing evolutionary
algorithms for the structural unit-testing of Object-Oriented programs.
Relevant contributions include the introduction of novel methodologies for
automation, search guidance and Input Domain Reduction; the strategies
proposed were empirically evaluated with encouraging results.
Test cases are evolved using the Strongly-Typed Genetic Programming
technique. Test data quality evaluation includes instrumenting the test object,
executing it with the generated test cases, and tracing the structures traversed
in order to derive coverage metrics. The methodology for efficiently guiding
the search process towards achieving full structural coverage involves
favouring test cases that exercise problematic structures. Purity Analysis is
employed as a systematic strategy for reducing the search space.
Algorithms
Strongly-Typed Genetic Programming
Optimization Approach
Generation
Issues
The focus of research is on employing evolutionary algorithms for the
structural unit-testing of object-oriented programs. Relevant contributions
include the introduction of novel methodologies for automation, search
guidance and input domain reduction.
Type of research
Experiment
Any Comparison?
No
Fitness metrics
Structural coverage
Factors effecting the
performance
--
Quality
score
6
Instrument
Reference No.
28
Title
An experimental study of four typical test suite reduction techniques
Authors
Hao Zhong, Lu Zhang, Hong Mei
Published
Abstract
In
Information and Software Technology, vol-50, pg 534-546.
Year
2007
In software development, developers often rely on testing to reveal bugs.
Typically, a test suite should be prepared before initial testing, and new test
114
cases may be added to the test suite during the whole testing process. This
may usually cause the test suite to contain more or less redundancy. In other
words, a subset of the test suite (called the representative set) may still satisfy
all the test objectives. As the redundancy can increase the cost of executing
the test suite, quite a few test suite reduction techniques have been brought
out in spite of the NP-completeness of the general problem of finding the
optimal representative set of a test suite. In the literature, there have been
some experimental studies of test suite reduction techniques, but the
limitations of these experimental studies are quite obvious. Recently proposed
techniques are not experimentally compared against each other, and reported
experiments are mainly based on small programs or even simulation data. This
paper presents a new experimental study of the four typical test suite
reduction techniques, including Harrold et al.’s heuristic, and three other
recently proposed techniques such as Chen and Lau’s GRE heuristic, Mansour
and El-Fakin’s genetic algorithm-based approach, and Black et al.’s ILP-based
approach. Based on the results of this experimental study, we also provide a
guideline for choosing the appropriate test suite reduction technique and
some insights into why the techniques vary in effectiveness and efficiency.
Algorithms
GRE
H
ILP
GA
Optimization Approach
Reduction
Issues
Recently proposed techniques are not experimentally compared against each
other, and reported experiments are mainly based on small programs or even
simulation data.
Type of research
Experiment
Any Comparison?
Yes



Fitness metrics
Fault detection
Factors effecting the
performance
Time
Quality
score
12
Instrument
Reference No.
Genetic algorithm-based approach, which performs much
weaker in this aspect, the other three can produce
representative sets of almost the same sizes.
Genetic algorithm-based approach also performs the worst in
respect of execution time.
The ILP-based approach can always produce the smallest
representative sets among all the four approaches.
29
115
Title
Strongly-Typed Genetic Programming and Purity Analysis: Input Domain
Reduction for Evolutionary Testing Problems
Authors
José Carlos, Mário Alberto, Zenha-Rela, Bregieiro Ribeiro, Francisco Fernandéz
de Vega
Published
In
GECCO’08, July 12–16, 2008, Atlanta, Georgia, USA.
Year
2008
Abstract
Search-based test case generation for object-oriented software is hindered by
the size of the search space, which encompasses the arguments to the implicit
and explicit parameters of the test object's public methods. The performance
of this type of search problems can be enhanced by the definition of adequate
Input Domain Reduction strategies.
The focus of our on-going work is on employing evolutionary algorithms for
generating test data for the structural unit-testing of Java programs. Test cases
are represented and evolved using the Strongly-Typed Genetic Programming
paradigm; Purity Analysis is particularly useful in this situation because it
provides a means to automatically identify and remove Function Set entries
that do not contribute to the definition of interesting test scenarios.
Algorithms
Strongly-Typed Genetic Programming
Optimization Approach
Generation
Issues
Search-based test case generation for object-oriented software is hindered by
the size of the search space, which encompasses the arguments to the implicit
and explicit parameters of the test object's public methods.
Type of research
Experiment
Any Comparison?
No
Fitness metrics
Function Coverage
Factors effecting the
performance
--
Quality
score
4
Instrument
The average percentage of test cases that accomplished full
coverage increased approximately 6%.
Reference No.
30
Title
Search-Based Test Case Generation for Object-Oriented Java Software Using
Strongly-Typed Genetic Programming
Authors
José Carlos Bregieiro Ribeiro
Published
In
GECCO’08, July 12–16, 2008, Atlanta, Georgia, USA.
116
Year
Abstract
2008
In evolutionary testing, meta-heuristic search techniques are used to generate
high-quality test data. The focus of our ongoing work is on employing
evolutionary algorithms for the structural unit-testing of object-oriented Java
programs.
Test cases are evolved using the Strongly-Typed Genetic Programming
technique. Test data quality evaluation includes instrumenting the test object,
executing it with the generated test cases, and tracing the structures traversed
in order to derive coverage metrics. The strategy for efficiently guiding the
search process towards achieving full structural coverage involves favoring
test cases that exercise problematic structures and control-flow paths. Static
analysis and instrumentation is performed solely with basis on the information
extracted from the test objects' Java Bytecode.
Relevant contributions include the introduction of novel methodologies for
automation, search guidance and input domain reduction, and the
presentation of the eCrash automated test case generation tool.
Algorithms
Strongly-Typed Genetic Programming
Optimization Approach
Generation
Issues
The focus of our ongoing work is on employing evolutionary algorithms for the
structural unit-testing of object-oriented Java programs.
Type of research
Experiment
Any Comparison?
No
The impact of this reduction on the test case generation process is
visible on the case studies.
The average percentage of test cases that accomplished full coverage
within a maximum of 100 generations increased approximately 6%.
Fitness metrics
Structural coverage
Factors effecting the
performance
Configuration of parameters
Quality
score
5.5
Instrument
Reference No.
31
Title
An Empirical Evaluation of Test Suite Reduction for Boolean Specificationbased Testing
Authors
Xiaofang Zhang, Baowen Xu, Zhenyu Chen, Changhai Nie, Leifang Li
Published
In
The Eighth International Conference on Quality Software
117
Year
2008
Abstract
As the cost of executing and maintaining a large test suite is always expensive,
many test suite reduction techniques have been proposed and some empirical
studies of test suite reduction techniques have already been reported. The
aim of this paper is to provide a guideline for choosing the appropriate test
suite reduction techniques for Boolean specification-based testing. Four
typical heuristic reduction strategies: G, GE, GRE, and H are introduced to be
compared empirically. Furthermore, testing requirement optimization is
combined to enhance these four reduction strategies. This paper presents the
empirical evaluation of these reduction strategies based on a set of Boolean
specifications from TCAS II.
Algorithms
G (greedy) and G’
GE and GE’
GRE and GRE’
H and H’
Optimization Approach
Reduction
Issues
The aim of this paper is to provide a guideline for choosing the appropriate
test suite reduction techniques for Boolean specification-based testing.
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
 1-1 reduction of requirement could always improve the four
reduction strategies both in reduction effectiveness and reduction
performance. Hence, it is recommended to be implemented before
all test suite reduction strategies, even before test case generation.
But increases the time cost.
 Since GRE’ is a combined strategy, it is more complex than other
reduction strategies. It is suitable for small sized and small
overlapped test suites. Hence, GRE’ is recommended to obtain a
minimum size of test suite, if the time resource is enough.
 H’ groups the test cases into various classes and then reduces test
cases. Hence it can reduce the cost especially for larger test suites
with acceptable impact on the size of representative set. H’ is
recommended to obtain a better reduction performance, if the
time resource is limited.
Requirement Coverage
Time cost
Factors effecting the
performance
Time
Quality
score
11.5
Instrument
118
Reference No.
32
Title
Test-Suite Reduction Using Genetic Algorithm
Authors
Xue-ying Ma, Bin-kui Sheng, and Cheng-qing Ye
Published
In
Springer-Verlag Berlin Heidelberg 2005
Year
2005
Abstract
As the software is modified and new test cases are added to the test-suite, the
size of the test-suite grows and the cost of regression testing increases. In
order to decrease the cost of regression testing, researchers have researched
on the use of test-suite reduction techniques, which identify a subset of test
cases that provides the same coverage of the software, according to some
criterion, as the original test-suite. This paper investigates the use of an
evolutionary approach, called genetic algorithms, for test-suite reduction. The
algorithm builds the initial population based on test history, calculates the
fitness value using coverage and cost information, and then selectively breeds
the successive generations using genetic operations. This generational process
is repeated until a minimized test-suite is founded. The results of studies show
that, genetic algorithms can significantly reduce the size of the test-suite and
the cost of regression testing, and achieves good cost-effectiveness.
Algorithms
Genetic Algorithm (GeA)
Greedy Algorithm (ADD-DELTA)
Optimization Approach
Reduction
Issues
This paper investigates the use of an evolutionary approach, called genetic
algorithms, for test-suite reduction
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
 The sizes of reduced test-suites achieved by GeA are slightly smaller
than those reduced by ADD-DELTA.
 The percentages of cost-reduction achieved by GeA are higher than
those achieved by ADD-DELTA
 GeA achieves a higher cost-effectiveness (in terms of time to
perform reduction) than ADD-DELTA does.
Block Coverage and cost
Factors effecting the
performance
Parameter setting
Quality
score
13
Instrument
Reference No.
33
Title
Black-box System Testing of Real-Time Embedded Systems Using Random and
119
Search-based Testing
Authors
Published
Andrea Arcuri, Muhammad Zohaib Iqbal, and Lionel Briand
In
Testing Software and Systems, Springer Berlin/Heidelberg. 6435:95-110
Year
2010
Abstract
Testing real-time embedded systems (RTES) is in many ways challenging.
Thousands of test cases can be potentially executed on an industrial RTES.
Given the magnitude of testing at the system level, only a fully automated
approach can really scale up to test industrial RTES. In this paper we take a
black-box approach and model the RTES environment using the UML/-MARTE
international standard. Our main motivation is to provide a more practical
approach to the model-based testing of RTES by allowing system testers, who
are often not familiar with the system design but know the application domain
well-enough, to model the environment to enable test automation.
Environment models can support the automation of three tasks: the code
generation of an environment simulator, the selection of test cases, and the
evaluation of their expected results (oracles). In this paper, we focus on the
second task (test case selection) and investigate three test automation
strategies using inputs from UML/MARTE environment models: Random
Testing (baseline), Adaptive Random Testing, and Search-Based Testing (using
Genetic Algorithms). Based on one industrial case study and three artificial
systems, we show how, in general, no technique is better than the others.
Which test selection technique to use is determined by the failure rate (testing
stage) and the execution time of test cases. Finally, we propose a practical
process to combine the use of all three test strategies.
Algorithms
Random testing (RT)
Adaptive Random Testing (ART)
Genetic Algorithm (as search based testing (SBT))
Selection
Optimization Approach
Generation
Issues
our goal is to make such environment modeling as easy as possible, and allow
the testers to automate testing without any knowledge about the design of
the RTES.
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
Failure rate and execution time
Factors effecting the
performance
One main advantage of ART and SBT is that it can be tailored to whatever time
and resources are available for testing: when resources are expended and
time is up, we can simply stop their application without any side effect.
We can see that no testing technique generally dominates the others.
GA gets to the error state 90% of the time, whereas RT reaches it only in 6% of
the time, i.e., GA is more time consuming.
120
Quality
score
Instrument
11.5
Reference No.
34
Title
Multi-Objective Test Suite Minimization Using Quantum-Inspired MultiObjective Differential Evolution Algorithm
Authors
A.Charan Kumari, K.Srinivas, M.P.Gupta
Published
In
IEEE
Year
2012
Abstract
This paper presents the solution for multi-objective test suite minimization
problem using Quantum-inspired Multi-objective differential Evolution
Algorithm. Multi-objective test suite minimization problem is to select a set of
test cases from the available test suite while optimizing the multi objectives
like code coverage, cost and fault history. As test suite minimization problem
is an instance of minimal hitting set problem which is NP-complete; it cannot
be solved efficiently using traditional optimization techniques especially for
the large problem instances. This paper presents Quantum-inspired Multiobjective Differential Evolution Algorithm (QMDEA) for the solution of multiobjective test suite minimization problem. QMDEA combines the preeminent
features of Differential Evolution and Quantum Computing. The features of
QMDEA help in achieving quality Pareto-optimal front solutions with faster
convergence. The performance of QMDEA is tested on two real world
applications and the results are compared against the state-of-the-art multiobjective evolutionary algorithm NSGA-II. The comparison of the obtained
results indicates superior performance of QMDEA.
Algorithms
 Quantum-inspired Multi-objective differential Evolution Algorithm.
 NSGA-II
Selection
Optimization Approach
Issues
This paper presents the solution for multi-objective test suite minimization
problem using Quantum-inspired Multi-objective differential Evolution
Algorithm.
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
 QMDEA out performed NSGA II in both the applications in
achieving high quality solutions and also QMDEA is able to
obtain more number of solutions on its Pareto front than NSGA
II, giving a good number of choices to the decision maker.
 QMDEA is able to converge faster than NSGA II.
Code Coverage
Execution time
Factors effecting the
--
121
performance
Quality
score
Instrument
11
Reference No.
35
Title
A new heuristic for test suite reduction
Authors
T.Y. Chen, M.F. Lau
Published
In
Information and Software Technology 40 (1998) 347-354
Year
1998
Abstract
A testing objective has to be defined in testing a program. A test suite is then
constructed to satisfy the testing objective. The constructed test suite contains
redundancy when some of its proper subsets can still satisfy the same testing
objective. Since the costs of executing test cases and maintaining a test suite
for regression testing may be expensive, the problem of test suite reduction
arises. This paper proposes a heuristic towards the optimization of a test suite.
Algorithms
GE
GRE
H
Optimization Approach
Reduction
Issues
Since the costs of executing test cases and maintaining a test suite for
regression testing may be expensive, the problem of test suite reduction
arises. This paper proposes a heuristic towards the optimization of a test suite.
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
 all three heuristics GE, GRE and H deliver representative sets of
nearly the same size when the ratio of overlapping is small
 when medium the representative sets found by GRE is smaller
than those of GE and H
 when large the performance of the heuristics GE and GRE are
nearly the same, and the size of the representative sets produced
by either GE or GRE is less than that produced by heuristic H.
Branch coverage
Factors effecting the
performance
--
Quality
score
9.5
Instrument
122
Reference No.
36
Title
Search Algorithms for Regression Test Case Prioritization
Authors
Zheng Li, Mark Harman, and Robert M. Hierons
Published
In
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 33, NO. 4,
Year
2007
Abstract
Regression testing is an expensive, but important, process. Unfortunately,
there may be insufficient resources to allow for the re-execution of all test
cases during regression testing. In this situation, test case prioritization
techniques aim to improve the effectiveness of regression testing by ordering
the test cases so that the most beneficial are executed first. Previous work on
regression test case prioritization has focused on Greedy Algorithms.
However, it is known that these algorithms may produce suboptimal results
because they may construct results that denote only local minima within the
search space. By contrast, metaheuristic and evolutionary search algorithms
aim to avoid such problems. This paper presents results from an empirical
study of the application of several greedy, metaheuristic, and evolutionary
search algorithms to six programs, ranging from 374 to 11,148 lines of code
for three choices of fitness metric. The paper addresses the problems of
choice of fitness metric, characterization of landscape modality, and
determination of the most suitable search technique to apply. The empirical
results replicate previous results concerning Greedy Algorithms. They shed
light on the nature of the regression testing search space, indicating that it is
multimodal. The results also show that Genetic Algorithms perform well,
although Greedy approaches are surprisingly effective, given the multimodal
nature of the landscape.
Algorithms
 Greedy
 Additional greedy
 2-optimal greedy
 Hill climbing
 Genetic
Prioritization
Optimization Approach
Issues
Q1. Which algorithm is most effective in solving the test case prioritization
problem for regression testing?
Q2. What factors could affect the efficiency of algorithms for the test case
prioritization problem for regression testing?
Type of research
Experiment
Any Comparison?
Yes




For small programs, the five algos show almost identical
performance
For small programs, The mean fitness value for each program
revealed that the greedy is worst and genetic is slightly better
then others
ANOVA results for small programs and small test suite are
consistent (same)
To explore further that where difference lie, a multi-comparison
123
procedure (LSD) is performed. There is significant difference
between all pairs of algos on small programs, except Genetic
and additional greedy, means provides best results.
 For large program and small test suite by boxplots: additional
greedy and 2-optimal show best results then hill climbing and
genetic, where as genetic is better then hill climbing.
 For large test suite there is no significant difference between
additional greedy and genetic for small programs and no
significant difference in additional greed and 2-optimal for large
program. And pair wise comparisons have significant
differences.
 When perform repeatedly(100 times) these algorithms with
same test suite, greedy, additional greedy and 2-optimal show
same results means these are deterministic, while by hill
climbing and genetic have vary results with each experiment.
Code coverage, including block coverage, decision (branch) coverage, and
statement coverage
Fitness metrics
Factors effecting the
performance
The size of the program does not directly affect the complexity of test case
prioritization, whereas the size of the test suite does since it determines the
size of the search space.
Quality
score
14
Instrument
Reference No.
37
Title
A Genetic Algorithm for Test-Suite Reduction
Authors
Xue-ying MA, Bin-kui SHENG, Zhen-feng HE, Cheng-qing YE
Published
Abstract
In
2005 IEEE International Conference on Systems, Man and Cybernetics
Year
2005
In order to reduce the cost of regression testing, researchers have investigated
the use of test-suite reduction techniques, which identify a reduced test suite
that provides the same coverage of the software according to some criterion
as the original test suite. Existing test-suite reduction techniques consider testcase coverage criteria, other criteria such as risk or fault–detection
effectiveness, or combinations of these criteria, but ignore the test-execution
cost because of the complexity. Firstly, this paper presents a mathematical
model for this test-suite reduction problem and transforms it into a linear
integer-programming form. Then the paper investigates the use of an
evolutionary approach, called genetic algorithm, for this test-suite reduction
problem. Unlike other algorithms, our algorithm uses a new criteria, which is a
combination of a block based coverage criteria and a test-execution cost
criteria, to make decisions about reducing a test suite.
Finally, the paper presents the results of the empirical studies of our
algorithm. The studies show that our algorithm can significantly reduce the
size and the cost of the test-suite for regression testing, and the test-execution
124
cost is one of the most important features that must be taken into
consideration for test-suite reduction.
Algorithms
Optimization Approach
 Genetic
 Simple genetic
 Greedy (MgrA)
Reduction
Issues
Consider test execution cost
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
Sizes of reduced test-suites achieved by GeA are significantly
smaller than those reduced by SGeA, and the sizes of reduced
test-suites achieved by MGrA are slightly larger than those
achieved by GeA.
 Percentages of cost-reduction achieved by GeA are significantly
higher than those achieved by SgeA. The percentage reductions
in cost achieved by GeA are higher than those achieved by MGrA.
GeA is much more effective in cost reduction than SgeA, and
achieves higher cost effectiveness than MgrA.
block based coverage criteria and a test-execution cost criteria,
Factors effecting the
performance
--
Quality
score
10.5
Instrument

Reference No.
38
Title
Automatic test case optimization using a bacteriological adaptation model:
application to .NET components
Authors
Benoit Baudry, Franck Fleurey, Jean-Marc Jezequel and Yves LeTraon
Published
Abstract
In
17th IEEE International Conference on Automated Software Engineering, 2002.
Proceedings. ASE 2002
Year
2002
In this paper, we present several complementary computational intelligence
techniques that we explored in the field of .Net component testing. Mutation
testing serves as the common backbone for applying classical and new
artificial intelligence (AI) algorithms. With mutation tools, we know how to
estimate the revealing power of test cases. With AI, we aim at improving
automatically test cases efficiency. So, we looked first at genetic algorithms
(GA) to solve the problem of test. The aim of the selection process is to
generate test cases able to kill as many mutants as possible. Then, we propose
a new AI algorithm that fits better to the test optimization problem we called
bacteriological algorithm (BA): Bas behave better that GAs for this problem.
However, between GAs and BAs, a family of intermediate algorithms exists:
125
we explore the whole spectrum of these intermediate algorithms to
determine whether an algorithm exists that would be more efficient than
BAs.: the approaches are compared on a .Net system.
Algorithms
Optimization Approach
 Genetic
 Bacteriological
Generation
Issues
Test case optimization is more an adaptation process than an evolution
process, we propose bacteriological algorithms, taken from the bacteria
adaptation phenomenon.
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
--
Factors effecting the
performance
Quality
score
Instrument
BA is far speedy then GA in convergence
memorization of best test case
10.5
Reference No.
39
Title
A Case Study on the Use of Genetic Algorithms to Generate Test Cases for
Temporal Systems
Authors
Karnig Derderian, Mercedes G. Merayo, Robert M. Hierons, and Manuel Nunez
Published
In
IWANN 2011, Part II, LNCS 6692, pp. 396–403
Year
2011
Abstract
Generating test data for formal state based specifications is computationally
expensive. In previous work we presented a framework that addressed this
issue by representing the test data generation problem as an optimisation
problem. In this paper we analyze a communications protocol to illustrate how
the test case generation problem can be presented as a search problem and
automated. Genetic algorithms (GAs) and random search are used to generate
test data and evaluate the approach. GAs show to outperform random search
and seem to scale well as the problem size increases. We consider a very
simple fitness function that can be used with other evolutionary search
techniques and automated test case generation suites.
Algorithms
Optimization Approach
 Greedy
 Genetic Single point crossover/mutation
 Genetic Multi point crossover/mutation
Generation
Issues
Generating timed feasible transition paths
126
Type of research
Case Study
Any Comparison?
Yes
Fitness metrics
State coverage
Factors effecting the
performance
Parameter setting
Quality
score
8.5
Instrument
GA1 is constant and perform well then other 2 whereas GA2 is
faster among them
Reference No.
40
Title
A non-pheromone based intelligent swarm optimization technique in software
test suite optimization
Authors
D.Jeya Mala, M.Kamalapriya, R.Shobana, V.Mohan
Published
In
International Conference on Intelligent Agent & Multi-Agent Systems, 2009.
IAMA 2009.
Year
2009
Abstract
In our paper, we applied a non-pheromone based intelligent swarm
optimization technique namely artificial bee colony optimization (ABC) for test
suite optimization. Our approach is a population based algorithm, in which
each test case represents a possible solution in the optimization problem and
happiness value which is a heuristic introduced to each test case corresponds
to the quality or fitness of the associated solution. The functionalities of three
groups of bees are extended to three agents namely Search Agent, Selector
Agent and Optimizer Agent to select efficient test cases among near infinite
number of test cases. Because of the parallel behavior of these agents, the
solution generation becomes faster and makes the approach an efficient one.
Since, the test adequacy criterion we used is path coverage; the quality of the
test cases is improved during each iteration to cover the paths in the software.
Finally, we compared our approach with Ant Colony Optimization (ACO), a
pheromone based optimization technique in test suite optimization and
finalized that, ABC based approach has several advantages over ACO based
optimization.
Algorithms
 Artificial Bee Colony Optimization (ABC)
 Ant Colony Optimization (ACO)
Reduction
Optimization Approach
Generation
Issues
Effectiveness in terms of time and cost
Type of research
Experiment
127
Any Comparison?
Yes
Fitness metrics
Path coverage
Factors effecting the
performance
ACO




Quality
score
Instrument
ABC is faster and effective then ACO
Process is sequential
Solution selection is done only at
the end, leads to computational
overhead and memory limit
problems
Computational overhead
May face memory limit issue
ABC
Parallel
behavior
(multi
threading)
 The final optimal solution is the
improvement done during each
iteration of the bees' exploration
process.
 Less computational overhead
 Overcome memory limit issue

13
Reference No.
41
Title
A concept analysis inspired greedy algorithm for test suite minimization
Authors
Sriraman Tallam, Neelam Gupta
Published
In
Proceedings of the 6th ACM SIGPLAN-SIGSOFT workshop on Program analysis
for software tools and engineering
Year
2005
Abstract
Software testing and retesting occurs continuously during the software
development lifecycle to detect errors as early as possible and to ensure that
changes to existing software do not break the software. Test suites once
developed are reused and updated frequently as the software evolves. As a
result, some test cases in the test suite may become redundant as the
software is modified over time since the requirements covered by them are
also covered by other test cases. Due to the resource and time constraints for
re-executing large test suites, it is important to develop techniques to
minimize available test suites by removing redundant test cases. In general,
the test suite minimization problem is NP complete. In this paper, we present
a new greedy heuristic algorithm for selecting a minimal subset of a test suite
T that covers all the requirements covered by T. We show how our algorithm
was inspired by the concept analysis framework. We conducted experiments
to measure the extent of test suite reduction obtained by our algorithm and
prior heuristics for test suite minimization. In our experiments, our algorithm
always selected same size or smaller size test suite than that selected by prior
heuristics and had comparable time performance.
Algorithms
 Delayed-Greedy heuristic (DelGreedy)
 classical Greedy heuristic
 HGS (Harrold, Gupta and Soffa)
 SMSP
Reduction
Optimization Approach
128
Issues
to obtain same or more reduction in the test-suite size as compared to the
classical greedy
Type of research
Experiment
Any Comparison?
Yes
Minimized suite by DelGreedy was of same or smaller size then that
generated by other algorithms
on an average, for branch coverage adequate suites, DelGreedy
produced smaller size suites than Greedy, HGS and SMSP in 35%,
64% and 86% of the cases respectively.
On an average, for def-use coverage adequate suites, DelGreedy
produced smaller size suites than Greedy, HGS and SMSP in 39%,
46% and 91% of the cases respectively.
Fitness metrics
Branch Coverage and def-use coverage
Factors effecting the
performance
--
Quality
score
9
Instrument
Reference No.
42
Title
An Empirical Study of Incorporating Cost into Test Suite Reduction and
Prioritization
Authors
Adam M. Smith and Gregory M. Kapfhammer
Published
Abstract
In
ACM, SAC ’09 March 8-12, 2009, Honolulu, Hawaii, USA
Year
2009
Software developers use testing to gain and maintain confidence in the
correctness of a software system. Automated reduction and prioritization
techniques attempt to decrease the time required to detect faults during test
suite execution. This paper uses the Harrold Gupta Soffa, delayed greedy,
traditional greedy and 2-optimal greedy algorithms for both test suite
reduction and prioritization. Even though reducing and reordering a test suite
is primarily done to ensure that testing is cost-effective, these algorithms are
normally configured to make greedy choices with coverage information alone.
This paper extends these algorithms to greedily reduce and prioritize the tests
by using both test cost (e.g., execution time) and the ratio of code coverage to
test cost. An empirical study with eight real world case study applications
shows that the ratio greedy choice metric aids a test suite reduction method
in identifying a smaller and faster test suite. The results also suggest that
incorporating test cost during prioritization allows for an average increase of
17% and a maximum improvement of 141% for time sensitive evaluation
metric called coverage effectiveness.
129
Algorithms
Harrold Gupta Soffa (HGS),
delayed greedy (DGR),
traditional greedy (GRD), and
2-optimal greedy (2OPT)
Optimization Approach
Reduction
Prioritization
Issues
When reducing based on coverage, the algorithm picks the tests that cover the
most additional requirements. For the cost GCM, the algorithm finds the tests
with the lowest execution time. Ratio choices allow the algorithm to pick the
tests that cover the most requirements per unit cost.
Type of research
Experiment
Any Comparison?
Yes
HGS and DGR are the best methods for reducing the size of a
regression test suite.
2OPT and GRD consistently obtain the highest CE scores and the
DGR prioritizer does better than HGS.
GRD and HGS tend to be the most efficient techniques with 2OPT
and DGR exhibiting both higher time overheads
Fitness metrics
Requirement coverage
Factors effecting the
performance
Cost in terms of time, coverage, ratio (coverage/time)
Quality
score
10
Instrument
Reference No.
43
Title
A New Algorithm to Test Suite Reduction Based on Cluster Analysis
Authors
S. Parsa, A. Khalilian and Y. Fazlalizadeh
Published
Abstract
In
2nd IEEE International Conference on Computer Science and Information
Technology, 2009. ICCSIT 2009.
Year
2009
Regression testing is a significant activity which occurs during the maintenance
level of a software lifecycle. However, it requires a large amount of test cases
to test new or modified parts of the software. To address the issue, Test Suite
Reduction techniques have been presented. An appropriate technique should
generate a minimized test suite which exercises different execution paths
within a program while retaining the fault detection capability of the suite
130
admissible. To achieve this, a heuristic algorithm is proposed in this paper. The
new algorithm clusters test cases based on the similarity of their execution
profiles and sample some representatives to form the reduced test suite. The
results of applying the new algorithm to the Siemens suite and comparing to
the H algorithm manifest interesting insights into the effectiveness ofhe
proposed algorithm.
Algorithms
H algorithm
Coverage-based
algorithm(clustering)
Reduction


Optimization Approach
Issues
and
distribution-based
heuristic
Coverage-based techniques and distribution-based techniques (Cluster
filtering, failure-pursuit):
methods perform rather well, but do not necessarily make full
coverage
 empirical studies show that their functionality varies with the size of
the test suite.
 Often do not select similar test cases and are complementary
because they find different faults.
To address the limitations of the coverage-based and distribution-based
approaches, in this paper a heuristic algorithm combining the ideas of the
both approaches, is proposed.

Type of research
Experiment
Any Comparison?
Yes


Fitness metrics
Branch coverage
Factors effecting the
performance
--
Quality
score
11.5
Instrument
When the percentage of size reduction of both algorithms
are closely equal, the percentage of fault detection for our
approach is less than the H algorithm.
When the percentage of fault loss is nearly equal, our
algorithm have greater suite size reduction.
Reference No.
44
Title
Study of Test Suite Reduction Based on Quantum Evolutionary Algorithm
Authors
Yi-kun ZHANG, Ji-ceng LIU, Xue-min YANG,Ying-an CUI,Bao-wei
Published
Abstract
In
2010 International Conference on Artificial Intelligence and Computational
Intelligence, IEEE
Year
2010
Test suite reduction aims at adequately testing all the test requirements with
the least number of test cases. Through the coverage relationship between
131
test requirements and test cases, test suite reduction problem can be
converted into the standard optimization problem. Quantum evolutionary
algorithm (QEA) is an intelligent algorithm based on quantum computation. In
QEA, chromosomes are encoded with quantum bits as the basic information
bits, and individual variation evolution is realized by quantum mutation based
on quantum probability gates. So in QEA, the convergence speed and ability to
search global best results are superior to the traditional evolutionary
algorithms. Motivated by this, we propose a novel test suite reduction method
using quantum evolutionary. Finally, experiments validate the technology with
higher efficiency.
Algorithms
Optimization Approach
 Quantum evolutionary Algorithm (quantum genetic algorithm)
 GE algorithm
 GRE algorithm,
 HGS algorithm,
 genetic algorithm
Reduction
Issues
This paper proposes a novel approach that applying quantum evolutionary
algorithm to test suite reduction problem, and the simulation results validate
that quantum evolutionary algorithm has a very good performance in test
suite reduction.
Type of research
Case Study
Any Comparison?
Yes

Fitness metrics
The genetic algorithm, the quantum genetic algorithm has better
reduction performance, especially when the test suite has a
larger size.
 The time spent on reduction when using quantum genetic
algorithm is much less than the other four algorithms.
Requirement coverage
Factors effecting the
performance
It is easy to maintain the population diversity when using the full crossinterference, and then it has a good convergence speed.
Quality
score
10.5
Instrument
Reference No.
45
Title
Automated GUI Test Coverage Analysis using GA
Authors
Abdul Rauf, Sajid Anwar, M. Arfan Jaffer, and Arshad Ali Shahid
Published
Abstract
In
2010 Seventh International Conference on Information Technology
Year
2010
A Graphical User Interface (GUI) is a graphical front-end to a software system.
A GUI contains graphical objects with certain distinct values which can be used
to determine the state of the GUI at any time. Software developing
organizations always desire to test the software thoroughly to get maximum
132
confidence about its quality. But this requires gigantic effort to test a GUI
application due to the complexity involved in such applications. This problem
has led to the automation of GUI testing and different techniques have been
proposed for automated GUI Testing. Event-flow graph is a fresh technique
being used in the field of automated GUI testing. Just as control-flow graph,
another GUI model that represents all possible execution paths in a program,
event-flow model, in the same way, represents all promising progressions of
events that can be executed on the GUI. Another challenging question in
software testing is, “How much testing is enough?” As development proceeds,
there are fewer measures available that can be used to provide guidance on
the quality of an automatic test suite. Genetic algorithm searches for the best
possible test parameter combinations that are according to some predefined
test criterion. Usually this test criterion corresponds to a “coverage function”
that measures how much of the automatically generated optimization
parameters satisfies the given test criterion. In this paper, we have attempted
to exploit the event driven nature of GUI. Based on this nature, we have
presented a GUI testing and coverage analysis technique centered on genetic
algorithms.
Algorithms
Genetic Algorithm
Optimization Approach
Generation
Issues
Requires gigantic effort to test a GUI application due to the complexity
involved in such applications. This problem has led to the automation of GUI
testing and different techniques have been proposed for automated GUI
Testing. Event-flow graph is a fresh technique being used in the field of
automated GUI testing
Type of research
Experiment
Any Comparison?
No
Fitness metrics
Path coverage
Factors effecting the
performance
No. Of generations
Quality
score
10
Instrument
--
Reference No.
46
Title
An Improved Quantum Genetic Algorithm for Test Suite Reduction
Authors
Yi-kun ZHANG, Ji-ceng LIU, Ying-an CUI, Xin-hong HEI, Ming-hui ZHANG
Published
In
IEEE International Conference on Computer Science and Automation
Engineering (CSAE), 2011
Year
2011
133
Abstract
Test suite reduction is necessary to reduce the testing cost and increase
efficiency, therefore, test suite reduction model is proposed, and test suite
reduction problem is converted into a standard optimization problem.
Additionally, an improved evolutionary algorithm is also proposed, which
encodes the chromosome with quantum bit as its basic information bit. It
achieves the individual evolution by improving the quantum rotating gates
using adaptive rotation, and its convergence speed and global optimization
ability is superior to the traditional evolutionary algorithm. Motivated by this,
we propose a novel test suite reduction method using quantum evolutionary.
Finally, experiments validate the technology with higher efficiency.
Algorithms
Optimization Approach
 IQGA
 greedy algorithm
 HGS algorithm
 GRE algorithm
 Genetic algorithm and
 QGA separately.
Reduction
Issues
Improve the performance
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
IQGA can effectively reduce the test suite size when
guaranteeing the coverage rate.
 When initial test case set is small, the above algorithms have
similar reduction effect.
 Test cases number obtained by IQGA is much smaller than the
traditional reduction algorithms, and it has more advantage
than the genetic algorithm and QGA.
 Quantum genetic algorithm and improved quantum genetic
algorithm spend much less time than genetic algorithm.
Requirement coverage
Factors effecting the
performance
Quantum genetic algorithm uses the full cross -interference, and thus it is easy
to maintain the population diversity, it also has a good convergence speed.
Quality
score
11
Instrument

Reference No.
47
Title
Designing Better Fitness Functions for Automated Program Repair
Authors
Ethan Fast, Claire Le Goues, Stephanie Forrest, Westley Weimer
Published
Abstract
In
Proceedings of the 12th annual conference on Genetic and evolutionary
computation, GECCO’10
Year
2010
Evolutionary methods have been used to repair programs automatically, with
promising results. However, the fitness function used to achieve these results
134
was based on a few simple test cases and is likely too simplistic for larger
programs and more complex bugs. We focus here on two aspects of fitness
evaluation: efficiency and precision. Efficiency is an issue because many
programs have hundreds of test cases, and it is costly to run each test on
every individual in the population. Moreover, the precision of fitness functions
based on test cases is limited by the fact that a program either passes a test
case, or does not, which leads to a fitness function that can take on only a few
distinct values.
This paper investigates two approaches to enhancing fitness functions for
program repair, incorporating (1) test suite selection to improve efficiency and
(2) formal specifications to improve precision. We evaluate test suite selection
on 10 programs, improving running time for automated repair by 81%. We
evaluate program invariants using the Fitness Distance Correlation (FDC)
metric, demonstrating significant improvements and smoother evolution of
repairs.
Algorithms
Random
Genetic Algorithm
Optimization Approach
Selection
Issues
This paper investigates two approaches to enhancing fitness functions for
program repair, incorporating (1) test suite selection to improve efficiency and
(2) formal specifications to improve precision.
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
average test case evaluations per successful repair
Factors effecting the
performance
Computational overhead of GA
Quality
score
11.5
Instrument
GA performs slightly worse because of its computational over head
and if it is ignored the performance of GA is equal or exceeds that of
random.
Reference No.
48
Title
Variable Strength Interaction Testing with an Ant Colony System Approach
Authors
Xiang Chen, Qing Gu, Ang Li, and Daoxu Chen
Published
Abstract
In
16th Asia-Pacific Software Engineering Conference
Year
2009
Interaction testing (also called combinatorial testing) is an cost-effective test
135
generation technique in software testing. Most research work focuses on
finding effective approaches to build optimal t-way interaction test suites.
However, the strength of different factor sets may not be consistent due to
the practical test requirements. To solve this problem, a variable strength
combinatorial object and several approaches based on it have been proposed.
These approaches include simulated annealing (SA) and greedy algorithms. SA
starts with a large randomly generated test suite and then uses a binary
search process to find the optimal solution. Although this approach often
generates the minimal test suites, it is time consuming. Greedy algorithms
avoid this shortcoming but the size of generated test suites is usually not as
small as SA. In this paper, we propose a novel approach to generate variable
strength interaction test suites (VSITs). In our approach, we adopt a one-testat-a-time strategy to build final test suites. To generate a single test, we adopt
ant colony system (ACS) strategy, an effective variant of ant colony
optimization (ACO). In order to successfully adopt ACS, we formulize the
solution space, the cost function and several heuristic settings in this
framework. We also apply our approach to some typical inputs. Experimental
results show the effectiveness of our approach especially compared to greedy
algorithms and several existing tools.
Algorithms
Optimization Approach
1. simulated annealing
2. greedy algorithms Density and ParaOrder
3. ant colony systems
Generation
Issues
The strength of different factor sets may not be consistent due to the practical
test requirements. To solve this problem, a variable strength combinatorial
object and several approaches based on it have been proposed.
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
--
Factors effecting the
performance
parameter settings
Quality
score
9
Instrument
Compared to greedy algorithms and tools, ACS can generate smaller
size for most inputs. In some ACS can even generate same size of
test suites as simulated annealing strategy.
inherited randomness in ACS
Reference No.
49
Title
Comparison of nearest point algorithms by genetic algorithms
Authors
Janne Koljonen
Published
In
Expert System with Applications
136
Year
2011
Abstract
When computational methods are developed, the efficiency of the novel
methods should be compared to the existing ones. This can be done using,
e.g., analytical methods and benchmark test patterns. In addition, the
comparison of the best and the worst case performance is commonly of
interest. In this paper, methodologies of genetic algorithm based software
testing are adopted to the comparative computational testing of three
varieties of dynamic two-dimensional nearest point algorithms. The extreme
performances of the algorithms are searched for by optimizing the shape of
two-dimensional Gaussian distributions, from which the test patterns are
drawn. In particular, an approach to pairwise comparisons of computational
complexities of algorithms is proposed. The test case algorithms can be sorted
with respect to their computational complexity by the proposed method.
Algorithms
Random search
Genetic algorithms
Nearest point algorithm
Prioritization
Optimization Approach
Issues
Even by making primary distribution as fixed there are intrinsic deviation of
the fitness values due to different realizations of the distribution.
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
--
Factors effecting the 
performance

Quality
score
Instrument
The three point algorithms proved to be superior as compared with
genetic algorithms how ever in same cases random search out
performs. Also the three point algorithms proved to have a log liner
complexity.
fitness function
genetic operators
13
Reference No.
50
Title
A Simulation Study on Some Search Algorithms for Regression Test Case
Prioritization
Authors
Sihan Li, Naiwen Bian, Zhenyu Chen*, Dongjiang You, Yuchen He
Published
Abstract
In
10th International Conference on Quality Software
Year
2010
Test case prioritization is an approach aiming at increasing the rate of faults
detection during the testing phase, by reordering test case execution. Many
techniques for regression test case prioritization have been proposed. In this
137
paper, we perform a simulation experiment to study five search algorithms for
test case prioritization and compare the performance of these algorithms. The
target of the study is to have an in-depth investigation and improve the
generality of the comparison results. The simulation study provides two useful
guidelines: (1) Two search algorithms, Additional Greedy Algorithm and 2Optimal Greedy Algorithm, outperform the other three search algorithms in
most cases. (2) The performance of the five search algorithms will be affected
by the overlap of test cases with regard to test requirements.
Algorithms
Optimization Approach
Total Greedy Algorithm
Additional Greedy Algorithm
2-Optimal Greedy Algorithm
Hill Climbing
Genetic Algorithms
Prioritization
Issues
To study five search algorithms for test case prioritization and compare the
performance of these algorithms. The target of the study is to have an indepth investigation and improve the generality of the comparison results.
Type of research
Simulation Experiment
Any Comparison?
Yes
When roverlap=1, there is a slight difference between the
performance of the three greedy algorithms.
Additional Greedy Algorithm and 2-Optimal Greedy Algorithm
perform better than the other three algorithms.
On the other hand, the Total Greedy Algorithm performs much
weaker than the other four algorithms overall.
And Hill Climbing Algorithm performs good but not as good as
Additional Greedy and 2-Optimal Greedy Algorithms.
As for GA, it is worse than Hill Climbing Algorithm overall and the
performance of GA dramatically declines with the increase of
overlap and sometimes it is the worst of these five algorithms.
Fitness metrics
Requirement Coverage
Factors effecting the
performance
--
Quality
score
11
Instrument
Reference No.
51
Title
On the Value of Combining Feature Subset Selection with Genetic Algorithms:
Faster Learning of Coverage Models
138
Authors
James H. Andrews, Tim Menzies
Published
In
Proceedings of the 5th International Conference on Predictor Models in
Software Engineering
Year
2009
Abstract
The next challenge for the PROMISE community is scaling up and speeding up
model generation to meet the size and time constraints of modern software
development projects. There will always be a trade-off between completeness
and runtime speed. Here we explore that trade-off in the context of using
genetic algorithms to learn coverage models; i.e. biases in the control
structures for randomized test generators. After applying feature subset
selection to logs of the GA output, we find we can generate the coverage
model and run the resulting test suite ten times faster while only losing 6% of
the test case coverage.
Algorithms
Genetic algorithms
Optimization Approach
Reduction
Issues
Here we explore that trade-off in the context of using genetic algorithms to
learn coverage models; i.e. biases in the control structures for randomized test
generators.
Type of research
Experiment
Any Comparison?
No
Fitness metrics
Code Coverage
Factors effecting the
performance
--
Quality
score
6.5
Instrument
Reference No.
52
Title
Search based constrained test case selection using execution effort
Authors
Luciano S. deSouza, Ricardo B.C.Prudêncio, Flavia deA.Barros, Eduardo
H.daS.Aranha
Published
Abstract
In
Expert Systems with Applications
Year
2013
Software testing is essential to guarantee high quality products. However, it is
a very expensive activity, particularly when manually performed. One way to
cut down costs is by reducing the input test suites, which are usually large in
order to fully satisfy the test goals. Yet, since large test suites usually contain
139
redundancies (i.e., two or more test cases (TC) covering the same
requirement/piece of code), it is possible to reduce them in order to respect
time/people constraints without severely compromising coverage. In this light,
we formulated the TC selection problem as a constrained search based
optimization task, using requirements coverage as the fitness function to be
maximized (quality of the resultant suite), and the execution effort (time) of
the selected TCs as a constraint in the search process. Our work is based on
the Particle Swarm Optimization (PSO) algorithm, which is simple and efficient
when compared to other widespread search techniques. Despite that, besides
our previous works, we did not find any other proposals using PSO for TC
selection, neither we found solutions treating this task as a constrained
optimization problem. We implemented a Binary Constrained PSO (BCPSO) for
functional TC selection, and two hybrid algorithms integrating BCPSO with
local search mechanisms, in order to refine the solutions pro- vided by BCPSO.
These algorithms were evaluated using two different real-world test suites of
functional TCs related to the mobile devices domain. In the performed
experiments, the BCPSO obtained promising results for the optimization tasks
considered. Also, the hybrid algorithms obtained statistically better results
than the individual search techniques.
Algorithms
Optimization Approach
 BCPSO (Binary Constrained PSO)
 BCPSO-FS (hybrid algorithm)
 BCPSO-HC (hybrid algorithm)
 HC (Hill-Climbing)
 FS (Forward Selection)
 BE (Backward Elimination)
 Random
Selection
Issues
We formulated the TC selection problem as a constrained search based
optimization task, using requirements coverage as the fitness function to be
maximized (quality of the resultant suite), and the execution effort (time) of
the selected TCs as a constraint in the search process.
Type of research
Experiment
Any Comparison?
Yes
Integration suite Results
Both BCPSO-FS and BCPSO-HC were better than BCPSO, FS
and HC
 BCPSO-FS technique obtained the best average results
 Good results were also achieved by the local search
techniques FS and BE by being simpler to implement and
less computationally expensive.
In regression suite, only the hybrid BCPSO-HC was statistically
superior to its individual components (BCPSO, HC).

Fitness metrics
Requirement Coverage
Factors effecting the
performance
Effort Limits (time)
140
Quality
score
Instrument
13
Reference No.
53
Title
Genes and Bacteria for Automatic Test Cases Optimization in the .NET
Environment
Authors
Benoit Baudry, Franck Fleurey, Jean-Marc Jézéquel and Yves Le Traon
Published
In
Proceedings of the 13th International Symposium on Software Reliability
Engineering IEEE
Year
2002
Abstract
The level of confidence in a software component is often linked to the quality
of its test cases. This quality can in turn be evaluated with mutation analysis:
faulty components (mutants) are systematically generated to check the
proportion of mutants detected ("killed") by the test cases. But while the
generation of basic test cases set is easy, improving its quality may require
prohibitive effort. This paper focuses on the issue of automating the test
optimization. We looked at genetic algorithms to solve this problem and
modeled it as follows: a test case can be considered as a predator while a
mutant program is analogous to a prey. The aim of the selection process is to
generate test cases able to kill as many mutants as possible. To overcome
disappointing experimentation results on the studied .Net system, we propose
a slight variation on this idea, no longer at the "animal" level (lions killing
zebras) but at the bacteriological level. The bacteriological level indeed better
reflects the test case optimization issue: it introduces of a memorization
function and the suppresses the crossover operator. We describe this model
and show how it behaves on the case study.
Algorithms
Genetic algorithms
Bacteriological algorithms
Optimization Approach
Reduction
Issues
The experiments with genetic algorithms were not satisfactory. Both because
of the slow convergence and the unusual proportions of crossover and
mutation operators.
Type of research
Case Study
Any Comparison?
Yes
GAs are not perfectly adapted to the test cases optimization problem. A
more adapted model should provide memory and remove the notion of
individual to concentrate on the genes (test cases).
The bacteriological algorithm converges much faster than the genetic
one.
141
Fitness metrics
Faults detection
Factors effecting the
performance
No. of Test cases
Memorization
Quality
score
12
Instrument
Reference No.
54
Title
Automated test data generation using a scatter search approach
Authors
Raquel Blanco, Javier Tuya, Belarmino Adenso-Díaz
Published
In
Information and Software Technology 51
Year
2009
Abstract
The techniques for the automatic generation of test cases try to efficiently find
a small set of cases that allow a given adequacy criterion to be fulfilled, thus
contributing to a reduction in the cost of software testing. In this paper we
present and analyze two versions of an approach based on the scatter search
metaheuristic technique for the automatic generation of software test cases
using a branch coverage adequacy criterion. The first test case generator,
called TCSS, uses a diversity property to extend the search of test cases to all
branches of the program under test in order to generate test cases that cover
these. The second, called TCSS-LS, is an extension of the previous test case
generator which combines the diversity property with a local search method
that allows the intensification of the search for test cases that cover the
difficult branches. We present the results obtained by our generators and
carry out a detailed comparison with many other generators, showing a good
performance of our approach.
Algorithms
 Scatter search
 TCSS (test coverage scatter search)
 TCSS-LS (test coverage scatter search-local search)
Generation
Optimization Approach
Issues
The test case generators use a large set of test cases to cover all branches,
because during the search process they generate test cases that reach
branches that had already been covered by other test cases; however not all
of them form the set of test cases used in the test process of the program.
Type of research
Experiment
Any Comparison?
Yes
TCSS diversifies the search of test cases by means of the diversity
function in order to cover a great number of branches, while TCSS-LS
diversifies the search in the first iterations and then intensifies the
search through a local search procedure to obtain test cases that cover
the most difficult nodes.
142
TCSS-LS outperform the TCSS and other test case generators.
Fitness metrics
Branch Coverage
Factors effecting the
performance
--
Quality
score
9.5
Instrument
Reference No.
55
Title
Quality Improvement and Optimization of Test Cases – A Hybrid Genetic
Algorithm Based Approach
Authors
Dr. D.Jeya Mala, Dr. V.Mohan
Published
In
ACM SIGSOFT Software Engineering Notes
Year
2010
Abstract
Software development organizations spend considerable portion of their
budget and time in testing related activities. The effectiveness of the
verification and validation process depends upon the number of errors found
and rectified before releasing the software to the customer side. This in turn
depends upon the quality of test cases generated. The solution is to choose
the most important and effective test cases and removing the redundant and
unnecessary ones; which in turn leads to test case optimization. To achieve
test case optimization, this paper proposed a heuristics guided population
based search approach namely Hybrid Genetic Algorithm (HGA) which
combines the features of Genetic Algorithm (GA) and Local Search (LS)
techniques to reduce the number of test cases by improving the quality of test
cases during the solution generation process. Also, to evaluate the
performance of the proposed approach, a comparative study is conducted
with Genetic Algorithm and Bacteriologic Algorithm (BA) and concluded that,
the proposed HGA based approach produces better results.
Algorithms
Hybrid Genetic Algorithm (HGA)
Genetic Algorithm (GA)
Bacteriologic Algorithm (BA)
Reduction
Optimization Approach
Issues
Both simple optimization techniques as well as population based search
heuristics have only less consideration on fault revealing capability of the test
cases in test case optimization problem however best qualities of GA, BA and
local search techniques can be combined to get a better algorithm.
Type of research
Experiment/ Case Study
Any Comparison?
Yes
The performance of GA, BA and HGA are evaluated based on the fault
revealing capability and path coverage of the test cases. GA produces
143
non linear suboptimal solution and usually strikes up at local optima,
BA provides linear and optimal solutions, and HGA produces near
global optimal and linear optimal solutions with rapid convergence.
Finally, it has been proved that HGA outperforms the other two
population based search approaches.
Fitness metrics
Path Coverage
Factors effecting the
performance
GA usually strike up in local optima
non linear suboptimal solution
Number of Generations
13
Quality
score
Instrument
Reference No.
56
Title
A variable strength interaction test suites generation strategy using Particle
Swarm Optimization
Authors
Bestoun S. Ahmed, Kamal Z. Zamli
Published
In
The Journal of Systems and Software
Year
2011
Abstract
This paper highlights a novel strategy for generating variable-strength (VS)
interaction test suites, called VS Particle Swarm Test Generator (VS-PSTG). As
the name suggests, VS-PSTG adopts Particle Swarm Optimization to ensure
optimal test size reduction. To determine its efficiency in terms of the size of
the generated test suite, VS-PSTG was subjected to well-known benchmark
configurations. Comparative results indicate that VS-PSTG gives competitive
results as compared to existing strategies. An empirical case study was
conducted on a non-trivial software system to show the applicability of the
strategy and to determine the effectiveness of the generated test suites to
detect faults.
Algorithms
Particle Swarm Optimization
Optimization Approach
Generation
Issues
The particles iteratively move around the search space. During this movement,
there is a chance to produce non-integer velocities for the particles. Thus, the
position of such particle dimension becomes a non-integer number. To solve
this problem, the algorithm rounds the position to the nearest integer
number.
Type of research
Experiment
Any Comparison?
No
Overall, VS-PSTG generates the optimal test size in most cases. It should
also be noted that even though in some cases VS-PSTG results may not
144
the best, the results obtained were still competitive enough.
Fitness metrics
Fault Detection
Factors effecting the
performance
--
Quality
score
6.5
Instrument
Reference No.
57
Title
Software Reliability Testing Covering Subsystem Interactions
Authors
Matthias Meitner and Francesca Saglietti
Published
In
Proceedings of the 16th international GI/ITG conference on Measurement,
Modeling, and Evaluation of Computing Systems and Dependability and Fault
Tolerance
Year
2012
Abstract
This article proposes a novel approach to quantitative software reliability
assessment ensuring high interplay coverage for software components and
decentralized (sub-) systems. The generation of adequate test cases is based
on the measurement of their operational representativeness, stochastic
independence and interaction coverage. The underlying multi-objective
optimization problem is solved by genetic algorithms. The resulting automatic
test case generation supports the derivation of conservative reliability
measures as well as high interaction coverage. The practicability of the
approach developed is finally demonstrated in the light of an interactionintensive example.
Algorithms
Genetic Algorithm
Optimization Approach
Reduction
Issues
Random generation is carried out only once therefore it cannot guarantee to
cover all required testing scenarios. In fact, even assuming an accurate
knowledge of the operational profile, the resulting randomly generated test
case sample may still deviate from statistical expectation
Type of research
Experiment
Any Comparison?
No
Fitness metrics
Def-use-pair Covered
Factors effecting the
performance
--
Quality
6
Instrument
--
145
score
Reference No.
58
Title
Testing an Intentional Naming Scheme Using Genetic Algorithms
Authors
Sarfaraz Khurshid
Published
In
T. Margaria and W. Yi (Eds.): TACAS 2001, LNCS 2031, pp. 358-372, 2001.
c Springer-Verlag Berlin Heidelberg
Year
2001
Abstract
Various attempts have been made to use genetic algorithms (GAs) for
software testing, a problem that consumes a large amount of time and effort
in software development. We demonstrate the use of GAs in automating
testing of complex data structures and methods for manipulating them, which
to our knowledge has not been successfully displayed before on non-trivial
software structures. We evaluate the effectiveness of our GA-based test suite
generation technique by applying it to test the design and implementation of
the Intentional Naming System (INS), a new scheme for resource discovery
and service location in a dynamic networked environment. Our analysis using
GAs reveals serious problems with both the design of INS and its inventors'
implementation.
Algorithms
Genetic Algorithm
Optimization Approach
Generation
Issues
We evaluate the effectiveness of our GA-based test suite generation
technique by applying it to test the design and implementation of the
Intentional Naming System (INS)
Type of research
Experiment
Any Comparison?
No
Fitness metrics
Statement Coverage
Factors effecting the
performance
--
Quality
score
6.5
Instrument
--
Reference No.
59
Title
Automatic, evolutionary test data generation for dynamic software testing
146
Authors
Anastasis A. Sofokleous, Andreas S. Andreou
Published
In
The Journal of Systems and Softwares
Year
2008
Abstract
This paper proposes a dynamic test data generation framework based on
genetic algorithms. The framework houses a Program Analyser and a Test Case
Generator, which intercommunicate to automatically generate test cases. The
Program Analyser extracts statements and variables, isolates code paths and
creates control flow graphs. The Test Case Generator utilises two optimization
algorithms, the Batch-Optimistic (BO) and the Close-Up (CU), and produces a
near to optimum set of test cases with respect to the edge/condition coverage
criterion. The efficacy of the proposed approach is assessed on a number of
programs and the empirical results indicate that its performance is
significantly better compared to existing dynamic test data generation
methods.
Algorithms
Genetic algorithms with Close-Up (CU) algorithm and Batch-Optimistic (BO)
algorithm
random
gradient descent
standard genetic and
differential genetic (diffGA)
Generation
Optimization Approach
Issues
A special form of fitness functions is required to reflect dataflow instead of
control flow based evolutions.
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
Path Coverage
Factors effecting the
performance
--
Quality
score
9.5
Instrument
The comparison suggested the superiority of the proposed scheme
over the rest of the algorithms
Reference No.
60
Title
Evolutionary mutation testing
Authors
J.J. Dominguez-Jimenez, A. Estero-Botaro, A. Garcia-Dominguez, I. MedinaBulo
147
Published
In
Information and Software Technology
Year
2011
Abstract
Context: Mutation testing is a testing technique that has been applied
successfully to several programming languages. However, it is often regarded
as computationally expensive, so several refinements have been proposed to
reduce its cost. Moreover, WS-BPEL compositions are being widely adopted by
developers, but present new challenges for testing, since they can take much
longer to run than traditional programs of the same size. Therefore, it is
interesting to reduce the number of mutants required.
Objective: We present Evolutionary Mutation Testing (EMT), a novel mutant
reduction technique for finding mutants that help derive new test cases that
improve the quality of the initial test suite. It uses evolutionary algorithms to
reduce the number of mutants that are generated and executed with respect
to the exhaustive execution of all possible mutants, keeping as many difficult
to kill and potentially equivalent mutants (strong mutants) as possible in the
reduced set.
Method: To evaluate EMT we have developed GAmera, a mutation testing
system powered by a co-evolutive genetic algorithm. We have applied this
system to three WS-BPEL compositions to estimate its effectiveness,
comparing it with random selection.
Results: The results obtained experimentally show that EMT can select all
strong mutants generating 15% less mutants than random selection in over
20% less time for complex compositions. When generating a percentage of all
mutants, EMT finds on average more strong mutants than random selection.
This has been confirmed to be statistically significant within a 99.9%
confidence interval.
Conclusions: EMT has reduced for the three tested compositions the number
of mutants required to select those which are useful to derive new test cases
that improve the quality of the test suite. The directed search performed by
EMT makes it more effective than random selection, especially as
compositions become more complex and the search space widens.
Algorithms
Genetic algorithm
Random
Optimization Approach
Reduction
Issues
We present Evolutionary Mutation Testing (EMT), a novel mutant reduction
technique for finding mutants that help derive new test cases that improve
the quality of the initial test suite. It uses evolutionary algorithms to reduce
the number of mutants that are generated and executed with respect to the
exhaustive execution of all possible mutants, keeping as many difficult to kill
and potentially equivalent mutants (strong mutants) as possible in the
reduced set
148
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
Branch coverage
Factors effecting the
performance
Parameter setting
Quality
score
14
Instrument
EMT is more effective than random selection at selecting all the
strong mutants, especially with complex compositions.
Reference No.
61
Title
Iterative automatic test generation method for telecommunication protocols
Authors
G. Kova´cs, Z. Pap, G. Csopaki, K. Tarnay
Published
In
Computer Standards & Interfaces
Year
2005
Abstract
Standardized languages used for protocol specification provide an excellent
basis for both automatic and manual test generation. Test generation is
composed of two steps, the derivation of test cases from the specification,
and the selection of the test cases to be included in the final test suite in order
to reduce its execution time. This article proposes a new method that aims to
decrease the total number of test cases generated automatically by a test
derivation algorithm and at the same time to reduce the computation
requirements for the test selection procedure. It creates an iteration cycle on
the model of evolutionary algorithms, where the test derivation and selection
are done simultaneously. In each cycle first a “small” test suite is derived then
optimized, evaluated and finally compared to the best suite so far. It is kept as
the best suite, if it is found better according to some well-defined evaluation
criteria and test suite metrics. This iteration condition is based on the test
selection criteria. The article presents an experiment where iterative
algorithms are compared to two simple test derivation methods from different
aspects.
Algorithms
random trace test derivation algorithm
random event sequences
the proposed iterative test generation method
its enhancement with the immediate test selection
its extension with memory
Selection /Generation
Optimization Approach
Issues
This article proposes a new method that aims to decrease the total number of
test cases generated automatically by a test derivation algorithm and at the
same time to reduce the computation requirements for the test selection
149
procedure
Type of research
Experiment
Any Comparison?
Yes
The fault detection ratio of the random trace method was similar to
the random sequence based iterative algorithms. Though the
iterative algorithms required more time for smaller protocols, they
were found less sensitive for the increase in system complexity.
In case of more complex protocols the iterative algorithms were
found to decrease the total generation time of an optimized test suite
compared to the methods performing test generation and selection
in two separate steps. The iterative techniques also provided similar–
or slightly better–fault coverage and similar number of final test
cases.
Fitness metrics
Fault detection
Factors effecting the
performance
Complex and simple test cases
Quality
score
10.5
Instrument
Reference No.
62
Title
Scatter Search in software testing, comparison and collaboration with
Estimation of Distribution Algorithms
Authors
Ramo´n Sagarna, Jose A. Lozano
Published
Abstract
In
European Journal of Operational Research
Year
2004
One of the main tasks software testing includes is the generation of the test
cases to be used during the test. Due to its expensive cost, the automatization
of this task has become one of the key issues in the area. The field of
Evolutionary Testing deals with this problem by means of metaheuristic search
techniques.
An Evolutionary Testing based approach to the automatic generation of test
inputs is presented. The approach developed involves different possibilities of
the usage of two heuristic optimization methods, namely, Scatter Search and
Estimation of Distribution Algorithms. The possibilities comprise pure Scatter
Search options and Scatter Search—Estimation of Distribution Algorithm
collaborations. Several experiments were conducted in order to evaluate and
compare the approaches presented with those in the literature. The analysis
of the experimental results raises interesting conclusions, showing these
alternatives as a promising option to tackle this problem.
150
Algorithms
Optimization Approach
Estimation of Distribution Algorithms (EDA)
Scatter Search (SS)
Combination of Scatter Search and EDA (SS-EDA)
Generation
Issues
The approach here exposed makes use of the optimization methods above in
two different ways. On the one hand, pure SS alternatives are presented in
order to evaluate their performance and compare them with the approaches
in previous works. On the other hand, an SS–EDA collaborative approach is
included. As far as we know, this is the first time that SS is applied to the test
case generation problem, and the first time that SS and EDAs are combined.
Type of research
Experiment
Any Comparison?
Yes
Fitness metrics
Branch Coverage
Factors effecting the
performance
Parameter value
Quality
score
13.5
Instrument
The SS and EDAs combination offered encouraging results. The
collaborative approach outperformed the results of the EDA generator
in five of the test programs, and the results of the SS generator in four
programs.
151