HOW TO IMPLEMENT A SUCCESSFUL
TEST AUTOMATION PROGRAM
ABSTRACT
According to IBM, software testing represents more than 50 percent of overall
software and research and development costs. Industry analyst’s research
indicates that software testing in best in class organizations ranges from 2030%. However, demands on software testing are increasing as software
development, deployment and delivery is getting increasingly complex with the
proliferation of mobile and cloud technologies. Having an effective automation
program is critical, now more than ever, to getting testing costs in line with
overall R&D spend while maintaining software quality and managing time to
market goals.
A proven way of attaining the right cost of quality is to achieve the optimum
level of test automation that is justified by the economic benefits of automation.
Effectively implemented automated frameworks that are aligned with
organizational objectives deliver substantial benefits. Objectives include:
Accelerated
Testing Cycle
Decrease time to market for new applications and products
Increased
Coverage
Enable more extensive and detailed testing
Repeatable
and Auditable
Support formal testing processes that provide evidence of
execution and are easy to execute
Accuracy and
Reporting
Not prone to human judgment or errors
Enhanced
Productivity
Execute efficiently, repeatedly and often; Future test cycles will
be more productive
Organizations over the past few years have implemented automation projects to
address these objectives. However, according to Alliance’s research, more than
60% of these automation initiatives either stall or fail within the first six months
before delivering any value on the initial investment.
There are several impediments that cause these failures. These impediments
arise from organizational and technical issues but can be overcome by following
proven and practical approaches.
This white paper outlines these approaches to successfully implement
automation projects while overcoming these impediments.
© Alliance Global Services 2012
Page 1
This paper contains three sections. The first section discusses how automation
projects fail and the impediments to implementing successful automation. The
next section addresses best practices to addressing these impediments. The
final section describes concrete examples from Alliance’s experience on
implementing these best practices.
WHY AUTOMATION PROJECTS STALL OR FAIL
Failure of automation initiatives occurs during both the implementation and
steady state phases. The common scenarios of failure for automation initiatives
are as follows:
IMPLEMENTATION
•
More Expensive than Planned: Automation initiatives are software
development initiatives. Test automation is a highly complex development
effort that requires project management, design and programming resources
similar to an application development project. Like any development project,
test automation efforts will fail without proper management or adequate
investment of resources, tools and infrastructure. Additionally these
initiatives need a “critical mass” of scripts to be developed for them to deliver
perceivable value within the organization. If these costs are not accurately
planned, organizations experience sticker shock and abandon automation
efforts midway.
© Alliance Global Services 2012
Page 2
•
Takes Longer: As described in the previous point, there are a significant
number of items that need to be implemented as part of automation
initiatives, for them to be successful. Often these need to be carefully
planned and focused to deliver value. Automation initiatives fail when
treated as a side project for junior resources or a back-up project done inbetween releases by the testing team.
•
Fewer Manual Testing Resources Replaced: Automation of testing processes
addresses only some aspects of testing. These are activities where manual
testing cannot scale in a limited time frame. However, there is always going
to be a dependency on manual expertise, judgment and interpretation in
software testing that cannot be economically replicated using automation.
Depending on the type of application, business domain, technologies used
and the pace of application changes, automation may be confined to only a
small portion of the overall testing component and may disappoint project
sponsors on the number of manual testing resources that are replaced.
•
ROI is not met: Automation initiatives are kicked off in several instances
with little or no definition of business expectations. Since test automation
initiatives seek to achieve exactly the same outcomes as a manual testing
process, these expectations are easier to quantify and measure than most
software development initiatives. In cases where the ROI expectations are
defined, they may be too aggressive or optimistic. Industry literature and
vendor claims often understate the effort and complexity needed to
implement automation while inflating the returns that automation can
deliver.
STEADY STATE
•
Costs More to Maintain than Planned: Often, very little consideration or
thought is put at the beginning of automation initiatives towards support
and maintenance leading to few resources planned for the steady state
phase. Invariably, the team members who are asked to support the
automation suite had very little to do with the original implementation and
are tasked with running and enhancing the test suite.
•
Never Seem to Run Successfully: Poorly implemented frameworks cause the
automation suite to be extremely brittle. Any small changes in the
underlying application, test data or test environments ripple into substantial
© Alliance Global Services 2012
Page 3
portions of the suite rendering them either unusable or requiring major
effort to remediate.
•
Out Of Sync with the Application: Poorly written or recorded scripts need to
be updated regularly as the functionality of the underlying application
changes. In several failed projects, scenarios or test cases are left to become
“stale” and throw lots of false positives. In these cases scripts highlight
“defects” in an application when in fact these are the result of functional
changes legitimately implemented by the development team. This induces a
lack of trust with the framework and discounting of results created by the
automation suite.
•
Development Loses Buy-In: As a result of the previous two items,
development, test and operations teams lose trust and interest in the
automation tool and leave it out of the testing process. This in turn makes
the automation suite further out of sync and error-prone and leads to a
complete loss of value.
COMMON IMPEDIMENTS
The causes of automation project failure in the earlier section are widely
experienced by most organizations that have attempted automation. There are
several recurring reasons for these failures. Often automation projects fail
because too much of the attention is focused on choosing a tool and writing a
few scripts while paying no attention to the “why” and “how” of automation. In
other cases not enough resources or the wrong resources are applied to the
project. The core reason for these failures stems from a few key impediments.
These impediments are:
•
Lack of Objectives: Lacking clearly defined objectives for the automation
initiative is an assured way to not achieving them. The goal of the project
should not be automation for automation’s sake, but the use of automation
to meet specific business objectives.
•
Inadequate Automation Strategy: Test automation is not a one-to-one
replacement of manual testing. In most cases manual test cases and
approaches will need to be adapted for automation testing. Additionally,
tools and infrastructure needs to be aligned with the technical needs of the
application. Given these complexities, a well-defined automation strategy is
vital to the success of automation.
© Alliance Global Services 2012
Page 4
•
High Start-Up Costs: Start-up costs for building successful automation
projects is high. This includes cost of tools, infrastructure, framework
development and suite development. The size of this investment will depend
on the scope of automation planned as well as the extent of existing tools,
resources and test artifacts that can be used in the effort. If existing
artifacts or systems such as test scripts, traceability, defect tracking, etc.
need to be updated to enable automation, this further increases the
investment.
•
Lack of an Effective Automation Framework: Software products and
applications that are tested using automation tools are highly complex with
dynamic data and environmental conditions. An effective framework is
needed to enable reusable objects, test data management and test
environment setup. Organizations often ignore this in the rush to develop
scripts resulting in poor automation investments.
•
Lack of Metrics and Reports: The final impediment is not having an integral
metrics and reporting process as part of the automation initiative. This
process is needed to measure the automation initiative’s effectiveness in
delivering on the objectives and enabling any course corrections to the
implementation plans.
IMPLEMENTING A SUCCESSFUL AUTOMATION PROGRAM
The key factors that should be considered for a successful automation program
is to overcome the impediments described in the earlier section.
DEFINE AUTOMATION OBJECTIVES
Define objectives for your automation program, which are aligned with business
objectives and used as a guidepost for managing your automation program.
Once the objectives are defined, these should be converted to metrics, which are
quantifiable and measurable and show where you were and where you want to
be after implementing an automation program. Automation objectives should be
tied to business objectives such as:

Time to market for the software product or application

Cost of support resulting from software defects

Overall cost of testing for a given application size and platform
© Alliance Global Services 2012
Page 5

Product quality as measured by software defects seeping into production
DEFINE STRATEGY FOR AUTOMATION
The automation strategy is crucial for achieving best in class automation as it
defines how you embark on your journey for a successful automation program.
Once the business objectives are defined, the automation strategy must outline
how you will implement your automation program from conception to steady
state. Your automation strategy must address the following key questions.

What should we automate?

What are the resource requirements (both human and others)?

What is the framework to be used?

What is the ROI for the automation program?
What to Automate?
Identifying automation candidates is the single most important step to a
successful automation program as it directly defines the achieved business
value of the automation implementation.
Most of the manual testing is done to test individual functionality thoroughly
and users leverage many test techniques like BVA (Boundary Value Analysis),
state transition testing (aimed to find the sources most likely to be defective),
risk based testing to prioritize the tests where there is a risk and likelihood of
failure, all aimed to prioritize testing efforts to focus on important areas of
individual modules. One of the most important manual testing requirements is
to perform tests at low levels of granularity. For example, manual test cases
might test for functionality such as user actions, data manipulation, search,
and screen navigations to name a few.
In order to identify what to automate, users must ensure that it results in
sufficient and efficient automation scripts (derived from automation candidates
to design automation scenarios) to validate all key interfaces across various
components and business critical functions. The factors that need to be
considered for identifying the automation candidates include:

More resource intensive and tedious job to execute manually

Work flow based with requirement of large data sets and combinations
© Alliance Global Services 2012
Page 6

Application functionality frequently and most used

Which have longer life span of usage and not the scenarios which are of
shorter life span usage

Across complex sub-systems and integration point between various modules

Application functionality which is used repetitively across the application
What are the Resource Requirements?
You must identify and document all resource requirements up front. An
automation program can simply fail to achieve its objectives because
automation scripts are not being developed against the correct environment or
test data which results in huge rework. Resource requirements for automation
can be broadly classified into three major categories.



Environment
o
Identify the hardware/software/network requirements
o
Access to the application
o
Process, governance and configuration management
Tools
o
Automation
o
Test management
o
Defect tracking
Test Data
o
Test data planning
o
Test data preparation
o
Test data generation
Start-up Costs
Automation initiatives require significant investment of money and time to
obtain skills, tools, frameworks and scripts. Investment considerations should
include the following:
•
Infrastructure Setup: Funding for obtaining automation tools, test
environments, and deployment tools.
© Alliance Global Services 2012
Page 7
•
Skills: Automation requires highly technical resources with expertise in
specific testing tools and underlying technologies to ensure automation
projects success. Even with sophisticated automation tools and script-less
automation frameworks, resources need an understanding of underlying
application technologies, data generation processes, and SDLC processes.
•
Test Suite Development: Development of critical mass of test scripts,
supporting frameworks, test data processes and environment integration
scripts form the core of initial development. Additionally automation scripts
themselves need to be tested and debugged before introducing into
production.
Automation Framework
An automation framework should be reliable, repeatable, reusable and datadriven as they can dramatically improve success and reduce the chance of
abandonment on automation projects.
•
Reliable: The biggest achievement of automation tests is that they should be
reliable so that they can be run unattended.
•
Repeatable: The tests should be repeatable with the ability to test on
multiple configurations.
•
Reusable: Implementing a framework should result in repeatability and
reusability for incremental software builds.
•
Data-Driven: The ability to execute the business scenarios using various test
data sets.
DESIGNING AUTOMATION SCENARIOS
One of the key factors in implementation is the design phase where we should
identify the scenarios which add business value by automating those scenarios.
Automation scenarios are to be derived such that they test applications at a
higher level of granularity to cover the interaction between modules or business
components. These scenarios tend to unearth existing bugs and detect more
bugs that are difficult to detect by testing individual modules, functionality, and
business processes.
© Alliance Global Services 2012
Page 8
Writing good automation scenarios can be considered an artistic science
(scientific approach plus artistic touch). Besides product knowledge, writing
automation scenarios requires an understanding of the business context of the
end user. Here are some tips on how to derive good quality automation
scenarios from requirements and automation candidates.

The testers need to think as the end users to figure out the real world
scenarios or use cases which can be performed on the application by the
end user – this can be achieved by working with various stakeholders or the
subject matter experts.

The scenarios identified should be independent of other test scenarios,
which have their own pre conditions and data sets identified. This ensures
that a scenario is a story in itself and the automated test is like usage of the
application by any end user.

The scenarios have to be derived such that the end-to-end functionality of
the application is covered and all the business flows of the application are
working correctly.

Most software applications are response based. Identify the events triggered
either by users or by the application (business rules).

Identify special business events related to the application. Identifying the
scenarios around these special business events, i.e. promotional sales
campaigns, can help find bugs in the underbelly of the application.

Review the bugs and defects list of the system. This will provide further
insight into the application usage.
DESIGNING AN AUTOMATION FRAMEWORK
In order for the automated tests to be reusable, repeatable and maintainable,
test development standards need to be defined and followed. These can be
achieved by defining the framework and guidelines that have to be taken as the
base and following the development architecture. There are some basic features
that a framework is expected to have including designing a modular framework
for easy maintenance and enhancements which support a high degree of reuse.
© Alliance Global Services 2012
Page 9
ALLIANCE’S APPROACH
With years of experience in implementing successful automation programs for
our clients, Alliance has developed a set of tools, accelerators, and processes
that are blended with our expertise in implementing best in class automation
programs for our clients.
OBJECTIVES
Alliance’s automation initiatives start with a Rapid Test Assessment. During
this assessment, explicit and implicit business objectives are identified and the
organization’s capabilities on various key dimensions are identified. The
outcome of this assessment is a capabilities dashboard as shown below.
Once the assessment is completed the business objectives are documented
along with the capabilities to ensure an appropriate automation strategy is
implemented.
© Alliance Global Services 2012
Page 10
In the example above the business objectives are categorized into Quality
focused objectives and Speed/Efficiency focused objectives. All of the objectives
were driven by the business and the quantitative measures were developed
during the assessment to ensure significant returns. These objectives drive the
automation strategy.
STRATEGIES FOR AUTOMATION
Identifying scenarios for automation is the single biggest component of creating
an automation strategy. As described in earlier sections automation requires
significant investment of time, money and resources and what to automate
needs to be carefully planned. As part of this we analyze application
requirements, existing application functionality, past defects and manual test
scripts to develop automation candidates. Additionally, Alliance uses a
proprietary method of analyzing application source code and data repositories
to glean business critical transactions and add them to automation candidates.
These candidates are prioritized based on business value, automate-ability and
longevity. Based on this prioritization, detailed end-to-end automation
scenarios are designed. These scenarios are then mapped to manual test cases
to establish traceability.
START-UP COSTS
Alliance utilizes TitaniumTM Managed Testing Services to flatten the start-up
costs to make is easier for organizations to implement automation. This
© Alliance Global Services 2012
Page 11
includes providing highly skilled resources with deep automation experience to
plan and implement automation projects. Alliance utilizes a partnership with
leading commercial tool vendors in a “lease” model. Where appropriate and
applicable Alliance leverages open source tools developed in-house and in the
open source community to decrease licensing impact. Additionally, Alliance has
also developed cloud-based deployment models for the tools to decrease test
environment costs. The chart below shows how Alliance’s model flattens the
high startup costs for an automation project.
AUTOMATION FRAMEWORK
Alliance’s TitaniumTM Automation Framework focuses on all aspects of testing
not just script development and execution to deliver substantial efficiencies in
automation projects. As shown in the image below, there are several key
components to make automation successful in addition to scripting and
frameworks.
© Alliance Global Services 2012
Page 12
Key features of this framework are that it:

Bridges the gap between the business and the technical teams to allow
collaborative test case design and development

Enables rapid Script Development and integrates into short development
cycles such as Agile Sprint cycles

Is automation dependent on an efficient framework and governance

Utilizes quality measurement and reporting to evaluate effectiveness

Executes and automates all aspects of testing from Framework to test lab

Leverages reusable test data and test environment management processes
METRICS & REPORTING
Alliance utilizes a comprehensive metrics dashboard that measures key metrics
to evaluate the effectiveness of automation. Some of these metrics are:

Cost benefit analysis: Measures the manual testing cost for comparable
automation tests and identifying the benefit accrued by executing the
automation suite.
© Alliance Global Services 2012
Page 13

Backlog of test cases: Measures the outstanding test cases that need to be
automated while providing a view towards the velocity and direction of test
case generation.

Cost incurred: Identifies the overall cost of testing, cost of manual testing,
and automation testing while providing a view towards the impact of
automation on manual testing costs.

Coverage gap: Measures the gap of coverage between what has been
identified vs. what has been executed in a given test execution cycle.
© Alliance Global Services 2012
Page 14
CONCLUSION
As the software revolution gathers pace and the complexity of software
increases, test automation needs to be an integral part of every testing
organization’s execution strategy. Test automation offers the only viable option
for most organizations to keep the cost of testing flat or decreasing while
providing faster time to market and rapid testing cycles.
Therefore, executing successful automation projects is critical to achieving
organizational business goals. The models for successful implementation are
derived from proven software development processes, and require focus on all
aspects of test automation development from setting objectives to developing
viable resource models to measuring the success of the initiatives.
About Alliance Global Services
Alliance Global Services is a software development firm that partners with software,
technology
and information-intensive businesses on their mission critical work. Alliance architects and
builds software applications, platforms and products that become primary drivers of
innovation and revenue growth for its clients and their businesses. Alliance is recognized for
driving quality and speed-to-market when business success depends on The Software
InsideTM. Founded in 1994, Alliance is headquartered in suburban Philadelphia,
Conshohocken, PA. For more information, visit www.allianceglobalservices.com.
Corporate Headquarters
Six Tower Bridge | 181 Washington Street, Suite 350 | Conshohocken, PA 19428
Phone: 610.234.4301 | [email protected] | www.allianceglobalservices.com
© Alliance Global Services 2012
Page 15