BUSINESS WHITE PAPER
Maximize QA Resources with TestDriven Development and Automation
Find and fix defects earlier and reduce the risk of
customer-facing defects with a development strategy
driven by automated testing.
Overview
TABLE OF CONTENTS
Overview........................................1
The Reality of Manual Testing....2
QA and Project Realities..............2
Automated Customer Experience
Testing.............................................3
Results of Automation.................3
Benefits of Automated Customer
Experience Testing........................4
Conclusion.....................................5
About Cyara...................................5
Quality Assurance (QA) is a critical role in any technology project. QA professionals
ensure that when new systems are rolled out, the customer experience works as
intended without defects that are visible to customers. Not getting it right costs
you money, customers, and reputation as increasingly companies are penalized for
customer experience issues. You can make sure you get the customer experience
right by effectively using your QA resources. The challenge for QA in any project is
two-fold. First, development delays lead to pressure to “make up time” in QA, which
compromises the thoroughness of testing. Second, critical defects that are identified
during testing further compress the time available to QA. Do you ever wonder how
products are released with critical defects?
Automated testing can help resolve the time crunch by reducing the time needed for
testing, decreasing the cost of testing, and testing more of the software. But there
is still a problem if automated testing is not deployed until the end of the project.
A poor customer experience, a drop in customer loyalty, and direct or indirect
financial losses are often the result of a project methodology where testing follows
development, instead of being an integral part of development.
This white paper explains how a test-driven strategy, one where test cases are created
using automated testing software in parallel with development, can help you find and
fix defects earlier, thus saving you both time and money and reducing the risk of
customer-facing defects. You will also learn how this strategy better prepares your
QA team for the quick changes and rapid regression cycles that occur when defects
are discovered. An example of time saved by a real Cyara customer is also included
to demonstrate the advantage of automated testing over traditional manual customer
experience testing.
BUSINESS WHITE PAPER
QA and Project Realities / Page 2
Benefits of automated
testing:
•
•
•
•
•
•
Testing starts earlier
Criteria is more objective
Saves time, up to 80%
Eliminates human error
Fix defect faster
Better defect
documentation
• Minimizes regression
testing
• Lowers risk of defects in
production
The Reality of Manual Testing
Many companies have embraced automated testing tools across the majority of
their IT infrastructure. Automated regression and load testing tools are available in
virtually any environment. But, this trend has been slow to take hold in customer
experience testing and many companies continue to rely on traditional QA practices,
particularly manual testing.
In most software development methodologies, QA’s work comes at the end of the
cycle, when the development is complete, and project timelines are established up
front before development begins. At that early point, it is difficult to estimate how
much time is needed for testing because the number of defects and their severity
is unknown. With manual regression testing, the project timelines are blown up
if there are many defects or many re-runs of the entire testing suite that can be
required for effective regression testing. As a consequence, projects extend beyond
the original delivery date. This additional time causes a delay and leads to frustration.
This is because the realities of cost and time narrow the scope of regression testing.
In a typical manual testing scenario, another risk factor is that when defects are
corrected, QA teams usually retest only the fix. Likewise, when existing products are
enhanced or changes are made, typically just that change or enhancement is tested.
If the “fix” or change affects any other systems or processes that weren’t tested,
there is a high risk of a customer-facing defect that will be discovered in production,
potentially causing downtime and service outages that hit your bottom line and
damage your reputation with your customers.
Relying on manual
testing not only
jeopardizes project
timelines, it also puts
quality at risk.
Performing QA in this traditional way near the end of the project also leads to
inefficient use of QA resources. In a typical project there is a waiting game. QA
has little to do while waiting on development to finish their work. Then, QA is
overwhelmed once development is done and there is a mad rush to the finish line
for the project. An ironic consequence is that often after QA has identified defects
and communicated them to development, QA can then end up playing the waiting
game again, while they wait for the development team to produce the fix.
QA and Project Realities
For many companies, product release dates are driven by factors far outside of
development and QA. Often things such as a promise to a customer, the need for
something new to drive sales, or pressure to keep up with competitors leads to
setting unrealistic project delivery dates that are unmovable. When reality hits with
a deadline that can’t be moved, project timelines become too short, and the release
date is not always able to accommodate all of the tasks in the project lifecycle.
When QA cycles are reserved for the end of the project, management looks to QA
to make up the time. Eliminating planned testing cycles essentially puts the entire
project at a heightened risk for errors that could be found by customers.
BUSINESS WHITE PAPER
QA and Project Realities / Page 3
Mid-project adjustments
to the testing plan that
reduce QA time in
response to schedule
pressures are the major
cause of critical defects
being released into
production.
This scenario is all too common in customer experience projects. By not carrying
out complete regression testing of the overall customer experience, the team is
making a decision to accept the risk of unanticipated impacts caused by the latest
enhancement. Typically, the QA team is fully aware of these risks, but there is no
other option. In many customer experience projects, the clock just runs out.
As systems age over time, the situation does not improve. With every project to
enhance or change existing systems, budget pressure continue to dictate that testing
be performed only on that enhancement. Manual regression testing is just too slow
and costly to thoroughly test the entire system, and, since it comes right before
release, there’s just not enough time for a complete testing cycle. Over time, the
limited, manual testing process all but assures that critical defects are released to
production and found by customers.
Automated Customer Experience Testing
There is an answer. Automated testing reduces the overall time needed to ensure
robust test coverage, and tests can be run throughout the entire project lifecycle.
Cyara offers a comprehensive automated customer experience testing platform to
support companies that want to implement capabilities more rapidly, ending the time
crunch and ensuring that the customer experience that is designed is the customer
experience that is delivered.
Recently, Cyara was approached by a major US airline that wanted to replace their
old voice self-service system with a new speech-enabled, sophisticated IVR to
deliver a uniquely pleasant and efficient customer experience. Anticipating the need
for multiple and ongoing regression testing cycles, the airline investigated automated
customer experience testing software to better utilize their QA team and ultimately
release a better product.
Results of Automation
During the first phase of the four-phase project, automation was not yet available
and the test cycle took two testers two weeks to complete. Automated customer
experience testing software was then introduced, and the phase one test cycle
was automated as a regression campaign including all of the functional test cases.
Running the regression campaign, it took one tester just four hours to complete
and validate the campaign. That’s four hours using automation instead of 160 hours
using manual testing as shown in the chart below.
Manual Regression Testing
Cyara Automated Regression Testing
2 Testers
160 hours
1 Tester
4 hours for identical task
BUSINESS WHITE PAPER
QA and Project Realities / Page 4
Thorough regression
testing of the customer
experience uncovers
defects that will not be
found by testing only
the “fix”.
Developing the regression test cases was an investment, but it paid dividends
not only over the balance of the project, but also for future QA testing changes
as well. The airline now has a full set of regression test cases, which they have
subsequently added to, to run against the product whenever new functionality is
added or changed. This thorough regression testing of the customer experience has
uncovered defects that would not have been caught if they had continued testing
only specific changes.
Benefits of Automated Customer Experience
Testing
The major benefit of automated testing for the airline has been the ability to
implement test-driven development. The airline’s QA team prepares and adjusts test
cases in parallel with development, which provides the following benefits:
• Functional tests are performed earlier in the project using exactly the same
objective criteria that will be used in final testing
• Automated testing is 80% faster than manual testing
• Defects are determined by the testing software based on agreed criteria
• Human error and subjectivity is eliminated in determining errors
• Time needed to fix defects is reduced because QA has more time to document
information on the defect to pass to development
• Time required to thoroughly regression-test the fix is minimal due to
automation
• Lower probability of releasing a critical defect to production
In addition, the airline is now in a position to move rapidly to adapt their customer
experience as business requirements and customer expectations change in the future.
The airline’s QA resources are now spending their time building test cases that
effectively ensure the design specifications are met, working with development to
ensure testing is available when capabilities are complete, and designing incremental
testing that can drive business results even higher. The majority of defects are now
found during development because those test cases are available sooner, resulting
in dramatically cleaner releases to user acceptance testing and periodic load testing
cycles.
BUSINESS WHITE PAPER
QA and Project Realities / Page 5
Conclusion
Automated customer experience testing supporting a test-driven development
strategy has resulted in tremendous benefits for this major US airline helping them
transform the IVR experience for their customers. It also enabled their critical need
to continue to evolve the customer experience to keep pace with business needs
while ensuring that they are releasing a higher quality product. They are now setting
the benchmark in the travel industry for delivering the finest caller experience,
helping them to achieve and maintain a leadership position in their industry.
About Cyara:
Cyara is the leading
provider of customer
experience testing for
organizations around
the world. The Cyara
Platform enables contact
centers and online
channels to optimize
the customer experience
rapidly and decrease
the risk of exposing
customers to a poor
experience. Every day,
millions of flawless
customer interactions are
delivered by organizations
utilizing the Cyara
Platform.
Many Fortune 100 companies trust Cyara and automated testing for their contact
centers. If you are planning a significant transformation of your contact center or to
quickly bring customer experience improvements to your customers, Cyara is here to
help.
www.Cyara.com
Corporate Headquarters: Level 1 / 1 Cook Street Hawthorn, VIC 3122 | AUSTRALIA
Tel. +61 3 9093 0815 | www.cyara.com
Cyara and the Cyara logo are trademarks or registered trademarks of Cyara Solutions, Pty Ltd. or its affiliates in the United
States of America and/or other countries.
All other company names or product names may be the trademarks of their respective owners.
© 2014 Cyara. All rights reserved.