• Cognizant Reports Why On-Demand Provisioning Enables Tighter Alignment of Test and Production Environments Executive Summary Steve Smith looks like a worried man, moving restlessly in his chair and making frantic phone calls late on a Friday evening, inquiring about what went wrong with the software his team had developed. All seemed well during testing, but errors surfaced as soon as the application was deployed, leaving Steve and his team in a quandary. Tales such as Steve’s abound in the software testing arena. From banking applications to mobile phones and airport systems, software defects affect organizations’ business-critical and customer-facing applications, impacting their daily business operations and resulting in huge losses. According to the National Institute of Standards and Technology (NIST), software flaws cost the U.S. economy a whopping $60 billion annually. One of the primary reasons for such unpleasant situations — in which errors are discovered in production or only after end-users are affected — is low-quality testing, often impacted by poorly built test environments. Test environments rarely match production environments due to the huge cost and effort involved in mimicking them. Further, issues such as the time taken to build test cognizant reports | november 2013 environments, unplanned environment downtime and inaccurate configuration of test environments impact testing effectiveness. These result in application roll-out delays and test managers signing off on projects with caveats, leading to applications issues once put into production. Organizations are currently exploring various ways to mitigate these risks. Attempts to use virtualization to overcome test environment challenges have met with limited success. Also on the rise is “on-demand provisioning” on the cloud and traditional environments, as well as build and deployment automation. Our experience shows that cloud-based test environments offer better results by providing testing teams with on-demand capacity. Cloud-based offerings also enable organizations to generate standard builds that are more closely and cost-effectively aligned with real production environments. On-demand provisioning has much to offer organizations seeking to instill greater discipline in managing builds and elevating code quality. Application quality depends on test environments that support functional testing requirements; moreover, it is also important to have a production-like environment to complete the testing process without an exception. This synchronization of test and production environments is achieved by building a service framework around the test environment, supported by cloud-based computing grids. This approach, in our view, has already delivered great benefits to organizations that are piloting and enacting fullfledged on-demand test environment strategies. Test vs. Production Environments According to a 2008 Aberdeen study, only 42% of over 200 organizations surveyed were happy with the performance of their business-critical applications.1 Primary reasons for this include the inability to identify issues before end-users are impacted and the challenges of testing application performance in pre-production stages (see Figure 1). Sadly, our experience with companies across industries indicates that not much has changed. At a time when companies are laser-focused on operational cost-cutting and time-to-market acceleration, software defects that emerge in production environments can severely undercut financial performance and cause reputational damage. With applications becoming increasingly complex, the scope of testing has expanded tremendously, creating unforeseen challenges for testing teams. For instance, to assess their likely performance in real-time, Web and mobile applications must be tested for multiple operating systems and updates, multiple browser platforms and versions, different types of hardware and a large number of concurrent users. Further, in the case of new applications, it may not be possible to predict all the likely scenarios in which the application will be deployed. The conventional approach of manually creating in-house testing environments that fully mirror these complexities and multiplicities consumes huge capital and resources. This has caused application developers to take the risky route of conducting testing in their production environments. However, the Sarbanes–Oxley Act, as well as other regulations, restricts access to production systems, which has forced organizations across industries to conduct testing in separate environments. Further, it is not uncommon for organizations that cannot afford large-scale test environments to use sizing/ extrapolation techniques during performance testing to understand how an application behaves in the real world. This approach, however, does not guarantee success. Poorly built test environments often limit the ability of testing teams to test applications for various scenarios. Test environments differ from production environments in terms of OS, patch levels, software version, configuration, etc. The wider the gap between test and production environments, the greater the chance of an application misbehaving once deployed or for a defect to leak into live systems. The following are among the most common issues that afflict test environments: Creating test environments can be time-consuming and delay-prone. It can sometimes take weeks or month to set up a simple test environment due to delays in Top Challenges for Application Performance Management Inability to identify issues before end-users are impacted 57% Increase in complexity of applications 51% Inability to measure SLAs around application performance 37% Inability to test application performance in pre-production stages 34% Increase in complexity of network traffic 32% Source: Aberdeen Group Figure 1 cognizant reports 2 finding servers, procuring licenses for programs and testing tools and installing them. Roughly 30% of defects arise due to inaccurate test environment configurations, studies show. Further, configuration changes made in response to errors often go untracked, causing errors when applications are moved into production. Scheduling of the test environment is crucial, especially when environments are shared by two projects. If not properly managed, this can lead to unnecessary delays. Testing teams often clone production databases and write extract scripts to create test data. This approach is resource consumptive and error prone due to a lack of audit trails. Simulating the production issue in the test environment to identify the root cause is challenging due to a lack of test environment alignment with production versions at all system component layers. Test environment downtime and changes are often not controlled and communicated properly. This poses challenges in managing testing projects and achieving KPIs. Lack of a single view makes the environment less flexible to support changes due to code modifications during the test cycle. Further, it is difficult to tune a complex test environment to suit the advancements in software development methods such as Agile. The test environment refresh is not considered while planning for the test project execution phase. This narrows the opportunity to maintain an up-to-date test environment. Loosely managed asset control necessitates undue investments in the infrastructure. Inadequate access control reduces confidence in the state of the environment and its configuration control. Together, this adds to the cost of the project and to the risk-based sign-off of the test plan. Automation: Production systems have adopted automation to a great extent. The non prods lacked automations relatively due to the higher cost of automation tools. Build/ deployment and orchestration solutions have greatly improved the test environment build timelines. This has also brought in agility features to test environments. The Impact Test environment issues can impact delivery schedules and result in the risk of subsequent software failures, as well as additional business expenditures. It is estimated that 40% of the effort involved in a software development lifecycle goes toward managing defects that arise due to environment-related issues.2 The ever-present pressure to meet service level agreements (SLAs) can force IT managers to resort to risk-based sign-off on a product/application release. This means that the product/application is tested for only those risks that have a high probability of occurrence. This risked-based sign-off can have the unintended consequence of putting the organization’s reputation and financial condition at risk (i.e., errors that cause customers to question product quality — or, worse, cause monetary harm — can be grounds for lawsuits). Further, with the cost of fixing defects in production being significantly greater than in the early stages (see sidebar below), testing teams Cost of Fixing Defects According to a study by dynaTrace on application performance management, about 50% of software problems are discovered in production only after they have affected end users and that resolving these issues can involve huge costs.3 Business losses from system downtime can exceed $1 million per hour. The majority of defects creep into code during the early stages of the development lifecycle. The longer a defect goes undetected, the greater the costs of fixing it. According to studies, the ratio of the cost to fix a defect at different stages of a project is 1:10:100. This means the cost of fixing a coding defect in production is 10 times the cost of fixing it during testing and 100 times the corresponding cost of fixing it during the design or requirements phase (see Figure 2, next page). According to Gartner, “The cost of fixing defects ranges from a low of approximately $70 (cost to fix a defect at the requirements phase) to a high of $14,000 (cost to fix a defect in production).”4 cognizant reports 3 Index of the Relative Cost of Fixing Defects 110 90 70 50 30 10 0 Requirements Design Code Test Production Source: “The Economics of Testing,” www.riceconsulting.com Note: Analysis based on base cost of fixing a defect in the requirements and design stages, measured as one unit (hour, dollar, etc.). Figure 2 must ensure that defects do not pass through to later stages. On-Demand Provisioning The criticality of business applications for today’s organizations goes without saying. As such, delivering quality applications requires a proper test strategy complemented by an efficient test environment. Testing helps minimize errors but does not guarantee their removal. Organizations strive to deliver quality applications by reducing the majority of defects through rigorous testing while preparing themselves to fix minor bugs during production. In the case of a new version of an existing application, organizations must ensure that the new version works without disturbing the current systems environment. This creates the need for a production-like test environment, which might not offer the same processing capacity but delivers the same functionality and enables all required components to interact as they would in a production environment. Organizations have tried numerous options to improve the test process. Recently, system virtualization has provided some respite, but it is not a panacea. It has met base-level operational and financial objectives by eliminating the need for large capital investments in hardware, software and related infrastructure; however, given the requisite set-up costs, many pioneering companies have yet to achieve the operational flexibility and scalability required to deliver the forecasted productivity and ROI on assets. cognizant reports We believe that on-demand provisioning of test environments via the cloud offers an effective solution for achieving better ROI on assets. Cloud computing minimizes the volume of issues that plague current approaches to test environments. Cloud-based testing leverages on-demand computing grids that can be more easily and flexibly deployed, eliminating the need for the upfront capital expenditures (Cap-Ex) required to procure hardware and software. The pay-peruse model provides several benefits, including providing testing organizations with freedom from owning assets; enhanced collaboration among testers, developers and managers; greater levels of efficiency; and, importantly, reduced time-to-market for key business applications. These benefits are delivered by leveraging access to cloud-based resources over the Internet without compromising security requirements. The pursuit to reduce time-to-market and the expense of application releases is often hampered by huge costs and long delays in setting up an appropriate test environment. Production-like environments at reduced costs can be achieved through on-demand provisioning, combined with efficient processes and governance that creates the discipline to develop quality build images and version control. To a large extent, the testing model can also eliminate configuration errors. This will help organizations achieve their test plan objectives 4 and minimize defect leakages when applications enter production due to incorrect test environment setup/configurations. Organizations would do well to combine on-demand provisioning with industry best practices like shift-left (see sidebar below) to improve the overall quality of software and reduce the cost of application development, maintenance and testing. Shift-Left Practice Finding coding flaws early in the development lifecycle is one way to reduce the number of defects that surface in production. Shift-left is a practice where efforts are made to identify defects early and fix them before submitting the code to a testing team. It also includes sharing the project’s progress with clients at regular intervals, which ensures that the project is heading in the right direction. This relieves testing teams from incurring additional costs and effort required to fix defects at later stages, when teams are hard-pressed for time. The process can be time-consuming, and care must be taken to ensure it does not cause unnecessary delays in product development. Automating tests and running automated tests with every build can help in identifying bugs quickly. Footnotes “Application Performance Management: The Lifecycle Approach Brings IT and Business Together,” Aberdeen Group, June 30, 2008. 1 Julian Brook, “Testing Environment Management Challenges With Virtualisation,” ComputerWorldUK, Sept. 6, 2008. 2 Alois Reitbauer, “Is There a Business Case for Application Performance?” Dynatrace, May 26, 2010. 3 4 “Hype Cycle for Application Development, 2011,” Gartner, July 27, 2011. Bibliography Martin Perlin, "Notes from the Trenches: Obstacles and Challenges to IT Environment Stability," www.evolven.com, October 2011. Wayne Ariola, "The Next Generation of Test Environment Management," Virtualization Journal, July 12, 2011. "Maximizing the Value of Good Testing Practice in an Agile Environment: Delivering On Time, In Scope, On Budget and At the Tight Level of Quality," Sogeti, September 2010. “Is Shift-Left Agile? And Death by Build,” AgileFall, March 22, 2010. Vinaya Kumar Mylavarapu, “Taking Testing to the Cloud,” Cognizant, March 2011. "Official Fix for the Zune 30 Fail," www.gizmodo.com, December 31, 2008. Bojan Simic, "Application Performance Management: The Lifecycle Approach Brings IT and Business Together," Aberdeen Group, June 2008. "Improving Software Quality to Drive Business Agility," IDC, June 2008. "The Economics of Testing," www.riceconsulting.com, 2001. cognizant reports 5 Author and Analyst Vinaya Kumar Mylavarapu, Cognizant Research Center Subject Matter Expert Saravanan Kumar, Associate Director, Integrated Services Leader Navin Rao, AVP, QE&A About Cognizant Cognizant (NASDAQ: CTSH) is a leading provider of information technology, consulting, and business process outsourcing services, dedicated to helping the world’s leading companies build stronger businesses. Headquartered in Teaneck, New Jersey (U.S.), Cognizant combines a passion for client satisfaction, technology innovation, deep industry and business process expertise, and a global, collaborative workforce that embodies the future of work. With over 50 delivery centers worldwide and approximately 166,400 employees as of September 30, 2013, Cognizant is a member of the NASDAQ-100, the S&P 500, the Forbes Global 2000, and the Fortune 500 and is ranked among the top performing and fastest growing companies in the world. Visit us online at www.cognizant.com or follow us on Twitter: Cognizant. World Headquarters European Headquarters India Operations Headquarters 500 Frank W. Burr Blvd. Teaneck, NJ 07666 USA Phone: +1 201 801 0233 Fax: +1 201 801 0243 Toll Free: +1 888 937 3277 Email: [email protected] 1 Kingdom Street Paddington Central London W2 6BD Phone: +44 (0) 207 297 7600 Fax: +44 (0) 207 121 0102 Email: [email protected] #5/535, Old Mahabalipuram Road Okkiyam Pettai, Thoraipakkam Chennai, 600 096 India Phone: +91 (0) 44 4209 6000 Fax: +91 (0) 44 4209 6060 Email: [email protected] © Copyright 2013, Cognizant. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the express written permission from Cognizant. The information contained herein is subject to change without notice. All other trademarks mentioned herein are the property of their respective owners.