EBS TESTING: HOW TO GET YOUR BUSINESS USERS ON-BOARD AMIR M. FARHI, PANAYA INC. ……………….………………………………………………… ABSTRACT: Are you getting the cooperation you need from your business users? Do you have real-time visibility into your key users' test activities? Do you have all the test evidence needed to pass an audit? This paper explains how you can reduce 70% of the time it takes to create and execute an Oracle® E-Business Suite (EBS) test plan, make it easy for your entire testing team (IT & Business) to manage and track all their testing activities and improve the collaboration between IT and key users. INTRODUCTION Oracle® E-Business Suite systems must evolve at the speed of business change. It is therefore imperative for Oracle E-Business Suite managers to adapt their EBS landscapes in a timely manner to address business needs such as process changes, rollouts to new sites, and regulatory updates. In addition, they must cater to the maintenance needs of their EBS systems and implement EBS support packs and enhancement package upgrades on a regular basis. Oracle E-Business Suite systems tend to be highly complex, built on hundreds of millions of lines of standard code and customizations. This means that even a small change to the system could trigger a series of inadvertent consequences. To prevent production failures, companies regularly perform a series of functional test cycles – which can be executed manually and/or automatically. Industry papers discussing approaches to test automation have been available for well over a decade. Little has been discussed, however, about the applicability of test automation to Oracle E-Business Suite. This document describes the specific challenges of EBS functional testing, when test automation is inappropriate, and how to perform manual testing faster, better, and at a lower cost in these cases. -1- WHY IS AUTOMATED TESTING SUCH A CHALLENGE FOR ORACLE E-BUSINESS SUITE? A survey performed by Panaya in 2012 of 100 enterprises revealed that even the most mature testing organizations perform only 15% to 25% of their functional regression testing using test automation tools; the remainder is manual. There are three main reasons, inherent to Oracle E-Business Suite, which explain why test automation coverage is relatively low: (i) Oracle E-Business Suite functionality is heavily data driven. EBS business logic is dependent on and functions differently with change in the input data; this translates to changes in screen flow, screen content and screen format that must be factored into the test automation script. (ii) For business processes that change frequently, the test automation scripts must be continually kept up to date. A challenging and costly proposition. (iii) Preparing the required test data for test automation is also challenging due to EBS’s complex data structure and the various dependencies that exist in an EBS business process. Think about an automatic test script for a procure-to-pay process that requires five different material types for full testing coverage. Maintaining integrity of the data required to support the end-to-end process for the five material types is a daunting task. EBS MANUAL TESTING IS UNDERSERVED Since it’s virtually impossible for a company to develop automation test scripts to cover every possible EBS business scenario, manual testing is heavily relied upon to ensure that all the needed functionality is covered. Yet manual testing is severely underserved in terms of tools that increase productivity, enforce processes, and provide control. Performing manual tests requires an intimate understanding of the business processes to be tested, and so, it is typically the functional experts in IT or key business users (sometimes referred to as “super users”) that are tasked with performing manual testing. They do their best to test, report defects, report progress, and document test runs, on top of their other daily duties. This is all performed manually, typically with tools such as Microsoft® Word and Excel. In organizations where a dedicated test manager is allocated to the EBS team, management of the testing cycle is performed by the test manager. In the majority of EBS shops, however, there isn’t a dedicated test manager; the test manager’s role is performed by project managers, in the scope of their respective projects, as an added duty. Test management includes defining the testing strategy, identifying the tests to be performed, assigning tests to the manual testers, chasing the testers for their individual status reports and consolidating them to a “bigpicture” testing progress and coverage status, managing defects, ensuring that testers perform the testing on time, and more. Approximately two thirds of EBS shops use general productivity software like spreadsheets in support of these test management activities. It is little surprise, therefore, that they lack visibility and control over the testing cycle, and put too much time and effort into managing the testing activities. -2- DECIDING WHAT TO TEST: COST VS. RISK The first pain point comes early in the testing cycle: determining what, exactly, to test. Following a technical or functional change in a massive application like Oracle E-Business Suite, it’s a challenge to determine where exactly in the system testing should be focused. Customers have two options. One is to try to test every possible EBS transaction that may be affected to minimize risk of production failures. That, of course, increases testing time and cost. The other alternative is to essentially ‘guess’ where the greatest impact will be and test those areas. The second option comes with lower costs but much more risk. In practice, most organizations seek a happy medium between the two extremes, but the costly nature of manual testing makes many lean toward less testing. Finding a method to test more functionality within a strict timeline is one of the biggest challenges associated with implementing changes in the ERP application, so finding a cost-effective resolution is extremely valuable. -3- CREATING TEST SCENARIOS FOR SCRIPTED TESTING Test scenarios describe the steps needed to test a given function or process and the anticipated results of every step. In theory, this should work well: each manual tester has a written script telling him or her what to do, how to do it, and what results to expect. But in practice, this is far from the case. Typically written in MS-Word or Excel, test scripts contain varying levels of detail, with no standardization as to what should and should not be included. It’s a time-consuming process, as each script can take from minutes to several hours to create. And there’s no guarantee that others will be able to reliably follow the script; what’s clear to whoever wrote the script may not be to those charged with following it. UNSCRIPTED “EXPLORATORY” TESTING In some cases, test scenarios don’t exist at all. Instead, organizations practice a form of ad-hoc testing, known as exploratory testing, where manual testers are tasked to perform a test as they see fit and report results. There are two main use cases for exploratory testing: 1. The company doesn’t have pre-defined scripts and there is no time to create them. Therefore, manual testers run a test cycle based on their knowledge of the way the application is used. While this reduces test preparation time, it adds risk that the business process will not be tested adequately. 2. The QA team or functional testers create test scripts and conduct tests based on these scripts. At the User Acceptance Test (UAT) phase the test manager doesn’t want the key user to repeat the exact same scripts, because they were successfully tested already. Furthermore, the test manager needs to ensure the application is tested exactly as the business users will use it in their daily work, so he asks the user to simply do their day-to-day tasks on the QA system and check that everything works properly. This method provides an additional assurance for the quality of the application. Problems may arise when exploratory testing uncovers issues that developers must address. With no record of what, exactly, the tester did, it’s difficult for the developers to accurately recreate problems that need fixing. What’s more, there’s no knowledge transfer of the testing process from the business user to IT to enable IT to create test scenarios for future use. CAPACITY OF BUSINESS KEY USERS Testing takes the business key users away from their pressing day jobs and limits their ability to fulfil their other duties, such as supporting end-users, delivering training, defining processes, or liaising with IT regarding functional needs. These key users are typically among the most experienced and highest paid in the user organization and have very little time to spare. Since the key users are forced to fit test activities around their normal workload, testing often has to wait. DOCUMENTING TESTS PERFORMED At the same time, these time-strapped key users are often required to collect evidence to prove whether or not these tests were successful. This is especially true for organizations subject to government regulations, such as Sarbanes Oxley (SOX), U.S. Food and Drug Administration (FDA) rules, and others. Users must collect detailed proof of what was tested, by whom, and how. It can be a painstaking, time-consuming process for testers to constantly take screen shots, then copy and paste them into a document to show which tests were performed. This process can take at least 20 to 30 minutes per test, which translates into several days of non-productive work per test cycle per key user. -4- MANAGING THE MANUAL TESTING PROCESS Finally, when testing is manual, the testing manager has no real-time visibility into the process. He can distribute detailed instructions to testers but has no way of knowing if, when, and how they are conducting the tests. As a result, s/he cannot unequivocally know the level of progress being made and the precise issues which have cropped up. When testers are located at multiple sites, the challenge becomes all the greater, forcing the manager to resort to emails, spreadsheets and phone calls to keep tabs on testers. OPTIMIZING THE MANUAL TESTING PROCESS So it’s clear that manual testing will always be required; there’s simply no way in sight around having experienced users verify that the system functions as it should. However, many aspects of the testing process can and should be accelerated and managed. These include:  Determining what to test (test scoping)  Creating detailed test scenarios  Executing manual tests  Reporting defects that can be easily reproduced  Documenting test run evidence  Tracking test run progress In short, manual testers require better tools to make the process more efficient, accurate – and less painful. Testing managers need visibility and control over manual testing cycles. This visibility will enable organizations to conduct more testing in the same time window and hence, find more problems before the code goes into production. Increasing testing efficiency also has the effect of driving down the cost of testing and potentially speeding up project implementation, thus increasing business agility. EBS IT teams seeking a solution to accelerate the different facets of the manual testing process should evaluate manual testing solutions on the basis of the following attributes: 1. AUTOMATED TEST SCOPING OF EBS-DRIVEN CHANGES Determining which functionality of the Oracle E-Business Suite to test following a major change – especially when implementing EBS-driven changes, such as patches, upgrades or custom code releases – is crucial to minimizing the risk and cost of change. Analyzing the production usage of your EBS transactions combined with an impact analysis of the changes on your system can yield a risk-based approach to test prioritization that can reduce testing efforts by as much as 70%. 2. ACCELERATED CREATION OF TEST SCENARIOS Writing test scenarios is a painstaking process but it’s also one that’s well suited to automation. Your testing tool should capture the business process to be tested while a functional expert or key user navigates the EBS application. The testing tool should translate the steps into structured human-readable test scenarios, including documentation of expected results for every step of the scenario. You’re essentially capturing the knowledge of that key user and making it available for all others to emulate in their own testing. At the same time, you ensure the scripts are highly accurate – but take far less time to create. -5- 3. ACCELERATED EXECUTION OF MANUAL TESTING Similarly, your testing solution should ease the burden on testers by offloading many manual tasks associated with testing, such as walking the testers through the steps required when manually executing the test scenarios. 4. ACCELERATED DOCUMENTATION OF TEST RUNS Another aspect of test execution is the process of capturing and documenting test evidence for every test run. Seek a testing solution that enables you to determine exactly which process the tester followed, what the results were and how those results compare to what was expected. This is a crucial step in terms of providing proof of testing for auditors, as well as tracing production failures to the gaps in your test scenarios. It also relieves testers of the tedious process of manually documenting test results. 5. ACCELERATED REPORTING OF DEFECTS The ability to effectively report functional failure is crucial in enabling IT address issues uncovered by testing. Developers should get an accurate, screen-by-screen and keystroke-by-keystroke depiction of how the tester reached a problem point, so they can recreate and fix the problem. This is exactly the kind of data that testers struggle to capture and communicate accurately. 6. DESIGNED AND BUILT FOR BUSINESS USERS Remember your key business users will ultimately have to use the testing tool as part of their (almost) daily routine. They are already pressed for time; the last thing they want is to spend time learning how to use a testing tool that they never wanted to use in the first place. So they need seamless access to the testing tool from their EBS application. It should require minimal, if any, training to perform such testing activities as: viewing the assigned test tasks, executing the test tasks, testing in collaboration with other -6- testers to complete a full business process, and reporting defects and progress. The tool should not require complex installation of client-side software, and must be intuitive and simple to use, without interfering with the non-testing tasks of the key user. 7. TEST MANAGEMENT Finally, your manual testing solution should provide you with ongoing visibility and governance over the end-to-end testing process. You should be able to plan test projects, assign tasks to all manual testers, determine how close the testers are to completing their assigned tasks, and receive alerts when problems arise or steps are completed. The solution should also enable you to track a distributed multi-site testing operation. 8. SIMPLE SET-UP, MINIMAL ONGOING MAINTENANCE You need a solution that doesn’t require cumbersome software installations, frequent patching or painful upgrade processes on the manual testers’ client machines. It should take IT less than a day to set up the full solution. KEEP UP THE PACE The pace of business change isn’t likely to slow down, so you can’t expect the Oracle E-Business Suite testing burden to diminish either. Organizations that take the initiative to optimize their manual testing processes will ease the burden on manual testers, enabling them to spend less time testing while allowing IT to complete more projects. This ultimately empowers the company to become more agile and competitive. Learn how Panaya can help you optimize your EBS testing processes. Visit us at: WWW.PANAYA.COM. -7- ABOUT PANAYA Panaya Quality Management Cloud (QMC) for SAP, Oracle E-Business Suite, and Salesforce combines change impact simulation, automated code remediation, collaborative test management and test execution, providing companies with the agility to address their constant flow of business application changes. IT departments use Panaya QMC to easily scope and accelerate the delivery of any business application change (such as upgrades, patches, chart of accounts changes, etc.), while minimizing costs and eliminating risk. More than 1,000 customers worldwide rely on the Panaya Quality Management Cloud, using “cloud wisdom” from thousands of projects, transforming common practices into best practices. DISCLAIMER AND TRADEMARK NOTICES This report is provided by Panaya Ltd. It is completely independent of and not affiliated with Oracle or SAP AG. Oracle is a registered trademark of Oracle. Oracle and other Oracle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Oracle. SAP is a registered trademark of SAP AG. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries. All other product and service names mentioned are the trademarks of their respective companies. DISCLAIMER OF WARRANTY Panaya Ltd. makes no representation or warranties, either express or implied by or with respect to anything in this document, and shall not be liable for any implied warranties of merchantability or fitness for a particular purpose or for any indirect special or consequential damages. COPYRIGHT NOTICE No part of this publication may be reproduced, stored in a retrieval system or transmitted, in any form or by any means, photocopying, recording or otherwise, without prior written consent of Panaya Ltd. No patent liability is assumed with respect to the use of the information contained herein. While every precaution has been taken in the preparation of this publication, Panaya Ltd. assumes no responsibility for errors or omissions. This publication is subject to change without notice. © 2014 Panaya Ltd. All rights reserved. -8-