1
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Managing Testing for a Major
Oracle Application and
Database Upgrades
Peter Yu, Senior Director, iTech
Oracle Asia-Pacific & Japan
Ruihua Ji, Senior Product Manager,
Oracle Application Testing Suite
The following is intended to outline our general product direction. It is intended
for information purposes only, and may not be incorporated into any contract.
It is not a commitment to deliver any material, code, or functionality, and should
not be relied upon in making purchasing decisions. The development, release,
and timing of any features or functionality described for Oracle’s products
remains at the sole discretion of Oracle.
3
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Program Agenda
 Foreword
 Overview of Products Covered
 Introducing Acolade Consulting
 Case Study 1: DPI Shared Services
 Case Study 1 Demo: Managing the Testing of EBS11i
4
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Program Agenda (continued)
 Case Study 2: Upgrading to Database 11g
 Summary
 Q&A
5
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Quality is key….
• In todays highly competitive market place
the competitive advantage is your
processes and the applications you are
using to support them.
• Packaged applictions are complex both
from a infrastructure and software point of
view.
• Poor application quality will reduce or
eliminate the competitive advantage that
the applications intended to provide...
6
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Oracle Application Quality Management
High quality testing for all tiers of application stack
• Unit, functional, and performance testing
• Heterogeneous applications support
Infrastructure Testing
• Optimized database tier response time and load testing
• 10x-15x reduction in testing effort
• Heterogeneous applications support
Application Performance Management
• Rich performance monitoring and tuning from application to disk
Secure Test Data & Lab Management
• Test system provisioning and management while maintaining data
security & compliance by obfuscating sensitive information
Test Management
• End to end management for entire testing process from
requirements to test case tracking to bug tracking
7
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Quality Mgmt. for Entire Application Stack
Application Testing
Custom
Apps
Prepackaged
Apps
Composite
Apps
Middleware
Database
Physical & Virtual Servers
Storage
eCommerce
Apps
Proper Approach to Quality Management
Middleware
DBMS
Storage
Application Replay
Infrastructure
Infrastructure
Testing
toolsTools
Testing
Real Application Testing
ApplicationTesting Suite
Application
Testing tools
O/S
Application Infrastructure Changes
Application Changes
• Test infrastructure changes separately from application changes and
test one change at a time
• Specialized tools can result in much higher quality and efficient testing
• Proper test systems and test data mandatory for good testing results
8
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Overview of Products
Covered
9
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Oracle Application Testing Suite
Oracleintegrated
Functional Testing
 Powerful,
solution for ensuring application quality.
Oracle Test Manager
10
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Oracle Load Testing
Insert Information Protection Policy Classification from Slide 12
Oracle Test Manager
Bring structure, repeatability and visibility to testing efforts
• Five integrated Test Modules provide
complete test process management
Design Test
process
Manage Test
Requirements
Manage/Define
Test Cases
Manage
Test Executions
Document &
Track Defects
 Test Plan, Test Requirements, Test Cases, Test
Execution and Issues (Defects)
• Web-based interface
 Runs within Weblogic Server
View
Reports
 Common repository for all assets
 Stores data into Oracle database
Document Test
Requirements
File defects
found by test
execution
 Manual and automated Test cases
Centralized
Repository
(Oracle DB)
 Test manually or run or schedule OpenScript scripts
 User-defined roles and fields
 Increase visibility of app readiness
 Pre-packaged reports & dashboards
11
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Plan
Tests
Insert Information Protection Policy Classification from Slide 12
Execute
Tests
Write Test
Cases
New Test Execution Module
Key Benefits
 Dedicated module for test
execution allows better
overview and structuring of
test execution
 Simple re-execution for
skipped or failed tests within
a TestSet
 Support for manual and
automated tests
12
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Integration with JDeveloper
Unit Testing from within Oracle Test Manager
Key Benefits
•Support for Unit testing (single unit tests
or groups of unit tests)
•Supports full Development Life Cycle
from Dev -> QA
•All tests centralized with unified reporting
Junit Test
Oracle Test Manager
13
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Oracle Functional Testing - OpenScript
• Automates functional & regression testing
for Web, SOA and packaged applications GUI View: Standard - Easy
• Also a load testing script generator
• Custom Accelerators for Oracle Packaged
Applications (eBS, Siebel,JDE, Fusion/ADF,
etc) & Web
Adobe Flex)
technologies (Webservices,
Java Code View: Technical Extensible
• GUI-based visual scripting interface
• Flexible scripting extensibility using Java
• Integrated Eclipse IDE for script
development & debugging
• Integrated with Oracle Load Testing &
Oracle Test Manager
14
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Eclipse View: Advanced - Optional
Oracle Load Testing
Oracle Load Testing is an easy-to-use scalability testing tool that lets users quickly
eliminate performance bottlenecks in Web application.
• Realistic load testing for Web, SOA and
packaged applications
• Scales up to thousands of concurrent
users to simulate peak production loads
• Delivers better accuracy by performing
functional content validation under load
Configure scripts as a part of Load
Testing Scenario
• Integrated server monitors help identify
performance bottlenecks
• HTML based Session report & Flex
based Graphical reports for Real time
and Post-Runs
Create Load
Testing Scripts
In OpenScript
15
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Flex Based Reports for
runtime and post runs
12.2 New Features and Updates: OLT Hardware Estimation
 Oracle Load Testing Hardware Estimation:
16
–
Provides an estimate of the hardware requirements for a given scenario.
–
Generates report that determines how much h/w will likely be required to run the specified load test
scenario,
–
and how to modify the session configurations to run more Virtual Users on one machine.
–
Supported load testing modules in 12.2: Web/HTTP, Oracle Fusion/ADF, Oracle Siebel, Oracle EBS/Forms.
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
12.2 New Features and Updates: OLT Support for Functional Test Scripts
 OLT Support for Functional Testing Scripts
–
Playback functional testing scripts as part of an Oracle Load Testing application scenario.
–
Linux w/ VNC and FF browser/ Windows 2008 R2 w/RDP service required
Using the feature you can..
View Functional
Scripts running in a
RDP window
Functional Testing
Scripts show up in
Blue icon
Load Testing Scripts
show up in Green icon
17
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
• Run multiple Test Cases
concurrently, in a single agent
machine. (vs. OTM runs
sequentially, or need to distribute
them to the multiple agent
systems)
• Measure end-user response
time (such as Stop watch Time)
by running FT script in a load
scenario
• Narrow down to find a cause of
VU failure (OLT agent vs.
application) by running FT script
in a load scenario
Optimized testing for Oracle Applications
OpenScript Accelerators (v.12.2)
Oracle Fusion Applications
Oracle Ebusiness Suite
Siebel
• Accelerators are the
script templates enable
better recording &
playback for specific
Oracle Package
Applications
JD Edwards Enterprise ONE
PeopleSoft
Hyperion
Functional Testing Modules: Record &
Playback Application Specific Components
Load Testing Modules: Correlate Application
Specific Dynamic Parameters
Accelerators
EBS/
FORMS
Siebel
Web /
HTTP
Web /
HTTP
Fusion/
ADF
Web /
HTTP
JDEOne
Web /
HTTP
Hyperion
Web /
HTTP
Eclipse Platform
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Web /
HTTP
Database
OpenScript Framework
18
PeopleSoft
Insert Information Protection Policy Classification from Slide 12
Web
Services
Web /
HTTP
Adobe ®
Flex ®
• Pre-configured APIs and
correlation libraries
specific to the selected
application
• Built on top of the
Web/HTTP module to
provide additional
capabilities
• Reduce manual scripting
work by introducing
more automation in the
script creation
Optimized testing for Oracle
Test Accelerators and Test Starter Kits
Test Accelerator extends the web testing functionality
Example EBS:
 Support for EBS 11i & R12 (socket & servlet mode)
 Supports both FORMS and Web actions (back-office/self service)
 EBS specific object identification rules
 EBS specific correlations
 Automatic test case creation
 EBS Test Starter Kit (~35 pre-built
scripts)
19
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Oracle Real Application Testing
• Database Replay
• Concurrent load testing using
production workloads
End-to-End Workflow
Capture
Workload
Create Test
System
• SQL Performance Analyzer
(SPA)
• SQL unit testing
• Identify & tune regressed SQL
20
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Replay
Workload
Insert Information Protection Policy Classification from Slide 12
Deploy Replay
Clients
Real Application Testing with
Database Replay
• Replay production workload in test environment
• Identify, analyze & fix potential instabilities before making changes to production
• Capture Workload in Production
• Capture full production workload with real load, timing & concurrency
characteristics
• Move the captured workload to test system
• Replay Workload in Test
• Make the desired changes in test system
• Replay workload with full production characteristics
• Honor commit ordering
• Analyze & Report
• Errors
• Data divergence
• Performance divergence
Analysis & Reporting
21
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Database Replay
Customer Usage Scenarios
Client
Client
Unsupported
Changes
…
Client
Middle Tier
Usage Scenarios
•
•
•
•
Database Upgrades
Patches
Schema and parameter changes
Configuration changes e.g. Single
node to RAC, ASM, compression,
partitioning
• Storage, network or Interconnect
changes
• Changes to OS platforms,
upgrades, CPU or memory
22
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Storage
Insert Information Protection Policy Classification from Slide 12
Recording of
External
Client
Requests
Oracle Real Application Testing
SQL Performance Analyzer (SPA)
SQL Plans + Stats
Pre-Change Trial
SQL Workload
STS
•
•
•
•
•
23
SQL Plans + Stats
Post-Change Trial
Compare SQL
Performance
Analysis Report
Test and predict impact of system changes on SQL query performance
Analyze performance changes for improvements or regression
Comprehensive performance analysis and reporting
Re-execute SQL queries in test environment
End-to-end solution: STS, SQL Plan Baselines, & SQL Tuning Advisor
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Oracle Real Application Testing
SQL Performance Analyzer (SPA) – customer usage scenarios
Changes in SQL execution plans such as
 Database upgrades
 Data patches
 Initialisation parameter changes
 New index creations
 Optimizer statistics refresh
 Schema changes such as partitioning or material views
 Configuration changes to Operating System, hardware or database
24
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
SPA Report Example
3
2
5
1
4
25
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Customer Reference: DPI Shared Services
“Oracle Application Testing Suite enables us to plan and implement system tests, and produce related reports
and statistics, faster, more efficiently, and in greater detail.”
Kim McGinnes, Manager, Change & Communication
 Company Overview
–
 Application Testing Suite Benefits
The Department of Primary Industries provides an integrated
HR, Payroll and financial support service to 5 agencies
/departments within the Victorian Government. DPI is one of
11 Victorian Government (Australian) departments, reporting
to the Minister for Energy and Resources and Minister for
Agriculture. We employ more than 2,600 staff who work from
82 locations across the State.
 Challenges
–
–
–
26
Replace decentralized, overly time-consuming financial,
project management, and HR system testing processes with
consolidated, automated planning and testing systems
Provide a solid testing infrastructure from which to customize
tests, such as payroll functionality, without having to reinvent
the business systems’ testing process each time the test is
required
Reduce the risk of human error, and cut the time taken to
complete complex testing processes that require every test to
be completed five times
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
–
Tightly integrated test management and test automation tool
–
Improved test management reporting capability for manual and
automated tests through dashboard utility
–
Successfully implemented a scalable test automation framework for
Oracle EBS Suite (HCM).
–
Ability to use same test scripts for EBS 11i to 12 upgrade
–
Laid the foundation for substantial reductions in time, effort, and errors
–
Better utilization and productivity of testing and business resources
 The following products are used
–
Oracle Functional Testing
–
Oracle Test Manager
–
Oracle EBS Accelerators for Functional Testing
 Implementation partner
–
Insert Information Protection Policy Classification from Slide 12
Acolade Consulting
www.acolade.com.au
Acolade
• Established in 2000
• Specialists Oracle Applications Testing
solution provider
• Specialist Solutions Partner for testing
products
• Specializing in these business domains
– Human Capital Management
– Financials
27
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Acolade’s slides were
originally presented by their
Director, Mr. Rajesh
Krishnan, at US OpenWorld
2012. He is shown here
being honored by Oracle
VP, Ms. Patricia Nance
Best Practice Testing Methodology
 Detailed test strategy with clearly defined entry and exit
criteria
 Thorough review & prioritization of business requirements
 Business functions broken down into modular and
manageable test scripts
 Creation of traceability matrix between tests & requirements
 Extensive Test data analysis and preparation to cater for
various scenarios
 Test script expected result clearly documented
28
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
More Best Practice Testing Methodology
 Exhaustive system and user acceptance testing done
 Parallel runs and result reconciliations prior to Go Live.
Detailed analysis of variance.
 Use of test management and automation tools
 Proper governance and project controls
29
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
High-level Testing Process
Wk1
Wk2
Wk3
Wk4
Test Strategy & Planning
Wk5
Wk6
Wk7
Wk8
Wk9
Wk10
SO
Test Strategy
Test Scenarios &
Cases
Test Data
Test Execution
Defect
Management
Fix & Re-test
Sign Off
30
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Go
Live
Unit / System Testing
Testing Process
UAT
Sign
off
Production Cut over
Parallel Runs Reconciliations
Sign
off
31
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Performance Testing
Project Case Study
Project Background
 Oracle 11i environment with HCM, Financials in a shared services
environment
 5 Govt agencies supported by shared services
 Multiple releases and patches in a year
 Many repeat regression tests to be done with
 Test cases were not re-usable
 Tests were conducted using traditional word and excel
 Test Management challenges
32
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Our Solution
 Tests organized, prioritized in a test management tool
 Combination of manual and automation solution
 Customer’s regression tests that used to take 4 weeks
now takes 1 week
 Quicker and more meaningful reports generated
 Staff are focussed on ‘business as usual’ tasks
 Efficient management of test assets and test cycles
33
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Demo…
34
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Project Case Study
Project Background
 Oracle 11i environment with Oracle Apps modules - HCM,
Financials Apps
 10g Database
 Significant number of transactional data within each module
 Resource intense processes sharing the same instance
 Requirement to upgrade database to 11g
 Functional testing of Apps and Database to be done
35
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Our Solution
 Tests organized, prioritized in a test management tool using Oracle
Test Manager
 Test automation solution for Apps functional module using Oracle
Functional Testing
 Database volume testing against actual production work loads using
Real Application Testing (RAT)
 SQL bottlenecks identified using RAT
 Predict performance issues prior to the actual upgrade
 Multiple testing cycles done and SQLs tuned & baselined prior to
actual upgrade
36
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Benefits using RAT
 Predict accurately the performance with high “actual” production workloads tested
 Create benchmarks, make changes & test against the benchmarks after fix is done
 Accurate database and SQL regression defects identified early on in the project
and rectified
 Reduced testing cycles considerably which was otherwise time consuming
 Repeatability of tests any number of times
 Quicker and more meaningful reports generated
 Staff were more focussed on ‘business as usual’ tasks
 Integration with other Oracle testing products e.g., test management and
automation products
 Higher ‘Go Live and Stay Live’ confidence
37
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
How Acolade can assist?
Consultancy Services
 End to end test management and
reporting during the upgrade
 Upgrade impact analysis
 Experience consultants can work
on or off-site prior to your upgrade
(DB/infrastructure/network)
 Run RAT processes and interpret
the reports generated
 Assist in identifying SQLs that
have potential issues
38
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Contacts
Web – www.acolade.com.au
Email – [email protected]
Blog – www.oracleats.com
Solution Videos:
AcoladeConsulting Channel
Summary
39
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Benefits of using Oracle testing tools
For upgrading Oracle Applications & Databases
 Early defect detection and rectification
 Reduce testing cycles considerably
 Automated and manual testing process for adequate test coverage
 ATS has specific accelerators for major ERP offerings - EBS 11, 12,
Fusion, Siebel, JDE
 Same language (Java) used for Functional and Load testing
 Repeatability of tests any number of times
 Integrated test management, automation and Load testing
 Integration with other Oracle testing products e.g., database upgrades
40
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Oracle ATS is the Standard for Testing At Oracle
 Examples of teams using ATS @ Oracle
–
Product teams: Fusion, EBS, Siebel, JDE, FMW, CS, ITT, Retail, FlexCube...
–
IT Operations: PDIT, AIT, MyOracleSupport...
 2000+ ATS users @ Oracle
Functional Testing
 E-Business Suite QA has 20K+ functional scripts created by OpenScript
 Fusion Apps QA has 400 QA engineers using OpenScript
Load Testing
 Siebel Load Testing team performed benchmark testing w/ 21K concurrent virtual users
 PDIT tested GSI (eBiz) environment with 24K concurrent virtual users
Test Management
 Fusion MiddleWare QA has: 2 million test cases and 900 users world wide using OTM
41
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
Key take-aways
 The goal of testing is to ensure the expected quality for your
application under test
 Test automation can help to ensure quality
 ATS has been co-designed and is being used by Oracle Apps QA
 RAT reduces your risk, lowers cost and enhances productivity
 Quality is key in today’s competitive market and is needed whether you
are upgrading your application, database or infrastructure
 Oracle offers multiple solutions for testing and quality management to
help you to optimize your applications testing
42
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
The preceding is intended to outline our general product direction. It is intended
for information purposes only, and may not be incorporated into any contract.
It is not a commitment to deliver any material, code, or functionality, and should
not be relied upon in making purchasing decisions. The development, release,
and timing of any features or functionality described for Oracle’s products
remains at the sole discretion of Oracle.
43
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
44
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12
45
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Insert Information Protection Policy Classification from Slide 12