Enterprise Performance Testing
Karuna Karnati and Ajay Kumar
Accenture Delivery Center for Technology in India
Copyright © 2008 Accenture All Rights Reserved. Accenture, its logo, and High Performance Delivered are trademarks of Accenture.
Enterprise Performance Testing
Failure is not an Option
Copyright © 2008 Accenture All Rights Reserved.
2
Agenda
• Why Performance Testing
• Types & Approaches
• Part of SDLC
• Environment
• Test Metrics
• Report Crafting
Copyright © 2008 Accenture All Rights Reserved.
3
Industry Facts
What it costs when your internal enterprise systems go offline.
• Industry Application Cost per Hour
• Financial Brokerage Operations $6,500,000
• Financial Credit Card Sales $2,600,000
• Media Pay-per-view $1,150,000
• Retail Home shopping (TV) $113,000
• Retail Catalog sales $90,000
• Transportation Airline Reservations $89,500
• For external web sites, from the same Standish Group study, the typical
costs were in the middle of the range, running anywhere from $1,000 to
$27,000 per minute of downtime, or $60,000 to $1,620,000 per hour.
Later figures, since 1998, are much higher as companies move more of
their business onto the web.
Copyright © 2008 Accenture All Rights Reserved.
Data from The Standish Group
4
Industry Facts
• 2002 Winter Olympics, an online ticketing page was
unavailable during a critical sales period placing a potential
of $1.5 million dollars of revenue at risk.
• Application problem with the State of Minnesota’s $27 million
online tax filing system delayed tax refunds and diminished
consumer confidence for greater than 60,000 tax filers.
• Enterprise systems that perform poorly, function incorrectly
or fail completely cost businesses money, reputation and
market share.
Copyright © 2008 Accenture All Rights Reserved.
5
Industry Facts
• Performance measurement and tuning have
become more rigorous over the past few years.
Worldwide Web-Load Testing Market Growth
Distributed Environments, 1998 - 2004
Copyright © 2008 Accenture All Rights Reserved.
6
Why Performance Testing ?
Failure is not an Option
Copyright © 2008 Accenture All Rights Reserved.
7
Why Performance Testing
• Speed
– Does the application respond quickly enough for the
intended users?
• Scalability/Capacity
– Will the application handle the expected user load and
beyond?
Copyright © 2008 Accenture All Rights Reserved.
8
Why Performance Testing
• Stability/Robustness
– Is the application stable under expected and
unexpected user loads?
• Confidence
– Are you sure that users will have a positive experience
on go-live day?
Copyright © 2008 Accenture All Rights Reserved.
9
FURPS - Performance
• F unctionality
• U sability
• R eliability
• P erformance
• S upportability
Copyright © 2008 Accenture All Rights Reserved.
Performance
Effectiveness
Scalability
10
Performance Testing
Performance Testing is the discipline concerned with
determining and reporting the current performance of a
software application under various parameters
Performance Testing Types
• Load & Stress
Performance Test approaches
• Scalability & Stability
Performance Testing terminologies
• Scenario /Business Flow
• Operational Profile
• Work load
Performance Metrics
• Client side & Server side
Copyright © 2008 Accenture All Rights Reserved.
11
Performance Testing - SDLC
• Performance testing as an integral part of SDLC
 IDE tools
• Eclipse – Rational Performance Tester
• Visual Studio Team – Microsoft
 Profiling and Probes
Copyright © 2008 Accenture All Rights Reserved.
12
Trends in performance testing
• Performance testing as an integral part of SDLC
 New IDE Tools
• Microsoft, Eclipse, etc..
– Object and Resource profiling
– Probes
• Architecture Framework evaluation against performance
requirements
 The Open Group Architecture Framework (TOGAF) (
http://www.developer.com/design/article.php/3374171)
 The Architecture Tradeoff Analysis Method (ATAM) (
http://www.sei.cmu.edu/architecture/ata_method.html)
Copyright © 2008 Accenture All Rights Reserved.
13
Performance Test Environment
Get scripts from Vugen
Load Agent
Load Balancer
Virtual User Generators
Application Servers
Copyright © 2008 Accenture All Rights Reserved.
Database Server
System Under Test
Web Servers
Simulate virtual
users from Load
Agent
LoadRunner Controller
14
What is a Counter?
A Counter is a unit that measures and gathers performance-relevant
events of the system
Client Side Metrics
Server Side Counters
• Response Time
• Memory
• Throughput
• Processor
• Hits Per Second
• Disk
• Pass/Fail Statistics
• Network
Copyright © 2008 Accenture All Rights Reserved.
15
Categories of Counters
Memory
Cache
RAM
HDD
Processor
Disk
Seek Time
RPM
Platter
Network
Copyright © 2008 Accenture All Rights Reserved.
16
Performance Test Tool
Tool
LoadRunner
SilkPerformer
WebLoad
Jmeter
Copyright © 2008 Accenture All Rights Reserved.
Vendor
Hp-Mercury
Borland
RadView
OpenSource
Language
Java/C
BDL
Javascript
Record& Play
17
Report Crafting
• Provide test results in tabular format and graphs
• Include all issues faced during testing
• Document all the findings & observations (performance testing is close
to research)
• Load and especially stress would sometime reflect the bad side of an
application. It throws errors, capture all of them for future analysis
• Any deviations/workarounds used should be mentioned
• Conclusions should be substantiated with proper analysis
• Raw data can be provided as an appendix
Copyright © 2008 Accenture All Rights Reserved.
18
Where to go for more information
• http://www.PerfTestPlus.com (My site)
• http://www.TestingReflections.com (QA Blog Collection)
• http://www.QAForums.com (Huge QA Forum)
• http://www.loadtester.com (Good articles and links)
• http://www.segue.com/html/s_solutions/papers/s_wp_info.ht
m
• (Good articles and statistics)
• http://www.keynote.com/resources/resource_library.html
• (Good articles and statistics)
Copyright © 2008 Accenture All Rights Reserved.
19
Final Message
Performance Testing is Insurance to the Business
Perf-surance
Copyright © 2008 Accenture All Rights Reserved.
20
Questions?
..
Copyright © 2008 Accenture All Rights Reserved.
21
Thank You
Copyright © 2008 Accenture All Rights Reserved.
22