Borland LQM and
Agile Development
Matt Brayley-Berger
[email protected]
Domain Specialist, Americas
July 2008
Agenda
• Agile Overview
• How Agile are You?
• Agile and Testing
• Role of the Tester, Automation, Test-Driven Development
• How Do We Move to More Agile Testing?
• Automation Techniques and Considerations
• How to Automate without a UI? Reporting
• Performance Testing and Agile
• Identify problems early, save time!
• Borland Journey into Agile
Copyright © 2008 Borland Software Corporation.
Confidential
2
Consequences of Poor Quality
• People get fired
• Companies lose money
• Customers sue
• Stock values go down
April
2004
Software bug was major contributor to the worst power system failure
in North American history. Economic loss ~$6B
Jan
2005
US Government scraps $170M IT project due to poor software testing.
Oct
2005
Toyota announces software glitch with 75,000 Prius hybrids. Gas engine can
shutdown unexpectedly. Auto industry spends $2-3B on software fixes.
July
2005
Faulty software update disables customer’s computers. Costs Trend Micro $8.1M.
Trend lowered its revenue forecast by 2.9% and net income forecast by 16.7%.
May
2006
QuadraMed settles MedCath lawsuit for $2M.
Complaint alleged QuadraMed's software products failed to perform.
Copyright © 2008 Borland Software Corporation.
Confidential
Agile Overview
Copyright © 2008 Borland Software Corporation.
Confidential
How agile are you?
• Knowledge
• Intent
• Application
Just 17 percent of North
American and European
enterprises use agile
development processes
(Forrester Research “Enterprise Agile Adoption in
2006” survey)
Copyright © 2008 Borland Software Corporation.
Confidential
Agile Roots
• Hirotaka Takeuchi & Ikojuri Nonaka
• The New New Product Development Game
• Harvard Business Review – Jan/Feb 1986
Takeuchi
Nonaka
The ‘Godfathers of SCRUM’
• Originated from the Toyota factories' processes of efficiency
improvements in the production environment (aka Lean Manf)
Copyright © 2008 Borland Software Corporation.
Confidential
Engineering Misfires
CPSC, xyz Sprinkler Company Announce
Voluntary Recall To Replace O-Ring Fire
Sprinklers
(July 19, 2001)
WASHINGTON, D.C.- The U.S. Consumer
Product Safety Commission (CPSC), and xyz
Sprinkler Company, are announcing a
voluntary replacement program...
...These factors could cause the sprinkler
heads not to activate in a fire.
Copyright © 2008 Borland Software Corporation.
Confidential
September 30, 1999
Web posted at: 1:46 p.m. EDT (1746 GMT)
(CNN) -- NASA lost a $125 million Mars
orbiter because one engineering team used
metric units while another used English
units for a key spacecraft operation.
Development processes
All development processes
Waterfall
Iterative
Agile
Scrum
Copyright © 2008 Borland Software Corporation.
XP
Confidential
DSDM
FDD
Etc.
Manifesto for Agile Software Development
Individuals &
Interactions
over
Processes &
Tools
Working Software
over
Comprehensive
Documentation
Customer
Collaboration
over
Contract
Negotiation
Responding to Change
over
Following a
Plan
That is, while there is value in the items on the right, we value the items on the
left more.” http://agilemanifesto.org/
Copyright © 2008 Borland Software Corporation.
Confidential
9
Agile Myths ….
• no design
• no testing
• no documentation
• no idea of progress
• poor quality
• no plan
• Auditors won’t allow it
• Agile is easy
Copyright © 2008 Borland Software Corporation.
Confidential
Agile and Testing
Role of the Tester, Automation, and TestDriven Development
Copyright © 2008 Borland Software Corporation.
Confidential
Let’s Look at the Role of Testing
• Testing is the projects ‘headlights’
• Where are we now? Where are we going?
• Testing providing critical information to the project
• A key input in making informed decisions
• Testing alone does not ensure quality
• Find ways to set goals rather than focus on mistakes
• Checks and Balances
• The tester provides insight from the customers perspective
Copyright © 2008 Borland Software Corporation.
Confidential
12
What is Agile Testing?
• The Agile Manifesto treats software development as the
customer of testing
• Involves testing from the customer perspective as early as
possible
• Since working software is delivered often, testing too occurs very
often; Automation is key
• Testing as a discipline involves two areas:
• Developer Unit Testing
• Acceptance Testing
• Manual testing in agile development likely results in
buggy software or schedule slip
Copyright © 2008 Borland Software Corporation.
Confidential
13
The Agile Tester
• “If Agile is like driving a car, the tester helps keep the
team on the right road and makes them stop and ask for
directions when necessary”
• The key is to treat test automation as a development
practice
• Maintainable automated test assets are not ‘record-and-playback’
dependent
• Test assets should ideally be written in the same language as the
application
• Quality is a part of the entire development process; from
Day One
Copyright © 2008 Borland Software Corporation.
Confidential
14
Tester’s Bill of Rights
• You have the right to bring up issues related to quality
and process at any time
• You have the right to ask questions of customers and
programmers and receive timely answers
• You have the right to make and update your own
estimates for your own tasks and have these included in
estimates for stories
• You have the right to the tools you need to do your job in
a timely manner
Adapted from Testing Extreme Programming, 2003., by Lisa Crispin and Tip House
Copyright © 2008 Borland Software Corporation.
Confidential
15
Example Road-Trip through an Agile Project
Day
Driving
General Agile
Activities
Agile Tester Activities
1
•Leave Chicago on I-90W,
lunch in Madison, WI
•Enter I-94W
•Check into the hotel in
Dickinson, ND
User Stories
Release Planning
•Helping with user stories
•Find missing assumptions
•Define and Estimate acceptance
tests
2
•Leave Dickinson on
I-94W, lunch in Billings
•Check into the hotel in
Spokane, WA
•Iteration planning
•Coding and unit testing
•Refactoring and Integration
•Estimate testing and test
infrastructure tasks
•Firm up acceptance test details
•Acceptance Write Executable tests
•Organize Acceptance tests
3
•Leave Spokane on
I-94W, lunch at Yakima,
WA
•Arrive in Seattle
•Acceptance testing
•End of iteration
•Next iteration release
•Refactor acceptance tests
•Get acceptance tests running
•Execute acceptance tests
•Adjust acceptance tests to run
through UI
•Perform regression testing
•Release
Copyright © 2008 Borland Software Corporation.
Confidential
16
Tester’s Considerations During the Road-Trip?
• We have a clear goal is in this example: Travel from
Chicago to Seattle
• Given that we’ve budgeted 3 days, we can gauge progress by
how far we’ve gotten at the end of each day
• Consider the varied terrain and fuel consumption
• Depending on the time of year - What about road closures in the
mountains? Rock-slides? Overheating engine?
• Navigating through construction, detours, traffic jams
Copyright © 2008 Borland Software Corporation.
Confidential
17
What is Test-Driven Development?
• Related to the Test-First concepts of Extreme
Programming
• TDD requires that an automated unit test, defining
requirements of the code, is written before each aspect
of the code itself
• These unit tests can later be migrated to some degree into
Acceptance (Regression) testing as well as Performance testing
• In test-driven development, each new feature begins
with writing a test
• The test will inevitably fail because it is written before the feature
has been implemented, but serves as concrete acceptance
criteria
Copyright © 2008 Borland Software Corporation.
Confidential
18
Test-Driven Development Cycle
1. Add a test
• Every feature begins with writing a test
2. Run all tests and see if the new one fails
• validates that the test harness is working correctly
3. Write Code
• The next step is to write some code that will cause the test to
pass (and only designed to make it pass)
4. Run the automated tests
• If all test cases now pass, the programmer can be confident that
the code meets all the tested requirements
5. Refactor
Repeat
Copyright © 2008 Borland Software Corporation.
Confidential
19
Key Automation Selections
• Regardless of which Agile method you employ,
automation support for the team is critical
• The rapid speed of artifact creation will demand the need
for levels of tool supported automation
• Defect Tracking
• Source Code Control
• Execute Automated Tests before the UI is ready; and then retrofit
those tests when the UI is ready
• “A Fool with a tool, is still a fool”
• Ultimately the teams success is dependant on the team
members; tools only accelerate progress
Copyright © 2008 Borland Software Corporation.
Confidential
20
How do we move to more agile testing?
Manual Testing, Automation Considerations
Confidential
The Reality: Myths Dispelled
• “Automated tools are not responsive enough for Agile”
• Tools automation enhances, improves and ensures monitoring of
responsiveness
• “Agile is about the people, not the process or the tools”
• The people need a process to know what to do and tools to allow them to
do it efficiently
• “Localized teams with individuals working next to each other means
that communication issues are resolved”
• Reference material, reporting, tracing and delivery between teams need
to be supported
• “It‘s only a small change so there should be very little impact. No
need for regression testing”
• The smallest change can have the biggest impact – regression test!
• “Agile is only useful to smaller development shops, not enterprise
scale organizations”
• BT have done it!
• Borland have done it!
• You can do it!
Copyright © 2008 Borland Software Corporation.
Confidential
Most Testing is Still Done Manually
• A good first step in the move to more agile testing is to
establish a management repository that will allow teams
to communicate stories and relate those to tests
• Once tests are introduced into the management system,
we have visibility into progress, trends and can better
determine where automation is necessary
• Ultimately the goal is to automate as much as possible,
where practical
Copyright © 2008 Borland Software Corporation.
Confidential
23
Automation Provides Visibility
• Automated Regression Reports vs. Requirements
• Execution Pass/Fail Trending
• Defect Resolution Trends
Copyright © 2008 Borland Software Corporation.
Confidential
24
Automation Provides Visibility
• Automated Regression Reports vs. Requirements
• Execution Pass/Fail Trending
• Defect Resolution Trends
Defects per Component
Copyright © 2008 Borland Software Corporation.
Confidential
25
Automation Provides Visibility
• Automated Regression Reports vs. Requirements
• Execution Pass/Fail Trending
• Defect Resolution Trends
Defects per Component
Automated Test Results Trend
Copyright © 2008 Borland Software Corporation.
Confidential
26
Automating Tests and Agile
• The Need for Speed
• Fast iterations need a fast turnaround of tests
• Being responsive to change
• The Need for Repeatability
• Testing the same thing the same way
• Regression Testing
• The Need for Accuracy
• Ensuring comprehensive code coverage
• Requirements Based Testing
• Feeding the metrics
• Manual Tests are not responsive enough for Agile
Copyright © 2008 Borland Software Corporation.
Confidential
How Do We Test Incomplete Code?
• Testing in Agile environments can be a lot like changing a
tire while the car’s driving
• Stories must be defined in terms of business value and
should not span iterations
• Incomplete story definition tends to impact testers more heavily
than developers
• Agile testing tends to work best trailing a development
iteration
Development
Testing
Copyright © 2008 Borland Software Corporation.
Confidential
28
Agile Testing Coverage
• Every User Story has an Acceptance Criteria
• Test it! – Unit Test
• Every User Story is just one of several in a sprint
• Test it! – Functional integrated testing
• Every User Story is going to change the content of the
iteration
• Test it! – regression testing
• It‘s only a small change in an integrated system so
there should be every chance of an impact. We must
do regression testing
Copyright © 2008 Borland Software Corporation.
Confidential
Test Automation Before the UI
• All object and window definitions are stored in a
centralized Include file, called a Frame file
• Business logic unit or acceptance tests are based on the
object model centralized within those same Frame files
• For any objects or controls yet under development, these
can be stubbed in the frame file and still included in the
test case
• When the UI elements or embedded objects are developed, they
can be re-defined in the frame file without impacting test cases
• Services without a UI can be automated through the
Borland Java and .NET Explorers
Copyright © 2008 Borland Software Corporation.
Confidential
30
SilkTest Architecture for Automation
Test Case 3
Test Case 2
Test Case 1
Window.
Textbox.()
Copyright © 2008 Borland Software Corporation.
Button.()
Confidential
31
Test Assets Evolve with the Application
• Features yet to be developed can be ‘stubbed’ in the
Frame file
Copyright © 2008 Borland Software Corporation.
Confidential
32
Retrofitting Tests When the UI is Ready
• Consider:
• SilkTest INC files are the source of the object definitions,
• stubs for the UI would already be included based on our historical
development of the test assets working with development
• The test-case business logic is in a separate file that references
the INC file library
• By modifying the INC files to reflect the actual UI
(updating object tags, and appending ‘stubbed’
functionality), Silk Test scripts can be rapidly be adapted
Copyright © 2008 Borland Software Corporation.
Confidential
33
Borland SilkTest Eclipse
Functional Test Scripting in Java
within Eclipse
Closely align the testing
effort with development
Copyright © 2008 Borland Software Corporation.
Confidential
34
SCRUM Details
Copyright © 2008 Borland Software Corporation.
Confidential
35
Automated Performance Testing in Agile
Confidential
Performance Testing is Critical
• A Jupiter Media Metrix survey of more than 2000 users found that
46% of customer permantly or temporarily abandoned a site where
they experienced poor performance
• Performance defects found late in the process can be 100 to 1000
times more expensive than found earlier in the process
Copyright © 2008 Borland Software Corporation.
Confidential
How Do We Performance Test Earlier?
• True, that a full business-level test will require most of the
system to be developed
• What about key components? Shares components?
Interfaces into common or already existing services?
• Performance testing effort must be included in story
estimations
• Evolving levels of Performance testing detail as the
application develops
• Visibility into the application development enables earlier testing
of key components
• Re-use of unit testing assets
• Automation tools that allow for the conversion of JUNIT and
NUNIT
Copyright © 2008 Borland Software Corporation.
Confidential
38
Performance Testing is a Specialty
• .. However, these specialists need to be a part of the
development team!
• Why?
• Visibility in the application’s purpose, development plan and
evolution allow Performance testing assets to evolve with the
project
• Assets in line with development enable the team to more easily
test, more often
Copyright © 2008 Borland Software Corporation.
Confidential
39
Automation Enables Rapid Troubleshooting
Copyright © 2008 Borland Software Corporation.
Confidential
40
Automation Enables Rapid Troubleshooting
Copyright © 2008 Borland Software Corporation.
Confidential
41
Automation Enables Rapid Troubleshooting
Copyright © 2008 Borland Software Corporation.
Confidential
42
Automation Enables Rapid Troubleshooting
Copyright © 2008 Borland Software Corporation.
Confidential
43
Unit Tests Migrated into Performance Tests
//@Test(expected=java.lang.IllegalArgumentException)
JavaRegisterException(500,"java.lang.IllegalArgumentException",
JAVA_OPTION_MATCH_SUBSTRING);
ErrorAdd(FACILITY_NATIVE_JAVA, 500,
SEVERITY_INFORMATIONAL);
JUnit4CallFunc(ghTestObj, "doFoo", "doFoo“, sBuffer);
ErrorRemove(FACILITY_NATIVE_JAVA, 500);
if (StrSearch(sBuffer,"java.lang.IllegalArgumentException",
STR_SEARCH_FIRST)= 0) then
RepMessage("method did not throw:java.lang.IllegalArgumentException",
SEVERITY_ERROR);
end;
Copyright © 2008 Borland Software Corporation.
Confidential
44
Unit Tests Migrated into Performance Tests
//@Test(expected=java.lang.IllegalArgumentException)
JavaRegisterException(500,"java.lang.IllegalArgumentException",
JAVA_OPTION_MATCH_SUBSTRING);
ErrorAdd(FACILITY_NATIVE_JAVA, 500,
SEVERITY_INFORMATIONAL);
JUnit4CallFunc(ghTestObj, "doFoo", "doFoo“, sBuffer);
ErrorRemove(FACILITY_NATIVE_JAVA, 500);
if (StrSearch(sBuffer,"java.lang.IllegalArgumentException",
STR_SEARCH_FIRST)= 0) then
RepMessage("method did not throw:java.lang.IllegalArgumentException",
SEVERITY_ERROR);
end;
Copyright © 2008 Borland Software Corporation.
Confidential
45
Unit Tests Migrated into Performance Tests
//@Test(expected=java.lang.IllegalArgumentException)
JavaRegisterException(500,"java.lang.IllegalArgumentException",
JAVA_OPTION_MATCH_SUBSTRING);
ErrorAdd(FACILITY_NATIVE_JAVA, 500,
transaction Tmain
SEVERITY_INFORMATIONAL);
JUnit4CallFunc(ghTestObj, "doFoo", "doFoo“, sBuffer);
begin
ErrorRemove(FACILITY_NATIVE_JAVA, 500);
// junit.framework.Test suite()
JavaCallMethod(JAVA_STATIC_METHOD,
if (StrSearch(sBuffer,"java.lang.IllegalArgumentException",
"JU4ImporterTestSimple.suite", "suite");
STR_SEARCH_FIRST)= 0) then
// hObj := JavaGetObject(JAVA_STATIC_METHOD);
RepMessage("method did not throw:java.lang.IllegalArgumentException",
// use the following function to execute JUnit
SEVERITY_ERROR);
testsuites
end;
// FRunTestSuite(hObj);
// JavaFreeObject(hObj);
end TMain;
Copyright © 2008 Borland Software Corporation.
Confidential
46
Borland’s Journey into Agile
Copyright © 2008 Borland Software Corporation.
Confidential
Agile vs Enterprise – The Borland Linz Journey
• Established the development teams
• Lesson: QA isolated
• QA incorporated into teams
• Lesson: documentation out of step
• Documentation incorporated – engineering teams
• Lesson: skills division
• QA Coach role created
• Lesson: teams now more efficient
• Splitting & re-locating teams
• Lesson: distributed teams – EQC Singapore
• Communicate, Communicate, Communicate...
• Lesson: enabling the communication
• Borland on Borland
• Lesson: ...........
Copyright © 2008 Borland Software Corporation.
Confidential
Borland on Borland - the environment we live in
Owner: Management
Project Management
Reporting
RBT Environment
Owner: Scrum Team(s)
Owner: Product Owner
Owner: Scrum Team(s)
Iteration Management
Requirements Management
Test & Defect Management
Owner: Scrum Team(s)
Owner: Scrum Team(s)
Source Code Management
Test Assets Management
Functional Test Automation
Load/Performance Testing
Owner:
Scrum
Team(s)
Unit Testing
Copyright © 2008 Borland Software Corporation.
Confidential
Agile vs Enterprise
• Not just Borland...
• BT Case Study
• Challenges
• Maintain industry and leadership position in the market
• Increase customer satisfaction by reducing time to market for new
products
• Improve and standardize development processes, unifying delivery
• Results
• 20% gain in productivity
• Reduced development spend
• Increased efficiency and improved control
• http://www.borland.com/resources/en/pdf/case_studies/bt_group.
pdf
• Agile is not only useful to smaller development shops, but also
enterprise scale organizations
Copyright © 2008 Borland Software Corporation.
Confidential
Summary
• Agile is real and growing in the market
• The role of the tester is very critical in Agile projects
• Test-Driven development is one of the many Agile tools in our
toolbox
• Automation is key!
• SilkTest provides an ability to easily develop functional tests without a UI, and then
retrofit those assets as the UI develops
• Performance testing earlier than later – make specialists a part of
the team
• Reporting on the testing progress is a key to measuring velocity
• Borland SilkTest and SilkPerformer support critical Agile practices
• More Information available at www.borland.com
Copyright © 2008 Borland Software Corporation.
Confidential
2008 Forrester Wave: Functional Testing
• Forrester evaluated commercial functional testing tools
• Borland was categorized as a “Strong Performer”
• “Test automation tools that slash the time and resources required for
test execution. Test automation tools let application development
organizations make an upfront investment in test script creation to
slash the time required for test execution from a matter of days to a
matter of hours.”
• “HP isn’t the only big name in testing. Test automation tools from
Borland and IBM are more popular with technical testers who prefer
to work with powerful scripting languages rather than graphical user
interfaces."
Copyright © 2008 Borland Software Corporation.
Confidential
52
Q+A
Copyright © 2008 Borland Software Corporation.
Confidential
Thank you!
Copyright © 2008 Borland Software Corporation.
Confidential