Document 288528
Borland LQM and Agile Development Matt Brayley-Berger [email protected] Domain Specialist, Americas July 2008 Agenda • Agile Overview • How Agile are You? • Agile and Testing • Role of the Tester, Automation, Test-Driven Development • How Do We Move to More Agile Testing? • Automation Techniques and Considerations • How to Automate without a UI? Reporting • Performance Testing and Agile • Identify problems early, save time! • Borland Journey into Agile Copyright © 2008 Borland Software Corporation. Confidential 2 Consequences of Poor Quality • People get fired • Companies lose money • Customers sue • Stock values go down April 2004 Software bug was major contributor to the worst power system failure in North American history. Economic loss ~$6B Jan 2005 US Government scraps $170M IT project due to poor software testing. Oct 2005 Toyota announces software glitch with 75,000 Prius hybrids. Gas engine can shutdown unexpectedly. Auto industry spends $2-3B on software fixes. July 2005 Faulty software update disables customer’s computers. Costs Trend Micro $8.1M. Trend lowered its revenue forecast by 2.9% and net income forecast by 16.7%. May 2006 QuadraMed settles MedCath lawsuit for $2M. Complaint alleged QuadraMed's software products failed to perform. Copyright © 2008 Borland Software Corporation. Confidential Agile Overview Copyright © 2008 Borland Software Corporation. Confidential How agile are you? • Knowledge • Intent • Application Just 17 percent of North American and European enterprises use agile development processes (Forrester Research “Enterprise Agile Adoption in 2006” survey) Copyright © 2008 Borland Software Corporation. Confidential Agile Roots • Hirotaka Takeuchi & Ikojuri Nonaka • The New New Product Development Game • Harvard Business Review – Jan/Feb 1986 Takeuchi Nonaka The ‘Godfathers of SCRUM’ • Originated from the Toyota factories' processes of efficiency improvements in the production environment (aka Lean Manf) Copyright © 2008 Borland Software Corporation. Confidential Engineering Misfires CPSC, xyz Sprinkler Company Announce Voluntary Recall To Replace O-Ring Fire Sprinklers (July 19, 2001) WASHINGTON, D.C.- The U.S. Consumer Product Safety Commission (CPSC), and xyz Sprinkler Company, are announcing a voluntary replacement program... ...These factors could cause the sprinkler heads not to activate in a fire. Copyright © 2008 Borland Software Corporation. Confidential September 30, 1999 Web posted at: 1:46 p.m. EDT (1746 GMT) (CNN) -- NASA lost a $125 million Mars orbiter because one engineering team used metric units while another used English units for a key spacecraft operation. Development processes All development processes Waterfall Iterative Agile Scrum Copyright © 2008 Borland Software Corporation. XP Confidential DSDM FDD Etc. Manifesto for Agile Software Development Individuals & Interactions over Processes & Tools Working Software over Comprehensive Documentation Customer Collaboration over Contract Negotiation Responding to Change over Following a Plan That is, while there is value in the items on the right, we value the items on the left more.” http://agilemanifesto.org/ Copyright © 2008 Borland Software Corporation. Confidential 9 Agile Myths …. • no design • no testing • no documentation • no idea of progress • poor quality • no plan • Auditors won’t allow it • Agile is easy Copyright © 2008 Borland Software Corporation. Confidential Agile and Testing Role of the Tester, Automation, and TestDriven Development Copyright © 2008 Borland Software Corporation. Confidential Let’s Look at the Role of Testing • Testing is the projects ‘headlights’ • Where are we now? Where are we going? • Testing providing critical information to the project • A key input in making informed decisions • Testing alone does not ensure quality • Find ways to set goals rather than focus on mistakes • Checks and Balances • The tester provides insight from the customers perspective Copyright © 2008 Borland Software Corporation. Confidential 12 What is Agile Testing? • The Agile Manifesto treats software development as the customer of testing • Involves testing from the customer perspective as early as possible • Since working software is delivered often, testing too occurs very often; Automation is key • Testing as a discipline involves two areas: • Developer Unit Testing • Acceptance Testing • Manual testing in agile development likely results in buggy software or schedule slip Copyright © 2008 Borland Software Corporation. Confidential 13 The Agile Tester • “If Agile is like driving a car, the tester helps keep the team on the right road and makes them stop and ask for directions when necessary” • The key is to treat test automation as a development practice • Maintainable automated test assets are not ‘record-and-playback’ dependent • Test assets should ideally be written in the same language as the application • Quality is a part of the entire development process; from Day One Copyright © 2008 Borland Software Corporation. Confidential 14 Tester’s Bill of Rights • You have the right to bring up issues related to quality and process at any time • You have the right to ask questions of customers and programmers and receive timely answers • You have the right to make and update your own estimates for your own tasks and have these included in estimates for stories • You have the right to the tools you need to do your job in a timely manner Adapted from Testing Extreme Programming, 2003., by Lisa Crispin and Tip House Copyright © 2008 Borland Software Corporation. Confidential 15 Example Road-Trip through an Agile Project Day Driving General Agile Activities Agile Tester Activities 1 •Leave Chicago on I-90W, lunch in Madison, WI •Enter I-94W •Check into the hotel in Dickinson, ND User Stories Release Planning •Helping with user stories •Find missing assumptions •Define and Estimate acceptance tests 2 •Leave Dickinson on I-94W, lunch in Billings •Check into the hotel in Spokane, WA •Iteration planning •Coding and unit testing •Refactoring and Integration •Estimate testing and test infrastructure tasks •Firm up acceptance test details •Acceptance Write Executable tests •Organize Acceptance tests 3 •Leave Spokane on I-94W, lunch at Yakima, WA •Arrive in Seattle •Acceptance testing •End of iteration •Next iteration release •Refactor acceptance tests •Get acceptance tests running •Execute acceptance tests •Adjust acceptance tests to run through UI •Perform regression testing •Release Copyright © 2008 Borland Software Corporation. Confidential 16 Tester’s Considerations During the Road-Trip? • We have a clear goal is in this example: Travel from Chicago to Seattle • Given that we’ve budgeted 3 days, we can gauge progress by how far we’ve gotten at the end of each day • Consider the varied terrain and fuel consumption • Depending on the time of year - What about road closures in the mountains? Rock-slides? Overheating engine? • Navigating through construction, detours, traffic jams Copyright © 2008 Borland Software Corporation. Confidential 17 What is Test-Driven Development? • Related to the Test-First concepts of Extreme Programming • TDD requires that an automated unit test, defining requirements of the code, is written before each aspect of the code itself • These unit tests can later be migrated to some degree into Acceptance (Regression) testing as well as Performance testing • In test-driven development, each new feature begins with writing a test • The test will inevitably fail because it is written before the feature has been implemented, but serves as concrete acceptance criteria Copyright © 2008 Borland Software Corporation. Confidential 18 Test-Driven Development Cycle 1. Add a test • Every feature begins with writing a test 2. Run all tests and see if the new one fails • validates that the test harness is working correctly 3. Write Code • The next step is to write some code that will cause the test to pass (and only designed to make it pass) 4. Run the automated tests • If all test cases now pass, the programmer can be confident that the code meets all the tested requirements 5. Refactor Repeat Copyright © 2008 Borland Software Corporation. Confidential 19 Key Automation Selections • Regardless of which Agile method you employ, automation support for the team is critical • The rapid speed of artifact creation will demand the need for levels of tool supported automation • Defect Tracking • Source Code Control • Execute Automated Tests before the UI is ready; and then retrofit those tests when the UI is ready • “A Fool with a tool, is still a fool” • Ultimately the teams success is dependant on the team members; tools only accelerate progress Copyright © 2008 Borland Software Corporation. Confidential 20 How do we move to more agile testing? Manual Testing, Automation Considerations Confidential The Reality: Myths Dispelled • “Automated tools are not responsive enough for Agile” • Tools automation enhances, improves and ensures monitoring of responsiveness • “Agile is about the people, not the process or the tools” • The people need a process to know what to do and tools to allow them to do it efficiently • “Localized teams with individuals working next to each other means that communication issues are resolved” • Reference material, reporting, tracing and delivery between teams need to be supported • “It‘s only a small change so there should be very little impact. No need for regression testing” • The smallest change can have the biggest impact – regression test! • “Agile is only useful to smaller development shops, not enterprise scale organizations” • BT have done it! • Borland have done it! • You can do it! Copyright © 2008 Borland Software Corporation. Confidential Most Testing is Still Done Manually • A good first step in the move to more agile testing is to establish a management repository that will allow teams to communicate stories and relate those to tests • Once tests are introduced into the management system, we have visibility into progress, trends and can better determine where automation is necessary • Ultimately the goal is to automate as much as possible, where practical Copyright © 2008 Borland Software Corporation. Confidential 23 Automation Provides Visibility • Automated Regression Reports vs. Requirements • Execution Pass/Fail Trending • Defect Resolution Trends Copyright © 2008 Borland Software Corporation. Confidential 24 Automation Provides Visibility • Automated Regression Reports vs. Requirements • Execution Pass/Fail Trending • Defect Resolution Trends Defects per Component Copyright © 2008 Borland Software Corporation. Confidential 25 Automation Provides Visibility • Automated Regression Reports vs. Requirements • Execution Pass/Fail Trending • Defect Resolution Trends Defects per Component Automated Test Results Trend Copyright © 2008 Borland Software Corporation. Confidential 26 Automating Tests and Agile • The Need for Speed • Fast iterations need a fast turnaround of tests • Being responsive to change • The Need for Repeatability • Testing the same thing the same way • Regression Testing • The Need for Accuracy • Ensuring comprehensive code coverage • Requirements Based Testing • Feeding the metrics • Manual Tests are not responsive enough for Agile Copyright © 2008 Borland Software Corporation. Confidential How Do We Test Incomplete Code? • Testing in Agile environments can be a lot like changing a tire while the car’s driving • Stories must be defined in terms of business value and should not span iterations • Incomplete story definition tends to impact testers more heavily than developers • Agile testing tends to work best trailing a development iteration Development Testing Copyright © 2008 Borland Software Corporation. Confidential 28 Agile Testing Coverage • Every User Story has an Acceptance Criteria • Test it! – Unit Test • Every User Story is just one of several in a sprint • Test it! – Functional integrated testing • Every User Story is going to change the content of the iteration • Test it! – regression testing • It‘s only a small change in an integrated system so there should be every chance of an impact. We must do regression testing Copyright © 2008 Borland Software Corporation. Confidential Test Automation Before the UI • All object and window definitions are stored in a centralized Include file, called a Frame file • Business logic unit or acceptance tests are based on the object model centralized within those same Frame files • For any objects or controls yet under development, these can be stubbed in the frame file and still included in the test case • When the UI elements or embedded objects are developed, they can be re-defined in the frame file without impacting test cases • Services without a UI can be automated through the Borland Java and .NET Explorers Copyright © 2008 Borland Software Corporation. Confidential 30 SilkTest Architecture for Automation Test Case 3 Test Case 2 Test Case 1 Window. Textbox.() Copyright © 2008 Borland Software Corporation. Button.() Confidential 31 Test Assets Evolve with the Application • Features yet to be developed can be ‘stubbed’ in the Frame file Copyright © 2008 Borland Software Corporation. Confidential 32 Retrofitting Tests When the UI is Ready • Consider: • SilkTest INC files are the source of the object definitions, • stubs for the UI would already be included based on our historical development of the test assets working with development • The test-case business logic is in a separate file that references the INC file library • By modifying the INC files to reflect the actual UI (updating object tags, and appending ‘stubbed’ functionality), Silk Test scripts can be rapidly be adapted Copyright © 2008 Borland Software Corporation. Confidential 33 Borland SilkTest Eclipse Functional Test Scripting in Java within Eclipse Closely align the testing effort with development Copyright © 2008 Borland Software Corporation. Confidential 34 SCRUM Details Copyright © 2008 Borland Software Corporation. Confidential 35 Automated Performance Testing in Agile Confidential Performance Testing is Critical • A Jupiter Media Metrix survey of more than 2000 users found that 46% of customer permantly or temporarily abandoned a site where they experienced poor performance • Performance defects found late in the process can be 100 to 1000 times more expensive than found earlier in the process Copyright © 2008 Borland Software Corporation. Confidential How Do We Performance Test Earlier? • True, that a full business-level test will require most of the system to be developed • What about key components? Shares components? Interfaces into common or already existing services? • Performance testing effort must be included in story estimations • Evolving levels of Performance testing detail as the application develops • Visibility into the application development enables earlier testing of key components • Re-use of unit testing assets • Automation tools that allow for the conversion of JUNIT and NUNIT Copyright © 2008 Borland Software Corporation. Confidential 38 Performance Testing is a Specialty • .. However, these specialists need to be a part of the development team! • Why? • Visibility in the application’s purpose, development plan and evolution allow Performance testing assets to evolve with the project • Assets in line with development enable the team to more easily test, more often Copyright © 2008 Borland Software Corporation. Confidential 39 Automation Enables Rapid Troubleshooting Copyright © 2008 Borland Software Corporation. Confidential 40 Automation Enables Rapid Troubleshooting Copyright © 2008 Borland Software Corporation. Confidential 41 Automation Enables Rapid Troubleshooting Copyright © 2008 Borland Software Corporation. Confidential 42 Automation Enables Rapid Troubleshooting Copyright © 2008 Borland Software Corporation. Confidential 43 Unit Tests Migrated into Performance Tests //@Test(expected=java.lang.IllegalArgumentException) JavaRegisterException(500,"java.lang.IllegalArgumentException", JAVA_OPTION_MATCH_SUBSTRING); ErrorAdd(FACILITY_NATIVE_JAVA, 500, SEVERITY_INFORMATIONAL); JUnit4CallFunc(ghTestObj, "doFoo", "doFoo“, sBuffer); ErrorRemove(FACILITY_NATIVE_JAVA, 500); if (StrSearch(sBuffer,"java.lang.IllegalArgumentException", STR_SEARCH_FIRST)= 0) then RepMessage("method did not throw:java.lang.IllegalArgumentException", SEVERITY_ERROR); end; Copyright © 2008 Borland Software Corporation. Confidential 44 Unit Tests Migrated into Performance Tests //@Test(expected=java.lang.IllegalArgumentException) JavaRegisterException(500,"java.lang.IllegalArgumentException", JAVA_OPTION_MATCH_SUBSTRING); ErrorAdd(FACILITY_NATIVE_JAVA, 500, SEVERITY_INFORMATIONAL); JUnit4CallFunc(ghTestObj, "doFoo", "doFoo“, sBuffer); ErrorRemove(FACILITY_NATIVE_JAVA, 500); if (StrSearch(sBuffer,"java.lang.IllegalArgumentException", STR_SEARCH_FIRST)= 0) then RepMessage("method did not throw:java.lang.IllegalArgumentException", SEVERITY_ERROR); end; Copyright © 2008 Borland Software Corporation. Confidential 45 Unit Tests Migrated into Performance Tests //@Test(expected=java.lang.IllegalArgumentException) JavaRegisterException(500,"java.lang.IllegalArgumentException", JAVA_OPTION_MATCH_SUBSTRING); ErrorAdd(FACILITY_NATIVE_JAVA, 500, transaction Tmain SEVERITY_INFORMATIONAL); JUnit4CallFunc(ghTestObj, "doFoo", "doFoo“, sBuffer); begin ErrorRemove(FACILITY_NATIVE_JAVA, 500); // junit.framework.Test suite() JavaCallMethod(JAVA_STATIC_METHOD, if (StrSearch(sBuffer,"java.lang.IllegalArgumentException", "JU4ImporterTestSimple.suite", "suite"); STR_SEARCH_FIRST)= 0) then // hObj := JavaGetObject(JAVA_STATIC_METHOD); RepMessage("method did not throw:java.lang.IllegalArgumentException", // use the following function to execute JUnit SEVERITY_ERROR); testsuites end; // FRunTestSuite(hObj); // JavaFreeObject(hObj); end TMain; Copyright © 2008 Borland Software Corporation. Confidential 46 Borland’s Journey into Agile Copyright © 2008 Borland Software Corporation. Confidential Agile vs Enterprise – The Borland Linz Journey • Established the development teams • Lesson: QA isolated • QA incorporated into teams • Lesson: documentation out of step • Documentation incorporated – engineering teams • Lesson: skills division • QA Coach role created • Lesson: teams now more efficient • Splitting & re-locating teams • Lesson: distributed teams – EQC Singapore • Communicate, Communicate, Communicate... • Lesson: enabling the communication • Borland on Borland • Lesson: ........... Copyright © 2008 Borland Software Corporation. Confidential Borland on Borland - the environment we live in Owner: Management Project Management Reporting RBT Environment Owner: Scrum Team(s) Owner: Product Owner Owner: Scrum Team(s) Iteration Management Requirements Management Test & Defect Management Owner: Scrum Team(s) Owner: Scrum Team(s) Source Code Management Test Assets Management Functional Test Automation Load/Performance Testing Owner: Scrum Team(s) Unit Testing Copyright © 2008 Borland Software Corporation. Confidential Agile vs Enterprise • Not just Borland... • BT Case Study • Challenges • Maintain industry and leadership position in the market • Increase customer satisfaction by reducing time to market for new products • Improve and standardize development processes, unifying delivery • Results • 20% gain in productivity • Reduced development spend • Increased efficiency and improved control • http://www.borland.com/resources/en/pdf/case_studies/bt_group. pdf • Agile is not only useful to smaller development shops, but also enterprise scale organizations Copyright © 2008 Borland Software Corporation. Confidential Summary • Agile is real and growing in the market • The role of the tester is very critical in Agile projects • Test-Driven development is one of the many Agile tools in our toolbox • Automation is key! • SilkTest provides an ability to easily develop functional tests without a UI, and then retrofit those assets as the UI develops • Performance testing earlier than later – make specialists a part of the team • Reporting on the testing progress is a key to measuring velocity • Borland SilkTest and SilkPerformer support critical Agile practices • More Information available at www.borland.com Copyright © 2008 Borland Software Corporation. Confidential 2008 Forrester Wave: Functional Testing • Forrester evaluated commercial functional testing tools • Borland was categorized as a “Strong Performer” • “Test automation tools that slash the time and resources required for test execution. Test automation tools let application development organizations make an upfront investment in test script creation to slash the time required for test execution from a matter of days to a matter of hours.” • “HP isn’t the only big name in testing. Test automation tools from Borland and IBM are more popular with technical testers who prefer to work with powerful scripting languages rather than graphical user interfaces." Copyright © 2008 Borland Software Corporation. Confidential 52 Q+A Copyright © 2008 Borland Software Corporation. Confidential Thank you! Copyright © 2008 Borland Software Corporation. Confidential
© Copyright 2024