Introducing Large Scale Agile testing John Coogan, Oct 2013 Ericsson oss › Operator Support System › 1 Billion Subscribers › 320 Operators › Over 1000 Deployments › PDU OSS Based in Athlone Complex Networks, Simple Management Ericsson Internal | 2012-01-31 | Page 2 Background › Incremental waterfall projects › 2 major releases per year › Back-ended test › Fault profile Ericsson Internal | 2012-01-31 | Page 3 Cumulative › Heavy handover process RFA RFS GA Why Agile? › New product and architecture › New opportunity Change › Culture › Environment › Increased output › Shorter TTM › Maintain high level of quality Speed › Decrease dependencies › Customer feedback Flexibility An Opportunity to Excel Ericsson Internal | 2012-01-31 | Page 4 Ericsson Internal | 2012-01-31 | Page 5 Test Strategy: Key points Quality in Teams › Development Team has test responsibility and ownership › Every sprint potentially releasable › Test and automated part of “done” Continuous Integration › Continuous Integration and test at all levels › Fast Failure, fast feedback Customer Collaboration › Reduced TTM › Customer focus - always on › Release Verification Clean Room and system level performance Agile Principles: Better Quality Ericsson Internal | 2012-01-31 | Page 6 Agile Test Quadrants Business Facing Automated Manual and Automated TAF Usability Tests Exploratory Tests Critique the Product Supporting the Team Acceptance Tests Security Tests Q2 Q3 Q1 Q4 Static Tests Performance Tests Unit Tests Load Tests Basic Integration Tests ility Tests Automated Tools Dev Env TAF Technology Facing Ericsson Internal | 2012-01-31 | Page 7 Team Structure APP1 SM Continuous Integration Release Verification APP2 SM e2e … System Performance Deployment APPn SM Ericsson Internal | 2012-01-31 | Page 8 Deployment Validation Challenges – Test Profile # of Resources › Teams from various backgrounds › Dedicated testers versus generalists › Different levels of adoption › Reliance on others › Mindset! Ericsson Internal | 2012-01-31 | Page 9 Pure Test Focus Development/Test Competence Test/Design Competence Pure Design Focus Challenges – Integration › Parallel development of CI Framework › Manual deployment of test servers › Early and frequent check-in › User story size › Big bang! Ericsson Internal | 2012-01-31 | Page 10 Challenges – Test Environment › Parallel development of platform and deployment procedures › Complex production environments › Evolving toolset › Competence level Ericsson Internal | 2012-01-31 | Page 12 Challenges - Scale › Large number of teams and rising › Strong focus on individual teams › Geographical diversity › Vision gets lost Ericsson Internal | 2012-01-31 | Page 13 BenEfits Ownership Automation and Efficiency Fast Feedback Customer Engagement Quality Improvement Knowledge Sharing and Collaboration Ericsson Internal | 2012-01-31 | Page 14 What have we Learned? › Full versus Incremental Devolution of Test Responsibility › Whole Team Test Approach is, sometimes, not enough › The Value of Release Verification › Strong central strategy, decentralized execution › Test Coaching, Support to Teams › Dedicated Testers in Development teams Ericsson Internal | 2012-01-31 | Page 15 What have we Learned? › Central provision of complex customer like test environment › Building a CI Framework and populating it with quantity and quality test cases takes time › Task force needed to drive key CI implementation › Performance test in development teams, small wins › Running Scrum ceremonies is the easy part… Ericsson Internal | 2012-01-31 | Page 16 Principles behind the Agile Manifesto. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. Business people and developers must work together daily throughout the project. Our highestBuildpriority to individuals. satisfy the customer projects around is motivated Give them the environment and support they need, trust them to get the job done. through earlyandand continuous delivery The most efficient and effective method of conveying information tosoftware and within a development of valuable, quality team is face-to-face conversation. Working software is the primary measure of progress. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. Continuous attention to technical excellence and good design enhances agility. Simplicity--the art of maximizing the amount of work not done--is essential. Ericsson Internal | 2012-01-31 | Page 17 Questions Ericsson Internal | 2012-01-31 | Page 18