Test How to Succeed in Test Automation Björn Hagström & Davor Crnomat, Testway AB Introduction • • • • • • The need for Automation Cost of Automation Plan to Automate Tools Test Approach Summary The Need for Automation • • • • Expanding regression suites Rapid release-cycles Test complexity Repetitive tasks degrade and divert attention • ... Cost of Automation Cost Manual Testing Automated Testing Test Cases run Plan to Automate • Treat an automation project as you would treat any other development • Start test development early • Allocate resources • Do not forget training Test Automation Project Requirements There need to be clear and verifiable requirements of what the automated tests should do Analysis How the tests are interacting with the AUT, how to obtain test data etc. Design Error handling and logging, robust and scalable tests Test Verify that the tests indeed find and logs defects Maintenance Scalability and knowledge transfer Identify areas with potential gain Documentation Create Test Data Test Setup Test Execution Go for the juiciest targets first! Tool Selection • Do we really need automation? What problems are we trying to solve? • Is test automation’s solution cost effective? • Write long list of potential tool candidates http://www.sqa-test.com/toolpage.html http://www.stickyminds.com/tools.asp • Prepare list of required features and use it to reduce the list • Contact the tool vendors • Evaluate tools (max 3-4 tools) Tool Requirements • • • • • • • • • • • • • • • Easy of use, easy of maintaining Platform support Data base support, File I/O Tool customization Test result logs and reports Scripting language (arrays, list, structures, conditional logic, loops, create and call functions…) UI element identifiers (identify UI elements as objects) GUI Map (name mapping of UI identifiers) Error handling, Unexpected error recovery The debugger Command line script execution Vendor (Maturity, financial stability, market share, support) Licensing, pricing The user community Etc. Contact Tool Vendors • Questions to ask tool vendors: 1. 2. 3. 4. • What does this tool do? What are benefits of using this tool? How can this tool solve my problems? Why is this tool better then other similar tools? Etc. Questions to ask tool users: 1. 2. 3. 4. How does this tool perform? Are there any bottlenecks? What is your impression of the vendor Which other tools did you consider when purchasing this tool? Etc. An extensive list of the questions you can find here: http://www.grove.co.uk/pdf_Files/STT_Questions.pdf • Ask vendors for demonstration Tool Evaluation • Read publications which have evaluated some of testing tools from your list • Summarize information that you got from vendors and users • Conduct an in-house competitive trial It should be run by the evaluation team. If you do it alone, don’t ”fall in love” and don’t let the license price be your main issue. • Make recommendation/ decision • Write a short written report Tool Implementation • Set up team • Start pilot project Pilot project is short-term, non-critical and significant. Implementation, maintenance and failure analysis is explored* • Evaluate pilot project • If pilot project is successful plan for phased installation * Software testing practitioner - Grove Consultants Automation Test approach Test automation is software development: • • • • • • • • • Code review Code standards Naming conventions Error handling Reusability Documentation Source control Maintainability Etc. Automation Test Automation test should be: • • • • • • Independent Maintainable Reliable Robust Readable Documented Capture/Replay Test automation tools vendors’ marketing: “Tests can be created quickly and easily by non-programmers.” ”Test Automation is very easy. Just record the scripts at any given point of time and replay them whenever you want to.” Do you really believe it? Good Automation Practices I • • • • • • • • Plan automation Educate developers on the automation requirements Start small and focus on modularity and reusability Build and document libraries Use wrapper functions Build initialization and clean-up routines Apply error handling Build error recovery Good Automation Practices II • • • • • • • • • Use GUI maps Make your code machine independent Avoid Bitmap comparisons, Use API Use data-driven testing technique Make test units short Log your tests Add comments to your script’s code Document your tests Keyword-driven testing Advantages: • Testers don’t need programming skills to build new test scenarios – involve more users • Modularity and reusability • Test scenarios can be used as test documentation Disadvantages: • Time to set up initial framework • Programmers dedicated to implementing and maintaining the framework are still needed. • It could be hard to control test cases (spread sheets) • Time consuming to learn all keywords Summary • Do not try to automate everything • Treat test automation as a software development • Tool selection is a small project • Consider alternative solutions ? Remember! Enter the evaluation form and be a part of making Øredev even better. You will automatically be part of the evening lottery