How to comply with ISO26262 efficiently – the case for highly automated testing By Steve Barriault Technical Sales & Marketing Manager – Asia © Vector Software Inc, all rights reserved. Company introduction • Founded in 1990 in Rhode Island by embedded engineers • First release of VectorCAST in 1994 • VectorCAST provides unit and integration testing, as well as system test coverage tools that are uniquely automated • These tools can be run on a host, a simulator or directly on a target • Have worldwide offices and representation Netherlands Sweden Rhode Island Japan Georgia Korea Arizona China London France Italy India Israel Software testing under ISO 26262 • ISO 26262 is IEC 61508 adapted to the needs of the Automotive Industry • Adopts a similar approach to software testing and code coverage requirements to other, longer-lived standards (such as DO-178B) • The challenge: meet its requirements in an industry where deadlines are coming up much faster than in Aerospace – In order to achieve success, efficiency will be key! Our experience with standards • Our roots are in the Aerospace industry, where our 15+ year of experience enabled us to work with all the leading organizations • Our Automotive business is expanding quickly, with a growing list of companies using VectorCAST • Requirementsbased testing • Interface testing • Fault injection test • Resource usage test(*) (*) Source: Table 12 and 15: Methods for software unit/integration testing Metric Activities • Unit test • Integration test • System test By Goal By Scope Tests in ISO 26262 • Structural code coverage • Testrequirement association Requirement-base testing • Ensures that the software fulfill its mission – Sometimes called functional test • Strongly recommended for all levels of ASIL, for both unit and integration testing • Stubbing can be performed to enhance your ability to test low-level requirements in isolation – They “replace” your existing code so you can better control your inputs and outputs in the code – But stubs can take a long time to be generated with scripting-based tools – With VectorCAST, the stubs are automatically generated in seconds, with no user input whatsoever Requirement-base testing • The link between the requirement and the test case should ideally be documented – In VectorCAST, it can be. Our unit and integration tests can be linked to specific requirements. – The test case data (PASS|FAIL) that demonstrate requirements can be uploaded to a requirement management system such as DOORS DOORS® requirements VectorCAST Test cases Execution on host, simulator or target Req. 1 Test 1 PASS/FAIL Req. 2 Test 2 PASS/FAIL! Req. 3 Test 3 PASS/FAIL External interface test • External Interface Testing is a subset of functional testing. • Highly recommended for all ASIL level, both unit and integration • It verifies that: – Functions sent data out in the appropriate format and delivery mechanism – Functions that receive data in the appropriate format perform correctly – That the behavior when receiving data that is not formatted directly is known • Can also be tested by VectorCAST Other types of test • Fault injection test: – Voluntarily inject arbitrary faults to test safety mechanisms (ex: by corrupting values of variables) – Recommended for unit/integration testing, strongly recommended ASIL D (and C in integration) – In VectorCAST, can provide test cases that have faulty values and verify that the defensive code gets invoked • Resource usage test: – – – – Often only doable on target or at least simulator Recommended for unit/integration testing Strongly recommended ASIL D Our superior degree of target integration can also help you do some of this, but perhaps not all Generating test case values • Based on requirements – Strongly recommended for all ASIL, unit/integration test • Equivalence classes: – This method may be used to partition possible input values of external interfaces – Strongly recommended ASIL B, C and D – VectorCAST has a facility to generate automatically such partitioned test cases • Error guessing: – Here, the tester tries to test errors that are suspected to be error prone – Only recommended all ASIL – Also easily possible in VectorCAST Generating test case values • Analysis of boundary values – Try values approaching, at, or crossing the boundaries, including out of range value – Can mean the type range or the functional range – Strongly recommended for ASIL B, C and D • VectorCAST has extensive tools to do this – Auto-generation of MIN-MID-MAX test cases for all the extreme variable type values – Import from CSV functional range values – and execution of these in test cases Code coverage • At decision point, both TRUE and FALSE to be executed MCDC • One test case minimum to execute one line of code Branch Statement • Lets you know when you have been “testing enough” • Different criteria that require more or less test cases to achieve • VectorCAST supports all three criteria recommended by ISO 26262 (and the “other criteria” – function/call coverage) • All operands must independently affect the outcome of the condition How coverage criteria stack up • Statement if((a || b) && c) – Line of code executed at least once 1 test case required • Branch if((a || b) && c) T 2 tests cases required – Both the TRUE and FALSE branches are executed • MC/DC F if((a || b) && c) RESULT – All operands can independently affect the outcome a b At least n+1 test case c required © Vector Software, all rights reserved 13 F F T F T F T T F T T T F T F F Knowing what needs to be done! Green: Fully covered (good) Red: Not covered (bad) Orange: Partially covered (?) +/- critical software Statement + Branch + MCDC very critical software What you get - Automaticity • Unit test environments are generated automatically: – All drivers and stubs generated with NO user input • Constructing a test case is done through point and click OR CSV – NO scripting of ANY kind • Both Black Box and White Box are allowed • Execution on target is done 100% automatically – You click a button, and the rest is done for you • Execution reports are generated 100% automatically • Both GUI and command line are available • Re-running any test environment in regression mode is automatic, even if the test cases are modified, or if the underlying code is modified – Regression testing is completely automated What you get - Flexibility • Full, guaranteed support for C/C++ of ANY complexity • Users can control the value of ALL parameters, return values, global data and data coming from stubs – even for pointers, exceptions, etc. • Can test individual values, special values (NAN, positive infinity, etc), range of values, list of values, even call code to generate Monte Carlostyle of test cases • Can create complex test cases that set state machines and test their transition from one state to another (compound test cases) • Creation of test cases from CSV • Automatic test case generation based on basis path analysis, MIN-MIDMAX, and more so as to give a leg up during structural coverage • Code coverage is displayed in an easy-to-understand way Other capabilities you get… • The ability to test libraries – even if you don’t have access to the code! • The capability to test as you develop (agile development), or even to first write test cases before writing code (test-driven development) • The debugger can be used to control test case execution (so the tool becomes a test vector generator for debugging too) • The ability to perform timing calculations, in some conditions can also be done while other processes from the OS are executing • The ability to stub library functions, if desired… • … and much more Quite simply, you are getting a complete test bench that enables you to comply with ISO 26262 efficiently, so you can still meet demanding deadlines! What about tool qualification? • ISO 26262 Part 8, Section 11 mentions tool qualification – The objective of the qualification of software tools is to provide evidence of software tool suitability for use when developing a safety-related item or element, such that confidence can be achieved in the correct execution of activities and tasks required by ISO 26262 • Depends on how critical the tool’s reliability is to the quality of the code • Process more flexible than DO-178B, but if you need tool qualification, Vector Software has a long experience of providing this service – We can provide you documents demonstrating the tool performance as adequate in your environment: same compiler version, board, debugger, and tool version Conclusion • ISO 26262 contains a number of recommendations that have been proven effective in other industries – It does not reinvent the wheel but builds on a rich heritage to customize an unique standard to the needs of the Automotive industry • Unit/integration testing and code coverage can be very timeintensive, which no one in this industry can afford! • The only way to meet ISO 26262, increase code quality and still meet your deadlines is to invest in test automation – Fortunately, VectorCAST tools have both the automation and flexibility required for you to achieve ISO 26262 compliance in a timely manner Questions?