QUALITY ASSURANCE
2
Quality Assurance
Can Refer To
• A set of activities designed to ensure software quality
• A department in a software development organization
3
Quality Assurance Overview
• Validation
• Are we building the right product?
• Focused on customer requirements
• Collaborative activity: Involves customer
• Activity: Testing
• Verification
• Does code meet spec?
• Internal activity: Developers/QA only
4
Verification
• Testing
• Execute program to check for bugs
• Tracing / Formal Inspection
• Review source code to check for bugs
• Formal Verification
• Prove source code correctly implements its specification
5
The Role of Quality Assurance
• An independent division in a development organization
• QA representatives participate in development process
from start to finish
• While developers design and code, QA plans tests
• Effective bug squashing requires ___________ between
developers and QA
6
TESTING
7
Testing
• Addresses the ultimate question: “Does it work?”
• Which mind set is correct?
• Test to show that the program meets requirements
• Test to find bugs
8
Why test?
• Purpose: Discover Errors
• Not to show that a program is “correct”
• A successful test reveals a previously undiscovered error
• How should a tester feel about a test in which the program
performs correctly?
• Why do developers make poor testers?
9
Qualities of a Good Tester
Do you have...
• A compulsive desire to break things?
• A feeling of joy and personal fulfillment when you uncover
hidden flaws?
• A dogged determination to slog through tedious specs
coupled with a twisted creative streak that enjoys
dreaming up unlikely scenarios allowed by those specs?
• The technical skill to write automated tests?
• An iron will to succeed through methodical, calculated
design and execution of deviously difficult test cases?
• Are you OCD?
10
White Box vs. Black Box Testing
• Black Box
• External Perspective
• Does not use knowledge of implementation details
• Feed in input, observe output
• White Box Testing
• Internal Perspective
• Select input based on knowledge of implementation details
• May use debugger to step through code
• Advantages / Disadvantages
11
Test Scopes
• Unit Testing
• Integration Testing
• Validation Testing
12
Regression Testing
• Checks whether code modifications introduced bugs that
were not present in the previous build
• Done by QA
13
Agile QA Techniques
• Test Driven Development
• Continuous Integration
14
Test Driven Development
• Key Agile Development Technique
• Write Unit Test Before Implementing Functionality
• Process:
1. Write spec for a new method to be implemented
2. Create stub method (empty body)
3. Create unit test for method
4. Run test (observe failure)
5. Implement method
6. Run unit test
7. Debug if fails
• Advantages?
15
TEST CASE DESIGN
16
Test Cases
• A test case consists of
• Input Data
• Expected Results
• Good test cases have a high probability of finding an
undiscovered error
• Goal for Test Case Design
• Design test cases that systematically uncover different classes of
errors with minimal effort
17
Specification Based Testing
Design test cases for a method using
• Specification: _________ box testing
• Implementation: ________ box testing
Specification based testing:
• Design test inputs that conform to precondition
• Compute expected test outputs according to spec
18
Specification Testing Example 1
Operation Transfer(clears S: Stack; replaces T: Stack);
ensures T = Reverse(#S);
Input:
#S = ?
#T = ?
Expected Output:
S=?
T=?
19
Specification Testing Example 2
Operation Mystery(clears Num: Integer;
updates S1, S2: Stack);
requires 0 <= Num and Num <= |S1|;
ensures S1 o S2 = #S1 o #S2 and #Num = |#S1| - |S1|;
Input:
#Num = ?
#S1 = ?
#S2 = ?
Expected Output:
Num = ?
S1 = ?
S2 = ?
20
Specification Testing Example 3
Operation Mystery(replaces E: Entry; updates S: Stack);
requires |S| > 1;
ensures E = DeString(Prt_Btwn(1, 2, #S)) and
S = Prt_Btwn(0, 1, #S) o Prt_Btwn(2, |#S|, #S);
Input:
#E = ?
#S = ?
Expected Output:
E=?
S=?
21
Designing Test Cases
• So far, we've picked test inputs somewhat randomly
• Want to design a set of effective test cases that
efficiently uncovers bugs.
• How?
22
Testing with Equivalence Classes
• Equivalence classes guide efficient test case design
• An equivalence class is a set of elements, all of which
share something in common
• Example:
• Most input is restricted in some way
• Thus, two equivalence classes for each input value:
• Legal input
• Illegal input
23
Case Study
• A screen requires input of a single numeric value
• The legal range is 0..100
• What are the equivalence classes?
• How many test cases needed?
24
Equivalence Classes, cont.
• Fence post errors are common
• Not all legal inputs are fully equivalent
• Consider these equivalence classes:
• Well within range
• Well out of range
• Just within range
• Just out of range
• What are the equivalence classes for the previous
example?
25
Illegal Inputs and Preconditions
• Two types of inputs
• From external sources (ex. User Interface, file, network)
• From internal sources (within program, via method parameters)
• Program specs should define how to handle invalid
external input
• What about invalid parameter values?
• Remember: Preconditions typically specify valid parameter values
26
Illegal Inputs and Preconditions
• Consider this method
// computes factorial of num
// precondition: 0 <= num <= 12
int factorial(int num) { … }
• Suppose num's value originates from external source
• What are the equivalence classes for factorial?
• Should factorial be tested with illegal inputs?
27
An Aside: Defensive Programming
• UI layer is responsible for validating all inputs and ensuring that
preconditions are met for all methods it calls
void btnFactorial_click(ActionEvent e) {
String numStr = txtNum.getText();
// convert numStr to int num and verify it meets factorial preconditions
int result = factcontroller.factorial(num);
// display result
}
• Application logic layer can assume that preconditions are met
class FactorialController {
int factorial(int num) {
if (num == 0) return 1; else return num * factorial(num-1);
}
}
28
Defensive Programming, cont.
But what if a bad user input "slips through" the UI validation
defense shield and violates a non-UI method precondition?
• How could this happen?
• Want to catch this case! (Why?)
• What should the non-UI method do if handed a bad
parameter value?
• Options?
• Don't want to write a lot of redundant validation code
• Bloats / complicates the code
• More code to test
Case study: Windows 3.1
29
Assertions and Preconditions
Use assertions to check preconditions
• assert causes runtime crash if condition is false
• UI Layer
void btnFactorial_click(ActionEvent e) {
String numStr = txtNum.getText();
// validate numStr and convert…
int result = factcontroller.factorial(num);
// display result
}
• Application Logic Layer
class FactorialController {
int factorial(int num) {
assert num >= 0 && num <= 12;
if (num == 0) return 1; else return num * factorial(num-1);
}
}
30
Assertions and Preconditions
• Assertions are a "last ditch" defense against bad
parameter values
• Benefits
• Shorter than if statements
• Help reveal bugs in calling code
• Can be disabled if desired when application ships
• Note: Never appropriate to use assertions to do UI-level
validation
• Why?
31
Summary: Designing Test Cases
1. Define equivalence classes for each input
2. Design test cases based on equivalence classes
This sounds academic... but it works!
32
Case Study
A program computes an employee’s pay based on
• Hourly rate
• Number of hours worked
• > 40 hours yields overtime
• Number of insured dependents
• > 1 insured dependent leads to deductions
33
FORMAL VERIFICATION
34
"Beware of bugs in the above code; I have
only proved it correct, not tried it."
- Donald Knuth, 1977
35
Formal Verification
• Proof of correctness of code
• Assuming preconditions are met, proof demonstrates that
the code ensures that the postconditions are met
• What if preconditions are not met?
• Example: Invalid parameter values
• Can a proof have errors?
36
Automated Formal Verification
• Holy Grail of Verification Research:
• Given code and specs, automatically prove that code meets specs
• Requirements:
• Specs must be written formally
• Approaches:
• Add annotations to existing programming languages
• Design “verifiable” programming language
37
RESOLVE Prover
Developer writes
• Specs
• Assertive Code
RESOLVE Prover
• Generates Verification Conditions (VC's)
• Mathematical assertions about code
• Attempts to prove VC’s
Demo: Int_Do_Nothing_Realiz
38
Assertive Code
Some RESOLVE statements (ex. While) can contain assertions
that assist the prover in verifying correctness
Operation Multiply(clears i: Integer; updates j: Integer);
Procedure
Var s: Integer;
While (i > 0)
changing i, s;
maintaining (s + i * #j = #i * #j) and (i >= 0);
decreasing i;
do
s := s + j;
i := i - 1;
end;
j :=: s;
end Multiply;
39
QA Summary
Consider Testing, Tracing, Formal Verification
• Which require program execution?
• Which can show absence of bugs?
• Which require developer involvement?