How to Build Process Performance Models (PPMs) using only five (5) datapoints AGENDA:  Concept of Process Performance Models (PPMs)  Challenges in PPM Building  Concept of Bayesian Belief Network (BBN) (including statistical concepts applied such as binomial distribution, conditional probability and bayesian probability)  PPM Building Process using BBN approach  Selection of Critical Sub-Processes using BBN-PPM 2 Concept of Process Performance Model (PPM)  The CMMI definition of Process Performance Model is “a description of the relationships among attributes of a process and its work products that are developed from historical process-performance data and calibrated using collected process and product measures from the project and that are used to predict results to be achieved by following a process.” Historical Data 3 Challenges in Process Performance Modeling Challenges:   LIMITED QUANTITATIVE DATA low variability of available data 4 Concept of Bayesian Belief Network (BBN)  Bayesian Belief Network (BBN) is a probabilistic graphical model that represents a set of variables and their probabilistic independencies (e.g. the probabilistic relationships between diseases and symptoms) Historical Data Experts Belief 5 Concept of Bayesian Belief Network (BBN)  Figure below is an example of a Bayesian Belief Network. It is a graphical representation of the underlying probabilistic relationships of a complex system  The calculation and analysis of a BBN took advantage of conditional and marginal independences among random variables Burglary P (b) Earthquake Alarm P (e) P (a I b,e) 6 Statistical Concepts Applied_Binomial Distribution Binomial Distribution (Probability of Success):  is the probability of the number of successes in the N outcomes (given N number of Yes/No experiments)  e.g. Determine the probability of obtaining exactly 3 heads if a fair coin is flipped 6 times? Given: r = 3, N = 6 and π = 0.05 Where: P(r) - is the probability of exactly r successes r – is the number of successes N - is the number of events π - is the probability of success on any one trial. Assumptions: - The number of observations n is fixed - Each observation is independent - Each observation represents one of two outcomes ("success" or "failure") - The probability of "success" p is the same for each outcome 7 Statistical Concepts Applied_Conditional Probability Conditional Probability:   is the probability of an event occurring given that another event has already occurred e.g. Figure below illustrates the conditional probability of A given B and B given A P(A|B) = A A,B P(A,B ) P(B) ; B P(B|A) = P(B,A ) P(A) 8 Statistical Concepts Applied_Bayesian Probability Bayes’ Probability:    Bayes' Theorem is a simple mathematical formula used for calculating conditional probabilities It is one of the different interpretations of the concept of probability and belongs to the category of evidential probabilities The Bayesian interpretation of probability can be seen as an extension of logic that enables reasoning with uncertain statements P(A|B) = P(BIA )P(A) P(B) where: P(A) : is the prior probability of A: the probability that A is correct before the data B are seen. P(B|A) : is the conditional probability of seeing the data B given that the hypothesis A is true. This conditional probability is called the likelihood. P(B) : is the marginal probability of B. P(A|B) : is the posterior probability: the probability that the hypothesis is true, given the data and the previous state of belief about the hypothesis. 9 High Level Overview on PPM development through Bayesian Belief Network (BBN) Identification of Response (Y) Identification of PPM Scope Identification of Influencing factors Data Collection (Quantitative Data and Qualitative Information) Probability Calculations Selection of Critical Sub-process 10 Model Building – Identification of Response  Identification of PPM response (Y)  Typical responses (Ys): Ticket Resolution Duration Ticket Resolution Effort Backlog Processing Efficiency 11 Model Building – Identification of Stages / Nodes   Identification of stages that impacts the response PPM scope are identified based from projects’ lifecycle Response (e.g. Ticket Resolution Duration) Projects’ Stages / Nodes: Stage 1 Stage 3 (e.g. Investigation) (e.g. System Integration) Stage 2 (e.g. Fixing) 12 Model Building – Identification of Influencing Factors  Identification of factors influencing the response through brainstorming with projects’ SMEs as well as its measures Ticket Allocation Skill of Resources Testing Process Ticket Complexity Review Process Stage 2 (e.g. Fixing) 13 Model Building – Collecting Historical Data  Collect project’s historical data Yes – if the factor / node is applicable to the ticket or requirement No – otherwise 14 Model Building – Probability Computations  Compute for the Binomial and Conditional probabilities Binomial Probabilities (Probability of Success) - represents the current probability of success of each factor Influencing Factors Probability of Success (Computed) Code Probability of Success (Experts’ Belief) Code Ticket Complexity Skill of Resources Ticket Allocation Testing Process Review Process 100% 40% 80% 60% 60% 100% 40% 80% 60% 60% Conditional Probabilities (Weights) - represents the impact of each factor to the response (for that specific stage) Influencing Factors Conditional Probability (Computed) Code Conditional Probability (Expert's Belief) Code Ticket Complexity Skill of Resources Ticket Allocation Testing Process Review Process 25% 17% 16% 34% 7% 25% 20% 15% 30% 10% 15 Model Building – Probability Computations  Compute for Stage Probabilities based from binomial and conditional probabilities of each factor Ticket Allocation P(F2) 40% P(F1) 100% Skill of Resources P(F3) 80% Testing Process Ticket Complexity P(F4) 60% Review Process P(F5) 60% Stage 2 (e.g. Fixing) Stage Probability 65% 16 Model Building – Probability Computations  Compute for End to End Probabilities using Conditional Probability calculations: Stage 1 Stage 2 Stage 3 Response (e.g. Investigation) (e.g. Fixing) (e.g. System Integration) (e.g. Ticket Resolution Duration) Stage Probability  77% Stage Probability 70% Stage Probability 79% End to End Probability 75% The End to End Probability represents the current probability of meeting the goal, given current project’s performance. It would help project to know how well they are performing with respect to their goal. 17 High Maturity – Selecting Critical Sub-processes  Select the priority stage based on the values of stage probabilities then select the critical sub-process based from binomial and conditional probability of each factor Stage 1 Stage 2 Stage 3 Response (e.g. Investigation) (e.g. Fixing) (e.g. System Integration) (e.g. Ticket Resolution Duration) Stage Probability 77% Stage Probability Influencing Factors 70% Stage Probability 79% End to End Probability 75% Ticket Complexity Skill of Resources Ticket Allocation Testing Process Review Process 100% 40% 80% 60% 60% 25% 20% 15% 30% 10% Code Probability of Success Conditional Probability Code Code 18 Benefits of using BBN Models • In summary: why should we use BBNs? • BBNs on their own enable us to model uncertain events and arguments about them. The intuitive visual representation can be very useful in clarifying previously opaque assumptions or reasoning hidden in the head of an expert. • With BBNs, it is possible to articulate expert beliefs about the dependencies between different variables and BBNs allow an injection of scientific rigour when the probability distributions associated with individual nodes are simply 'expert opinions'. • BBN will give an idea of what stage and factors are greatly impacting the response. • BBN can be used to quantify how much improvement in the selected critical sub- process is required for the project to meet their goal. 19 References:  PPQA Website Link: http://ppqc.blogspot.com/2008/08/help-understanding-process-performance.html  Bayesian Network – Psychology Wiki Site Link: http://psychology.wikia.com/wiki/Bayesian_network  SEI Website Link: http://www.sei.cmu.edu/library/assets/practitioner-viewattach.pdf  HyperStat Online Website Link: http://davidmlane.com/hyperstat/A2301.html 20 Contact Information  Jeanette O. Ruiz Accenture Philippines Email Address: [email protected]  Amir Khan KPMG Consultant Email Address: [email protected] 21