How to Forecast Repairable System Failures? ASA SPES-QP Webinar Nov. 14, 2011 Larry George Problem Solving Tools 1 11/11/2011 Learning Objectives Make actuarial forecasts of E[failures] for repairable systems Quantify uncertainty Recommend warranty reserves Use available data, fully 2 11/11/2011 3 11/11/2011 Eggs Delimit Sections: Background Section 4 11/11/2011 Want Better Service and Spares Forecasts? “Original data should be presented in a way that will preserve the evidence in the original data for all the predictions assumed to be useful.” Rule #1, Walter Shewhart Preserve all relevant information in data Make no unwarranted assumptions Quantify randomness, approximations, and uncertainty Hindcast = Observations 5 11/11/2011 Forecast SOTA Please enter your forecast_____ Naïve: installed base*ARR (Oracle) Simple: demand rate = ARR, 1/MTBF, or SWAG (theorem when hell freezes over) Fancy: Power law, NHPP or “Weibull process” These are expected values, not demand distribution or parameters 6 11/11/2011 Repairable System SOTA Renewals: N(t) = failures in [0,t) E[N(t)] ≡ M(t), approx. M(t) and Var[N(t)] Walter Smith: Renewal with trend Lajos Takacs: dependent availability Crow-AMSAA-Duane, RG-6, Nelson, Ascher-Feingold, Trindade et al. ∂M(t)/∂t ~ αt-β For one system! 7 11/11/2011 ReliaSoft SOTA http://www.weibull.com/hotwire/issue58/relb asics58.htm “Monitoring Warranty Returns using SPC” Assume: iid errors, Weibull mle, CI on E[N(t)] is SPC limit, extrapolation Month Ships Jul-05 Aug-05 Sep-05 Jun-05 100 3 3 Jul-05 140 Aug-05 150 2 5 Weibull Lse fit to npmle α 12.82 Mle Β 2.58 6.1 4 4 1.54 8 11/11/2011 9 11/11/2011 Invalid Assumptions Constant failure rates, independent increments, memoryless, iid renewal NHPP or Weibull: different ages, CI on power law exponent Demand ~ Poisson, normal, negative binomial (patented by Robin Roundy) Confidence interval = prediction interval Need ages at failures 10 11/11/2011 Can’t Estimate CI on Power Law Exponent “…though you may have a high p value, all it says is that the power law hypothesis is a valid explanation for the data, nothing more. The uncertainty in the tail exponent is such that the estimate becomes quite meaningless…“ “…huge outlier changes tail behavior completely.” (Konstantin Litovsky) 11 11/11/2011 Nevada Table Highlighted data are statistically sufficient 12 11/11/2011 US AFLC Forecast Σn(t)a(t) 13 11/11/2011 Actuarial Forecast = Σn(t)a(t) n(t) is installed base (survivors) a(t) is actuarial failure rate In forecast calendar interval, of ages t = 1,2,…,k+1, where k is the oldest unit Compared with N/MTBF it’s still simple Ulpianus ca 220 AD Roman legion pensions Requires extrapolation of a(k+1) only Reliability R(t) = exp[-Σa(t)] 14 11/11/2011 Advantages & Disadvantages Advantages: information Unbiased, more precise and less uncertainty, no unwarranted assumptions Quantify randomness, sample uncertainty, and extrapolation uncertainty Disadvantages: cost and work N/MTBF requires only N and MTBF Actuarial forecast requires installed base and actuarial rate estimates 15 11/11/2011 Nevada Table Highlighted data are required by GAAP 16 11/11/2011 Mike says, Warren Buffet acts Why make actuarial warranty reserve forecast? “For book purposes, it [warranty reserve] is also a deduction against income but, alas, not for tax purposes… The only enterprise that I can think of that can set up a reserve and deduct it is an insurance company.” (Mike) Warren Buffet bought GEICO So forecast warranty reserves actuarially and expense them 17 11/11/2011 Job Interview and Σn(t)d(t) 18 11/11/2011 Repairable Forecast = Σn(t)d(t) Renewal process: TBF, TBF,… iid E[N(t)] = Σ Fn(t) (convolution), so d(t) = E[N(t)] − E[N(t−1)] d(t) = a(t) if failures stay dead Companies track TTFF, TBF1, TBF2,… by serial number, unnecessarily 19 11/11/2011 Nevada Table Highlighted data statistically sufficient, even for repair processes 20 11/11/2011 Generalized Renewal Forecasts Generalized renewal process is TTFF, TBF, TBF,… TBFs are iid E[N(t)] = Σ Fo*Fn-1(t) (convolution), TTFF ~ Fo(t), sum over n Demand rate d(t) = E[N(t)]−E[N(t−1)] Actuarial Forecast = Σn(t)d(t) 21 11/11/2011 Repair Process Distributions 22 11/11/2011 Repairable Forecast = Σn(t)d(t) Repair process is TTFF, TBF1, TBF2,… indep. but not identical M(t) = Σ Fo*F1*F2*…*Fk-1(t) TTFF ~ Fo(t), TBF1 ~ F1(t), TBF2 ~ F2(t),… (∂M(t)/∂t)∆t = d(t) Test F1(t) ~ F2(t) ~… KolmgorovSmirnov test for censored data 23 11/11/2011 Nevada Table Highlighted data statistically sufficient, even if Oracle dbs hide repair counts of same parts 24 11/11/2011 Some Extrapolation Required 25 11/11/2011 Component D Failure Rate Extrapolations http://www.fieldreliability.com/QPMeeker.doc 26 11/11/2011 Component D: Nonparametric vs. Weibull Estimation P[Life ≤ 36 months] npmle & linear extrapolation nplse & linear extrapolation mle Weibull 0.0100 (95% UCL =0.027) 0.0100 lse Weibull 0.0079 (95% UCL =0.014) 0.0094 27 11/11/2011 Forecast Requires Extrapolation Only for oldest unit in installed base a(k+1), d(k+1), and perhaps n(k+1) Stepwise, piecewise linear extrapolation a la Test Conconi ≈ ___/ maybe \____/? Vary knot point and minimize Σ[â(t) – piecewise linear(t)]2 â(t) 28 11/11/2011 Test Conconi 29 11/11/2011 PLFR parameters Test Conconi for PLFR Estimated a(t) Winner Extrapolat ed a(t) 30 11/11/2011 PLFR Actuarial Rate Extrapolations 31 11/11/2011 Your Brain on Math 32 11/11/2011 Math Topics Confidence and prediction limits Estimate variance-covariance matrix Σa of â(t) Estimates are NOT uncorrelated! Extrapolate Σa entries Randomness vs. sample uncertainty Fix, ECO Dependence of TTFF, TBF1, TBF2,… 33 11/11/2011 Forecast Confidence Interval “Any measurement that you make without knowledge of its uncertainty is completely meaningless” Walter Lewin, MIT â(t) estimators are correlated! Var[Σn(t)d(t)] ≡ σ2 = nΣdn’ Simulate â(1), â(2),…, â(k) from N(â, Σa) Compute d(1), d(2),…,d(k), Σd 1-α% CI is ~Σn(t)d(t) ± σzα/2 34 11/11/2011 Sample Extrapolate Σ Using PLFR WIP σt 2 = Σ WIP 0 0 Var[a+bt] = Var[a]+t2Var[b] Var[] = “standard error” squared σk+12 σk+22 σk+32 Extrapolate 35 11/11/2011 Extrapolate VarianceCovariance Matrix Regression extrapolation gives standard errors of coeffs => variance of a(k+1) and cov(a(k+i),a(k+j)) k is age of oldest units cov(a(i),a(k+j)) for i ≤ k is WIP Assume cov(a(k+i),a(k+j)) = 0 (neutral) 36 11/11/2011 Cov(â(t), â(t’)) ≠ 0 Estimate Empirical, from broom charts Estimate â(t) from population subsets (Jerry Ackaret’s broom charts) Compute their covariances (test for stationarity!) Cramer-Rao lower bound Nelson-Aalen Σa ≈ E[∂2lnL/∂a(t)∂a(t’)]-1 37 11/11/2011 Prediction Limit on Demand Not same as confidence interval on expected demand (aka forecast) Simulate demands from each age interval N[{d(t)}, Σd] d(t) N[n(t)d(t), √n(t)d(t)(1-d(t))] Repeat. Prediction limit = %ile Sample and extrapolation uncertainty > randomness 38 11/11/2011 Hindcasts, Forecasts, and Prediction Limits 39 11/11/2011 What if there’s a fix, ECO? → d(t;new) at some calendar time Forecast = Σn(t;old)d(t;old) + Σn(t;new)d(t;new) + Σn(t;repaired)d(t;new) d(t;old) First sum is before fix, second and third are after fix Delete old repaired units from n(t;old): put in last sum! 40 11/11/2011 Repair Process with a Fix 41 11/11/2011 Dependence Bounds on distributions of order statistics for dependent TTFF, TBF1,… => bounds on forecasts Copulas generate marginals with specified correlations Distribution-free bounds on EOQ! 42 11/11/2011 Ancient Math Still Works 43 11/11/2011 Recommendations Do risk, uncertainty, and cost justify more than ships and returns counts? Actuarial warranty reserves may be expensed Preserve all relevant info. in data. Use it. Quantify its uncertainty. Don’t assume. Forecast = ΣΣn(j;t)a(j;t) so hindcasts = observations 44 11/11/2011 Learning Objectives Make actuarial forecasts of E[failures] for repairable systems Quantify uncertainty Recommend warranty reserves Use available data, fully 45 11/11/2011 Some References Agrawal, Vipul and Sridhar Seshradi, “Distribution-Free Bounds for Service-Constrained (Q,r) Inventory Systems,” NRL Vol. 47, 2000 Ascher, H. and Feingold, H. (1984). Repairable Systems Reliability. New York, NY: Marcel Dekker, Inc. Caraux, G. and O. Gascuel, “Bounds on distribution functions of order statstics for dependent variates,” Stats & Prob. Letters, Vol. 14, pp. 103105, 1992 Hunter, David, “Approximating percentage points of statistics expressible as maxima,” TIMS Studies in Management Science, Vol. 7,pp. 25-36, 1977 Levi, Retsef, Georgia Perakis, and Joline Uichanko, “The Data-Driven Newsvendor Problem: New Bounds and Insights,” submitted to Operations Research, http://web.mit.edu/uichanco/www/data-driven.pdf Smith, Walter L., “Remarks on renewal theory when the quality of renewals varies (1),” 36th Session of the International Statisistics Institute, Sidney, Aust., Sept. 1967 46 11/11/2011 Rotten Egg Nebula 47 11/11/2011 Progress in Artificial Stupidity: “Sampling theory is needed for exploration and ultimate criticism of an entertained model in light of current data, while Bayes’ theory is needed for estimation of parameters conditional on the adequacy of the entertained model.” G. E. P. Box NUREGs say P[Failure] ≈ p+λ vs. real P[Failure] = p+(1−p)(1−e−λ) (assuming independence) P[2-out-of-2 fail] ≈ (p+2λ)2 vs. (1−p)2e−2λ−λcc (λcc is a common cause) Earthquake initiating event, Sample distributions with estimated p and λ ∫P[Failure|Eq of mag. u]dF(u) Exchangeability: Does (Y1,...,Yn) have same distribution for all permutations? Is “posterior” p+λ relevant to a single plant? 48 11/11/2011