Accendo Reliability

Your Reliability Engineering Professional Development Site

  • Home
  • About
    • Contributors
  • Reliability.fm
    • Speaking Of Reliability
    • Rooted in Reliability: The Plant Performance Podcast
    • Quality during Design
    • Critical Talks
    • Dare to Know
    • Maintenance Disrupted
    • Metal Conversations
    • The Leadership Connection
    • Practical Reliability Podcast
    • Reliability Matters
    • Reliability it Matters
    • Maintenance Mavericks Podcast
    • Women in Maintenance
    • Accendo Reliability Webinar Series
    • Asset Reliability @ Work
  • Articles
    • CRE Preparation Notes
    • on Leadership & Career
      • Advanced Engineering Culture
      • Engineering Leadership
      • Managing in the 2000s
      • Product Development and Process Improvement
    • on Maintenance Reliability
      • Aasan Asset Management
      • CMMS and Reliability
      • Conscious Asset
      • EAM & CMMS
      • Everyday RCM
      • History of Maintenance Management
      • Life Cycle Asset Management
      • Maintenance and Reliability
      • Maintenance Management
      • Plant Maintenance
      • Process Plant Reliability Engineering
      • ReliabilityXperience
      • RCM Blitz®
      • Rob’s Reliability Project
      • The Intelligent Transformer Blog
    • on Product Reliability
      • Accelerated Reliability
      • Achieving the Benefits of Reliability
      • Apex Ridge
      • Metals Engineering and Product Reliability
      • Musings on Reliability and Maintenance Topics
      • Product Validation
      • Reliability Engineering Insights
      • Reliability in Emerging Technology
    • on Risk & Safety
      • CERM® Risk Insights
      • Equipment Risk and Reliability in Downhole Applications
      • Operational Risk Process Safety
    • on Systems Thinking
      • Communicating with FINESSE
      • The RCA
    • on Tools & Techniques
      • Big Data & Analytics
      • Experimental Design for NPD
      • Innovative Thinking in Reliability and Durability
      • Inside and Beyond HALT
      • Inside FMEA
      • Integral Concepts
      • Learning from Failures
      • Progress in Field Reliability?
      • Reliability Engineering Using Python
      • Reliability Reflections
      • Testing 1 2 3
      • The Manufacturing Academy
  • eBooks
  • Resources
    • Accendo Authors
    • FMEA Resources
    • Feed Forward Publications
    • Openings
    • Books
    • Webinars
    • Journals
    • Higher Education
    • Podcasts
  • Courses
    • 14 Ways to Acquire Reliability Engineering Knowledge
    • Reliability Analysis Methods online course
    • Measurement System Assessment
    • SPC-Process Capability Course
    • Design of Experiments
    • Foundations of RCM online course
    • Quality during Design Journey
    • Reliability Engineering Statistics
    • Quality Engineering Statistics
    • An Introduction to Reliability Engineering
    • An Introduction to Quality Engineering
    • Process Capability Analysis course
    • Root Cause Analysis and the 8D Corrective Action Process course
    • Return on Investment online course
    • CRE Preparation Online Course
    • Quondam Courses
  • Webinars
    • Upcoming Live Events
  • Calendar
    • Call for Papers Listing
    • Upcoming Webinars
    • Webinar Calendar
  • Login
    • Member Home

by Larry George 1 Comment

Variance of the Kaplan-Meier Estimator?

Variance of the Kaplan-Meier Estimator?

The well-known variance of the Kaplan-Meier reliability function estimator [Greenwoood, Wikipedia] can drastically under-or over-estimate variance. The covariances of the Kaplan-Meier reliability pairs at different ages are ignored or neglected. Variance errors and covariance neglect bias the variance of actuarial demand forecasts. Imagine what errors and neglect do to confidence bands on reliability functions.

The Kaplan-Meier reliability function estimator is used when time-to-failure data are censored and grouped by start period cohorts. References about its variance-covariance matrix are scarce. “Survival and Event History Analysis…” page 91 lists a slight difference from the Greenwood variance for the Nelson-Aalen estimator cumulative failure rate function [Aalen et al.]. Wayne Nelson sent me citations of his paper and an earlier paper by Odd Aalen [Nelson]. They show the cumulative failure rate function estimates are independent. However, the Kaplan-Meier reliability and actuarial failure rate function estimates are not. I use Mathematica to compute the Cramer-Rao bound on variance-covariance matrix for small numbers of cohorts. See reference by Paul Tune for computation methods that avoid inversion of Fisher information matrix.

Pointwise confidence limits on reliability estimates use the Greenwood variance formula [Sawyer, Freedman]. There are legitimate confidence bands on the Kaplan-Meier reliability function estimator [Hall and Wellner], based on the covariance function of estimator’s asymptotic multivariate normal distribution. The confidence bands are a function of unknown underlying distributions of failure and censoring times. “Programs for the calculation of the bands are available from the authors.” [In C, https://sites.stat.washington.edu/jaw/RESEARCH/SOFTWARE/software.list.html/] I need the variance-covariance matrix of actuarial failure rates corresponding to the Kaplan-Meier estimator for the variance and standard deviation of actuarial failure forecasts, for logistics, spares stock levels, availability, etc. 

In a previous article I used the Cramer-Rao bound on the variance-covariance matrix to compute the standard deviations of actuarial failure forecasts from ships and returns counts [“ESG and Reliability?” George]. I used Mathematica to do the same for the first example in tables 1 and 2, because the Greenwood variance is a Cramer-Rao bound for maximum likelihood estimators.

Greenwood Variance vs. Empirical Variance?

The Greenwood variance bounds reliability function estimates under some regularity conditions (positive definite matrixes and existence of Jacobian matrix). Maximum likelihood estimator variance converges asymptotically to the Cramer-Rao lower bound under some conditions. What happens if the data are not asymptotic? If your product is reliable, then failures are scarce and the Kaplan-Meier reliability function estimate is not asymptotic; if data are censored then the reliability function estimate is even farther from asymptotic [Khan et al.]. The following examples show empirical standard deviation estimates for comparison with Greenwood standard deviations.

The Kaplan-Meier estimator uses periodic ships (cohorts) and grouped, censored failures from each cohort. Table 1 and 3 show typical data in “Nevada” tables, so-called because the grouped failure data look like Nevada on its side. 

The ships and grouped failures in table 1 gives the same reliability estimates from each cohort as the Kaplan-Meier estimate (table 2). The standard deviations (table 2) and covariances of cohort reliability estimates are zero, because the reliability estimates didn’t vary. 

Table 1. Stupid example of ships (cohorts) and grouped failures

PeriodShips123
110051015
2100 510
3100  5

Table 2. Kaplan Meier reliability estimate and standard deviation estimates. Residuals are the differences between cohort reliability estimates and the K-M reliability estimate.

AgeK-M RelGreenwood StdevEmpirical StdevResiduals Stdev
10.950.012600
20.850.02400
30.700.40300

Table 3. Ships and grouped failures from “Weibull Analysis of Perplexing Field Data,” by James McLinn

WeekShips12345678
12010101010
250 1010101
370  101010
4100   10101
5100    1010
6100     101
7120      10
8120       1

Table 4. Reliability estimates from each cohort in table 3. Estimate the cohort empirical reliability functions as, R(t)*(1‑deaths(t)/Survivors(t‑1)); t=1,2,…. 

Table 4. Kaplan Meier reliability estimate and empirical reliability estimates from each cohort in table 3. 

Age, weekK-MRel.12345678
10.9890.9500.9500.8970.8970.8420.8420.7830.783
20.9890.9800.9800.9600.9600.9390.9390.9170.917
30.9770.9860.9860.9710.9710.9570.9570.942 
40.9770.9900.9900.9800.9800.9700.970  
50.9620.9900.9900.9800.9800.970   
60.9620.9900.9900.9800.980    
70.9400.9920.9920.983     
80.9400.9920.992      

Estimate the sample variance-covariance matrix by treating each cohort’s grouped failure counts as independent random samples. Compute the variance-covariance matrix from table 4 columns. There are alternative variance-from-the-mean formulas: variance from the averages  or variance from the Kaplan-Meier estimates from all cohorts (residuals). Neither is exactly the mean, because the average has sample variability and the Kaplan-Meier estimator is asymptotically unbiased. 

Table 5. Variance-covariance matrix of the empirical reliability function estimates from each cohort. Covariances are not negligible!

Age, Weeks11345678
10.0002       
20.00020.0002      
30.00040.00040.0008     
40.00040.00040.00090.0009    
50.00070.00070.00150.00150.0023   
60.00080.00080.00160.00160.00250.0025  
70.00110.00110.00230.00230.00350.00350.0049 
80.00100.00100.00210.00210.00330.00330.00450.0045

Note: the Excel VAR() function does not compute what you may think it should, ∑(observations‑average)^2/Sample size! If you want that variance, use the VARP() function. VAR() computes ∑(observations‑average)^2/(n-1). I also checked the COVAR() function; it produces what you should think it should, 
∑∑(observations(i)-average(i))*(observations(j)-average(j))/Sample size.

Figure 1. Greenwood and jackknife standard deviation {“JKstdev”) estimates
Figure 1. Compare alternative reliability function variance estimators. SSE variance is variance from the Kaplan Meier estimator

Figure 2. Compare alternative reliability function variance estimators. SSE variance is variance from the Kaplan Meier estimator
Figure 2. Compare alternative reliability function variance estimators. SSE variance is variance from the Kaplan Meier estimator

Table 6 cohort ships and grouped failure counts appear to have been simulated from the same distribution for every cohort. The ships and returns process seems suspiciously stationary. The broom chart of reliability estimates from each cohort coincide (figure 3). 

Table 6. Data from https://www.weibull.com/hotwire/issue119/relbasics119.htm “Predicting Warranty Returns in Weibull++7”

MonthSales123456789
Mar-1016231357911121517
Apr-103723 27111720253033
May-101319  03467910
Jun-103600   2612152025
Jul-103298    26101419
Aug-101333     0346
Sep-101584      035
Oct-104508       29
Nov-104463        2
Figure 3. Broom chart of reliability estimates from each subset are the same (superimposed), just for shorter durations.
Figure 3. Broom chart of reliability estimates from each cohort are the same, just for shorter durations. 
Figure 4. Greenwood standard deviation is much smaller than the empirical standard deviation estimates. The simulated standard deviation estimates are close to Greenwood.
Figure 4. Greenwood standard deviation is much smaller than the empirical standard deviation estimates. The simulated standard deviation estimates are close to Greenwood.

So What?

The actuarial forecast (or hindcast) of failures in period t is ∑a(s)*n(t-s); s=1,2,…,t, where n(t-s) is the installed base of age t-s, and a(s) is the actuarial failure rate conditional on survival to age s. The actuarial forecast variance is 

∑Var[a(s)]*n(t-s)^2+∑∑Covar[a(t-s),a(s)]*n(s)*n(t-s).

The first term depends on Var[a(s)] so using Greenwood’s variance could lead to biased forecast. The covariance is also needed; the second term could be significant.

The covariances of the Nelson-Aalen cumulative failure rate function estimates are zero [Wellner, Nelson, Aalen], but actuarial failure rates could have covariance. Neglecting actuarial rate covariances under-or over-estimates the variance of actuarial forecasts (figures 2 and 4 bar-charts). The jackknife estimators of the “survival function integrals” may interest you [Azarang et al.], because the survival function integral is the MTBF, if lifetime data are uncensored. If data are censored, extrapolation is required. Bootstrap resampling of the Kaplan-Meier estimator yields data for estimation of the variance-covariance matrix, but does not capture the variation between cohorts.

Table 7. Variance-covariance matrix of Kaplan-Meier reliability estimates from table 1 data

Age12Act. hindcastStdev
10.000163550.0000047751.11E-14
20.000004770.001758814.50
1NA()NA()22.325NA()

Table 8. Variance-covariance matrix of actuarial failure rate function from table 4 cohort reliability estimates from Jim McLinn’s data

Week12345678
10.000164       
200      
30.00021400.000241     
40000    
50.00029200.00033100.000378   
6000000  
70.00037700.00043200.00049800.00058 
800000000

Table 9. Hindcasts and their variances. The last column is the square root of hindcast variances, the standard deviation of the actuarial hindcast. A hindcast is a forecast for a period in the past

WeekShipsAct. rateHindcastObserved∑VAR…∑∑COVAR…Stdev
1200.010.2210.07 0.26
2500.000.5410.070.000.26
3700.010.6321.250.601.36
41000.001.2721.250.001.12
51000.021.5235.035.223.20
61000.001.5235.030.002.24
71200.022.29413.3921.025.87
81200.002.75413.390.003.66

Each cohort input to the Kaplan-Meier estimator includes grouped failures at ages up to the oldest cohort’s age. Use the empirical reliability estimates from each cohort to compute an estimate of the variance-covariance matrix. They’re independent samples even though they are from cohorts of different sizes, with different numbers of grouped failure counts, with different maximum ages. If you want to forecast future returns, extrapolate the ships and actuarial failure rates. I use regression, which gives me some indications of standard deviations to plug into table 4, the reliability function estimates for each cohort. 

Recommendations?

Don’t believe the Greenwood variance of the Kaplan-Meier reliability function estimate. You may argue that the empirical reliability estimates from each cohort are not of the same size, because each successive cohort is one age-interval shorter. I agree with that and that the variance-covariance estimates of the oldest units may suffer from small failure counts. I am not referring to the cohort sizes (ships) but to the numbers of reliability estimates in each cohort and to small failure counts of reliable products. Perhaps I should use weighted variance-covariance estimates [Khan et al.]. Perhaps I should derive weights that minimize distance of the empirical variance-matrix from the Cramer-Rao bound. I don’t know. Help? 

Can I help you with the variance of actuarial forecasts? Need to set spares inventories? Confidence bands on reliability function estimates? Reliability or survival function estimation without life data? Suppose all the available data was the ships counts and the sums of failure counts in each column of the Nevada table. That’s population data required by GAAP from revenue and service costs. Kaplan-Meier requires lifetime data. Ships and returns counts are population data. Is your Kaplan-Meier data a sample? How much does it cost to track lifetimes? How many errors?

You may ask, “How are you going to estimate the variance-covariance of the nonparametric estimator from ships and returns counts? You used the Cramer-Rao bound in previous article on the variance-covariance matrix” [“ESG and Reliability?” George]. Send me some field reliability ships and returns counts data and you’ll see.

References

Aalen, O. O., “Nonparametric Inference for a Family of Counting Processes,” Annals of Statistics, Vol. 6, 701726, 1978

Odd O Aalen, Ørnulf Borgan, and Håkon K. Gjessing, Survival and Event History Analysis, A Process Point of View, Springer, 2008

Leyla Azarang, Jacobo de Uña-Álvarez and Winfried Stute, “The Jackknife Estimate of Covariance of Two Kaplan–Meier Integrals with Covariables,” Statistics, Vol. 49, No. 5, pp. 1005-1025,DOI:10.1080/02331888.2014.960871, 2015

D. A. Freedman, “Greenwood’s Formula,” https://www.stat.berkeley.edu/~freedman/greenwd.pdf

Larry George, “ESG and Reliability?” may appear in “Weekly Update,” www.accendoreliability.com, 2023

Greenwood, M., “The natural duration of cancer. Reports on Public Health and Medical Subjects,” Vol. 33, pp. 1–26, His Majesty’s Stationery Office, London, 1926

W. J. Hall and Jon A. Wellner , “Confidence Bands for a Survival Curve from Censored Data,” Biometrika, Vol. 67, No. 1, pp. 133-143, April 1980

Habib Nawaz Khan, Qamruz Zaman, Fatima Azmi, , Gulap Shahzada, and Mihajlo Jakovljevic, “Methods for Improving the Variance Estimator of the Kaplan–Meier Survival Function, When There Is No, Moderate and Heavy Censoring-Applied in Oncological Datasets,” Frontiers in Public Health, May 2022

James McLinn, “Weibull Analysis of Perplexing Field Data,” ARSymposium, 2010

Wayne Nelson, “Theory and Applications of Hazard Plotting for Censored Failure Data,” Technometrics, Vol. 42, No. 1, February 2000

S. Sawyer, “The Greenwood and Exponential Greenwood Confidence Intervals in Survival Analysis,” September 4, 2003

Paul Tune, “Computing Constrained Cramer-Rao Bounds,”  IEEE Transactions on Signal Processing, Vol. 60, No. 10, pp. 5543-5548, doi: 10.1109/TSP.2012.2204258, Oct. 2012

Bhanukiran Vinzamuri, Yan Li, and Chandan K. Reddy, “Calibrated Survival Analysis using Regularized Inverse Covariance Estimation for Right Censored Data,” IEEE Transactions on Knowledge and Data Engineering, DOI:10.1109/TKDE.2017.2719028, June 2007 

Jon A. Wellner, “Notes on Greenwood’s Variance Estimator for the Kaplan-Meier Estimator,” Univ. of Washington, January, 2010

Filed Under: Articles, on Tools & Techniques, Progress in Field Reliability?

« Improving Fatigue Resistance
FMEA Occurrence Risk- Insights and Advices »

Comments

  1. Larry George says

    March 11, 2023 at 1:01 PM

    Stupid me. I computed the Greenwood standard deviation wrong in Table 2. Kaplan-Meier reliability differs from cohort reliabilities too. Cohor reliabilities do not vary.
    Age KM Rel Greenwood Cohort Reliabilities
    1 0.95 0.0126 0.95
    2 0.85 0.024 0.8947
    3 0.70 0.403 0.8235
    Greenwood standard deviations should have been
    1 0.0689
    2 0.11091
    3 0.11878

    Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Articles by Larry George
in the Progress in Field Reliability? article series

Join Accendo

Receive information and updates about articles and many other resources offered by Accendo Reliability by becoming a member.

It’s free and only takes a minute.

Join Today

Recent Articles

  • Risk Prioritization in FMEA – a Summary
  • What Are Best Practices for Facilitating Qualitative Assessments?
  • So, What’s Still Wrong with Maintenance
  • Foundation of Great Project Outcomes – Structures
  • What is the Difference Between Quality Assurance and Quality Control?

© 2023 FMS Reliability · Privacy Policy · Terms of Service · Cookies Policy

This site uses cookies to give you a better experience, analyze site traffic, and gain insight to products or offers that may interest you. By continuing, you consent to the use of cookies. Learn how we use cookies, how they work, and how to set your browser preferences by reading our Cookies Policy.