Reliability Estimation Efficiency? Without Lifetime Data?

What is the efficiency of your reliability estimators and failure forecasts? Compare naïve forecasts: guess, AFR, MTBF, or times series extrapolations such as ARIMA, GMDH, and AI, vs. forecasts based on reliability estimators. With grouped lifetime data, the Kaplan-Meier estimator is statistically efficient, under a condition often ignored! With lifetime data you could use maximum likelihood estimators like Weibull or Kaplan-Meier from (a sample of?) grouped lifetimes. Without lifetime data you could estimate reliability from population ships and returns counts from data required by GAAP! This article compares efficiency and cost of forecasts based on alternative data and estimators.

Efficiency and cost depend on who is talking: 

• Statistical efficiency = ratio of estimator variance to Cramer-Rao bound 
• Cost-benefit of data: (sample?) lifetimes vs. (population!) ships (cohorts) and failure counts.
• DOGE goal: “…modernizing federal technology and software to maximize governmental efficiency and productivity” [The White House]. Output/Zero inputs => Infinite productivity and efficiency!

Google AI says, “In a government context, efficiency means achieving the same or greater outputs with fewer resources, while productivity measures how many units of output are produced from one unit of inputs, with improved productivity being a means to achieve greater efficiency.” 

Leontief says… “In the context of Leontief’s input-output analysis, efficiency refers to the optimal use of resources to produce goods and services.” Leontief models the interconnectedness of different sectors in an economy. Changes in one sector can affect the efficiency and productivity of other sectors [Wikipedia]. Optimal means balance costs of data, costs of analyses, and costs of errors, risks,…!

Kaplan-Meier Might Not be a Maximum Likelihood Estimator!

If data are periodic, grouped failure counts (by ship period or cohort), in the form of a Nevada table, then statistical software uses the Kaplan-Meier nonparametric reliability estimator [SAS, MiniTab, ReliaSoft, CRAN R-“Survival”, XLSTAT, and others?]. I wrote a spreadsheet to do that and published an article describing it in ASQ “Reliability Review” [George 2005].  

The Kaplan-Meier estimator assumes that censoring is non-informative, i.e., that censoring doesn’t depend on the survival time. If ships counts (cohorts) are a nonstationary random process, then the Kaplan-Meier estimator does not use all available information and is neither the maximum likelihood estimator nor statistically efficient. If data are periodic, grouped failures (by ship period or cohort) in the form of a Nevada table, then the ships cohorts and period failure sums (from all current and prior cohorts) are statistically sufficient to make nonparametric maximum likelihood reliability estimates and quantify their sample uncertainty.

Try simple tests to see whether ships cohorts are a nonstationary Poisson process [Nelson and Leemis]. Is mean=variance? View Excel Sparklines or trend? Skew? Kurtosis? If ships cohorts are a nonstationary random process, then use maximum likelihood or least-squares reliability estimators from periodic ships and returns (failure counts) account for cohort variability, with or without grouped lifetime data.

Without the need for lifetime data, DOGE could lay off people who track subjects by name or serial number, collect lifetime data, and manage databases that contain lifetimes (or samples of lifetimes). NOT tracking subjects from birth to failure, by name or serial number, will surely save money!

What would you do without the reliability estimates from lifetime data? Without forecasts that account for age-specific field failure rates? The USAF ordered the use of MTBF estimates or failures/time unit instead of actuarial failure rate estimates and actuarial forecasts [AFMAN20-116_AFGM2025-01 15 January 2025]. 

Would you like nonparametric, population reliability estimates and actuarial forecasts, without lifetime data, from population data required by GAAP? For service parts too? (A little work is required to convert revenue or sales, BoMs, and service costs into product and parts’ installed base by age; (Gozinto theory, [George Sept. 2023]) even for generalized renewal process data without counts of prior failures!!! [George Nov. 2021] SAS declined our proposal to program PROC RELIABILITY without lifetime data. Send field reliability data if you would like an example.

How to Forecast Without Lifetime data?

What if you just use AFR (Annual Failure Rate), averages, regression, or AI on failure counts or returns? Saves thinking! What if you use time series failure forecasts? Past failure data should be available? Extrapolation is easy to explain to management!

People say lifetime data requires tracking subjects, products, or parts, by name or serial number, from birth to death or censoring. FAA requires tracking ~75 “safety critical” parts per aircraft; FDA requires serial numbers on implantable devices for tracking their lifetimes. Pacemakers are removed on death. Given censored lifetime data, people use the Kaplan-Meier reliability estimator.

Medtronic used to publish periodic pacemaker implants and failure counts along with their Kaplan-Meier reliability estimates based on their lifetime data. I sent them reliability estimates based on their published periodic implants (cohorts) and cohort failure counts, without lifetime data. Our estimates agreed. Medtronic quit publishing sales and failure counts and their lifetime-based reliability function estimates.

Statistical (Asymptotic) “Efficiency”?

Statistical efficiency is the ratio of an unbiased parameter estimator’s variance to the Cramer-Rao lower bound on variance. The Cramer-Rao variance bound is (the diagonal of…) the inverse of the Fisher “information matrix,” the expected values of all second derivatives of the log-likelihood function with respect to all parameters, such as reliability or failure rate function values. With grouped, censored failure time data, the Cramer-Rao bound is achieved asymptotically by the Kaplan-Meier estimator, and the Cramer-Rao variance is Greenwood’s variance formula, if censoring doesn’t depend on the survival time!

The Kaplan-Meier likelihood function is ∏Binomial[d(t), n(t), a(t)] [Wikipedia], where d(t) are deaths at age t, n(t) is number of survivors to age t, and a(t) is actuarial failure rate function conditional on survival to age t. The likelihood function is often simplified to ∏(1-a(t))^n(t) because the Binomial combinatorial term would drop out of maximization, if cohorts were not random. With regression or least-squares, min[∑(observed(t)-hindcast(t))²], and likelihood is approximately ∏Normal(μ(t),∑(t)), where ∑(t) is the variance-covariance matrix of reliability or failure rate function estimates. With time series analyses or AI, likelihood could be a function of multivariate polynomials, e.g. www.GMDH.net/ [Farlow, Goshulka]. With ships (cohorts) at rate λ(t) and periodic returns or failure counts data, likelihood could be ∏Poisson(λ(t)*G(t)) where G(t) is cdf=1-exp(-∑a(s)), s=1,2,…,t [Mirasol]. This models lifetimes as service times in and M/G/infinity self-service system and the ships and returns data correspond to inputs and outputs of M/G/infinity self-service systems.

How do ratios of Cramer-Rao bounds help? Make you feel better? Statistical efficiency usually refers to parameter estimator(s). What about nonparametric distribution function estimator efficiency [Stein]? The Kaplan-Meier reliability function estimator is asymptotically “statistically efficient”, under some conditions, but Greenwood’s variance estimator (C-R bound) can be way off for finite samples or if the conditions for the Kaplan-Meier estimator are not met [George, May 2021].

Compare Alternative Forecasts

• Please enter forecast _______?
• Guess demand based on graphs or pictures of past demands ______?
• AFR = Annual failure rate = failures/installed or failures/survivors, and forecast=AFR*installed base [Jan. 2025 USAF order to compute AFR=failures/time].
• Time Series forecasts: moving average, ARIMA, Holt-Winters, seasonal SARIMA, etc. based on past returns or failure counts [e.g., www.smartcorp.com], forecast = Excel’s FORECAST.ETS(); no reliability estimation required
• AI [www.predii.com/]? GMDH [Farlow, www.gmdh.net]?, Streamline? [Goshulko]
• Actuarial forecasts ∑[a(s)*n(t-s)] from nonparametric maximum likelihood estimator of a(s) from ships and returns. 

An econometrician for the automotive aftermarket company Triad Computer Systems tried to forecast the demands for auto parts with regression, ∑β(s)*n(t-s), s=1,2,…,t, where n(t-s) is vehicle counts of age t-s and β(s) are regression coefficients. That didn’t work due to auto-correlation (no pun intended). I pointed out that that regression model was an actuarial forecast, the regression coefficients were actuarial failure rates, and that we could estimate them from vehicle installed base, Triad’s catalog of which parts go into which vehicles, and auto parts’ sales by part number. He asked me, “What if the demands for auto parts were second, third, or ??? renewals?” Over the weekend, I revised the actuarial rates to be estimated from generalized renewal processes using least squares.

An actuarial forecast (or hindcast) is ∑a(s)*n(t-s), s=1,2,…,t, where a(s) are actuarial failure or demand rates, and n(t-s) are product or parts counts of age t-s. Actuarial rates could be derived from generalized renewal processes of parts’ failures, with or without lifetime data! The distribution of an actuarial forecast distribution is approximately normal according to martingale theory. The USAF Logistics Command assumed Poisson demands from M/G/infinity self-service queuing system.

In the late 1990s, Triad was so successful selling actuarial forecasts and recommending parts’ stock levels that a competitor bought Triad and laid off me and the Econometrician. Triad is now www.Epicor.com and uses time series or AI forecasts from www.smartcorp.com/ or www.predii.com/.

Compare forecasts by SSE= ∑(Forecast-Observed)² or by ∑(Forecast-Observed)²/Forecast, the Chi-square goodness-of-fit test statistic (Table 2). How? Compute forecasts from Nevada table (Table 1) for period 2 using only data from period 1, for period 3 using data from periods 1 and 2, etc. Extrapolate ships (nonstationary Poisson?) and actuarial rate estimates with mean, regression, Excel FORECAST.ETS, or whatever seems appropriate.

Table 1. Example Data for Alternative Forecasts from “Nevada Charts to Gather Data,” Schenkelberg https://accendoreliability.com/nevada-charts-gather-data/

Month	Ships	Jan	Feb	Mar	Apr	May	Jun
Jan	3519	3	6	3	7	10	3
Feb	6292		3	8	20	35	24
Mar	7132			8	13	25	31
Apr	5633				4	13	6
May	4222					5	8
Jun	4476						6
Sums	31274	3	9	19	44	88	78

Kaplan-Meier nonparametric reliability function estimator from grouped lifetime data (body of Table 1) assumes failures are dead forever. Compute actuarial failure rate function a(s) from the Kaplan-Meier reliability function and make actuarial forecast = ∑a(s)n(t-s), s=1,2,…,t for period t in the future.

Ships (cohorts) and bottom row sums from Nevada table 1 are statistically sufficient to make nonparametric reliability and actuarial rate function estimates. Compute the nonparametric maximum likelihood failure rate function estimator and actuarial forecasts from periodic ships (cohorts) and returns or failures (S&R), from ships cohorts and bottom row sums in Nevada table 1 [George April 2021]. 

Table 2. Ships and returns, S&R Actuarial forecast Chi-Square goodness-of-fit Beats Alternatives! Small is better.

Period	Failures	AFR*Survivors	FORECAST.ETS	K-M Actuarial	S&R Actuarial
Jan	3
Feb	9	3	3	3	8.36
Mar	19	11.09	15	30.86	13.83
Apr	44	29.07	26.48	40.34	33.71
May	88	30.48	54.47	56.43	53.79
Jun	78	35.97	99.73	56.43	120.96
SSE		5397	1956	2086	3149
Chi-Square		182.99	50.03	47.19	42.13

Don’t use Kaplan-Meier estimator for renewal process data in a Nevada table; failure counts may include units that failed earlier! Use nonparametric least squares from Nevada table [George Nov. 2024} or from ships and return counts for renewal processes. (Not in this article. Send data if you have renewal or generalized renewal counts and don’t know how many prior failures there were.) 

K-M Goodness-of-Fit Chi-square Is Worse??? 

There’s more data in a Nevada table than in its margins: ships and sums of monthly returns, but is there more information? No! Kaplan-Meier estimator ignores randomness in ships cohorts!!! Entropy tells how much information is in a distribution function. Compare the entropy of Kaplan-Meier estimate vs. nonparametric maximum likelihood reliability function estimates. (Note this is partial entropy for ages t=1,2,…,6.) Entropy = -∑p(t)LOG(p(t),2) bits summed over range of random variable

Kaplan-Meier entropy = 0.1147 bits and K-M R(6) = 0.98662. Ships and returns npmle entropy = 0.1389 bits and npmle R(6) = 0.98091. That entropy difference may be due in part to differences in reliabilities R(6)? The nonparametric maximum likelihood estimate from ships and returns spans more probability 0.0191 than the Kaplan-Meier, 0.0134. The ships and returns nonparametric maximum likelihood estimator and its corresponding actuarial forecasts capture more information than Kaplan-Meier and its actuarial forecasts!

Ignoring random ships could explain why Kaplan-Meier estimator forecast’s chi-square is bigger. The Kaplan-Meier likelihood from a Nevada table of grouped failure counts is  ∏Binomial[d(t), n(t), a(t)] t=1,2,… The nonparametric maximum likelihood from ships and returns is ∏Poisson(λ*G(t)), or ∏Poisson(λ(t))*Poisson(λ(t)*G(t)) if ships cohorts are nonstationary. The second likelihood’s Poisson(λ(t)) terms account for random cohorts assuming lifetime is service time in an M/G/infinity service system [George Nov. 2024].

Here is the maximum likelihood replacement for the Kaplan-Meier reliability estimator to deal with random cohorts. Maximize the likelihood function ∏Binomial[d(t), n(t), a(t)] FOR EACH COHORT!

Non-stationarity of Ships (Cohorts) Process

Take another look at the ships in table 1: 3591, 6292, 7132, 5633, 4222, 4476. A stationary Poisson mean equals its variance. Mean ships is 5212, and variance is 1,883,617. Sample skew is 0.253, and Poisson skew is 0.0139. Sample kurtosis is -1.43, and Poisson kurtosis is 3.00. Ships are definitely not a stationary Poisson process. But it is hard to disprove that ships are individual single observations of a nonstationary Poisson process [Nelson and Leemis] and that outputs are also Poisson outputs from an M/G/infinity service system [Mirasol, and Eick, Massey and Whitt].   

Replacement for Kaplan-Meier Likelihood?

The likelihood function for Nevada-table grouped lifetimes is ∏Binomial[d(t), n(t), a(t)] t=1,2,…. You’re entitled to derive the maximum likelihood reliability estimator. I simply maximize the likelihood of each member of the product, simultaneously. Consider this alternative? Estimate reliability and failure rate functions R(t;i)=1-G(t;i) and a(t;i) for each cohort i = 1,2,…,6, and estimate their variance, using the binomial likelihood estimator for each cohort, ∏Binomial(d(t;i),n(t;i), p(t;i)), t=1,2,… where p(t;i) is the pdf of G(t;i) [George Nov. 2024]. Fortunately, that relieves the assumption of equal censoring, because all members in a cohort get censored the same way! 

Table 3. Actuarial Forecasts, SSE, and chi-square for Jan-June combining successive Cohort Info: Jan. alone, Jan-Feb, Jan-Feb-Mar, etc.

Month	Forecast	Observed	Chi-Square
Jan	3.26	3	0.0212
Feb	12.14	9	0.8111
Mar	26.31	19	2.0287
Apr	48.25	44	0.3738
May	60.46	88	12.541
June	59.98	78	5.415
SSE, Chi-sq		1164.34	21.19

Actuarial forecasts using cohort likelihoods beat ships and returns. Chi-square is 21.19 compared to actuarial from ships and returns of 42.13, and beats Kaplan-Meier chi-square 47.19. Cohort maximum likelihood estimators also provide empirical estimates of the variance-covariance matrix.

Variances of Demand Forecasts from Example data

Var[AFR*(installed base N)]=N*AFR*(1-AFR) (binomial variance) assuming AFR is proportion failing<=1.0. For example: monthly AFR = 241/31274= 0.0077 so forecast = 5212*0.0077=40 and variance = 40 (If data were from a stationary Poisson ships process.)

Variance of actuarial forecast is ∑Var[a(s)]*n(t-s)²+∑∑Covar[a(s),a(t-s)]*n(s)*n(t-s). Send data and I’ll compute cohort reliability estimates, variance-covariance matrix, hindcasts and forecasts, and their variances.

Optimize Costs, the sum of all these:

• Errors: ∑(Observed-Forecast)*(cost per unit error)?
• Variance-covariance of actuarial forecasts: Var[∑a(s)n(t-s)]=
  ∑Var[a(s)]n(t-s)²+∑∑Cov[a(s), a(t-s)]*n(t)*n(t-s) 
• P[Demand>stock level] (P[backorder]) is asymptotically normally distributed depending on service level, mean demand, and Var[∑d(s)n(t-s)]] with age-specific demand rate d(s)
• E[backorders(t))|cohort size]=Cohort size*P[Backorder]
• Duration of backorder
• Costs of tracking products, parts? Maintaining db?
• Costs of statistics: people and software?
• Costs of model error: AFR? AI? Weibull? AFR? TSA? Kaplan-Meier? Other???
• Cost of failure(s) to maintain service level, 1-P[backorder]

Optimum, budget constrained, resource allocation equates bang-per-buck for marginal costs, 
(dTotal cost/d Resource cost)*(d Resource cost/d $$$).

Financial Efficiency Afterthoughts?

Collecting lifetime data incurs the costs of ∑deaths counted*lifetimes+∑censored counted*survivor lifetimes. Consider sample lifetime vs. population ships and returns reliability estimates and forecasts. What if you track a sample of censored lifetimes vs. population ships and returns counts? Variance of reliability or survival function estimators costs money depending on estimator and use of estimates e.g., forecasts, stock levels, warranty reserves. Variance of actuarial demand forecast is used in inventory stock level, backorder, and service level. Accountants ignore actuarial forecasts of warranty reserves, because they can take a charge or credit for the error in warranty reserves. What happens to the money accountants squirrel away in warranty reserves if not spent?

Ships and returns counts can be derived from population data required by GAAP. Some work is required to convert product installed base by age (P) to parts’ installed base by age using BoM and Gozinto theory, MRP: P(I-N)^-1 P=vector of product installed base by age, I=identity matrix, N=Next assembly matrix how many parts gozinto product(s).

Recommendations?

Nonparametric actuarial forecasts from period ships and returns have better chi-square goodness-of-fit than actuarial forecasts from the “statistically efficient” Kaplan-Meier reliability estimator, from Fred’s Nevada table data,. But that doesn’t mean ships and returns estimators will always win. Fred’s ships data appear to be nonstationary random process variables. Don’t trust Kaplan-Meier estimator if ships cohorts are random! Maximize binomial likelihood on each cohort simultaneously! That uses information from ships as well as grouped failure counts. Don’t trust Greenwood’s variance of Kaplan-Meier reliability function! It’s an asymptotic bound and inaccurate for finite lifetime data with random ships.. 

Optimize the trade among with or without lifetime data, statistical methods, and costs of variability. Trade cost of data for cost of demand forecasts’ errors: forecast, stock levels, back orders, holding costs and obsolescent inventory. Use variance-covariance of forecasts, including dependence among actuarial rate estimates to estimate costs of backorders, service level, etc. 

Why collect lifetime data, even samples, when ships and returns counts are available in data required by GAAP? For free. If you’re tracking parts in products for parts’ lifetime data, why not make bivariate or multivariate reliability estimates, forecasts, and failure analyses, for warranty reserves, product reliability centered maintenance, and? [Tesla Model S, George 2025]

References

AFMAN20-116_AFGM2025-01 15 January 2025, https://static.e-publishing.af.mil/production/1/saf_aq/publication/afman20-116/afman20-116.pdf/ revises “AFMAN20-116 PROPULSION LIFE CYCLE MANAGEMENT FOR AERIAL VEHICLES, 13 April 2022

S. G. Eick, W. A. Massey and W. Whitt, “The Physics of the Mt/G/∞ Queue,” Ops. Res. , 41, 731-742, 1993

Stanley Farlow, Self-Organizing Methods in Modeling, CRC Press, 1984

Greenwood, M., “The natural duration of cancer. Reports on Public Health and Medical Subjects,” Vol. 33, pp. 1–26, His Majesty’s Stationery Office, London, 1926

E. L. Kaplan and P. Meier, ”Nonparametric Estimator From Incomplete Observations,” J. Amer. Statist. Assn., Vol. 53, pp. 457-481, 1958

Wassily Leontief, https://en.wikipedia.org/wiki/Input%E2%80%93output_model/

Barry Nelson and Lawrence M. Leemis, “The Ease of Fitting but Futility of Testing a Nonstationary Poisson Processes From One Sample Path,” Proceedings of the 2020 Winter Simulation Conference, IEEE, 2020

Noel M. Mirasol, The Output of an M/G/infinity Queuing System is Poisson,” Operations Research, 11, 282-284, 1963

Lance Serating or Alex Goshulko, “Demand Forecasting and Inventory Planning with GMDH Streamline-Webinar,” YouTube, gmdhsoftware.com/

Charles Stein, “Efficient Nonparametric Testing and Estimation,” Third Berkeley Symposium on Mathematical Statistics and Probability, pp. 187-195, 1954 and 1955

The White House, “Establishing and Implementing The President’s “Department of Government Efficiency,” Jan. 20, 2025

References by George

“Kaplan-Meier reliability estimation spreadsheet,” ASQ Reliability Review, Vol. 25, No. 2, pp. 6-12, June 2005

“Estimate Field Reliability Without Life Data,” https://accendoreliability.com/estimate-field-reliability-without-life-data/, April 2021

“Gozinto Theory and Parts’ Installed Base (by age),” Problem Solving Tools News, June 1999 and https://accendoreliability.com/gozinto-theory-parts-installed-base/, May 2021

“Renewal Process Estimation Without Life Data,” https://accendoreliability.com/renewal-process-estimation-without-life-data/, Sept. 2021 

“Bivariate Reliability Estimates from Survey Data” (Tesla Model S) https://accendoreliability.com/bivariate-reliability-estimates-from-survey-data/ Dec. 2021

“What Price Required Data?” Problem Solving Tools Nwsltr6.doc, June 1999 and https://accendoreliability.com/gozinto-theory-parts-installed-base/, Nov. 2022

“Gozinto Theory and Parts’ Installed Base,” https://accendoreliability.com/gozinto-theory-parts-installed-base/, Sept. 2023

“Kaplan-Meier Estimator Ignores Cohort Variability,” https://accendoreliability.com/kaplan-meier-ignores-cohort-variability/, Nov. 2024

“Multiple-Failure-Mode Reliability Estimation,” (Tesla Model S)  https://accendoreliability.com/multiple-failure-mode-reliability-estimation/, Jan. 2025 

“Progress in USAF Engine Logistics?  https://www.linkedin.com/feed/update/urn:li:activity:7287540108554092547?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAAbJp0BbUlroX_HoCMwKCQJbTECYdUtHXA/, Feb. 2025