Lifetime data is nice to have, but lifetime data is not necessary! Generally Accepted Accounting Principles require statistically sufficient data to estimate nonparametric reliability and failure rate functions. Some work is required!
ISO 14224 “Petroleum, Petrochemical and Natural Gas Industries—Collection and Exchange of Reliability and Maintenance Data for Equipment” requires lifetime data to estimate exponential or Weibull reliability functions! Sales or ships and returns or failure counts are statistically sufficient to make nonparametric estimates of reliability and failure rate functions, without unwarranted distribution assumptions or lifetime data!
LILCO 1911-1998: Long Island Lighting
I was in the maintenance manager’s office when someone interrupted; a disgruntled employee was making threats. Manager disappeared. Later the manager returned. I explained how to estimate reliability from installed base and failure data required by GAAP (Generally Accepted Accounting Principles). Manager said, “We don’t even have records of our equipment, even purchase records!”
CMMS Software Implements Standards
At a Houston reliability meeting, 1977, oil company consultants bragged about their yachts. Has there been progress since then?
“…the Achilles heel of these technologies (AI and ML) is poor data quality. By some estimates, at best 3% of historical equipment performance data meet basic quality standards.” [Ciliberti]
“Depending on the sector in which your business operates, you may be subject to regulatory compliance standards that you must adhere to.” “Inspections and audits assess the business’s adherence to the different standards.” “CMMS Software can help a business meet regulatory compliance standards more painlessly.” [Morsillo]
Field reliability data led to selling failure rate estimates, which led to writing standards for data collection, which led to auditing to assure compliance with standards, which led to sales of CMMS software to implement standards. For example: OREDA (Offshore REliability Data) started ISO 14224 to get more data so it could publish more failure rates in the OREDA Handbook [Gjerstad] and now in the “cloud” https://oreda.com/. This is known as “Materialist Capture” or “Financial Capture”.
ISO 14224 Attempts to be Constructive?
“As of today, monitoring and follow-up of technical status of SIS components require considerable manual effort by extracting and interpreting maintenance and failure data from various systems and sources.” [Hauge et al.]
Ali Bashir says, “Results from reliability analytics and usage of codes are often not communicated back to the shop floor. It is important for people to realise that the codes that are used and the information put into CMMS is referenced to make data driven decisions.”
In “ISO 14224-Implementation in 6 Easy Steps,” Roy Milne says, “…how reliability will consume the data, analyze it, validate it and provide recommendations.”
ISO 14224 requires reliability lifetime data, which makes CMMS database programming easy; it also recommends estimation of constant failure rates or Weibull reliability functions. ISO 14224 avoids ISO 31000 risk, uncertainty, and PRA (Probabilistic Risk Analysis); it focuses on reliability as probability based on field data [Terje Aven]. ISO 14224:2016 introduced “Uncertainty” [Selvik et al. quibble].
What is My Problem with ISO 14224?
ISO 14224 requires reliability lifetime data. That data is sufficient but not necessary to make nonparametric reliability estimates. Lifetime data costs a lot, requires a lot of work to track every part by name and serial number, and incurs errors and missing data.
ISO 14224 recommends constant failure rate or Weibull estimation. Nonparametric reliability estimation, without life data, has been done for 35 years. Data required by GAAP is free, contains lifetime information, is not supposed to have accounting errors, and is population data!
ISO 14224 recommends baseball reliability statistics: Paretos, RCA (Root-Cause Analysis), RGA, MCF, and Markov or Simulation.
- Paretos are OK statistics. RCA is OK too. Assuming exponential or Weibull reliability requires justification. Lifetime data is unnecessary for exponential, Weibull or nonparametric reliability estimation. ISO 14224 requires data sufficient for nonparametric reliability estimation but never mentions nonparametric statistics, multivariate statistics, or dependence.
- RGA (Reliability Growth Analysis a la AMSAA, Larry Crow) is really MTBF growth analysis. Want reliability function growth analysis [https://accendoreliability.com/statistical-reliability-control/#more-522710/]?
- MCF (Mean Cumulative Function) is for renewal processes. MCF gives averages only, [Nelson, https://accendoreliability.com/predicting-repair-rates-plots/#more-289657/] not reliability functions [George 2004].
- Markov analysis requires transition rates. Markov transition rates are nonparametric, actuarial failure or repair rates.
- Simulation depends on correctness of distribution assumptions and dependence (copulas).
What People Say About ISO 14224?
“…ISO 14224 was developed for the Petroleum, Natural Gas and Petrochemical industries, the same process can and should be applied in any organization” [Kovacevic, italics are mine]. Organizations that use GAAP have reliability statisticians that use accounting data to collect periodic product installed base by age, failure counts or spare parts’ usage, and engineers have BoMs of service parts that go into product installed base by age to convert into service parts’ installed base by age. Gozinto theory or MRP (Materials Requirements Planning) says P*(I-N)-1 tells parts installed base from P = the vector of product installed base by age and N is the next-assembly matrix that tells how many of each part gozinto each next assembly [https://accendoreliability.com/gozinto-theory-parts-installed-base/#more-417514/].
“I have personally wasted hundreds of hours of my work life sifting through a poor CMMS structure trying to find records” [Ferrari]. Sorry, you might have looked in wrong place(s). Installed base and returns or failure counts can be derived from data required by GAAP. Revenue = Σships*price and Service Cost = Σreturns*cost per return. You may have to work to dig up the ships and returns counts, but accountants have to report revenue and cost elements. I confess that I had to look up Apple computer sales in industry publications tp convert computer sales to parts’ installed base using the gozinto formula.
Compare ISO 14224 lifetime data requirements vs. ships and returns counts. ISO 14224 requires lifetime data, which may be summarized. Table 1 (“Nevada table”) contains cohorts and grouped failure counts or equivalent for units tracked by name and serial number: start time or date, failure time or date for each product or part. Grouped lifetime data is OK, but a Nevada table is not necessary. The bottom row of table 1 contains period ships and failures or returns without cohort id. Table 2 data is statistically sufficient to make nonparametric estimates of reliability and failure rate functions.
Table 1. Nevada table of periodic ships and grouped failure or return counts. Bottom row sums monthly failure counts.
| Month | Ships | Jan | Feb | Mar | Apr | May | Jun |
| Jan | 3519 | 3 | 6 | 3 | 7 | 10 | 3 |
| Feb | 6292 | 3 | 8 | 20 | 35 | 24 | |
| Mar | 7132 | 8 | 13 | 25 | 31 | ||
| Apr | 5633 | 4 | 13 | 6 | |||
| May | 4222 | 5 | 8 | ||||
| Jun | 4476 | 6 | |||||
| Returns | 3 | 10 | 19 | 45 | 88 | 78 |
Table 2. Statistically sufficient data from table 1.
| Month | Ships | Returns |
| Jan | 3519 | 3 |
| Feb | 6292 | 10 |
| Mar | 7132 | 19 |
| Apr | 5633 | 45 |
| May | 4222 | 88 |
| Jun | 4476 | 78 |
Table 1 contains more information than table 2, because the table 1 grouped failure counts are identified by cohort. Table 2 failures could have come from any prior cohort. Kullback-Leibler divergence measures this information: Σf(t|KM)ln(f(t|KM)/f(t|S&R)) = -0.00087 bits for t=1,2,3,4,5. (f(6|S&R)=0 blows up the formula.) The Kullback-Leibler divergence for the Weibull fits is -0.004 bits. Those are small differences in information.
The OREDA handbook 2015 edition costs NOK 4000, (~$384 USD). SuperSmith Weibull software costs 974 Euro (~$1052 USD). “Entry-level CMMS software costs $29-$79/user/month, while subscription-based CMMS onboarding spans $600-$2,500 in upfront expenses” [softwareconnect.com/learn/best-cmms-software-pricing/]. How much work would it take to get periodic ships and returns or failure counts from your company’s accounting system? Is it worth 0.00087 bits of information from CMMS or 0.004 bits if you use Weibull software?
ISO 14224 Sections vs. Corresponding GAAP data
Numbers refer to ISO sections.
1. ISO 14224 requires costs but GAAP + work supplies cost data.
2. ISO 14224 requires parts’ start times but GAAP supplies product sales or installations by period, revenue=Σprice*sales. I admit I looked up Apple computer, Firestone tire, and Ford monthly sales in industry publications. Ron Salzman, Ford, sent me actual production, when I sent him field reliability estimates.
3. ISO 14224 requires hierarchical parts’ classification but BoMs (Bills of Materials) supply parts counts and next-assembly matrix N for MRP or gozinto; P(I-N)-1 converts product installed base P into parts installed base by age [George, https://accendoreliability.com/gozinto-theory-parts-installed-base/#more-417514/].
4. ISO 14224 requires failure modes, which may be apparent from specifications? FMEA? FRACAS? ISO 14224 doesn’t recognize dependent failure modes or multivariate reliability.
7. ISO 14224 requires fail dates but GAAP requires periodic reporting of service costs, spare parts’ sales, complaints, etc.; J&J Glucometer complaints were nearly always led to replacements.
Published sales and failure data? Medtronic used to publish pacemaker sales, failure counts, and their reliability estimates,…until I sent them nonparametric field reliability estimates that agreed with theirs from their lifetime data.
Do the best you can with…
Nonparametric estimation of reliability from lifetime data grouped by cohort uses the Kaplan-Meier estimator. Nonparametric estimation from ships and returns uses maximum likelihood or least squares [George 2023]. Markov analysis requires actuarial (nonparametric, age-specific) transition rates. Simulation is GIGO if distributions are wrong. Multivariate reliability statistics quantifies dependent failure modes; e.g. seismic, e.g., automotive aftermarket customers who bought A also bought B.
André-Michel Ferrari asked how to estimate reliability from incomplete renewal process failure data for bearings started in 2008 from CMMS data started in 2015 (table 3). Thanks for permission to use the example. Bearings started service 1/1/2008. CMMS started 7/1/2015 collecting failures (“change”) dates. What about lost failures prior to CMMS?
Table 3. Bearing change records TTF is since 1/7/2015. Some are successive failures of at same location. (“Date of change” is in Canadian format.)
| Bearing Id | TTF months | TTF months | Date of change |
| F1 | 53 | 53 | 1/12/2019 |
| F1 | 22 | 23 | 2/11/2021 |
| F2 | 71 | 72 | 7/27/2021 |
| F3 | 71 | 72 | 7/27/2021 |
| F3 | 5 | 4 | 11/2/2021 |
| P1 | 60 | 60 | 26/06/2020 |
| P4 | 31 | 32 | 21/02/2018 |
| P5 | 39 | 39 | 15/10/2018 |
| P5 | 16 | 16 | 16/02/2020 |
| P5 | 26 | 27 | 12/5/2022 |
| P6 | 35 | 36 | 23/06/2018 |
| P8 | 87 | 88 | 18/10/2022 |
| P6 | 94 | 59 | 17/05/2023 |
Bearing manufacturers claim Weibull reliability [Xintao et al.], so assume a Weibull renewal process describes times between failures; TBF reliability is Exp[‑(t/η)β]. Simulate successive lifetimes from 1/1/2008 for each bearing Id and estimate Weibull parameters to match quarterly observed CMMS failure counts. Include zero-failure counts from 7/1/2015 to first recorded failure in 2018.
Find Weibull parameters eta and beta to minimize the sum of squared errors (SSE) between observed and simulated 2015-2023 quarterly failure counts. Entries in table 4 are Excel COUNTIFIS() functions, mostly 0 and 1, that tell how many simulated Weibull renewals are in each calendar quarter. There are two “Simulated” rows in table 4, because I replicated the simulation, two times for each bearing: F1, F2, F3, P1, P4, P5, P6, and P8. I used Solver several times (because simulation changes TBFs) to find eta and beta to minimize SSE. Column B contains latest solution. Column C contains a copy of previous Solver solution. I ran Solver six times to get the mean, standard deviation and coefficient of variation for the Solver solutions. Replicating the Solver solution reduced the coefficient of variation from ~6.2% for eta and 5.3% for beta.
Table 4. Simulated quarterly failure counts from data in table 3, start date 1/1/2008 and CMMS start date 7/1/2015.
| Quarter | 7/1/2015 | 10/1/2015 | Etc. | 4/1/2023 | 7/1/2023 | 10/1/2023 |
| Failures | 0 | 0 | 1 | 0 | 0 | |
| Simulated | 0 | 0 | 0 | 0 | 0 | |
| Simulated | 0 | 0 | 0 | 0 | 0 |
Table 5. Estimation of Weibull parameters. Column B contains current Solver solution; column C contains copy of previous solution.
| SSE | 37 | 36 | Etc. | mean | sd | cv |
| Eta | 7962.4 | 7962.4 | 7924.7 | 58.18 | 0.73% | |
| beta | 1.855 | 1.855 | 1.80 | 0.075 | 4.18% | |
| mean | 7071.6 | 7071.6 | 7049.9 | 52.07 | 0.74% |
The connection between Weibull parameters and SSE may not be obvious. Table 4 row labeled “Failures” counts observed quarterly renewals in table 1, including zeros. Table 2 rows labeled “simulated” count simulated quarterly renewals. SSE is SUM(observed-simulated)2 summed over all quarters since 7/1/2015. Solver adjusts eta and beta to minimize SSE. (I could have used months in table 4 instead of quarters, but I felt like aggregating into quarterly counts might help Solver.) Perhaps Bayesian methods would help supplement data in table 3 [Dykstra and Laud].
New Reliability Methods?
Estimating reliability from ships and returns counts is not new! People have been saying, “You need lifetime data to estimate reliability functions.” I estimated maximum likelihood nonparametric reliability functions for Apple Computer service parts from computer sales and parts’ returns counts (1990). I started doing that for auto parts’ renewal processes using least squares (1995).
Consider multivariate reliability function estimation in different failure modes [Maathuis, George]. E.g., COVID-19 daily case, death, and recovery counts. HIV+->AIDS->death counts? With multiple end-point or failure modes, the renewal process likelihood function is L=Πf*k(t(j)), where (f*k(.) is convolution) for k-th failure of j-th component f(t(j)) different for each (kind of) component.
Compare ISO 14224 Required Lifetime Data reliability estimates vs. ships and returns reliability estimates using the Kullback-Leibler divergence to quantify difference between alternative data. There is more information in lifetime data, even censored, than in ships and returns counts. Some of that information is wiped out by assuming constant failure rate or Weibull reliability. K-L divergence is bits of information (difference). Use cost data to convert bits to $$$ [George 2019, “Statistical Reliability Control”].
Are you paying for your CMMS to implement ISO 14224? Are you paying to put data into the CMMS? Are you paying for Weibull S/W? Are you paying auditors to certify compliance with ISO 14224? OREDA cloud data costs EURO 300/year+.
GAAP data is free, but you have to work to dig ships and returns counts out of company financials! Do the best you can with available data. I’ll help; send data to pstlarry@yahoo.com., describe it, and ask for nonparametric field reliability estimates. If your installed base is products, send their BoMs or gozinto matrix N along with product ships and parts’ failure counts.
ISO 14224 Contacts
ISO TC67/WG4, “Reliability Engineering and Technology”, https://www.iso.org/committee/49506.html/
ISO TC67 US representative is Tony Ciliberti, Reliability Dynamics LLC, tony.ciliberti@rd-eam.com,www.reliabilitydynamics.com/
Oil and gas industries including lower carbon energy, www.iso.org/committee/49506.html/
Tony.Krakenes@SINTEF.no, https://www.sintef.no/en/projects/2015/oreda-handbook2/
References
ISO 14224(2006)E, “Petroleum, Petrochemical and Natural Gas Industries — Collection and Exchange of Reliability and Maintenance Data for Equipment,” the International Organization for Standardization, 2006
Terje Aven, “The Flaws of the ISO 31000 Conceptualisation of Risk,” Editorial, J. Risk and Reliability, Vol. 231, Issue 5, pp. 467-468, Oct. 2017
Terje Aven, “Forget the traditional risk matrix – better alternatives exist,” Risk analysis, 2nd. Ed. Wiley. https://drive.google.com/file/d/1X0GiZsiUAsmtuHh4U6ujLftxy2QdG5uB/view/, 2015
Ali Bashir, “Implementing ISO 14224 in Six Steps,” https://www.linkedin.com/pulse/implementing-iso-14224-six-steps-ali-bashir/, May 2017
Tony Ciliberti, “Use ISO 14224 Methods to Optimize Equipment Performance Data Quality and Results from Artificial Intelligence and Machine Learning,” ANSI Blog
R. L. Dykstra and Purushottam Laud, “A Bayesian Nonparametric Approach to Reliability, The Annals of Statistics, Vol. 9, No. 2, pp. 356-367, https://doi.org/10.1214/aos/1176345401/, 1981
André-Michel Ferrari, “10 Crucial Attributes to Optimize Your CMMS Set-up for Reliability Analysis,” 10 Crucial Attributes to Optimize your CMMS set-up for Reliability Analysis (accendoreliability.com)/, Weekly Update, Feb. 24, 2024
L. L. George, “Actuarial Forecasts for the Automotive Aftermarket,” SAE Transactions, Vol. 113, Section 5: Journal of Materials and Manufacturing, pp. 697-701, 2004
L. L. George, “Random-Tandem Queues and Reliability Estimation, Without Life Data,” RandTand.pdf – Google Drive/, 2019
L. L. George, “Gozinto Theory and Parts’ Installed Base,” Weekly Update, https://accendoreliability.com/gozinto-theory-parts-installed-base/#more-417514, Sept. 2023
Torkell Gjerstad, “The OREDA Handbook and Its Role In Offshore Risk Analysis,” https://oreda.com/handbook/, 2009
Stein Hauge, Solfrid Håbrekke, Mary Ann Lundteigen, and Lars Bodsberg, “Standardised Failure Reporting and Classification of Failures of Safety Instrumented Systems,” Proceedings of the 30th European Safety and Reliability Conference and the 15th Probabilistic Safety Assessment and Management Conference, 2020
James Kovacevic, “Understanding ISO 14224: Your Guide to Sustainable Defect Elimination,” https://accendoreliability.com/understanding-iso-14224-guide-sustainable-defect-elimination/#more-204782/, April 2017
Marloes H. Maathuis, “Nonparametric Maximum Likelihood Estimation for Bivariate Censored Data,” Thesis submitted to the Delft University of Technology for the degree of Master of Science/Wiskundig Ingenieur, January 2003
Roy Milne, “ISO 14224 – Implementation in 6 Easy Steps”, https://www.linkedin.com/pulse/iso-14224-implementation-6-easy-steps-roy-milne/, Jan. 2019
Tony Morsillo, “6 Ways CMMS software will Help you Breeze Through Audits,” https://zoidii.com/blogpost/cmms-audits/, March 2023
Wayne Nelson, “Predicting Repair Rates with Plots,” Quality Progress, pp 34-39, Nov. 2018
J. T. Selvik, E. B. Abrahamsen, and K. J. Engemann, “Definition of Reliability and Maintenance Concepts in Oil and Gas – Validity Aspects,” Safety and Reliability, Vol. 39, no. 2, pp. 134-164, DOI:10.1080/09617353.2020.1759258, 2020
Xia Xintao, Chang Zhen, Zhang Lijun, and Yang Xiaowei, “Estimation on Reliability Models of Bearing Failure Data,” Mathematical Problems in Engineering, Vol. 2018, Article ID 6189527, https://doi.org/10.1155/2018/6189527/, March 2018
Leave a Reply