From ASQ Reliability Review, Vol. 16, No. 3, Oct. 1996. Revised 2002, 2018, and Jan. 4, 2026.
The Apple Computer Reliability Department manager, Wayne Smith, told me (circa 1991), “We make sure Apple doesn’t sell a product that doesn’t work,” (by in-house product tests during design phase). (I worked in Apple’s Service Department.) Testing helps, but real, age-specific reliability is determined in the field, in the hands of customers, in their environments. Companies could benefit from using age-specific, field reliability to eliminate waste, improve customer satisfaction, and reduce uncertainty. The Field Reliability Applications Award is a clone of the Malcolm Baldrige National Quality Award modified to quantify reliability maturity. The field reliability applications evaluation in this article supplements Fred’s podcast on “Reliability Maturity” [Schenkelberg].
These facts motivate field reliability and its applications:
- Field reliability is pervasive. The definition of reliability includes the phrase “for a specified time or to specified age.” That means the product continues to work throughout its useful life. If the product doesn’t have good field reliability, customers will know. If you know your products’ field reliability, you’ll know whether improvement is justified and how to improve it.
- Field data is the source of reliability. The definition of reliability also includes the phrase “under specified conditions.” That means field conditions. What other conditions could possibly interest customers?
- Potential field reliability applications abound. Solutions to many real problems using field reliability await your use. People unafraid of mathematics, statistics, computer programming, and data analyses need to implement these solutions.
- Benchmarking is an elaborate form of making excuses. Work on improvement instead of surveying competitors who don’t use field reliability. “We’re doing as well as the best in class” is a poor excuse for not doing the best you could do.
- Uncertainty is scary. Convert management from managing uncertainty to managing randomness. Any manager can manage uncertainty; only the best can manage randomness, with the help of field reliability.
Should You Use This Evaluation?
You have a choice of formats: paper evaluation at the end of this article or Google sheet “NFRAA”, https://docs.google.com/spreadsheets/d/1AQfRt2GWLZ6aZ3pEh5bZgkgRkRF7ceEAQWl5Z0_Ik14/edit?usp=sharing/.
For your evaluating your company or organization, include its deficiencies and the pressures that result in subpar performance. The NFRAA sheet lists productive applications of field reliability and estimates the potential profit from these applications. It suggests how to change reliability into a conduit for field data throughout organizations and production cycle; it indicates reliability actions and suggests tactics to improve for achieving what management wants: more reliability and less uncertainty.
This evaluation is not an administrative brushfire that draws people from productive work. It does not require management to change. There is no national publicity, no sanctions for failure, no federal agency or professional society involved, no qualification plan for award examiners, no written application, and no filing fee.
Use this evaluation if you do any of the following:
- Reliability applications. Measure your current level of field reliability applications, to see what to do next and estimate the profit from doing so. Reliability managers should use this evaluation to measure the extent of their organization’s reliability applications and to estimate profit lost by not applying field reliability to the fullest extent.
- Engineering. Learn where to get information that’s relevant for design and what information manufacturing and field organizations need for new products and what fixes for old ones. Specifications may be incomplete or imperfect. You can have 500,000-hour MTBF and still have unacceptable warranty returns. Manufacturing, shipment, training, and service can only degrade inherent reliability. Customers may use products in unintended ways.
- Manufacturing. Learn what you can do to reduce the impact of manufacturing, testing, packaging, shipping, installation, and training on inherent reliability and what you can do to achieve inherent reliability as nearly as possible cost effectively.
- Field service. obtain early warnings, forecast service requirements, implement age-specific diagnostics, and provide field service as well as possible. It also helps forecast logistics demands: tools, facilities, manpower, and spares, throughout the support life of products. Age-specific failure rates indicate onset of retirement, so nobody is stuck with obsolescent spares.
- Quality. Managers know that reliability is part of quality and contributes to customer satisfaction and company profit.
- Finance and accounting. Accountants estimate costs and then change numbers when actual costs become known. What could your company do with unnecessarily reserved funds?
Key Concept and Characteristic?
Reliability is determined in the field. Reliability predictions don’t matter except perhaps for technologically new products. Most products are, at least in terms of field reliability, evolutionary. Even if a product is new, its reliability may be the same as previous products’ field reliability. What the customer says-determines failure-not what you say failure is, not how you think the customer should use the product.
Fundamental knowledge of statistics; management science; and operations research, also known as reliability, is applicable to using field data for decision making. This evaluation organizes that fundamental knowledge along organizational functional lines. Each topic contains steps that indicate the extent to which organizations know and apply field reliability. Your organization doesn’t have to do every step in the evaluation. These steps can cost more than they’re worth, but if you don’t know that, that’s no excuse for doing nothing.
This evaluation stresses the use of age-specific field reliability of products and service modules. If your organization doesn’t know age-specific reliability, how can it make reliability-based improvements?
Scoring
Pick the answer under each topic that best describes your organization’s current level of field reliability applications. The answers are ordered from disservice to the highest level of service. Each level assumes that the lower levels of service have been implemented, superseded, or obviated by the current level.
The answers have a value from -1 to +7. Add the values from all the topics. If the sum is negative, divide it by -10. If the sum is positive, divide it by 45/2, (45 is the maximum possible score.) Multiply your annual service profit by the result. If your service organization is not a profit center, then divide your annual service cost by the score; if the score is negative, then consider your organization lucky that service costs aren’t worse. The result estimates how much field reliability applications could contribute to profit or reduce costs. (The Google Sheet does the math for you.)
The reasoning behind the transformation from score result into profit is as follows. The typical 80-20 manager has someone do 20% of the work that yields 80% of potential revenue. Assuming that profit is 40% of potential revenue, the 80-20 manager garners profit equal to 20% of potential revenue, or half of the profit. The forgone half of the profit results from estimating age-specific field reliability and implementing its applications. Sure, the forgone profit requires 80% of the work, but work is what people are paid for.
If your service organization is not a profit center, then the reasoning is as follows. Service costs can be cut in half by complete application of field reliability (score = 2) compared with the state of the art (score = 0).
Your mileage may vary depending on conditions, but two results won’t vary:
- You won’t get any value from field reliability if you don’t use it.
- You can’t use field reliability if you don’t know it.
Discrete Improvement
Quality evangelists advocate continuous improvement. Improvements from field reliability result from discrete changes. Implement new methods, while maintaining old methods, until you’re comfortable enough to abandon the old methods. Then notice the discrete increase in profit.
Is your reliability organization really doing the steps you claimed in the evaluation; thoroughly to the extent justified by cost and value. Then do the next steps to move your organization down each topic list, in order of profitability. Estimate the revenue and cost from the next step in each topic. Do the step that has the highest profit minus cost first. In other words, check each topic for the question one step beyond your current level. Consider how to implement them, and compute how much revenue they’ll produce and how much they cost. Implement the next level for the topic that has the greatest revenue minus cost.
Field Reliability Maturity Evaluation
Please enter you choice as a number corresponding to your choice in the third, empty column: one entry in each table.
What is the field reliability of your product and its service parts?
| A. We predict MTBF; we don’t use field data. | -1 | |
| B. Don’t know field failure rate, product hasn’t been in the field long enough. | 0 | |
| C. We collect monthly failure counts. | 1 | |
| D. We compute monthly MTBF or failure rate per unit shipped. | 2 | |
| E. We estimate age dependent field reliability function R(t) or field failure rate function for all ages t up to the age of the oldest product in the field and use the estimates to estimate service requirements and life cycle costs. | 3 | |
| F. Same, extrapolated using field data from previous, comparable products. Result is obtained using credibility or some weighting of field data from previous products and current product. | 4 | |
| G. We advertise the reliability function and age specific field failure rates of our products. The same information on major service modules is available for potential customers. | 5 | |
| H. We compute Paretos, the proportions of each failure type. | 6 | |
| I. We compute Paretos conditional on ages at failures. | 7 |
What are expected service requirements under warranty?
| A. New product won’t fail. It has been designed not to fail. | -1 | |
| B. Don’t know, warranty is not enforced, just changed, or should change, because product changed or because competitors changed their warranty. | 0 | |
| C. Past returns were n per month. Wiggly line graphs on the wall in our manufacturing department show monthly returns. | 1 | |
| D. Past proportions of warranty returns were p%. | 2 | |
| E. Probability of failure under warranty is p% with confidence level nn%. | 3 | |
| F. The proportions of failures are … for causes …, and the warranty costs for FY24, FY25 were…and… | 4 | |
| G. The age dependent failure rate under warranty is … The proportion DoA is… These measures are used to determine optimal burn-in. | 5 | |
| H. Sell through time distribution is… | 6 |
Do you have an Early Warning system?
| A. Dealers and third-party service providers take care of problems, so we never hear about field performance unless it’s really bad news. | -1 | |
| B. We listen to the field and to manufacturing. Our problem escalation process deals with field problems. Our telephone answering team handles x% more phone calls each year. | 0 | |
| C. We set burn-in to balance its cost with the cost of infant mortality field failures that escape burn-in. We use burn-in failures and ongoing reliability test samples to verify burn-in duration, validate field reliability estimates and determine sell-through time. | 1 | |
| D. We forecast expected cumulative failures since intro and watch confidence limits on forecast of cumulative failures. | 2 | |
| E. If some service module drives cumulative failures above the confidence limit, we estimate the size of the problems and prepare accordingly. | 3 | |
| F. We set warning limits at the level of intervention, i.e. at the level where it becomes cost effective to respond with changes, warranty extensions, recalls or whatever is appropriate. Of course we have evaluated alternatives and have responses prepared. | 4 |
Do you use field-reliability-based forecasts to recommend spares stocks?
| A. We ship 95% of most orders within one day, for most parts, if we stock them, when we can, and if it doesn’t conflict with production requirements. | -1 | |
| B. We extrapolate past spares requirements just like B-school taught us and produce spares stocks so we ship most orders from stock. | 0 | |
| C. We estimate the AFR and multiply it by next year’s installed base to forecast demand, then produce spares stocks so we ship most orders from stock. | 1 | |
| D. We estimate the actuarial failure rates and multiply them by next year’s installed base to forecast demand, then recommend stock levels assuming Poisson demands and (S-1,S) periodic review order policy given customers’ service level requirements. | 2 | |
| E. We compute the forecast demand distribution from the superposition of renewal processes, then set stock levels given demands computed from estimated actuarial rates and (s,S) continuous review order policy given customers’ service level requirements. | 3 | |
| F. We set stock levels using multi-echelon, lateral resupply policy allowing vendor partners to supply some needs given customers’ service level requirements. | 4 | |
| G. We set stock levels taking into account dependence among product demands. | 5 |
Do you base warranty reserves on age-specific field reliability?
| A. Our accounting department recommends warranty reserves, then adjusts them as actual warranty returns become known. Near the end of production, we take a credit for unspent funds squirreled away in warranty reserves. Warranty extensions, recalls, and other surprises are come out of profits if any. | -1 | |
| B. Reliability department estimates warranty failure probability, and we choose warranty reserves based on past experiences and guesses. | 0 | |
| C. Reliability department estimates warranty failure probability, and we choose warranty reserves based on the variation in past experiences and on the expected costs and variations. | 1 | |
| D. We balance the warranty cost with lost sales cost to determine the optimal warranty. We try to equitably share costs of after warranty failures with customers to balance their cost and ours. | 2 | |
| E. Interdepartmental accounting charges engineering and manufacturing for the reliability problems attributable to premature wearout (design defects) and to infant mortality (process defects). | 3 |
Do you know reliabilities of your suppliers’ products in your products; in others’ products?
| A. Suppliers say their products are very reliable. | -1 | |
| B. They say they met our MTBF specification. | 0 | |
| C. Their life test shows they met our MTBF specification with confidence 99%. I. e. their observed MTBF was within the upper 99% confidence limit of our spec although observed MTBF was low. | 1 | |
| D. Our field data and their life test shows they met our MTBF specification with confidence 90%, and their MTBF was greater than our specification. | 2 | |
| E. Their reliability in our applications has been R(t) for all t within the useful life of our product. | 3 | |
| F. They share field failure data with us, and, overall, the reliability has been R(t) for all t within the useful life of our product. | 4 |
Is vendor selection based on vendors’ field reliability and life cycle cost alternatives including service costs?
| A. Management estimates service costs. It’s not the role of reliability department to estimate service costs. Anyway, our purchasing department makes vendor decisions. | -1 | |
| B. If vendors meet our MTBF specification, they’re chosen for other reasons, like cost. | 0 | |
| C. We use the field reliability experience of vendors’ products in our previous products to get some idea of how reliable they’ve been in the past. | 1 | |
| D. We and the vendors test some of their new products in our prototypes to adjust failure rate estimates observed from field data in previous products and other applications. | 2 | |
| E. Vendor selection is based on life cycle costs to us and to customers, among other factors. Life cycle cost includes service costs, and it is based on vendors’ population field data when vendors share with us in return for learning how their products fare in ours. | 3 |
Do your service people use reliability-based diagnostics, service kits, and optimal opportunistic replacement?
| A. Our service people have so much experience they don’t need diagnostics. They do opportunistic replacement whenever they can to generate more service revenue. | -1 | |
| B. Our service manual recommends diagnostics, and our service staff issues a standard service kit to cover all contingencies. We buy larger service trucks every couple of years. | 0 | |
| C. Our service manual contains diagnostic flow charts to several levels based on past experience, expert opinion, and repair costs. | 1 | |
| D. Our expert system diagnostics are based on part failure probabilities | 2 | |
| E. Our fault tree-based diagnostics use part failure rates and times or costs to recommend optimal diagnostic sequences. Our service kits are based on the job, the history of the unit to be repaired, and the part failure rates and part costs. | 3 | |
| F. Our diagnostic system also copes with intermittent failures with an optimal shotgun replacement recommendation based on the age of the problem machine as well as on age specific part failure rates and diagnostic times, spares costs, intermittent failures,… | 4 |
Do you verify fixes? Do you ever have to fix problems more than once?
| A. When we decide on a fix we have a party. | -1 | |
| B. We test a few units before release just to make sure the fix works. | 0 | |
| C. We test a sample of units to failure, with accelerated life tests if we have to. | 1 | |
| D. We also monitor a sample of production with the fix. | 2 | |
| E. We estimate population change in age specific failure rates following release of fixes and test hypotheses regarding success of fixes. | 3 | |
| F. We estimate how fast fixes percolate through the population, if fixes are voluntary or made only upon failure, to estimate rates of acceptance or adoption of fixes. | 4 |
Do you ever get stuck with obsolescent spares?
| A. What spares? Customers can get parts from OEMs or buy a new product from us or our competition. | -1 | |
| B. We stock only high-turn parts so we only get stuck with them. | 0 | |
| C. Engineering recommends spares stocks when they release a new product and we stock them for x years. | 1 | |
| D. We agree to provide spares for x years, and at end of production we produce the expected spares required for x years, at the current demand rates. | 2 | |
| E. We agree to provide spares for the useful life of the product, as determined from age dependent failure rates, and at end of production we produce the expected spare parts required for the useful remaining life, counting spares already in the field, using actuarial forecasts. | 3 | |
| F. We agree to provide spares for the useful remaining life, and at end of production we produce the expected spare parts required, counting spares in the field, using age dependent field failure rates, and we redistribute spares as required among service providers. | 4 |
Scoring. Use Google Sheet to do math if you prefer.
| Please enter annual service profit before taxes. Negative denotes cost of service operations. | $100,000 | |
| Potential profit from field reliability = IF(Service Profit>0,(Service Profit)*(1+IF(SUM(Scores)<0,SUM(Scores)/10, ELSE SUM(scores)/10,SUM(Scores)/22), ELSE Service Profit (negative)) | $100,000 | |
| A. Value answers from -1,0,1,2,… Add the values. If the sum is negative, divide it by 10. If the sum is positive divide it by 44. Take the antilog base 10 of the quotient. Multiply the result by your annual profit from service before taxes. The result is an estimate of forgone profit. | ||
| B. The average 80-20 manager has someone do 20% of the work that yields 80% of potential revenue. Assuming potential profit is 40% of potential revenue, the 80-20 manager garners 20% profit, or half of the potential profit. The other half of the potential is lost | ||
| C. That is the fraction of potential profit you’re probably getting as a result of your current level of reliability. Mileage may vary due to conditions, but two things that won’t vary are that you won’t get any value from field reliability if you don’t learn age-specific field reliability or field actuarial failure rates and use them! |
Discrete Improvement?
Are steps being done that detract from current profit (negative scores)? For Pete’s sake stop!
Continuous improvement is advocated by quality evangelists; one of Demings 14 points is “Improve constantly and forever” [Deming, Google AI]. Unfortunately, the only continuous improvement made by most companies is do the same old things with fewer people, more cheaply. Is your reliability organization really doing everything you claimed? Are all the steps below your current level really being done, really contributing to the bottom line of your organization, except of course the steps that detract from potential profit. Is your reliability organization really doing the step(s) you claimed, fully and thoroughly to the extent justified by cost and value? Check the list for the item one step beyond your current level under each topic. Consider how to implement them, how much they cost and how much they’ll produce. If your reliability engineer doesn’t know your products’ and service modules’ age dependent field failure rates, don’t hold it against him. Most companies don’t know their products’ and service modules’ field reliabilities, so benchmarking wouldn’t be much help. Do the steps and move down each list, in order of profitability.
If you have questions on how to do the steps, ask your reliability staff or ask them to ask me. I do some of them already, and I think I can help do the rest with help from your field data and from references. Most analytical problems have already been solved. All we need to do is find the solutions and apply them.
If you have more suggestions for this reliability maturity evaluation, or, if you have additions, omissions, or criticisms for this evaluation, please send them to pstlarry@yahoo.com.
If your reliability people don’t know your products’ and service parts’ age-specific field reliability, benchmarking wouldn’t indicate any problem. Send me your field data, and I’ll estimate your age-specific field reliability and failure rate functions and quantify variance-covariance of estimates.
References
Fred Schenkelberg, “Fundamentals of Reliability Maturity,” Podcast, https://accendoreliability.com/podcast/arw/fundamentals-of-reliability-maturity/, Dec. 9, 2025
W. Edwards Deming, Out of The Crisis, MIT Press, pp. 23-24, 1982
Leave a Reply