Gap Assessment Questions

Appendix A: List of Generic Questions to Consider When Conducting a Gap Assessment

The following is an except from The Process of Reliability Engineering, a book by Carl S. Carlson and Fred Schenkelberg. Within the book see section 6.3 How is a reliability gap assessment performed? for more information on tailoring questions in preparation for an assessment.

The following are potential questions to consider when performing a reliability gap assessment. This list is not meant to be a template and should only be used as thought-starters. The list of questions used in a gap assessment should be suitable for the circumstances and scope of the individual project.

Questions relating to the reliability strategic vision (step 1)

What products or services will need reliability strategy and planning?
Is there a stated organization-wide vision for reliability?
1. What is it?
2. How was it developed?
How long should products last in service?
How trouble-free should products be in service?
From a reliability standpoint, where does the organization need to be in one year? Three years? Five years?
What kind of reliability organization should there be in the future to support the needed reliability of products in terms of staffing, training, and organization?
What business processes need to be in place to support the needed reliability of products?
What is the scale of the product that will be developed, in terms of quantity per year, cost per item, and any other planning elements that affect the type of reliability tools and methods that will be used in the reliability plan?
What other questions about reliability strategy, appropriate for the unique circumstances of the company, need to be considered?
Who owns reliability? When does management get involved with reliability issues? Is management involvement reactive or proactive?

Questions relating to reliability gaps (step 2)

What are the known gaps in the reliability strategic vision?
What has been the customer feedback about reliability of current products or services?
What current products and processes represent the biggest reliability challenge to the company? What data support this conclusion?
What is the capability of current products and processes to achieve the reliability requirements?
What are the known high-risk areas?
How well does the reliability of current products compare to those of the competition?

Questions relating to reliability methods

As can be seen from The Process of Reliability Engineering Chapter 8, the methods break down into 4 categories of tasks—requirements, risk reduction, assurance, and organizational—and a set of 11 subcategories. We’ll use these same breakdowns to outline potential reliability method questions.

Questions relating to reliability targets

What are the current reliability requirements for systems, subsystems, and/or components? How were these set?
Are reliability requirements measurable at system, subsystem, and component levels?
Are reliability requirements verifiable during the product development timeframe?
Are reliability requirements converted into actionable technical specifications?
What are customer expectations for the reliability of products or services?
How would you characterize the maturity of the requirements process?
1. How well are requirements based on customer requirements or standards?
2. How well do requirements include environment and use profiles?
3. Are system requirements properly flowed down to lower levels?
4. What is the utilization of distributions for environmental and use conditions?
What other questions about reliability requirements, appropriate for the unique circumstances of the company, need to be considered?

Questions relating to reliability data needs

How is reliability of current products measured? How is the cost of unreliability measured?
What is the quality of field failure data? How much noise does it contain? Are root causes and failure mechanisms identified from field failures? How is field failure data managed?
What definition of failure is used to measure reliability?
How is reliability data analyzed?
What other questions about reliability data, appropriate for the unique circumstances of the company, need to be considered?

Questions relating to designing-in reliability

When questioning about specific DFR methods, the following progression of questions is recommended for understanding capabilities and potential gaps:

To what extent do you use a selected reliability method?
How do you use a selected reliability method?
Why do you use the selected reliability method?
What examples do you have that demonstrate application of the selected reliability method?

Selecting the reliability methods to consider

The idea here is not to examine every possible reliability method or tool but to focus on the more important methods or tools. Select the specific reliability methods that will be included in the reliability gap assessment, based on the scope and circumstances of the project or program being assessed. The following is a list of potential reliability methods to consider, aligned to the stages of a product development process.

Examples of reliability methods used during the “Concept Development” stage

System conditions of use and operating profiles
System-level reliability requirements
Reliability requirements flow-down to subsystems and components
System reliability modeling
System FMEA
Identification of reliability-critical components and subsystems

Examples of reliability methods used during the “Design” stage

Design margin analysis
Design FMEAs on selected subsystems and components
Fault tree analysis
Root cause of known reliability problems
Product design guides
Design for manufacturability
Derating
Reliability input into design reviews
Robust design methods (e.g., DOE or physics of failure)
HALT
Review of supplier FMEAs for critical components

Examples of reliability methods used during the “Assurance” stage

Reliability test methods
ALT methods
A reliability test plan
System reliability growth testing
Verification that suppliers meet supplier reliability requirements
Management reviews to include test failure data

Examples of reliability methods used during the “Manufacturing” stage

Process FMEAs
Process control plans
Screening and monitoring plans
Statistical process control
A field test plan
Verification that reliability requirements are met before launch
Product lessons learned documentation

Examples of reliability methods used during the “Field” stage

Field reliability performance (via FRACAS)
Analyze and fix field problems
Reliability-centered maintenance
Reliability monitoring until product is deactivated

Questions relating to supplier reliability

What processes are in place to assure the reliability of supplier parts?
1. Are reliability specifications incorporated into supplier bid packages?
1. Are selected reliability tasks incorporated into supplier bid packages?
1. Are suppliers selected who are capable of achieving reliability objectives?
1. Is there a process in place to identify critical supplier parts?
1. Does someone in the organization review and approve supplier tasks for critical parts before shipment?
1. Does someone in the organization review and approve the achievement of supplier reliability requirements for critical parts before shipment?
1. What other questions about supplier reliability, appropriate for the unique circumstances of the company, need to be considered?

Questions relating to manufacturing reliability

Are current manufacturing processes documented with a process flow diagram? What is the current use and effectiveness of process FMEAs?
Is there a process in place to effectively utilize process FMEAs?
How well are manufacturing operations controlled? What process control methods are being used?
Are key product characteristics being identified and translated into key process characteristics? Are key process characteristics controlled through process control plans?
What methods are being used to assure that manufacturing processes are stable and capable?
Do operators have well-written work instructions that ensure consistency and capability of their respective operations? What other questions about manufacturing reliability, appropriate for the unique circumstances of the company, need to be considered?

Questions relating to reliability validation

What is the current use and effectiveness of analytical simulation methods for modeling the reliability of systems?
1. What is the current use and effectiveness of analysis to verify that reliability requirements are met before product launch?
1. What is the current use and effectiveness of physical testing methods to verify that reliability requirements are met before product launch?
1. What is the current use and effectiveness of ACT to verify that reliability requirements are met before product launch?
1. How well do reliability predictions match observed field reliability?
1. What other questions about reliability assurance, appropriate for the unique circumstances of the company, need to be considered?

Questions relating to reliability growth

What is the current use and effectiveness of appropriate reliability growth models to improve test and field reliability?
1. What is the current use and effectiveness of life data analysis to analyze test and field data?
1. Is there a failure review system in place, such as FRACAS? How well does it work?
1. Is the effectiveness of corrective actions tracked over time? If so, how is this done?
1. Are identified failure modes addressed in other product lines? If so, how is this done?
1. What other questions about reliability growth, appropriate for the unique circumstances of the company, need to be considered?

Questions relating to service and maintenance

Are service needs identified early in product development and designed into the product? What service gaps exist?
Are maintenance requirements anticipated and implemented with adequate preventive maintenance plans?
Is reliability-centered maintenance properly used on current programs? What are the results?
What is the customer feedback on product servicing?
What limited-life components need to be maintained or replaced? What other questions about service and maintenance, appropriate for the unique circumstances of the company, need to be considered?

Questions relating to organizational resources

What personnel are currently supporting reliability activities? What are their qualifications?
How is the reliability activity currently organized? What is the reporting structure? Is there a homeroom for reliability support activities, such as training, common processes, and reliability task support? If so, how well does it work?
What functions and disciplines are currently involved in reliability planning? What skills and disciplines need to be involved?
Who needs to be involved in the reliability planning process? What communication links will need to be set up?
Is there a “champion” in management who actively provides support for reliability activities at management level? If so, how well does this work?
What training is available for reliability engineers? What is the content of the training?
What reliability training is provided for design engineers? What is the content of the training?
What reliability training is provided for managers? What is the content of the training?
How well is reliability training supported by management? What is its priority?
What procedures document the current reliability practices? How were they developed?
What business processes support the current reliability tasks? How were they developed?
What is management’s attitude toward reliability? What is their level of support for reliability activities and tasks? What are considered management’s roles and responsibilities regarding reliability planning and execution?
What organizational problems need to be avoided to ensure success in reliability activities?
What other questions about reliability organizational resources, appropriate for the unique circumstances of the company, need to be considered?

Questions relating to integration with engineering processes

1. What level of responsibility does engineering have to achieve reliability objectives? Is this understood and implemented in daily work?

2. Are reliability methods integrated into ongoing engineering procedures and tasks?

3. What other issues exist that will need to be considered in the development of a comprehensive reliability plan?

4. What other questions about integration of reliability activities with engineering processes, appropriate for the unique circumstances of the company, need to be considered?

Questions relating to the maturity matrix

Questions related to the maturity matrix are posed as a check step afterward to confirm that the details all make sense to the stakeholders.

Questions relating to reliability plans

What are the current reliability plans? How are they developed, documented, tracked, and executed?
How would you characterize the maturity of the reliability planning process?

Do reliability plans have required activities?
Do reliability plans have an appropriate level of detail?
Is the status of reliability plans regularly reviewed? How well are roadblocks and other execution issues addressed?
How well are reliability plans uniquely crafted for each project and projected risk?
Is there a way to update reliability plans based on contingencies?
How well are business or market considerations integrated into reliability plans?
Is the reliability plan properly integrated into strategic business plans of the company?

What reliability plans have been developed in the past for similar products? How well have these been executed and what have been the results?
What other questions about reliability planning, appropriate for the unique circumstances of the company, need to be considered?

Questions relating to executing reliability tasks

What infrastructure and strategy in the organization support the execution of reliability plan tasks?
Is the current infrastructure and strategy sufficient to ensure timely execution of reliability plan tasks?
How well does the organization execute tasks in general?
What roadblocks or issues are anticipated that would impede the timely execution of reliability plan tasks?
Who is responsible for achieving reliability in the organization? Do they have sufficient authority to ensure that identified tasks are fully executed?
When certain tasks are completed, such as HALT or FMEA, what are the next steps?
What constraints could derail the execution of the reliability plan? What about funding, lead times, and competing priorities?
What assets exist to support the execution of a reliability plan (e.g., lab analysis)?
What reviews exist to follow-up on reliability plan tasks?
How well are the individual tasks in the reliability plan integrated with other reliability tasks?

How well are the tasks integrated with the overall product development program?