Accendo Reliability

Your Reliability Engineering Professional Development Site

  • Home
  • About
    • Contributors
  • Reliability.fm
    • Speaking Of Reliability
    • Rooted in Reliability: The Plant Performance Podcast
    • Quality during Design
    • Critical Talks
    • Dare to Know
    • Maintenance Disrupted
    • Metal Conversations
    • The Leadership Connection
    • Practical Reliability Podcast
    • Reliability Matters
    • Reliability it Matters
    • Maintenance Mavericks Podcast
    • Women in Maintenance
    • Accendo Reliability Webinar Series
    • Asset Reliability @ Work
  • Articles
    • CRE Preparation Notes
    • on Leadership & Career
      • Advanced Engineering Culture
      • Engineering Leadership
      • Managing in the 2000s
      • Product Development and Process Improvement
    • on Maintenance Reliability
      • Aasan Asset Management
      • CMMS and Reliability
      • Conscious Asset
      • EAM & CMMS
      • Everyday RCM
      • History of Maintenance Management
      • Life Cycle Asset Management
      • Maintenance and Reliability
      • Maintenance Management
      • Plant Maintenance
      • Process Plant Reliability Engineering
      • ReliabilityXperience
      • RCM Blitz®
      • Rob’s Reliability Project
      • The Intelligent Transformer Blog
    • on Product Reliability
      • Accelerated Reliability
      • Achieving the Benefits of Reliability
      • Apex Ridge
      • Metals Engineering and Product Reliability
      • Musings on Reliability and Maintenance Topics
      • Product Validation
      • Reliability Engineering Insights
      • Reliability in Emerging Technology
    • on Risk & Safety
      • CERM® Risk Insights
      • Equipment Risk and Reliability in Downhole Applications
      • Operational Risk Process Safety
    • on Systems Thinking
      • Communicating with FINESSE
      • The RCA
    • on Tools & Techniques
      • Big Data & Analytics
      • Experimental Design for NPD
      • Innovative Thinking in Reliability and Durability
      • Inside and Beyond HALT
      • Inside FMEA
      • Integral Concepts
      • Learning from Failures
      • Progress in Field Reliability?
      • Reliability Engineering Using Python
      • Reliability Reflections
      • Testing 1 2 3
      • The Manufacturing Academy
  • eBooks
  • Resources
    • Accendo Authors
    • FMEA Resources
    • Feed Forward Publications
    • Openings
    • Books
    • Webinars
    • Journals
    • Higher Education
    • Podcasts
  • Courses
    • 14 Ways to Acquire Reliability Engineering Knowledge
    • Reliability Analysis Methods online course
    • Measurement System Assessment
    • SPC-Process Capability Course
    • Design of Experiments
    • Foundations of RCM online course
    • Quality during Design Journey
    • Reliability Engineering Statistics
    • Quality Engineering Statistics
    • An Introduction to Reliability Engineering
    • An Introduction to Quality Engineering
    • Process Capability Analysis course
    • Root Cause Analysis and the 8D Corrective Action Process course
    • Return on Investment online course
    • CRE Preparation Online Course
    • Quondam Courses
  • Webinars
    • Upcoming Live Events
  • Calendar
    • Call for Papers Listing
    • Upcoming Webinars
    • Webinar Calendar
  • Login
    • Member Home

by Dianna Deeney Leave a Comment

QDD 033 How Many Do We Need To Test?

How Many Do We Need To Test?

In this episode we review sampling for design tests. We talk through a generic thought process for choosing a statistically relevant sample size and propose some basics that we can all learn about to better understand sampling.

Our goal is for us to be able to better talk through a sampling scenario with our quality and reliability engineering friends, and to better prepare for the information that they’re going to want to know when asked, “How many do we need to test?”

 

View the Episode Transcript

 

The types of things we consider:

  1. What is our acceptance criteria?
  2. Is the data going to be continuous or discrete?
  3. How are we expecting this product to perform for this test? Do we have any historical data?
  4. How confident do we need to be in the result?
  5. What is the smallest difference we want to be able to notice?
  6. What is the test method? Is it validated? What is its precision and accuracy?

Even if we don’t have all the answers, it’s good to include quality and reliability engineers in sample size calculations. They may have input into the requirement or test method. This plus knowing that each requirement needs its own sampling study (like we did in this episode) shows that getting them involved with the iterative drafts of our test plan could be a benefit to the project.

Citations

Understanding the basics of hypothesis testing will set you up for a better understanding of many sampling methods. Any basic statistics book would provide a good introduction!

Episode Transcript

In this episode we’re going to be talking about sampling. How many do we need to test? “42 is the answer to the ultimate question of life, the universe and everything,” according to The Hitchhiker’s Guide to the Galaxy, a science fiction adventure. But figuring out a statistically relevant sample size is not so straightforward. Statistics and probabilities are not an intuitive subject to understand or explain. And, I know, you ask a statistician or quality or reliability engineer a question about something like sampling and you’re likely to never get a straight answer. “It depends,” is what you’ll probably hear most often, and if you do get an answer and ask, “why?” you’ll get a lot of mathematical explanations and graphical plots.

Let’s talk through a generic thought process for choosing a statistically relevant sample size. Let’s also talk about some basics that you can look into for your own knowledge growth that will help you to meet the statistic-minded folks at least halfway. Our goal today is so you can better talk through a sampling scenario with your quality and reliability engineering friends, and you can better prepare for the information that they’re going to ask about. Let’s get started after this brief introduction.

Hello and welcome to quality during design the place to use quality thinking to create products others love, for less. My name is Dianna. I’m a senior level quality professional and engineer with over 20 years of experience in manufacturing and design. Listen in and then join the conversation at QualityDuringDesign.com.

Today we’re talking about sample sizes for testing our designs. There is a lot to consider when choosing a sample size that is statistically relevant, where we have a certain confidence that the performance of our product is meeting its requirements or specifications. There is no one perfect answer for how many to sample. It depends on what we’re measuring, how confident we want to be in the result, and a whole host of other things.

Why are there so many questions going into what sample size to choose? It’s because we want to limit our chances of making errors. We’re collecting samples, testing them, and then fitting the results of our sample to a mathematical model (like a probability distribution). We use that model to make predictions or conclusions about how the greater population is going to perform, and we make decisions from it. If we don’t test enough then we could fail to act when we should. That’s called a Type II error. In design, this could mean we missed an opportunity for improvement. If we have too much data, then we could want to act when we shouldn’t, which is a Type I error.

The ways we can position ourselves to use statistical analysis to make the best decisions is understanding statistics and also considering historical information and how we’ll collect the data. There are some rules of thumb, but they don’t apply to every test scenario. And, if we blindly use them, we may end up testing more than we really need. If we test 30 samples, that allows us to be confident to use some standard statistical tests where we can make conclusions about our data with a confidence level (you can search up the Central Limit Theorem to find out more about this). However, if we know more about our product and its requirements, we can make a better assessment of the sample size needed. There are many cases where we don’t need as many as 30 samples which will save on test costs, in that there’s less products to test and less test time needed.

If we have a brand-new design that’s never been tested before and we want to get to know how it will perform a little better, then a good rule of thumb to test is 5. Where this is in no way a sample size that is statistically relevant, it is enough data points to plot to get a sense of the data. From this limited test, we can make better assessments for what our sample size should be for our formal verification or validation testing. It’s better than a complete guess. In my experience, testing 5 as a screening-like test plus the samples we’ll need for our official, statistically relevant tests is still less than 30 samples total. In designed experiments like DOE (design of experiments) and ANOVA (analysis of variance), there are some rules of thumb that suggest that 3 samples is okay. If we’re under development and doing HALT (highly accelerated life test), then doing 1 to 2 samples might be enough.

Let’s talk about some of the things we start thinking about when someone asks, “How many samples do I need to test?” We’re about to go through that generic thought process for choosing a statistically relevant sample size.

One of the first questions is, “What is our acceptance criteria?” This is getting into what it is we’re exactly testing and what type of analysis we may need to perform with the data. Are we comparing the performance of test samples against a requirement, or are we comparing two things? Is a requirement a minimum or maximum value, or is it based on an average? What is considered a success and what is considered a failure? Do our requirements include the operating and environmental conditions of our product? Was this information considered when the requirement was developed? Are there any reliability requirements, that the design needs to function at a certain performance level after a set period? When requirements are being developed, it’s good to think about how we’re going to test this requirement to make sure that our product meets it. We may have a draft requirement, but then our quality or reliability engineer on our team may give us input like environmental conditions, reliability targets, or even confidence. And that input may be a good addition to a requirement.

Back to choosing a sample size, we have a good understanding of the requirement. Now, “Is the data going to be continuous or discrete?”Continuous data is measured and can be any value in a range. Discrete data is counted and can only be certain values. Whether our data is continuous or discrete narrows down our choices for how would determine a sample size. There are different probability distributions for each type of data.

We’ll also wonder, “How are we expecting this product to perform for this test? Do we have any historical data?” We could use results from benchtop testing, engineering confidence testing, or testing of similar products. What we’re looking for is a hint of statistical measures that might represent the formal data we’re going to be collecting through our sample. What we learn could affect our sample size. We’ll consider the type of probability distribution this data may take and the measures of location and dispersion. Typical measures of location are things like mean and median, and for dispersion are things like range, variance, and standard deviation. With our historical data, we’ll also look at the types of failure modes seen in a test. How do the observed failure modes align with the requirement? Is there more than one failure mode?

Still thinking about what we might need for a sample size: we also think about, “How confident do we need to be in the result?” We covered this in a previous episode titled The Five Aspects of Good Reliability Goals and Requirements. Here’s what I said about confidence levels in that episode: “If we don’t specify a confidence level, then we can assume a 50% confidence level. I’ve never had a team to find their confidence level in anything at a 50% level. Who wants to be half confident? We can stay our desired confidence level as part of our reliability requirement. The confidence level that we choose can be dependent upon customer perception, the effect on the overall function of our product, or how serious it is. If our product doesn’t work, name a few. Why do we add a confidence level? Because there’s variation in everything, both in how we make product and how we measure it. Setting a confidence level accounts for the variability we’re going to see in our test data.”

Another question we ask to help determine a statistically relevant sample size is about difference to detect. “What is the smallest difference we want to be able to notice?” The difference to detect is our ability to detect a difference between our requirement and what we measure, or the difference between a hypothesize value and the actual value. It’s related to a Type II error (that’s the error where we fail to act when we really should or fail to improve our design when it needs it). For continuous data, the difference to detect is typically related to the number of standard deviations, but it could be just based on a choice that we make. For discrete data, the difference to detect is usually based on a proportion of the population.

To choose a difference to detect, we consider everything we know about the situation at this point, starting with how our requirement (or acceptance criteria) compares with how we’re expecting our results to look from our historical data. Are we expecting our test results to be close to our target spec? If not, then maybe we’ll choose a different to detect that’s large. If we do expect the results to be close to the spec, then we’ll likely choose a small difference to detect. For example, if we’re testing for tensile strength with a minimum requirement of 5N, and our benchtop tests of some initial samples are indicating a performance of 70 to 100N, then we might not be at all worried about being able to detect a small difference. Our product is performing with a factor of safety that’s pretty large, so we decide that our risk of making an error is pretty low. So, we’ll choose a large difference to detect, like 4N. Choosing a large difference to detect will reduce our sample size requirements. However, if our benchtop results are indicating that a performance that range between 6 to 10N; that’s a lot closer to our 5N minimum requirement and there’s not much room for a factor of safety. We’ll want to detect a much smaller difference. We may want to be able to detect a difference that’s less than the sample standard deviation of our historical measures. How much less? That depends on how critical this component or feature is to the performance, functionality, reliability, and safety of our product. Choosing a small difference will increase our sample size requirement.

Finally, we’ll look at the way we’re planning to test. “What is the test method? Is it validated? What is its precision and accuracy?” We may have a historical test method we want to use, but is it going to be able to detect the difference that we need to? Is its precision and accuracy good enough for our test? Depending on how our requirement is stated, a reliability engineer may recommend a different test. There are reliability life tests where we’re estimating the durability of our component. Or we could use accelerated life testing where we apply stresses on our design to bring about and accumulate damage. Different test designs will require different sample sizes.

Now, we’re equipped with the information to be able to choose a sample size that is statistically relevant. We start with understanding the acceptance criteria, knowing any historical information, choosing how confident we need to be, and ensuring the test and data collection methods match the requirement.

What are topics you could study a bit to get a better understanding of sampling? I’d recommend looking at hypothesis tests as a topic, where we’re evaluating a null hypothesis an alternate hypothesis. This type of analysis is used in science, engineering, and testing. Getting a good handle on hypothesis testing will set you up for a better understanding of many sampling methods. Any basic statistics book would provide a good introduction to hypothesis testing.

Our goal today was to be able to better talk through a sampling scenario with our quality and reliability engineering friends and have a better appreciation of the types of things that are considered when a sample size is picked. We stepped through a generic thought process for choosing a statistically relevant sample size. And I recommended learning more about hypothesis testing.

What’s today’s insight to action? Before jumping to ask, “How many do I need to test?” – as a team, we can equip ourselves with as many answers as we can to the questions we asked today. I’ll include a list of these questions on the podcast blog. Even if we don’t have all the answers, it’s good to include quality and reliability engineers in sample size calculations, so we’ll be confident enough to approach them about our test. They may have an alternative test method like accelerated life testing. This plus knowing that each requirement needs its own sampling study (like we did today) shows that getting them involved with the iterative drafts of our test plan could be a benefit to the project.

Please visit this podcast blog and others at qualityduringdesign.com. Subscribe to the weekly newsletter to keep in touch. If you like this podcast or have a suggestion for an upcoming episode, let me know. You can find me at qualityduringdesign.com, on LinkedIn, or you could leave me a voicemail at 484-341-0238. This has been a production of Denney Enterprises. Thanks for listening!

Filed Under: Quality during Design, The Reliability FM network

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Quality during Design podcast logo

Tips for using quality tools and methods to help you design products others love, for less.


by Dianna Deeney
Quality during Design,
Hosted on Buzzsprout.com
Subscribe and enjoy every episode
Google
Apple
Spotify

Recent Episodes

QDD 101 Quality Tools are Legos of Development (and Their 7 Uses)

QDD 100 Lessons Learned from Coffee Pod Stories

QDD 099 Crucial Conversations in Engineering, with Shere Tuckey (A Chat with Cross-Functional Experts)

QDD 098 Challenges Getting Team Input in Concept Development

QDD 097 Brainstorming within Design Sprints

QDD 096 After the ‘Storm: Compare and Prioritize Ideas

QDD 095 After the ‘Storm: Pareto Voting and Screening Methods

QDD 094 After the ‘Storm: Group and Explore Ideas

QDD 093 Product Design with Brainstorming, with Emily Haidemenos (A Chat with Cross Functional Experts)

QDD 092 Ways to Gather Ideas with a Team

QDD 091 The Spirits of Technical Writing Past, Present, and Future

QDD 090 The Gifts Others Bring

QDD 089 Next Steps after Surprising Test Results

QDD 088 Choose Reliability Goals for Modules

QDD 087 Start a System Architecture Diagram Early

QDD 086 Why Yield Quality in the Front-End of Product Development

QDD 085 Book Cast

QDD 084 Engineering in the Color Economy

QDD 083 Getting to Great Designs

QDD 082 Get Clarity on Goals with a Continuum

QDD 081 Variable Relationships: Correlation and Causation

QDD 080 Use Meetings to Add Productivity

QDD 079 Ways to Partner with Test Engineers

QDD 078 What do We do with FMEA Early in Design Concept?

QDD 077 A Severity Scale based on Quality Dimensions

QDD 076 Use Force Field Analysis to Understand Nuances

QDD 075 Getting Use Information without a Prototype

QDD 074 Finite Element Analysis (FEA) Supplements Test

QDD 073 2 Lessons about Remote Work for Design Engineers

QDD 072 Always Plot the Data

QDD 071 Supplier Control Plans and Design Specs

QDD 070 Use FMEA to Design for In-Process Testing

QDD 069 Use FMEA to Choose Critical Design Features

QDD 068 Get Unstuck: Expand and Contract Our Problem

QDD 067 Get Unstuck: Reframe our Problem

QDD 066 5 Options to Manage Risks during Product Engineering

QDD 065 Prioritizing Technical Requirements with a House of Quality

QDD 064 Gemba for Product Design Engineering

QDD 063 Product Design from a Data Professional Viewpoint, with Gabor Szabo (A Chat with Cross Functional Experts)

QDD 062 How Does Reliability Engineering Affect (Not Just Assess) Design?

QDD 061 How to use FMEA for Complaint Investigation

QDD 060 3 Tips for Planning Design Reviews

QDD 059 Product Design from a Marketing Viewpoint, with Laura Krick (A Chat with Cross Functional Experts)

QDD 058 UFMEA vs. DFMEA

QDD 057 Design Input & Specs vs. Test & Measure Capability

QDD 056 ALT vs. HALT

QDD 055 Quality as a Strategic Asset vs. Quality as a Control

QDD 054 Design Specs vs. Process Control, Capability, and SPC

QDD 053 Internal Customers vs. External Customers

QDD 052 Discrete Data vs. Continuous Data

QDD 051 Prevention Controls vs. Detection Controls

QDD 050 Try this Method to Help with Complex Decisions (DMRCS)

QDD 049 Overlapping Ideas: Quality, Reliability, and Safety

QDD 048 Using SIPOC to Get Started

QDD 047 Risk Barriers as Swiss Cheese?

QDD 046 Environmental Stress Testing for Robust Designs

QDD 045 Choosing a Confidence Level for Test using FMEA

QDD 044 Getting Started with FMEA – It All Begins with a Plan

QDD 043 How can 8D help Solve my Recurring Problem?

QDD 042 Mistake-Proofing – The Poka-Yoke of Usability

QDD 041 Getting Comfortable with using Reliability Results

QDD 040 How to Self-Advocate for More Customer Face Time (and why it’s important)

QDD 039 Choosing Quality Tools (Mind Map vs. Flowchart vs. Spaghetti Diagram)

QDD 038 The DFE Part of DFX (Design For Environment and eXcellence)

QDD 037 Results-Driven Decisions, Faster: Accelerated Stress Testing as a Reliability Life Test

QDD 036 When to use DOE (Design of Experiments)?

QDD 035 Design for User Tasks using an Urgent/Important Matrix

QDD 034 Statistical vs. Practical Significance

QDD 033 How Many Do We Need To Test?

QDD 032 Life Cycle Costing for Product Design Choices

QDD 031 5 Aspects of Good Reliability Goals and Requirements

QDD 030 Using Failure Rate Functions to Drive Early Design Decisions

QDD 029 Types of Design Analyses possible with User Process Flowcharts

QDD 028 Design Tolerances Based on Economics (Using the Taguchi Loss Function)

QDD 027 How Many Controls do we Need to Reduce Risk?

QDD 026 Solving Symptoms Instead of Causes?

QDD 025 Do you have SMART ACORN objectives?

QDD 024 Why Look to Standards

QDD 023 Getting the Voice of the Customer

QDD 022 The Way We Test Matters

QDD 021 Designing Specs for QA

QDD 020 Every Failure is a Gift

QDD 019 Understanding the Purposes behind Kaizen

QDD 018 Fishbone Diagram: A Supertool to Understand Problems, Potential Solutions, and Goals

QDD 017 What is ‘Production Equivalent’ and Why Does it Matter?

QDD 016 About Visual Quality Standards

QDD 015 Using the Pareto Principle and Avoiding Common Pitfalls

QDD 014 The Who’s Who of your Quality Team

QDD 013 When it’s Not Normal: How to Choose from a Library of Distributions

QDD 012 What are TQM, QFD, Six Sigma, and Lean?

QDD 011 The Designer’s Important Influence on Monitoring After Launch

QDD 010 How to Handle Competing Failure Modes

QDD 009 About Using Slide Decks for Technical Design Reviews

QDD 008 Remaking Risk-Based Decisions: Allowing Ourselves to Change our Minds.

QDD 007 Need to innovate? Stop brainstorming and try a systematic approach.

QDD 006 HALT! Watch out for that weakest link

QDD 005 The Designer’s Risk Analysis affects Business, Projects, and Suppliers

QDD 004 A big failure and too many causes? Try this analysis.

QDD 003 Why Your Design Inputs Need to Include Quality & Reliability

QDD 002 My product works. Why don’t they want it?

QDD 001 How to Choose the Right Improvement Model

© 2023 FMS Reliability · Privacy Policy · Terms of Service · Cookies Policy

This site uses cookies to give you a better experience, analyze site traffic, and gain insight to products or offers that may interest you. By continuing, you consent to the use of cookies. Learn how we use cookies, how they work, and how to set your browser preferences by reading our Cookies Policy.