Even Stand-Up Comedians Understand Statistical Sampling

We were chatting with some coworkers about stand-up comedians, and someone mentioned that even the most popular comedians try out their new material in small venues before doing a big show for a larger audience. They do this to collect feedback and fine-tune their performance before reaching thousands of people across different cities.

My reliability engineering brain immediately reacted: “This means even stand-up comedians understand the importance of sampling.” They perform a small-scale version of their show to understand how the audience reacts. Based on that feedback, they decide whether the show is ready for a wider audience. In other words, they want to estimate the population’s reaction based on a sample. To me, this is a perfect example of statistical sampling.

Why Do We Bother with Sampling?

What Even Is Sampling?

In brief, sampling is a way to draw conclusions about a whole population by looking at a smaller sample. The need for sampling in engineering starts with practicality. For example, if you want to understand the reliability of 1 million smartphones you’re about to produce, you can’t realistically test all 1 million devices. This is where sampling and inference come in. They allow us to draw conclusions about a population by looking at the performance of a smaller, more manageable sample. Statistical inference is the engine that makes this possible, and it’s extremely useful in engineering.

We use this concept all the time in reliability testing. We take a sample of the product or component we plan to manufacture, test it, and draw conclusions about the reliability of the whole batch. Even though this sounds simple, there are principles behind it that make it legitimate and powerful. In frequentist statistics (which we’re assuming here), we rely on the Law of Large Numbers and the Central Limit Theorem to make valid inferences. (I’ll probably write about the difference between the frequentist and Bayesian approaches in a future article.)

What Does the Law of Large Numbers (LLN) Tell Us?

Simply put, LLN says that as you collect more samples, your measurement of interest (like the mean) will get closer to the true value for the entire population. Let’s say you want to estimate the mean of a population, so you draw a sample and calculate its mean. Then you draw another sample and calculate the mean of both samples together. You repeat this process, adding more samples each time. As the number of samples increases, the sample mean converges to the true population mean. (See Figure 2 for an illustration.)

What Is the Central Limit Theorem (CLT) For?

The CLT works hand in hand with LLN. According to the Central Limit Theorem, the distribution of sample means will be approximately Gaussian (normal), and the mean of that distribution will equal the true population mean.

Let me explain more clearly: you draw a sample and calculate its mean. You keep that value, draw another sample, and calculate its mean. You repeat this many times. The CLT tells us that the distribution of all those sample means will form a normal curve centered around the true mean — even if the original population distribution was not normal.

This is incredibly useful because it allows us to create confidence bounds around calculated reliability estimates.

And just like with LLN, as you collect more samples, the distribution of these means gets narrower — meaning less variance, meaning you’re gaining more confidence in your estimates.

These two principles (LLN and CLT) are not just useful — they’re fundamental. They’re used everywhere: in medicine, insurance, manufacturing, engineering simulations, and much more.

How Do You Draw a Good Sample?

You’ve probably heard the phrase “garbage in, garbage out.” That applies perfectly to sampling. A sample should truly represent the population you’re trying to understand — otherwise, even the most advanced analysis will lead you to the wrong conclusions.

So, what makes a good sample when you’re trying to estimate the reliability of the product you’re building? Here are three key principles to follow:

1. Random selection: You have to select your test units randomly. That means every unit in your production lot should have an equal chance of being picked. No “choosing what’s convenient” — randomness is what prevents you from introducing accidental bias.
2. Representative of the population: Pull samples from different batches, shifts, and even different factories if you manufacture in multiple locations. This helps ensure your sample reflects the true variability in your overall production — not just a small slice of it.
3. No pre-screening or bias: Don’t cheat the system by testing only the “good-looking” or pre-inspected units. If you’re filtering the units before you sample them, you’re not testing reliability — you’re testing your selection bias.

A good sample isn’t just a smaller version of your production — it’s a fair and unbiased reflection of it.

I hope you enjoyed the content and that it offered some useful takeaways. Engaging with this post by commenting, or sharing helps it reach others who may benefit as well. Please follow me on Linkedin.