Motivation for Structured Experimentation

When planning and designing an experiment, it may be tempting to try and accomplish all the objectives is a single experiment.  The thinking is often that experimentation is time consuming and expensive, so one experiment must be better than multiple experiments.

However, in general, it is a good idea to plan for multiple experiments which often is a much more efficient approach.  We like to think of experimentation as a methodology that is best implemented in phases.  We define experimental phases as:

1. Screening Phase
2. Knowledge Building Phase
3. Optimization Phase

 

Let’s understand the purpose of the 3 phases.

Screening studies are used when we are starting with many factors that we want to investigate.  We may not have a lot of understanding of exactly which factors are important, so screening studies help us do this in an efficient manner.   The goal is to determine which factors significantly affect the response(s) of interest.  We are not yet worried about developing a very precise model, but rather who is important and who isn’t.

I’m a tennis player and I’m always trying to improve various aspects of my game.  Recently, I’ve been trying to develop a better serve.  Tennis players know that holding serve is important for winning matches.  Suppose I am trying to improve two key aspects of the serve:  Speed and Accuracy.  These are the outputs or responses.  The key to winning points on the first serve is to have high velocity in the right location.  Some of the factors I have been playing around with in an effort to improve velocity and accuracy are:

1. Height of Ball Toss
2. Location of Ball Toss (how far into the court)
3. Grip location
4. Degree of Racquet Drop
5. Eye placement during impact
6. Initial Stance at Baseline
7. Degree of Body Twist during Serve motion

It’s likely that some of these factors have very big impacts on one or both responses and some may have little or no impact.  The key of the screening study is to determine exactly which factors are important and are worth modeling.  In order to develop a predictive model, I may have to investigate many levels for each factor (e.g. ball tosses at say 3-4 different heights), but I would be wasting a lot of time if I’m running all of these various trials with factors that are not even relevant.  To illustrate, suppose I was going to run trials so that each factor is investigated at 3 different levels.  To run all possible combinations, I would need 3⁷ combinations which equals 2187!  That would be an awfully long day on the tennis court.

By keeping each factor at 2 levels the total number of combinations would be 2⁷ = 128 combinations which is still quite a lot.  Fortunately, there is a very nice technique that allows us to run fewer trials without sacrificing much information.  This technique is called a fractional factorial experiment and is commonly used in screening studies.  We’ll talk more about fractional factorials in a later article.  What’s the downside for running only 2 levels of each factor?  With each factor run at only 2 levels, we don’t have the ability to model non-linear relationships, but that is a price worth paying, since the initial objective is only to identify which factors are relevant.  Once we narrow down the list of factors to the most important ones, we can do a much smaller experiment that allows non-linear relationships to be understood and modeled.  This occurs in the optimization phases, which is the 3^rd phase identified above.  Optimization experiments all a precise determination of the mathematical relationship between factors and the responses and non-linearities may be modeled.

In between the screening phase and optimization phase, there may be experiments that we call Knowledge Building studies.  This middle phase may or may not be required depending on the type of screening study that was initially run.  When many factors are present, we may elect to do a screening study that has low “resolution” (we’ll discuss this later in the article series), and we may want to follow-up with a study to improve the resolution after eliminating some of the insignificant factors.  This middle phase is not quite optimization but builds in the initial screen studies to gain more information about which factors should be selected to optimize responses.

Keep in mind that not all phases are required in every situation.

In some cases, only one experiment is warranted.  For example, if there are only a couple of factors that are to be studied, then it would make sense to go straight to the optimization phase since only a few factors can be efficiently studied with optimization-type experiments.  It would be wasteful to do screening when we only have a few factors to start with.

Also, sometimes a screening study gives us such good information about the factors and model that further studies are not necessary.

In summary, a phased approach to experimentation is recommended since it results in the most efficient path to cross the finish line.