Statistical Design of Experiments Training for AOCS Journal Editors Frank Rossi, Associate Director Statistics, Kraft Foods May 3, 2015.

Statistical Design of Experiments Training for AOCS Journal Editors

Frank Rossi, Associate Director Statistics, Kraft Foods

May 3, 2015

Kraft Foods Group, Inc.

Cake Baking Experiment

• Cake formula has been determined

• Need to determine the cooking time and temperature levels that produce the best cake

• Temperature can be varied from 300 to 450 degrees F

• Time can be varied time from 30 to 50 minutes

• Previous research has shown that a baked cake with an internal temperature of 200 degrees is ideal

Objective

Kraft Foods Group, Inc.3

Cake Baking ExperimentStep 1

• Bake some cakes where cooking time and temperature are systematically varied across the specified ranges. Measure the internal temperature on each one.

• The time and temperature combinations form a square.


Cake Baking ExperimentDesign with the data collected

• What have we learned so far?• Somewhere around here seems likely to deliver• How do we get to this exactly?



• Bake some more cakes to better define the cooking time and temperature space. Measure the internal temperature on these too.

• The added points form a circle

Step 2


Cake Baking ExperimentComplete data set

191

189186

• These points are sufficient to build a model to predict the internal temperature

• Why did I run two more at the center?• Predictions are valid only within the circle!



• A surface plot indicates the time and temperature combinations that are expected to deliver the desired internal temperature of 200 degrees

What does the model look like?


Cake Baking ExperimentSome Thoughts on the Cake Baking Experiments

• We don’t particularly care about the individual experimental runs

• We use the set of them to understand the effects of time and cook temperature on internal temperature

• Since the design is created so that we can independently quantify the effects on time and cooking temperature in the most efficient manner, we need all of the experimental runs to accomplish this

• We use the repeatability of replicate runs as a ruler to assess the significance of the effects of cooking time and temperature

• By randomizing the order of experimental runs we can make sure that the effects of any uncontrollable factors are not mixed up with the effects of cooking time and temperature.


Statistically Designed Experiments

• In a statistically designed experiment, factors (often ingredient levels and/or processing conditions) are systematically varied so that their effects can be quantified in an efficient way

• Statistical analysis of designed studies focuses on the development of models that relate how factors affect the responses

• These models can provide a wide variety of information, such as:

an ordering of important formulation/processing variables

an area of interest for further study

optimal product formulations (subject to cost and production constraints, if applicable)

specifications

How are they different?


Experimental Design Types

• Screening designs are used to: reducing the number of factors under investigation

identifying factor ranges further study

• Response Surface designs are used to: develop a model of the space defined by the factor ranges (optimization)

• Mixture designs have factor levels that add to a fixed total objective can be screening or optimization

• Robust (Taguchi) designs are used to: determine the factor settings that reduce variability due to factors that are costly,

difficult or impossible to control

There are several types of statistically designed experiments, each focusing on a different objective

Step 2 in my cake baking was a response surface design


Screening Design

• Designs are factorial combinations – the complete set or a fraction• Example: 8 factors are studied in a 16 run Fractional Factorial design

Most often have 2 levels for each factor

emc2 rdx frodex lacitol nfdm wpc lactose acidity-1 -1 -1 -1 -1 -1 -1 -1-1 -1 -1 1 1 1 1 -1-1 -1 1 -1 1 1 -1 1-1 -1 1 1 -1 -1 1 1-1 1 -1 -1 1 -1 1 1-1 1 -1 1 -1 1 -1 1-1 1 1 -1 -1 1 1 -1-1 1 1 1 1 -1 -1 -11 -1 -1 -1 -1 1 1 11 -1 -1 1 1 -1 -1 11 -1 1 -1 1 -1 1 -11 -1 1 1 -1 1 -1 -11 1 -1 -1 1 1 -1 -11 1 -1 1 -1 -1 1 -11 1 1 -1 -1 -1 -1 11 1 1 1 1 1 1 1


Screening Design

• Independently vary only 2 levels of each factor

Step 1 in my cake baking design was a screening design


Response Surface Designs

• Common design types have all continuous factors : Central Composite Box Behnken

• Custom designs can be created to include discontinuous factors D-optimal design

• Example: five factors are investigated in a 28 run Central Composite design

Most often have a small number of factors varied over 3-5 levels

lactose frodex salt alginate lactic acid2 0 0 0 0-1 -1 1 -1 -10 0 -2 0 00 0 0 0 -2-1 1 -1 1 10 0 0 0 0-1 1 -1 -1 -1-1 -1 1 1 11 1 -1 1 -10 0 0 0 21 1 -1 -1 10 0 0 0 01 -1 1 1 -11 -1 -1 -1 -11 -1 1 -1 10 0 0 0 00 0 2 0 00 2 0 0 00 0 0 2 0-2 0 0 0 0-1 1 1 -1 1-1 1 1 1 -10 0 0 0 01 1 1 1 10 0 0 -2 0-1 -1 -1 -1 11 1 1 -1 -11 -1 -1 1 1-1 -1 -1 1 -10 -2 0 0 0


Response Surface DesignsStep 2 in my cake baking was a response surface design

• There are 5 levels for each factor

• This is the minimum required to fit a response surface model• Covering the design space more densely is sometimes done


Mixture DesignsFactors sum to a fixed total

15

Tequila Triple Sec Lime Juice0.40 0.30 0.300.50 0.30 0.200.50 0.20 0.300.60 0.20 0.200.55 0.20 0.250.45 0.30 0.250.55 0.25 0.200.45 0.25 0.300.50 0.25 0.25

0.1

0.1

0.1

0.2

0.2

0.2

0.3

0.3

0.3

0.4

0.4

0.4

0.5

0.5 0.5

0.6

0.6

0.6

0.7

0.7

0.7

0.8

0.8

0.8

0.9

0.9

0.9

Triple Sec

Tequila

Lime Juice

• Common design types: Simplex Lattice

• Example: mixture design varying tequila, triple sec, and lime juice to determine the optimal Margherita


Robust Designs

• Noise is variation that is costly, difficult or impossible to control• Example: make a macaroni and cheese product robust to consumer

preparation variation– Prep 1 - overheating the sauce and under-draining the pasta (thin consistency)– Prep 2 - under-heating the sauce and over-draining the pasta (thick consistency)

• Both average and variation across the noise is analyzed• Screening designs are typically used

Objective is robustness to “noise”:

Prep 1 Prep 2 Mean Variability inemc2 rdx frodex lacitol nfdm wpc lactose Consistency Consistency Consistency Consistency

-1 -1 -1 -1 1 1 1-1 -1 1 1 -1 -1 1-1 1 -1 1 -1 1 -1-1 1 1 -1 1 -1 -11 -1 -1 1 1 -1 -11 -1 1 -1 -1 1 -11 1 -1 -1 -1 -1 11 1 1 1 1 1 1


Analysis of Experimental Designs

• Expressed in the form of an equation• Example – simple model for a 3 factor experiment:

• The X’s represent the factor settings, the betas are coefficients that are estimated based on the data from the experiment.

• A small beta for a specific factor indicates that the factor has a small effect on the response. A large beta for a specific factor indicates that the factor has a large effect on the response.

• The sign of the beta indicates the direction of the effect

Models relate design factors to responses

332211 XXXresponse


Experimental Design Models

• Simple model - main effects model:

• More complex model - main effects and pairwise interactions model:

• Even more complex model - full quadratic model:

Models can be very simple or much more complex

332211 XXXresponse

332211 XXXresponse

322331132112 XXXXXX 222333222111 XXX

332211 XXXresponse

322331132112 XXXXXX


Objectives, Designs and Models

• Screening designs are based on simpler models – main effect models or ones containing some or all possible factor interaction pairs

• Response surface designs are based on more complex models – full quadratic models or variants of these

• More data is needed to fit more complex models.

• The key to models that fit well and are easy to explain and understand is having a design that makes the factor main effects and interactions independent of each other.

Designs are linked to project objectives, models are linked to design types


Independence

• A poor design:

• We cannot separate the effect of factor 1 and factor 2 – we say that these factors are confounded

Varying factors independently means that we can separate their effects in the analysis model

Run Factor 1 Factor 2

1 50 .5

2 60 1

3 70 1.5


Independence

• A better design:

• We can now separate the effects of factor 1 and factor 2 – they are no longer confounded

What does this mean?

Run Factor 1 Factor 2

1 50 .5

2 50 1.5

3 70 .5

4 70 1.5


Independence

• A full factorial design (all possible factor level combinations) guarantees that we will be able to separate out the main effects and pairwise interactions of the factors

• In some instances a fraction of the set of all possible factor level combinations will allow separation of all main effects and pairwise interactions

• We can also use a fraction if fitting a simpler model is what is desired

• If even just one design run is missing the design may be compromised as the independence of the factors will likely be lost!


Design Resolution

• Resolution 3 designs – main effects are not confounded with other main effects but are confounded with pair-wise interactions

• Resolution 4 designs – main effects are not confounded with other main effects or pair-wise interactions; pair-wise interactions are confounded with each other

• Resolution 5 designs – there is no confounding between main effects, between pair-wise interactions, or between main effects and pair-wise interactions

A design is characterized its by resolution – how complicated a model can we fit to the response data?


Design Strategy

• Designs that provide the most amount of information require the most runs

• Designs with smaller numbers of runs may not provide as much information as we would like

• Example: to investigate 6 factors you could use…

an 8 run resolution 3 design.

a 16 run resolution 4 design.

a 32 run resolution 5 design.

Trade off between available resources and information


Replication

• Would you make the same conclusions about the experimental results for both of these studies?

• Replication variability is a ruler that we use to judge the significance of the size of the factor effects.

Replicating design points informs us about the reliability of the results

250

300

350

400

450

500

169

165

170

185

191189

25 30 35 40 45 50 55time

250

300

350

400

450

500

169

165

170

185

185205

25 30 35 40 45 50 55time

The factor effects are not larger than the variability in the repeated runs.

The factor effects are much larger than the variability in the repeated runs.


Randomization

• Factors external to the design may influence the results, for example: several raw material batches may be needed to complete the design runs

there may be a time sequence effect from the start to the end of the experiment

• Randomization helps keep the effects of any external factors separate from the factors in the experiment

The order in which we make the experimental runs can be important


Randomization

• 4 runs can be made from each batch of raw material.

• Randomized order: the effect of raw material batch is spread out across the factor main effects and interactions.

Example: 3 factor 8 run full factorial experiment

Run Batch Factor 1

Factor 2

Factor 3

1 1 1 1 -1

2 1 1 -1 -1

3 1 -1 1 1

4 1 1 1 1

5 2 -1 -1 1

6 2 1 -1 1

7 2 -1 -1 -1

8 2 -1 1 -1


Blocking

• Certain factors external to the design may have such strong effects that they need to be considered in the experimental design rather than dealt with through randomization

• Including these factors as blocks means confounding the block levels with higher order factor interactions

• More runs may be needed to accommodate a blocking factor

Sometimes randomization isn’t enough!


Blocking

• 4 runs can be made from each batch of raw material, which we know alone will have a large effect.

• Batch is confounded with the three way interaction of Factor 1, Factor 2 and Factor 3.

Example: 3 factor 8 run full factorial experiment

Run Batch Factor 1 Factor 2 Factor 3

1 1 -1 1 1

2 1 1 -1 1

3 1 -1 -1 -1

4 1 1 1 -1

5 2 -1 1 -1

6 2 -1 -1 1

7 2 1 1 1

8 2 1 -1 -1


Model Fitting

• Example – model fitting results from a 3 factor central composite design

• Factors: steam pressure, pump pressure and line speed

• Response: viscosity

• The model for this design is:

Regression analysis is used to fit the design model to the response data

332211 XXXresponse

322331132112 XXXXXX 222333222111 XXX


Model FittingRegression model results

Percentage of response variability explained by the model.

Total response variability. Response variability explained by the model.

Significance test for the model. P-values < .05 are often considered significant


Model FittingRegression model results Significance test for

P-values < .05 are often considered significant

1

Estimate of 1

Significance test for

P-values < .05 are often considered significant

Estimate of 12 12

Estimate of 11


Model FittingPairwise interactions represented in a matrix plot

Significant interaction between factor 1 and factor 2


Model Assessment

• Residuals appear to be normally distributed

• No patterns in residual plots:

Typical assessment tools used for other linear regression models apply here

2001000-100-200

99

95

90

80

70

60

5040

30

20

10

5

1

Residual

Perc

ent

Normal Probability Plot(response is Viscosity)

5000480046004400420040003800

150

100

50

0

-50

-100

Fitted Value

Res

idua

l

Versus Fits(response is Viscosity)

16012080400-40-80-120

4

3

2

1

0

Residual

Freq

uenc

y

Histogram(response is Viscosity)

16151413121110987654321

100

50

0

-50

-100

Observation Order

Res

idua

l

Versus Order(response is Viscosity)


Model Refinement

• Often regression models are refined to remove terms that are not statistically significant

• This is not as critical for designed experiments – why?– Designs are created so that the factors are independent of each

other– The conclusions of factor significance will not change very much – Model predictions will be similar for the full and refined models

• So is it necessary?– Maybe preferred– Sometimes software makes this difficult to work with multiple

responses as they may have different significant effects

Is this a critical next step?


Model RefinementAnalysis results for full and refined models

Full Model

Refined Model

Estimates are the same or very nearly so

P-values are directionally similar


Model Verification

• This is a critical step!• Example: I want to use my viscosity model to identify factor level

combinations that achieve a target of 4300• There are many possibilities!

How do we know we can depend on a model’s predictions?

Line speed = 31Line speed = 22 Line speed = 40


Model Verification

• Factor level combinations should be run, responses measured and then compared to the model predictions to verify the model’s accuracy

• Model verification is helped when some of the original runs from the experiment are repeated– This verifies that the new runs are matching the original– Maybe differences are not the model but the second execution!

It is critical that factor level combinations of interest be verified!

steam pressure

pump pressure line speed

Original Viscosity

Pred Formula Viscosity

Verification Viscosity

200 370 22 4858 4725200 420 22 4826 4853200 420 40 4684 4664280 420 40 4254 4340173 395 31 4909 4953307 395 31 4031 4061250 405 31 4297228 380 31 4302260 400 22 4299240 385 22 4309220 385 40 4294240 410 40 4302


How to Judge a Paper With a DOE

• Does the design match the objective (or stated conclusions)? You cannot optimize with a screening design Need at least three levels for each continuous factor for

optimization Need to be able to fit at minimum a full quadratic model for

optimization• How was the design executed?

Randomization or a restricted randomization would be best If not is there a compelling reason? This should be mentioned or discussed in the paper’s text

• Is there replication? This helps demonstrate the validity of the results

Things to consider about the design


How to Judge a Paper With a DOE

• How much of the analysis is shared in the paper? Response data ANOVA or regression output (p-values) Visualization of the factor effects

• Is it clear that the authors understand the output in the analysis? Is there a focus on just one or two things in the output (like R2)

• Is there any assessment of the models and how well they fit the data? Residual analyses Plot of observed by predicted

• Have the models been refined (insignificant terms removed from the model)

• Is there any verification of the models’ ability to predict? Are predictions made within the design space?

Things to consider about the analysis


Assessments of Individual Papers

Criteria S. DEMIRKOL ET AL. (2006)

S.E. LUMOR AND C.C. AKOH (2005)

C.F. TORRES ET AL. (2002)

J.R. ORIVES ET AL. (2014)

JAOCS-14-0174

JAOCS-15-0155

N. ANARJAN ET AL. (2014)

Appropriate DesignReplication

Randomization

Output PresentationOutput DiscussionModel AssessmentModel Validation

Scored as: Good (G), Fair (F), Poor (P), None (N)


Example 1

• Design – faced centered cube (central composite design) with 3 centerpoint replicates is appropriate for optimization

• Observed response data, predictions and residuals are included in the paper

• No discussion of design execution (randomization)

(2006) OPTIMIZATION OF ENZYMATIC ALCOHOLYSIS OF SOYBEAN OIL


Example 1

• Statistical analysis results include model coefficients and p-values

• The authors have refined the model by removing statistically insignificant terms

• The model results are visualized in contour plots



Example 1

• Only the barest assessment of the fit of the model: plot of observed by predicted (the residuals are differences between the points are the line)

• No validation of the predicted optimum and it is out of the design space Optimum level for enzyme/oil weight ratio is .09, the range of this

factor in the design is .10 to .20




Criteria Example 1 Example 2 Example 3 Example 4 Example 5 Example 6 Example 7

Appropriate Design GReplication GRandomization NOutput Presentation GOutput Discussion GModel Assessment PModel Validation N


45


Example 2

• Design – three factor central composite design (in a sphere) with 9 (OMG!) centerpoint replicates is appropriate for optimization

• Two design runs were “excluded from model fitting” – explained as outliers. These are critical for fitting the model!

• No mention of randomization

(2005) INCORPORATION OF STEARIC ACID INTO PO:PKO


Example 2


• The authors have refined the model by removing statistically insignificant terms

• The model results are visualized in contour plots



Example 2

• Model assessment: normal probability plot of residuals and observed by predicted plot



Example 2

• Model validation with 5 additional runs





Appropriate Design G PReplication G GRandomization N NOutput Presentation G GOutput Discussion G GModel Assessment P FModel Validation N G


50


Example 3

• Design: 22 full factorial designs with center-points at each of 3 levels of a third factor – this is not appropriate for a true optimization since curvature cannot be fit for two of the factors

• Replication and randomization included in the design

(2002) OPTIMIZATION OF THE ACIDOLYSIS OF FISH OIL WITH CLA


Example 3


• There is no evidence that any model assessment was performed



Example 3

• There is a detailed discussion of the modeling results and they are visualized in a number of informative plots. The authors use and description of desirability functions to assess multiple responses demonstrates understanding of their modeling

• There was no validation of the optimized conditions





Appropriate Design G G/P PReplication G G GRandomization N N GOutput Presentation G G GOutput Discussion G G GModel Assessment P F NModel Validation N G N


54


Example 4

• Design - Simplex centroid design is appropriate for mixture optimization

• Replication of center-point is included in the design• No mention of randomization

(2014) EXPERIMENTAL DESIGN APPLIED FOR COST AND EFFICIENCY OF ANTIOXIDANTS IN BIODIESEL


Example 4

• Only very basic analysis results are presented in the paper

• No mention of either model assessment or refinement



Example 4

• Visualization of the response surface in a three dimensional plot is more sexy than informative (a contour plot will be more useful)

• The authors do a nice job of describing in the text the factor effects and their implications on both the response of most interest and cost



• The authors use multi-response optimizer functionality in their software to determine a cost effective blend

• There is no validation of the blend though there is a reference that the prediction is close to the observed response in a design point nearby

Example 4(2014) EXPERIMENTAL DESIGN APPLIED FOR COST AND EFFICIENCY OF ANTIOXIDANTS IN BIODIESEL




Appropriate Design G G/P P GReplication G G+ G FRandomization N N G NOutput Presentation G G G FOutput Discussion G G G GModel Assessment P F N NModel Validation N G N P


59


Example 5

• Design - Box-Behnken design limits the area where the model can be used to predict (no corner points in the 4-dimensional hypercube)

• There is no mention of randomization

JAOCS-14-0174 - METHANOLYSIS OF HEVEA BRASILIENSIS OIL BY SO3H-MCM-41 CATALYST: BOX-BEHNKEN EXPERIMENTAL DESIGN


Example 5

• There are no statistical analysis results presented in the paper, only the regression equation

• The equation looks to be in coded units for the factors (the coefficients are similar in scale though the factors themselves are not)

• There is no evidence of a statistical assessment of the significance of the factor effects



Example 5

• Only the barest assessment of the fit of the model: plot of observed by predicted (the residuals are differences between the points are the line)



Example 5

• The discussion of the factor effects is based only on a visual evaluation of the factor effects from viewing 3-dimensional plots of the response surface

• Main effect and interaction plots would make it easier for readers to understand the nature of the factors and their interactions

• The optimum is validated with three replicated runs





Appropriate Design G G/P P G FReplication G G+ G F GRandomization N N G N NOutput Presentation G G G F POutput Discussion G G G G PModel Assessment P F N N PModel Validation N G N P G


64


Example 6

• The 2 factor Central Composite Design (the authors call it a 3x3 factorial which is also correct) is appropriate for optimization

• There is no mention of randomization

JAOCS-15-0155 - TRANSESTERIFICATION OF SANITATION WASTE FOR BIODIESEL PRODUCTION USING RESPONSE SURFACE METHODOLOGY


Example 6

• ANOVA results for all responses are provided in the paper. It is evident that the authors have refined the models be removing non-significant terms and they provide the final prediction equations

• There is no assessment of the fit of the models (not even R2!)so we do not know how accurately the equations will predict



Example 6

• The identified optimum values are all design points so it appears that the developed models were not used to determine these

• There appears to be an error in this one since 40 is outside the design space but there is a data point at Temp = 60 that corresponds to this value (and neglects to mention that responses at the replicate points are 91.9 and 92.2)



Example 6

• Visualization of the response surface in a three dimensional plot is more sexy than informative (a contour plot will be more useful)

• The discussion of the factor effects is based only on a visual evaluation of the factor effects from viewing 3-dimensional plots of the response surface

• Main effect and interaction plots would make it easier for readers to understand the nature of the factors and their interactions





Appropriate Design G G/P P G F GReplication G G+ G F G GRandomization N N G N N NOutput Presentation G G G F P FOutput Discussion G G G G P FModel Assessment P F N N P NModel Validation N G N P G P


69


Example 7

• 3 Factor Central Composite Design in a sphere is appropriate for optimization

• Though not explicitly mentioned in the paper the design execution appears to be randomized

• Observed and predicted response values are included in the table

(2014) OPTIMIZATION OF MIXING PARAMETERS FOR A-TOCOPHEROL NANODISPERSIONS PREPARED USING SOLVENT DISPLACEMENT METHOD


Example 7

• ANOVA results for all responses are provided in the paper. It is evident that the authors have refined the models be removing non-significant terms and they provide the final prediction equations



Example 7

• There is discussion of model assessment though R2, a lack of fit test and observed versus predicted plots



Example 7

• Contour plots and a multi-response optimization plot help the readers understand how the optimum factor levels were determined

• The predicted optimum was validated with an additional experimental run





Appropriate Design G G/P P G F G GReplication G G+ G F G G GRandomization N N G N N N GOutput Presentation G G G F P F GOutput Discussion G G G G P F GModel Assessment P F N N P N FModel Validation N G N P G P F


74

Statistical Design of Experiments Training for AOCS Journal Editors Frank Rossi, Associate Director Statistics, Kraft Foods May 3, 2015.

Documents

cooking temperature

kraft foods group

best cake temperature

temperature levels

temperature combinations

effects of time

temperature space

cake baking experiment