Exploring Best Practises in Design of Experiments: A Data Driven Approach to DOE Increasing Robustness, Efficiency and Effectiveness

Copyright © 2014, SAS Institute Inc. All rights reserved.

Exploring best practises in Design of Experiments A Data Driven Approach to DOE Increasing Robustness, Efficiency and Effectiveness



Julie

Who’s here from jmp

Bernard Luke

Malcolm Phil


jmp helps you make better decisions, faster


We will show you how you can

§  Simplify and make DoE work for more people in more situations

§  Make use of existing data to have better informed experiments

§  Make better decisions in less time


What we will cover today Time Topic Speaker 0940 Introduction to Design of Experiments (DoE) Malcolm Moore

1025 Identifying key factors and optimising processes using the key factors Phil Kay

1100 Break 1130 Example of DOE in Service Industries Malcolm Moore

1155 Effective experimentation when we have constraints on the factor combinations Phil Kay

1220 Data Driven DoE and Choice Experiments Malcolm Moore 1250 Summary and close Malcolm Moore 1300 Adjourn for lunch


Help us to help you . . .


How often is DoE used in your organisation?

(Select one)

1.  Never

2.  Rarely

3.  Often

4.  The default approach for experimentation


What is you organisation’s general view of DoE (not your view which can be different)?

(Select one)

1.  Committed to it

2.  Unsure what it is

3.  Not really bothered

4.  Tried it but it didn’t work

5.  Against it


Are your experimental problems ever complex (factor constraints, disallowed combinations)?

(Select one)

1.  Never

2.  Rarely

3.  Often

4.  Always

5.  Don’t know


Do you have existing data that you would like to use to inform future experiments?

(Select one)

1.  Never

2.  Rarely

3.  Often

4.  Always


Contents

§  Background to DOE

§  Why Use DOE?

§  Tips for Effective DOE with Classical Designs

§  Definitive Screening

§  Case Studies 1-3

§  Role of Statistical Modelling and DOE in Learning

§  Data Driven DOE

§  Case Study 4


BACKGROUND TO DESIGN OF EXPERIMENTS (DOE)


FATHER OF DOE RONALD A. FISHER

Rothamstead Experimental Station, England – Early 1920’s


FISHER’S FOUR DESIGN PRINCIPLES

1.  Factorial Concept - rather than one-factor-at-a-time 2.  Randomization - to avoid bias from lurking variables 3.  Blocking - to reduce noise from nuisance variables 4.  Replication - to quantify noise within an experiment


AGRICULTURAL IMPACT

US corn yields

Cornell University, http://usda.mannlib.cornell.edu/MannUsda


WHY USE DOE?


Typical Process The properties of products and processes are often affected by many factors:

In order to build new or improve products and processes, we must understand the relationship between the factors (inputs) and the responses (outputs).

Typical Process

Machine

Operator

Temperature

Pressure

Humidity

Yield

Cost

…

Inputs Factors

Outputs Responses


Traditional One-Factor-at-a-Time §  A common approach is one-factor-at-a-time experimentation.

§  Consider experimenting one-factor-at-a-time to determine the values of temperature and time that optimise yield.


Traditional One-Factor-at-a-Time








Traditional One-Factor-at-a-Time §  One-factor-at-a-time

experimentation frequently leads to sub-optimal solutions.

§  Assumes the effect of one factor is the same at each level of the other factors, i.e. factors do not interact.

§  In practice, factors frequently interact.


Interaction between factors


Experimental Design §  Most efficient way of investigating relationships. §  Runs (factor combinations) chosen to maximize the information §  Ideally balanced for ease of analysis and interpretation


ITERATIVE AND SEQUENTIAL NATURE OF CLASSICAL DOE


TIPS FOR EFFECTIVE DOE WITH CLASSICAL DESIGNS


Stages of Experimental Design

§  Designing an experiment involves much more than just selecting the sequence of experimental runs:

§  Historically, improper planning is the most common cause of failed experiments.

Plan Design Conduct Analyse Confirm


Some Planning Steps

§  Review what we know •  Have peer discussions

§  Determine new questions to answer

§  Identify factors and ranges to investigate

§  Define responses •  Easy and precise to measure


Common Experimental Objectives

Identify Important Factors

Screening Design

Classical Fractional Factorial

Optimise Process RSM Design

Classical Central

Composite

Optimise Ingredients Mixtures

Classical Simplex & Extreme Vertices


Sequential Experimentation Reduces Total Cost



Screening Design



Classical Central

Composite




Sequential Experimentation



Screening Design



Classical Central

Composite



Definitive Screening Design Simplifies Experimental Workflow


Sequential Experimentation



Screening Design



Classical Central

Composite



Definitive Screening Design

Optimal Design Manages Experimental Constraints


Determining the Appropriate Factors §  Determining the factors to be included in your experiment is a

critical part of planning. •  Exploring too many factors may be costly and time

consuming. •  Exploring too few may limit the success of your experiment.

§  Prior knowledge and analysis of existing data are useful aids to identifying and prioritising factors for study. Other methods may include: •  Brainstorming •  Ishikawa


Selection of Factor Range is Critical With Two Level Designs …



By experimenting at the two settings in yellow, X would be declared unimportant



By using half and often times much less than than half the factor range X is declared important



By using half and often times much less than than half the factor range X is declared important

Often leads to narrow factor ranges to force linear relationships but

consequence is high risk of determining sub-optimal solution


Determining the Appropriate Responses

§  Selection of your responses will also be critical to the success of your experiment. Whenever possible: •  Choose variables that correlate to internal or external

customer requirements •  Find responses that are easy to measure •  Make sure your measurement systems are precise, accurate,

and stable

§  Analysis of current data, prior knowledge, measurement systems analysis are useful aids.


DEFINITIVE SCREENING


Fractional Factorials: Complex workflow from many factors to optimum settings

Tempting to miss out middle step which can result in selection of

wrong factors and decisions


Definitive Screening Design

§  Identifies active main effects, uncorrelated with other effects.

§  May identify significant quadratic effects, uncorrelated with main effects and at worst weakly correlated with other quadratic effects.

§  If few factors turn out to be important, can identify significant two-way interactions uncorrelated with main effects and weakly correlated with other higher order effects.

§  One stage experiment if three or fewer factors important: •  progress straight to full quadratic model •  optimise process with no further experimentation •  otherwise augment DSD for optimization goals


New Class of Screening Design

§  Three-level screening design •  2m + 1 runs when m is even •  2m + 3 runs when m is odd •  1 additional run for

categorical factors •  based on m fold-over pairs

and an overall center point, where m is number of factors

•  the values of the ±1 entries in the odd-numbered runs are determined using optimal design.

2 BRADLEY JONES AND CHRISTOPHER J. NACHTSHEIM

IV fractional factorial design is a standard alterna-tive. However, these designs require twice as manyruns as the resolution III design and again lead, fre-quently, to substantial ambiguity. If an interactioncontrast is identified as active, the experimenter hasno way to definitively identify which of the interac-tions in the set of confounded two-way interactionsare active. Again, follow-up work is required to iden-tify the active e!ects.

Another limitation of resolution III and IV de-signs is that they have no capability for capturingcurvature due to pure-quadratic e!ects. Of course, itis traditional to add center runs to two-level screen-ing designs to get a global assessment of curvature.Still, these runs do not allow for separate estimationof the quadratic e!ects of each factor. So an indica-tion of curvature in the analysis leads to still moreambiguity that can only be resolved with additionalruns.

Our new class of three-level screening designs hasthe structure illustrated in Table 1. We use xi,j todenote the setting of the jth factor for the ith run.For m factors, there are 2m + 1 runs based on mfold-over pairs and an overall center point. Each run(excluding the centerpoint) has exactly one factorlevel at its center point and all others at the ex-tremes. As described in the next section, the val-ues of the ±1 entries in the odd-numbered runs of

TABLE 1. General Design Structure for m Factors

Factor levelsFoldover Run

pair (i) xi,1 xi,2 xi,3 · · · xi,m

1 1 0 ±1 ±1 · · · ±12 0 !1 !1 · · · !1

2 3 ±1 0 ±1 · · · ±14 !1 0 !1 · · · !1

3 5 ±1 ±1 0 · · · ±16 !1 !1 0 · · · !1

......

......

.... . .

...

m 2m " 1 ±1 ±1 ±1 · · · 02m !1 !1 !1 · · · 0

Centerpoint 2m + 1 0 0 0 · · · 0

Table 1 are determined using optimal design; theeven-numbered values (!1) result from the fold-overoperation. These designs have the following desirableproperties:

1. The number of required runs is only one morethan twice the number of factors.

2. Unlike resolution III designs, main e!ects arecompletely independent of two-factor interac-tions. As a result, estimates of main e!ects arenot biased by the presence of active two-factorinteractions, regardless of whether the interac-tions are included in the model.

3. Unlike resolution IV designs, two-factor inter-actions are not completely confounded withother two-factor interactions, although theymay be correlated.

4. Unlike resolution III, IV, and V designs withadded center points, all quadratic e!ects areestimable in models comprised of any numberof linear and quadratic main-e!ects terms.

5. Quadratic e!ects are orthogonal to main e!ectsand not completely confounded (though corre-lated) with interaction e!ects.

6. With 6 through (at least) 12 factors, the de-signs are capable of estimating all possible fullquadratic models involving three or fewer fac-tors with very high levels of statistical e"-ciency.

We use the term “definitive screening” because ofpoints one through five above. These are small de-signs that, unlike resolution III and IV factorial de-signs, permit the unambiguous identification of ac-tive main e!ects, active quadratic e!ects, and, in thepresence of a moderate level of e!ect sparsity, activetwo-way interactions.

In our view, another practical advantage of thedesigns we propose is the explicit use of three levels.It has been our experience that engineers and scien-tists often feel some discomfort using two-level de-signs for two reasons. First, statisticians advise themto experiment boldly by choosing a substantial inter-val between low and high values of each factor. Buttheir scientific training inculcates the notion that thefunctional relationship between independent and de-pendent variables is usually nonlinear, particularlyover a wide range. This leads to some cognitive dis-sonance in considering the use of two-level designs.Second, even in the early stages of a study, investiga-tors frequently have an opinion regarding the “best”

Journal of Quality Technology Vol. 43, No. 1, January 2011


Use of Three Level Designs Advantageous §  Scientists and engineers are uncomfortable using two-level designs

•  Restricting factor ranges may result in sub-optimal solutions •  Scientific/engineering judgment suggests relationships nonlinear over

wide ranges

§  Investigators frequently have an opinion regarding the “best” levels of each factor for optimizing a response •  Experimental region centered at these levels. •  Two-level design might screen out an important factor when

experimental region centred at “best” •  Adding centre points allows test for curvature •  However ambiguity over factors causing curvature •  DSD avoids ambiguity by making it possible to uniquely identify the

source(s) of curvature.


CASE STUDIES


Case Study 1: Optimising a Chemical Process

Why Consider Definitive Screening Designs?


Background

§  Five factors

§  One response yield

§  Goal optimise yield

§  Keep total cost of experimentation to minimum

§  Contrast traditional approach of main effect screening design plus augmentation to RSM with DSD


§  Traditional screening approach correlates main effects with two factor interaction effects

§  Cost constraint and inexperience with such designs can lead to missed DOE steps

§  Investigator missed step of augmenting main effect design to separate correlated interaction effects from assumed important main effects

§  Resulted in wrong set of factors selected for RSM design which results in wrong solution

Background


Traditional Approach with Missed Step


Resolution III Design Perfectly Correlates Main Effects With Interaction Effects


Model Interpretation

§  Fitted Model

Y = b0 + b1*X1 + b2*X2 + b3*X3 + Error

§  Correct Interpretation of Fitted Model

Y = b0 + b1*(X1+X2X3) + b2*(X2+X1X3) + b3*(X3+X1X2) + Error


Missed Step Augments Initial Design to Separate Main Effects From Interactions


Model Interpretation of Augmented Design §  Correct Interpretation of Model Fitted to Augmented design

Y = b0 + b1*X1 + b2*X2 + b3*X3 + b12*X1X2 + b13*X1X3 + b23*X2X3 + Error

§  Allows clear separation of main and interaction effects

§  This step was missed in case study prior to modelling curvature


§  DSD results in correct identification of important factors due to non correlated main and two factor interaction effects

§  Because just three factors are important DSD results in one step design: •  In addition to correctly identifying correct factors •  DSD requires no augmentation to identify optimal

settings of important factors

Background


CASE STUDY 1


Conclusions

§  Fractional factorial designs can lead to selection of wrong factor set

§  Complex workflow for avoiding this risk which may be misunderstood or not applied by users new to DOE

§  May lead to conclusion that DOE does not work for us!

§  DSD simplifies DOE process and removes risk of selecting wrong factor set

§  Provides one step DOE when three or fewer important factors •  Sufficient to identify correct factor set and determine best

settings of selected factors


Case Study 2: Optimising Marketing Response Rate and Profitability

Definitive Screening Design for Efficiency


Background §  Goal is to maximise return from credit card

marketing campaigns. Two outputs: •  Response rate - percentage mailed a credit card offer

who accept the offer; •  Indexed usage – average profit per individual over a

twelve month period.

§  Factors are balance transfer period, interest free period for new purchases and %APR at end of any introductory offers.

§  Goal: determine characteristics of credit card offer that maximises response rate and profitability.


CASE STUDY 2


Conclusions

§  DSD can be cost effective with few factors when cost of experimental run is high

§  Tradeoff is greater uncertainty (reduced power) in decisions


CASE STUDY 3


Case Study 3: Optimising Yield

What About Constrained Factor Spaces?


Background §  From chapter 5 of Goos &

Jones

§  Chemical reaction

§  Goal: maximise yield

§  2 factors: Temperature and Time


Background §  Expert knowledge tells us

•  Certain conditions will give poor results (hence, constraints)

•  Behaviour very non-linear

§  We will show •  Design where prior

knowledge is ignored. •  Fitting the design to the

problem


Example of Process Constraint


Shrink Experimental Range to Factorial






Optimal Design: Use Actual Factor Range


… optimal designs allow investigation of complete factor space properly adjusted for constraints

Typical Process

Machine

Operator

Temperature

Pressure

Humidity

Yield

Cost

…

Inputs Factors

Outputs Responses

Optimal Design: Fit to Model

Model

Y = f(X)

The process is not seen as a black box anymore…


CASE STUDY 3


Conclusions

§  Custom Design permits studying any: •  combination of factors with or without constraints, •  number of factor levels, •  blocking structure.

§  Build your design to suit the problem instead of fitting the problem into a design


Case Study 4: Designing Products People Want to Buy

Data Driven DOE


ROLE OF STATISTICAL MODELLING AND DOE IN LEARNING


LEARNING IN THE FACE OF UNCERTAINTY

76 Adapted from Box, Hunter and Hunter

What is really happening

What we think is happening

Measurement and Data Collection

Analysis

Situation Appraisal


Design

Able to Consistently Meet Customer Requirements

Model Real World

Unable to Consistently Meet Customer Requirements

Data Driven DOE Integrates Incremental Learning Across DOE and Observational Sources of Data

Y = F(X) + Error


Situation Appraisal

Situation Appraisal


Simple Process of Statistical Learning

DOE Data ….…. Observational Data


Data Sources §  DOE and/or observational (historical)

§  Potential problems with observational data: •  X’s are correlated – identification of “best” model

difficult •  Outliers (potential or real) - bias model estimation •  Missing data cells – result in loss of whole data rows

with traditional least squares based analysis •  Range over which X’s varied may be limited –

restricting model usefulness •  May not have measured all relevant X’s

§  In some situations these can also be issues with DOE datasets


WHAT IS DATA DRIVEN DOE?


Data Driven DOE: Integrating Statistical Modelling and DOE

§  Learning is incremental and effective statistical modelling of observational data aids design of next experiment.

§  Analysis approach needs to manage real (messy) data simply •  Correlated X’s, outliers, missing cells •  Quickly deliver “best” current model to revise with new DOE data •  Aid better analysis of new experimental data when unexpected

occurs •  Build models based on individual datasets and aggregated data

§  Good statistical modelling integrated with DOE helps reduce total learning time, effort and cost

§  It would be a shame to not use pre-existing data that comes for free


JMP Statistical Discovery: Integrating Statistical Modelling with DOE

Effectiveness Of Learning

Speed of Learning

Statistical Discovery

Traditional Approaches

§  Integrated methods §  Ease of use §  Manage messy data §  Wide array of DOE

approaches

§  Satisfy (customer) needs

§  Reduce learning time §  Save effort and cost


DATA DRIVEN DOE EXAMPLE


Background

§  PC retailer is observing appreciable retail price variation in its laptop computer line.

§  Goals: •  Investigate factors associated with retail price variation. •  Perform further experimentation in key factors to

optimise and standardise pricing across stores.


CASE STUDY 4


Conclusions

§  Analysis of prior data helps identify factors and ranges to use in next DOE.

§  Analysis of prior data helps reduce risk and increase efficiency and effectiveness of future experiments.

§  Exploit prior data that comes for free to inform next experiment.


Data Driven DOE: Integrated Statistical Modelling and DOE

§  Supports wide range of user skills

§  Exploratory analysis and statistical modelling of historical messy data simplifies and shortens whole DOE process.

§  Next generation DOE enables more staff to apply DOE with reduced learning and implementation effort

§  Interact with model predictions to build consensus

§  Integrated simulation capabilities enables rapid progression from models to decisions

§  Manage risk better by correctly identifying signal from noise


QUESTIONS?


We have shown you how you can

§  Reduce the risk of wrong decisions •  Make DoE work for more people in more situations

§  Fit the best design to your problem •  Find the best solution while managing system

constraints

§  Mine your “messy” data to inform future experiments •  Make better decisions in less total time using Data

Driven DOE


Make better decisions, faster with jmp


Supplier of Digital Printing Materials §  Needed to double capacity of a product line to meet

growing demand.

§  Poor understanding of key process step responsible for increasing capacity.

§  Large number of potentially important variables and limited budget for experimentation.

§  Definitive Screening Design enabled screening of all factors and process optimisation in a small number of runs to achieve doubling of production rate without additional capital investment.

§  Saved £100,000s off development budget and enhanced the credibility of the site as a location for cost-effective high-value manufacturing within a multi-national organisation.


Large Multi-National Chemical Company

§  Losing market share to start-ups who were faster at introducing new products and more agile at adapting to changing customer requirements.

§  Needed to get more products to market faster.

§  Instituted a culture of experimentation with JMP Pro for variable selection and DOE to accelerate cycles of learning, enabling more new products to be introduced faster.

§  Helped retain and grow market share, facilitating increased dividend growth to shareholders and increased staff retention and satisfaction.


What are you going to do next?

Ask us to help you Download a trial of JMP §  Visit our website: www.jmp.com

Join our Design of Experiments Webcasts: §  Exploring Best Practise in DoE: 14:00 on 20

November

§  Invite your colleagues

§  Mastering JMP on DoE: 1400 on 14:00 on 14 November

Register on our website

Exploring Best Practises in Design of Experiments: A Data Driven Approach to DOE Increasing Robustness, Efficiency and Effectiveness

Data & Analytics

sas institute

use doe

doe increasingrobustness

doe work

father of doe ronald

time experimentation

use of existing data

stages of experimental