IMPRINT Developer’s Workshop December 6-7, 2005 Meta-analytic Reviews of the Effects of Temperature and Vibration on Performance J.L. Szalma & G. Conway.

IMPRINT Developer’s Workshop December 6-7, 2005

Meta-analytic Reviews of the Effects of Temperature and Vibration on Performance

J.L. Szalma & G. Conway

University of Central Florida


Performance under Stress

• Stress Degrades Performance

• Relation of Stress to IMPRINT: predict the degree of performance degradation under specific forms of stress

• Currently estimates may not be representative of the entire literature

• Estimates may not reflect interactions with other variables


Performance under Stress

• Goal for Current work: Based on a quantitative review of the literature, derive estimations of the effect of specific stressors on human performance

• Outcome: Improve IMPRINT as a performance prediction tool by increasing the accuracy of the stress algorithms


Why Use Meta-Analysis?

• Advantages

• Quantitative synthesis

• Controls for sampling error and low power

• Permits examination potential moderator variables

• Disadvantages

• Generally have to discard many published studies because they do not meet inclusion criteria

• Analysis is only as good as the data that go into it


Significance Tests• Significance tests are useful, but finding a statistically

significant difference depends on the power of the analysis

• Many publications in behavioral science include statistical analyses of relatively low power

• It would be helpful to have a standardized measure of mean differences that indicates the strength of the effect independent of the results of a significance test and that can be compared across different measures


The Effect Size

Cohen’s d (Hedge’s g): Standardized Mean difference between two conditions

Range: -∞ --------------- 0 --------------- +∞

s

XXd CE


Magnitude of Effect

Magnitude of Effect (Cohen, 1988)

Large: .80

Medium: .50

Small: .20

These are only meant to be guidelines

The magnitude of ‘large’ and ‘small’ effects can vary by domain


General Procedures in Meta Analysis

• Literature Search

• Weighted Average Effect Size

• Variance Partitioning

• Moderator Analysis

• Hierarchical Moderator Analysis

• Criteria for inclusion


Weighted Effect Sizes

Not all effect sizes are equally accurate

Analogy: Accuracy of a sample mean

The sampling error associated with a given effect size can be estimated

To obtain a more precise estimate of the average effect size, each d is weighted by the reciprocal of its error variance (Hedges & Olkin, 1985).


Weighted Effect SizesThus,

iiw dwd *

2

1

iiw where


Error varianceVariance of the d’s

Estimate of the ‘true’ variance in the population of effect sizes (random vs. fixed effects)

Variance Partitioning

σ2d = σ2

δ + σ2e


Heterogeneous Effect Sizes and Moderator Variables

If the true weighted average effect size were constant across environments, then

02

22ed

and


If

02 Then other variables are likely to moderate the magnitude and/or direction of the effect size associated with the variable of interest (temperature/vibration)

Analogous to interaction effects in ANOVA

Moderator Analyses


Moderator Variable: Taxon

• Task Category (IMPRINT Taxon)

•Perception/discrimination

• Information Processing/Problem Solving

•Numerical

•Fine Motor Continuous

•Fine Motor Discrete


Moderator Variable: Dependent Measure

• Speed

• Accuracy


Moderator Variable: Intensity

• Range of Temperature, Heat (Intensity)

• Less than 85°F ET

• Greater than 85°F ET

• Range of Temperature, Cold (Intensity)

• Less than 49°F ET

• Greater than 49°F ET


Rise in Core Body Temperature as a Function of Effective Temperature


Moderator Variable: Duration

• Less than 1 hour exposure

• Between 1 and 2 hours exposure

• Between 2 and 3 hours of exposure

• Greater than 3 hours of exposure


Hierarchical Moderator Analysis

• Nesting moderator variables within levels of other moderating variables

• Analogous to multiple interactions in ANOVA

• Precision of effect size estimates decreases as one moves down the hierarchy


Example: Thermal Stress

Global Effect size

HeatCold

Perception Information Processing

Perception Information Processing

The number of available studies decreases as one moves down the hierarchy


Study Criteria

• Empirical study involve the application of a heat or cold stress.

• A control group at room temperature

• Enough data to convert temperature into WBGT

• WBGT = 0.567(DB) + 0.393(RH) +3.94• WBGT = (ET-13.1)/0.823• WBGT = 0.7(WB) + 0.3(DB)


Study Criteria

• Whole body exposed to ambient air heat/cold stress

• The study must involve human performance measures

• Enough information obtain effect size statistic


Meta Analytic Results

• Literature search resulted in 291 papers published between 1925 and 2003.

• Criteria for Inclusion rules resulted 49 papers

• 57 useable primary studies

• 567 effect sizes

• Independence Problem (analogous to ANOVA)


The Effect of Thermal Stress on Performance

-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

Thermal Stress

Eff

ect

siz

e (d

)

Global Heat Cold


Thermal Effects as a Function of Task Type

-4.5

-3.5

-2.5

-1.5

-0.5

0.5

1.5

2.5

3.5

4.5

1

Tasks

Eff

ect

Siz

e (d

)

Heat Cold

PIP/PS FMC

P

IP/PS

FMDFMC

NN

FMD


Thermal Effects as a Function of Speed and Accuracy

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

Dependent Variable

Eff

ect

size

(d

)

Global Accuracy RT


Effect Size as a Function of Temperature Range: Heat

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Temperature Range

Eff

ect

size

(d

)

Heat (overall)

ET<=85 F

ET>85 F


Effect Size as a Function of Temperature Range: Cold

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Temperature Range

Eff

ect

size

(d

)

Cold (overall)

ET<=49F

ET>49F


Effect Size as a Function of Task and Exposure Duration

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1

Duration on Task

Eff

ect

Siz

e (d

)

<1

1<=t<2

2<=t<3

3<=t


Duration by Intensity Interaction: Heat

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

1

Duration

Eff

ect

Siz

e (d

)

Heat ET<85F Heat ET>85F

time<2hr

time>2 hr

time<2hr

time>2 hr


Duration by Intensity Interaction: Cold

-6.5

-5.5

-4.5

-3.5

-2.5

-1.5

-0.5

0.5

1.5

2.5

3.5

4.5

5.5

6.5

Duration

Eff

ect

Siz

e (d

)

Cold ET<49F Cold ET>49F

time<2hr

time>2 hr

time<2hrtime>2 hr


Conclusions and Implications

• Temperature clearly impacts performance

• Effects are different across Taxons

• Task type interacts with other moderating variables

• Implication: Need consider moderating variables (and interactions) when estimating effect of temperature on performance.

IMPRINT Developer’s Workshop December 6-7, 2005 Meta-analytic Reviews of the Effects of Temperature and Vibration on Performance J.L. Szalma & G. Conway.

Documents

effect cohen

effect independent

given effect size

weighted effect sizesthus

eheterogeneous effect

effect of specific stressors

effects of temperature

moderator variablesandifthen