Web vs. Lab John H. Krantz Hanover College. Outline To web or not to web? Deciding to use the web What has been done and how General findings.

Web vs. Lab

John H. Krantz

Hanover College

Approach

Not focus on technology Focus on the research question What is best way to answer question

Approach

Web is a tool to use in determining best way to ask question

Much as field experiments have become a tool: e.g. Milgram, S. , Bickman, L., & Berkowitz, L.

(1969).  Note on the drawing power of crowds of different sizes.  Journal of Personality and Social Psychology, 13(2), 79-82.

While this example is trivia it shows an example of a study not possible in the laboratory

Approach

What this implies: For some questions web is a superior method to

lab May depend on your situation as we all use what

we have available

Characteristics of Web vs. Lab Web

Distributed User or remotely

controlled environment and location and some factors are unknowable

Accidental sampling of large population

Potentially anonymous Requires computer

Lab Confined Experimenter controlled

environment or at least determinable

Accidental sampling generally but random sampling of restricted population is possible

Experimenter is known to be present or near

May not have computer

Characteristics of Web vs. Lab Similarities

A certain act of belief is needed to accept self-report responses

Cannot control state and previous state or events of participant This is an issue in control

Random assignment is possible

Reasons for Use – Musch & Reips (2000)

In descending order (7 very important – 1 not important)

Large N 5.5 High statistical power 4.5 Speed data collection/participants

from other countries 3.6 External validity 3.4 Low cost 3.2

Reasons for Use

Possible responses not possible on paper Dynamic and interactive (but any computer would

do) Access to special populations More precise replication Reduction of experimenter effects Comparability to local sample New set of variables to study

Reasons for Concern

Ethical issues Informed consent/debriefing Data security

Data validity Subject falsification/collaboration

Loss of experimental control Equipment/environment Variation

Reasons for Concern

Computers are necessary Lack of ability to clarify instructions and

informed consent New class of variables that may impact

responses Does this last one sound familiar?

Importance of Problems Faced Musch & Reips (2000) In descending order

Lack of control of behavior 3.6 Lack of control of motivation 3.4 Inability to ask questions 3.3 No control over hardware 2.9 Nonrepresentative sample 2.8 Ethical problems 1.5

Issues regarding the sample

Web is stated to be not representative Questions

How many labs have representative sample? Do we know what person variables are significant

or do we control out of the assumption of a possible effect?

The greater diversity of samples (Krantz & Dalal, 2000 and “College Sophomores”) may allow this to become a research question instead of assumption

The Web Study In Context

Relevant Dimensions Methodological Issues Subject-Related Issues Ethical Issues

A Decision Guide

Relevant Dimensions – Method Statistical Power

Refers to noisy data Not in all cases will noise be better on Web

Ruppertsberg et al (2000). Web, n = 151, s = 12.5, SE = 1.02 Control, n = 14, s = 7.4, SE = 1.9

Effect Size or Robustness of the Effect Basically is it easy to obtain the result

Relevant Dimensions – Method Sample Bias

Is the effect sensitive to the characteristics of the sample?

External Validity Does this effect or relationship happen outside of

laboratory Does this effect or relationship happen away from

a computer

Relevant Dimensions – Method Need for Control

Stimulus Problematic Dimensions

Color Luminance Temporal If can keep on one page

Relative spatial dimensions are fine

Example Study: Column Taper Description:

Doric columns appear to lean when vertical Apparently due to taper of column width

Method Three columns. Adjust so out two same position

as inner (vertical or horizontal) Validation Results

Manipulation Validation

y = 0.99x

0

1

2

3

4

5

6

0 1 2 3 4 5

Target Angle

Mea

sure

d A

ngle

Vertical Results

y = 386.88x - 3.36r2 = 0.87

y = 325.75x - 11.72r2 = 0.99

-200

-150

-100

-50

0

50

100

150

200

250

-0.4 -0.2 0 0.2 0.4

Log Column Proportion

Tilt

at

Perc

eiv

ed V

ert

ical (a

rcm

in)

Web Vert

Lab Vert

Linear (Web Vert)

Linear (Lab Vert)

Relevant Dimensions – Method Need for Control

Measurement Need will be sensitive to type of measurement Think of all question order effects Now type of response objects

Environment Effect of Web Known

Has the method been examined on Web Time for data collection

Response time for results

Relevant Dimensions – Subject Special Populations

Ethnic groups Age Education Other targeted populations (Buchanan, 2000)

Self-Monitoring: found groups predicted high and low on scale to validate response

Relevant Dimensions – Subject Interaction with Subjects

Need for Clarification As part of research design

Demand Characteristics Web removes us, but our design might still

provide some hints

Relevant Dimensions – Ethics

Sensitivity of Ethical Issues Deception Potential for harm

Effects of Fraud by Participants on Data Faking responses Participant Collaboration

A Decision Guide - Method

Low High

Statistical Power Either Web

Effect Size Depends1 Either

Sample bias Either Usually Lab2

Ecological Validity

Either Depends3

Sensitivity of Study to/Need:

Decision Guide – Method

Low High

Control Either Lab

Known Valid on Web

Lab at least as check

Web

Time for Data Web Either

Decision Guide – Subject

Low High

Special Populations

Either Web, in most cases

Subjects from other countries

Either Web

Interaction with subjects

Either Lab

Demand Possibility

Either Web

Decision Guide – Ethics

Low High

Sensitivity to Ethics

Either Lab

Effects of Fraud Either Lab

Factors not Included in Decision Guide Did not include ease or cost

Also not important factor in early researchers, Musch & Reips, 2000

These factors should not drive design criteria.

Conclusions

Web and lab distinct environments Each method has strengths Match method to research question Still useful to examine methods

Study locally as well as globally Differences do not necessarily imply web is less

valid than lab Differences could be meaningful Differences could imply problem in lab envirnment