Issues in experimental design for the study of atypical language development

08/10/2013

1

Issues in experimental design for the study of

atypical language development

David Saldaña

DESIGNING YOUR STUDY: CONTROL

08/10/2013

2

Controlled experiments This type of experiment is conducted in a well-controlled environment – not necessarily a laboratory – and therefore accurate measurements are possible.

Easier to replicate

Precise control of extraneous and independent variables.

The artificiality of the setting and lack of generalization

Demand characteristics or experimenter effects may bias the results

McLeod, S. A. (2012).

Field experiments The experimenter still manipulates the independent variable, but in a real-life setting (so cannot really control extraneous variables).

More likely to reflect real life because of it natural setting

Less likelihood of demand characteristics affecting the results

Can be used in situations in which it would be ethically unacceptable to manipulate the independent variable, e.g. researching stress.

Less control over extraneous variables

Less replicable

McLeod, S. A. (2012).

08/10/2013

3

Natural experiments Natural Experiments are conducted in the everyday environment of the participants but here the experimenter has no control over the IV as it occurs naturally in real life.

More likely to reflect real life because of it natural setting

Less likelihood of demand characteristics affecting the results

Can be used in situations in which it would be ethically unacceptable to manipulate the independent variable, e.g. researching stress.

Less likelihood of demand characteristics affecting the results, as participants may not know they are being studied.

No control over extraneous variables

Less replicable

Possible more expensive

McLeod, S. A. (2012).

A perhaps “silly” research question

Does vocabulary influence the comprehension of questions more than the comprehension of imperatives?

08/10/2013

4

A question that is a little more complicated

Do children with SLI understand questions better than typically developing children?

Do children with SLI understand questions better than imperatives?

DESIGNING YOUR STUDY: CONDITIONS

08/10/2013

5

How is your IV distributed?

Independent Measures

Repeated Measures

Matched Pairs

Independent measures (between-group)

• Each group gets one condition

• Different participants in each group • Avoids practice

• More people needed

• Participant variables could affect results.

08/10/2013

6

Task 1

• Measurement

Task 2

• Measurement

Repeated measures (within-group)

- Less participants - Precision determined by variation within same

subject - May be the only design that answers the

questions of interest. - Order effects: practice and fatigue effect

Counterbalancing

Moment 1

A

B

Moment 2

B

A

08/10/2013

7

Back to our “silly” study

Lets design the tasks and the data collection

Does vocabulary influence the comprehension of questions more than the comprehension of imperatives?

A little bit more interesting (perhaps) research question

Does vocabulary influence the comprehension of questions more than the comprehension of imperatives in SLI?

08/10/2013

8

Task 1

• Measurement

Task 2

• Measurement

Mixed-design

Task 1

• Measurement

Task 2

• Measurement

Gro

up

1

Gro

up

2

Some imaginative results

70

77 73 75

90 90 95

99

92 90

100

90

0

20

40

60

80

100

120

IQ Vocabulary ComprehensionQuestions

Word Reading SES ComprehensionImperatives

Stan

dar

d S

core

s

SLI Typical

08/10/2013

9

MATCHING

Matching your groups: why?

70

77 73 75

90 90 95

99

92 90

100

90

0

20

40

60

80

100

120

IQ Vocabulary ComprehensionQuestions

Word Reading SES ComprehensionImperatives

Stan

dar

d S

core

s

SLI Typical

08/10/2013

10

Matching your groups: on what?

Imp.Comp

6 8 10 2 4 6 8

24

68

68

10

IQ

QuestComp

510

15

24

68

SES

2 4 6 8 5 10 15 4 6 8 10

46

810

Voc

Your groups: three group design

• Group of interest

• Control group, matched on chronological age

• Control group, matched on variable of interest

08/10/2013

11

Some issues

• Match on variables of interest to the study

• Match average and distribution

• Study carefully your exclusions

Recruit clinical sample

Decide variables of

matching

Recruit age controls

Recruit “level” controls

Let’s do it! Matching exercise

Some issues

• Match on variables of interest to the study

• Match average and distribution

• Study carefully your exclusions

Recruit clinical sample

Decide variables of

matching

Recruit age controls

Recruit “level” controls

Check and report

08/10/2013

12

What statistical test?

An ideal world

Facon et al. (2011)

What statistical test?

An ideal world

08/10/2013

13

What statistical test?

An ideal world The real world

Non-normal distributions Different range and variance Different distributions

What statistical test?

None: look at the picture first!

Back-to-back histogram Back-to-back stem-and-leaf

08/10/2013

14

What statistical test?

What statistical test?

Mann-Whitney?

Two-sample Kolmogorov-Smirnov test

For small sample sizes: exact version

• Does not assume prior “shape” in distributions

• Tests for the differences in distribution

08/10/2013

15

Matching your groups: how much? The easy-peasy way

Matching your groups: how much?

http://intuitor.com/statistics/T1T2Errors.html

A brief recall of significance testing

Match No Match

I say No Match

I say Match

08/10/2013

16

“Casual acceptance of the null hypothesis” (Harcum, 1990)

< .2 too low

.2 to .5 ambiguous

> .5 fine

Frick (1995)

http://intuitor.com/statistics/T1T2Errors.html

Some things to remember • Population comparisons are group comparisons

(Paradis, 2010)

08/10/2013

17

Some things to remember

• Each group must be there for a reason

(Paradis, 2010)

An example: why each group?

OCS 20 adolescents (13-21) task with sound

OSS 20 adolescents (13-21) task with no sound

NO 20 children (6-11) oral language equivalent to SAL

Two groups of deaf adolescents

SAL 16 deaf (13-21) good oral language

SBL 16 deaf (13-21) poor oral language

Three groups of hearing adolescents

Torres (2013)

08/10/2013

18

Some things to remember

• Check matching while you are recruiting

(Paradis, 2010)

Keep your lab notes up to date

•Some important decisions to record: •Changes and discussions on criteria •Subjects in – subjects out. •Transcriptions and criteria if using MLU

(Paradis, 2010)