Threats to internal and external validity

THREATS TO INTERNAL AND EXTERNAL VALIDITY

By Bieger and Gerlach(A RodSaz Report)

INTRODUCTION

The results of a research study are only useful to the extent that they can be accurately and confidently interpreted.

The issue of accurate and confident interpretation of results is at the center of any discussion of validity.

VALIDITY

From validus, meaning “strong” Refers to the degree with which correct

inferences can be made from the results of a research study.

TWO ASPECTS OF VALIDITY

•INTERNAL VALIDITY•EXTERNAL VALIDITY

INTERNAL VALIDITY

Refers to the extent to which the results obtained in a research study are a function of the variables that were systematically manipulated, measured, and /or observed in the study.

For example, a researcher in interested in determining which of two instructional methods is superior for teaching a history concept. Suppose further that the researcher asked two teachers to each use one of the methods of instruction and then compared the mean test scores of each class following the instructions.

Ways the two classes differred:

Teachers’ teaching effectiveness and enthusiasm

Class interest or preparation One class may have interruptions (fire

drills)

POTENTIAL THREATS TO INTERNAL VALIDITY

A. HISTORY

Refers to the occurrence of events that could alter the outcome or the results of a study.

A. HISTORY

Refers to the occurrence of events that could alter the outcome or the results of a study. Previous history – occurs before the study

A study of the effectiveness of a new method for teaching a unit on the biology of a cell, many watched a television documentary about “The Cell”

A. HISTORY

Refers to the occurrence of events that could alter the outcome or the results of a study.

Concurrent history – occurs during the study Eg. Studying the effectiveness of using musical

activities to teach mathematics concepts. While one teacher uses the standard curriculum, another teacher is using the musical activities curriculum

B. MATURATION

Pertains to any changes that occur in the subjects during the course of the study that are not part of the study and that might affect the results of the study.

B. MATURATION

Pertains to any changes that occur in the subjects during the course of the study that are not part of the study and that might affect the results of the study. Biological ( growth processes)

Eg. Weight gain or increase in height due to breakfast or lunch program

B. MATURATION

Pertains to any changes that occur in the subjects during the course of the study that are not part of the study and that might affect the results of the study. Psychological (learning or development)

Eg. Effects of certain instructional techniques on concept learning of sixth graders, attainment of certain operational thought during that period has to be considered

C.TESTING

Relates to the possible effects of a pretest on the performance of participants in a study on the posttest. May alert subjects to the fact that they are

being studied May affect performance on later

administrations

D. INSTRUMENTATION

Is concerned with the effects on the outcome of a study of the inconsistent use of a measurement instrument (what the instrument is measuring changes during the duration of the study).

Eg. The effects of fatigue on an achievement test

E. STATISTICAL REGRESSION

Refers to the tendency of extreme scores to move (or regress) toward the mean score on subsequent retesting. Eg. Students scoring below 25% (lowest

extreme) on an IQ test are given a posttest. A higher posttest score is expected.

F. MORTALITY

Refers to the loss of subjects from a study due to their initial nonavailability or subsequent withdrawal from the study. Eg. More high-scoring people drop out from

the experimental group than from the control group

G. SELECTION

Pertains to the possibility that groups in a study may possess different characteristics and that those differences may affect the results. Differences in age, ability, gender or

racial/ethnic composition, or any of an almost unlimited number of ways.

PROCEDURES FOR MAXIMIZING INTERNAL

VALIDITY

Fraenkel and Wallen suggests four general ways:

1. Standardization of the conditions under which the research study is carried out will help minimize threats to internal validity from history and instrumentation.

Fraenkel and Wallen suggests four general ways:

2. Obtaining as much information as possible about the participants in the research study aids in minimizing threats to internal validity from mortality and selection.

Fraenkel and Wallen suggests four general ways:

3. Obtaining as much information as possible about the procedural details of the research study, for example, where and when the study occurs, minimizes threats to internal validity from history and instrumentation.

Fraenkel and Wallen suggests four general ways:

4. Choosing an appropriate research design can help control most other threats to internal validity.

SOME SPECIFIC SUGGESTIONS FOR MINIMIZING POTENTIAL

THREATS

history

The use of a control group, selected from the same population as the experimental group(s) and which experiences the same concurrent history as the experimental group(s), can help eliminate most of the effects of history. Also, the shorter the duration of an experiment, the less likely history will be a threat.

Maturation

The effects of maturation, like the effects of history, can be minimized by the use of a control group, selected from the same population as the experimental group(s). Also, like the effects of history, the effects of maturation tend to be minimized in studies of short duration.

Testing

The use of a research design that does not include a pretest can eliminate testing as a potential threat to internal validity. If baseline or pretreatment or data are needed, the use of unobtrusive measures (data collection techniques about which the experimental participant is unaware) may minimize the effects of testing. It also may help for a researcher to use different equivalent forms of a test for pretesting and posttesting.

instrumentation

Careful specification and control of the measurement procedures can eliminate most instrumentation threats. Standardized instruments, administration or data collection procedures, and the training or observers are among the procedures that help control the instrumentation threat.

Statistical regression

Avoiding the use of extreme scorers, when average scorers are excluded, will minimize the threat.

mortality

Choosing large groups and ensuring that they are representative of the population from which they were selected can minimize mortality threats. The use of follow-up procedures with a portion of those who leave the study or who were initially unavailable can further minimize mortality as a threat

selection

Random selection and random assignment of subjects minimize selection as a threat to internal validity. If random selection and assignment are not possible, the use of certain statistical techniques, used as part of a careful quasi-experimental design, can adjust for group differences and thereby minimize selection as a threat.

EXTERNAL VALIDITY

EXTERNAL VALIDITY

Refers to the extent to which the results of a research study are able to be generalized confidently to a group larger than the group that participated in the study. (Bracht & Glass)

EXTERNAL VALIDITY

Refers to the extent to which the results of a research study are able to be generalized confidently to a group larger than the group that participated in the study. (Bracht & Glass)

The researcher must have reason to believe that the variables used in the study are similar to those aspects as they exist in the larger population.

Population-related threats

The extent to which a sample is representative (or not representative) of the population from which it was selected.

Ecology-related threat

The extent to which characteristics of the setting or context of the research study are representative (or not representative ) of the setting and context to which the results are to be generalized.

POTENTIAL THREATS TO EXTERNAL VALIDITY

A. Effect of Testing

Refers to the fact that the administration of a test (for example, a pretest) may affect the responses or the performance of the participants in a research study.

Its results may not be generalizable to situations where pretesting will not occur.

Multiple-treatment interference

It pertains to the situation in which participants in a study receive more than one treatment. In such a case, the effects of the multiple treatment may interact.

Selection-treatment interaction

It is concerned with the possibility that some characteristic of the participants selected for the study interacts with some aspect of the treatment. May include prior experiences, learning,

personality factors, or any traits that might interact with the effect of the treatment

Effects of experimental arrangements

It pertains to situations where participants become aware that they are involved in a study, and , as a result of that awareness, their response or performance is different from what it would have been otherwise.

Effects of experimental arrangements

It pertains to situations where participants become aware that they are involved in a study, and , as a result of that awareness, their response or performance is different from what it would have been otherwise.

“Novelty effect” (newness) or belief on the part of the participants that they are receiving some “special” treatment (sometimes known as Hawthorne Effect.)

“Placebo effect” or to the participants’ belief in the effectivenesss of the treatment

Experimenter Effects

They refer to the possibility that an experimenter may sometimes unintentionally influence the performance of participants in a study.

Experimenter Effects

They refer to the possibility that an experimenter may sometimes unintentionally influence the performance of participants in a study.

Experimenter Effects

They refer to the possibility that an experimenter may sometimes unintentionally influence the performance of participants in a study. PASSIVE

Gender, race, or personal attributes of the researcher

Experimenter Effects

They refer to the possibility that an experimenter may sometimes unintentionally influence the performance of participants in a study. PASSIVE

Gender, race, or personal attributes of the researcher

ACTIVE The expectations of the researcher or observer

are communicated to the participant in a manner that affects performance

Specificity of variables

It is concerned with the extent to which the variables in a study are adequately described and operationally defined. Variables can be defined too specifically.

PROCEDURES FOR MAXIMIZING EXTERNAL

VALIDITY

EFFECT OF TESTING

Use research designs that do not include pretests.

Solomon four-group design is useful in determining the extent to which pretesting may have influenced the results of the study.

Multiple-treatment interference

The researcher should try to choose a design in which only one treatment is assigned to each subject. If such a design is practical, the researcher should try to control and/or measure the effects of all relevant treatments and incorporate them into a multiple-treatment design.

Selection-treatment interaction

Use random selection and assignment of participants.

If the above is not practical, use statistical techniques such as analysis of covariance, used in conjunction with a quasi-experimental design, can take into account differences due to measurable attributes of the individual.

Effects of experimental arrangements

Have a control group and a placebo group

Both should be treated the same in all respects, except in regard to the treatment itself

All of the participants should believe they are being treated comparably

Experimenter effects

Use “blind” data collection procedures. This means that the researcher does not collect data or make observations but instead trains a “naïve” observer to do so.

The person collecting the data or making the observations should be unaware of the purpose of the study and should be unaware of which participants are receiving the experimental treatment.

Specificity of Variables

Must operationally define variables carefully in a way that is meaningful in settings beyond that in which the study is being conducted.

Use of widely agreed upon definitions or multiple competing definitions should be considered.