Reliability and Validity Threats to Internal Validity Da Lee Caryl, Fall 2006.

Reliability and ValidityThreats to Internal Validity

Da Lee Caryl, Fall 2006

Internal Validity

• Defending against sources of bias arising in research design.

• When there is lack of internal validity, variables other than the independent(s) being studied may be responsible for part or all of the observed effect on the dependent variable(s).

An experimenter observes an increase in the probability of salivating after presenting the smell of delicious food

Leads to the conclusion that the smell of food produces salivation.

For this conclusion to be internally valid, the experiment must be designed so that conditions other than the food smell are ruled out as potential causes for salivation.

Does the sight of food along with the smell of food cause salivation?

Suppose:

Threats to Internal Validity

• Selection • Experimental Mortality• Location • Instrumentation • Testing• History • Maturation • Attitude of Subjects• Regression• Implementation

Selection

Participants in a study may have different characteristics and those differences may affect the results. Such as:

Age Intelligence Vocabulary

Strength Attitude Fluency

Maturity Fluency Ethnicity

Coordination SpeedSocioeconomic status

Experimental Mortality

Loss of subjects from the study due to:• Illness• Family relocation• Requirements of other activities• Absent during collection of data or fail to complete

tests

Location

Place in which data is collected, or an intervention is carried out, may influence the results.

Instrumentation

Inconsistent use of the measurement instrument.

• Instrument Decay – instrument changes in some way • Data Collector Characteristics – affects the nature of

the data they obtain. Such as gender, age, ethnicity, age, language patterns, etc.

• Data Collector Bias – unconsciously influence the outcome of the data

Testing

Pretests may influence the result of the posttest.

A group may perform better on a posttest because the pretest primed them to perform better.

History

Occurrence of events that could alter the outcome or the results of the study

• Previous – events that occurred prior to the study

• Concurrent – happening during the study

Maturation

Any changes that occur in the subjects during the course of the study that are not part of the study and that might affect the results of the study; such as changes due to aging and experience.

Attitude of Subjects

• Subjects opinion and participation can influence the outcome.

• Observing or studying subjects can affect their responses a.k.a. Hawthorne effect.

• Subjects receiving experimental treatment may perform better due to “receiving” treatment.

• Subjects in the control group may perform more poorly than the treatment group.

Regression

Groups that are chosen due to performance characteristics, either high or low, will on the average score closer to the mean on subsequent testing regardless of what transpires during the experiment.

Example: A group of students who score low on a mathematics test are given additional help. Six weeks later they are given an exam similar problems on another test, their average score has improved. Is it due to the additional help or other influences?

Implementation

Personal bias in favor of one method or another. Preference for the method may account for better performance by the subjects.

Individuals assigned to implement different methods, differ in ways related to the outcome.

Methods to Minimize Threats

Standardization of the conditions under which the research study is carried out will help minimize threats from history and instrumentation.

Obtain as much information as possible about the participants in the research study minimizes threats from mortality and selection.

Obtain as much information as possible about the procedural details of the research study, for example, where and when the study occurs, minimizing threats from history and instrumentation.

Choose an appropriate research design which can help control most other threats.

Journal of Agriculture Education

Miller, Greg & Pilcher, Carol(2002). Can Selected Learning Strategies Influence the Success of Adult Distance Learners in Agriculture? Journal of Agricultural Education, 43,34-43. Retrieved October 15, 2006 from the World Wide Web: http://pubs.aged.tamu.edu/jae/pdf/vol43/43-02-34.pdf

Bibliography

Fraenkel, J.R. & Wallen, N.E. (2006). Internal Validity. How to design and evaluate research in education (6th ed.,pp.168-186). New York: McGraw Hill.

Martin, Wendy (1997). Single Group Threats to Internal Validity. Retrieved October 15, 2006 from the World Wide Web: http://www.socialresearchmethods.net/tutorial/Martin/intval1.html.

Trochim, William (2006). Internal Validity. Retrieved October 15, 2006 from the World Wide Web: http://www.socialresearchmethods.net/kb/intval.html.