Transcript
Chapter 6: RESEARCH VALIDITY
9/27/2012
Roadmap
• Research study announcement• Exam 1 update• Reflection Assignment #1• Quick review & Ch. 5 wrap-up• Begin Chapter 6: Research Validity
Reflection Assignment #1
• Due Tuesday (next class)• Hard copy due at beginning of class– STAPLE YOUR BUSINESS. I’m so serious.
• Cover page: Assignment document (rubric for grading)
Quick Review
Random SamplingSimple Random
Stratified Random
Cluster Random
Systematic Sampling
Nonrandom SamplingConvenience
Quota
Purposive
Snowball
Quick Review
• Random selection vs. Random assignment
• Difference = purpose
How do we determine sample size?
• If population <100, measure them all– Special term for this?
• In general, get as big a sample as possible
• Sample size calculator: G*Power
• Depends on lots of factors
RESEARCH VALIDITYChapter 6
MORE Validity—yay!
• Remember: validity has to do with drawing accurate inferences
• So far: validity in the context of measuring variables
• Now: validity in the context of setting up studies
Research Validity
• Refers to the truthfulness of inferences made from a research study
• Think of validity on a continuum rather than all-or-none
• 4 major types of research validity
• Must prioritize
Types of Research Validity
• Statistical Conclusion Validity• Construct Validity• Internal Validity• External Validity
Statistical Conclusion Validity
• Validity with which we can infer that the IV and DV covary– Covary = vary together
• The validity of the inferences we make from our analyses
Stats Refresher: “Statistically Significant”
• p <.05
• What does it mean?– The observed relationship is probably
NOT due to chance alone
• Sometimes our stats are just wrong• Chance• Too little power (sample size)• Type 1 / Type 2 error
Construct Validity
• Refresh: construct = ?
• Validity of the inferences we make about constructs based on how we measure them
• What does this sound like?– The chapter 5 validity topics!
• Which constructs/operationalizations do we need to consider for construct validity?
• All of them!– IV– DV– Population– Setting
How do we assess Construct Validity?
• Content validity
• Criterion validity
• Convergent validity
ACTIVITY
Group Activity: 5-7 minutes• You’re applying for a grant to fund a
research project
• Identify research idea– IV - operationalize– DV - operationalize– Hypothesis
• Explain how you will gather evidence of construct validity in your measurements
Threats to Construct Validity
• Factors that impact how well our operationalizations actually represent constructs
• Pg 171, Table 6.2 – long list of threats
• We will focus on two major ones:– Participant reactivity to the experimental
situation– Experimenter effects
Reactivity to the Experimental Situation
(From the participant angle)
• Participants’ motives and perceptions
• Demand characteristics
• Positive self-presentation
Instruction set #1
We want to see how well you are able to learn the following sets of letters. Letters will appear in groups of 3 to 7, and each letter will appear on the screen for 1 second. Following the presentation of the letters, …
Instruction Set #2
In the following task, you will be presented with groups of letters ranging from 3 to 7 letters. Each letter will appear on the screen for one second. Your task is to…
Experimenter effects
• Researcher actions and characteristics that influence the responses made by the research participant
• Expectancies– Clever Hans
• Attributes– Biosocial– Psychosocial– Situational factors
Clever Hans
I Math
Internal Validity
• The extent to which we can accurately infer that the independent and dependent variables are causally related
Observed Effect (DV)
Independent Variable
Causally Related
Independent Variable
Observed Effect (DV)
Cause must precede effect
Cause and effect are related (covary)
No other explanation is plausible
No other explanation is plausible
Special Considerations
• Extraneous variables
• Confounding extraneous variables
Threats to Internal Validity
• History • Maturation • Instrumentation • Testing • Regression Artifact • Attrition • Selection • Additive and Interactive Effects
History
• Any event occurring after the study begins that could produce the observed outcome
• Differential history: only one group experiences history event
Maturation
• Changes in biological and psychological conditions that occur with the passage of time – Factors within the individual
• Example: Head Start program and achievement over a school year
Instrumentation
• Changes in the assessment/measurement of the dependent variable
• Example: multiple observers and interviewers
Testing
• Changes in a person’s score on the second administration of a test a result of previously having taken the test
• Example: pre-test and post-test on memory task
Regression Artifact
• A.k.a. regression toward the mean
• The tendency for extreme scores to become less extreme on a second assessment
Attrition
• Participant drop-out– Don’t show up for appointment– Decide to discontinue study
• Differential attrition is especially problematic
Selection
• The choice of participants for the various treatment groups based on different criteria – NOT random assignment
Additive & Interactive Effects
• The combined effect of several threats to internal validity
top related