Modeling Political Modeling Political Phenomena Phenomena Using Control Variables and Gauging Validity
Modeling Political Modeling Political PhenomenaPhenomena
Using Control Variables and Gauging Validity
Face ValidityFace Validity
Face validity means the measurement of a concept is consistent with an agreed definition.
It does not mean however that this is the best measurement to capture the concept.
Construct ValidityConstruct Validity
construct validity: The match between the land of theory and the land of observation
How effective do our operationalized variables represent the mental image of a concept into the public manifestation of that world?
ReliabilityReliability
We may want, or need to, test for reliability, which is to ask if our variables consistently provide the same results.
To be sure we can measure/test repeatedly or even use multiple measures for the same variable. For development we can also use GDP per capita besides energy consumption per capita.
Key question of Internal ValidityKey question of Internal Validity
When we test a hypothesis and either accept it or reject it, how do we know that we made the right decision?
What about alternative explanations that we did not account for?
What should we do to gain confidence?
Internal ValidityInternal Validity
Are there other causes for what I am observing?
If so, a study will lack internal validity if it cannot rule out plausible alternative explanations.
Internal Validity of a StudyInternal Validity of a StudyInternal Validity of a StudyInternal Validity of a Study
What you measured and what you saw?Your program and your observations?
ObservationObservationObservationObservation
ProgramProgram ObservationsObservations
What you What you dodo What you What you seesee
Is the relationship Is the relationship causalcausal between... between...
AlternativeAlternativecausecause AlternativeAlternative
causecause
AlternativeAlternativecausecause
AlternativeAlternativecausecause
In In thisthis study study
Program-outcome RelationshipProgram-outcome Relationship
The Purpose of Control VariablesThe Purpose of Control Variables
We use control variables to account for possible alternative explanations we can think of.
For example, when I examined whether democracies are generally more peaceful than autocracies I included several control variables.
Explaining Pacifistic DemocracyExplaining Pacifistic Democracy
Peace (Y) = Democracy (X1) + State Power (X2) + Development (X3) + # of Bordering States (X4)
In the model above, I have more confidence that Democracy is related to peace considering I control for the other variables that may skew my test.
• We need to take care that our theory is not missing other factors that may undermine the validity of our theory and tests.
• Our inferences will be flawed if we are actually capturing other processes through our variables.
• This means that the validity of our measures would be undermined.
• Several possible problems arise that are related to model misspecification and spurious relationships.
• Thus, we need to control for confounding factors and alternative explanations!!!
Model Misspecification and Model Misspecification and SpuriousnessSpuriousness
Antecedent variable: A variable that indirectly affecting the relationship between two other variables.
For example, Ivy league education increases income.
However, parental wealth and legacy admissions affect Ivy league education. Thus, income of graduates from Ivy League schools may not be random.
Here Ivy League Parents is an antecedent variable
Ivy League Parents Ivy League Kids high income kids
Hence, admission to Ivy schools clearly not random or pure merit-based, and thus the income earned by these people.
Model Misspecification and Spuriousness
Intervening Variable: These may be spuriously related to another relationship.
How can states fight each other if they are not contiguous with each other? Only the strongest, with large navies, bases, etc., could do so.
Hence, geographic contiguity or distance is an intervening variable. States may or may not be more peaceful, but it is hard to avoid conflict when it is on your borders.
Model Misspecification and Spuriousness
Alternative Variables: We also want to control for variables that would bias our results if omitted.
In this case, the X variables in a model would produce biased estimates, undermining their validity and producing error that leads to inaccurate inferences.
Here is a spurious relationship from my research
IGOs + conflicts
+ +
Powerful states
Powerful states both in more IGOs and conflicts, but these two variables not directly related but a function of state power.
Classic Spurious Case
Ice Cream Consumption Crime
Summer Temperatures
???
+ +
+
Hence we see that despite the fact that ice cream consumption is correlated with crime, the real cause is that summer temperatures increase both ice cream consumption and crime.
Assessing your knowledgeAssessing your knowledge
If your scientific study has taken care to make sure that your variables are measured correctly, used the appropriate control variables, and used proper tests, then what is next?
Conclusion ValidityConclusion ValidityConclusion ValidityConclusion Validity
What you did and what you saw?
Your program and your observations?
ObservationObservationObservationObservation
ProgramProgram ObservationsObservations
What you What you dodo What you What you seesee
Is there a Is there a relationshiprelationship between... between...
In In thisthis study study
Program-outcome RelationshipProgram-outcome Relationship
Group Work
Identify the level on which variables are measured.
Identify problems of construct validity, internal validity, and biased samples
External ValidityExternal Validity
Now that you are confident of what you found in your study, how well does my study or sample relate to the general population?
In other words, how strong is my study able to generalize to other cases?
Research Designs and Research Designs and SamplingSampling
In most studies what is examined are some cases, not an entire population. For example, in Presidential Election polls not every voter is asked how they will vote but still polls can be very accurate.
How does that happen?
Population vs. SamplePopulation vs. Sample
Research in the social sciences typically uses sampling methods.
We draw a sample of subjects from a greater population.
We then draw an inference from the sample about the greater population.
In other words, we are generalizing about a population from a subset (the sample).
Validity and BiasValidity and Bias
In order to draw an accurate inference from a sample, the sample needs to be reflective of the population from which it is drawn.
If a sample is not reflective of the population, then it is biased in some manner and the greater study will lack validity.
Types of SamplingTypes of SamplingNonrandom: snowballing, various improper selection techniques or limited data. Measurement error is greater.
Random: pure chance of lottery and should reflect population the larger the sample. Measurement error decreases.
Quota or stratified: Selecting on groups to form sample so as to reflect greater population. Measurement error is lower.
Census: Includes entire population. No measurement error.
Mathematical PrincipleMathematical Principle
The larger the sample size, the more it will reflect the population estimates/values.
Thus, the larger the sample, the less chance of measurement error.
External Validity of a StudyExternal Validity of a Study
TheoryTheoryTheoryTheory
CauseCauseconstructconstruct
EffectEffectconstructconstruct
What you What you thinkthink
Cause-effect ConstructCause-effect Construct
ObservationObservationObservationObservation
ProgramProgram ObservationsObservations
What You What You DoDo What You What You SeeSee
program-outcome relationshipprogram-outcome relationship
ObservationObservationObservationObservation
ProgramProgram ObservationsObservations
What You What You DoDo What you What you seesee
program-outcome relationshipprogram-outcome relationship
ObservationObservationObservationObservation
ProgramProgram ObservationsObservations
What You What You DoDo What You What You SeeSee
program-outcome relationshipprogram-outcome relationship
ObservationObservationObservationObservation
ProgramProgram ObservationsObservations
What You What You DoDo What You What You SeeSee
program-outcome relationshipprogram-outcome relationship
Can we generalize to other persons, places, times?
External Validity of a StudyExternal Validity of a Study
The last graphic is meant to convey the principle that external validity is gained by additional observations/tests in other studies.
This is why pundits will compare several election polls to see how well they compare. If they do not, then somebody is doing something wrong or different.