Lab 9—Last Lab Statistical Analyses & Threats to Validity.

Lab 9—Last Lab

Statistical Analyses & Threats to Validity

Background

Recall from last lab that there are two major types of designs. If you are interested in the association between

continuous variables, you should use a correlational design.

If you are interested in comparing two (or more) groups, it is likely you will use an experimental design.

The design you use will determine: statistical analysis you will perform threats to validity you will most likely need to address conclusions you can make from your research.

Correlational/Observational Design In this design, you are not comparing

groups. Rather, you are examining the association between two continuous variables.

You can use either correlation or regression analysis to analyze your data.

Correlation

Used when interested in assessing the strength of association between two variables

Used with continuous variables

Example research hypothesis: are higher levels of self-esteem associated with lower levels of depression (what direction correlation would be expected?)

Regression

Simple Linear Regression is a statistical method concerned with the amount of variance that one variable accounts for in another variable (that is, the amount of shared variance between variables).You also want to use regression when you are predicting one variable from the other.Used when you have continuous variables (e.g., motivation measured on a Likert scale, body weight).

Regression

Variable YVariable X

Shared variance

http://www.math.csusb.edu/faculty/stanton/m262/regress/regress.html

Correlation vs. Regression

Both correlation and regression are concerned with the amount of variance that variables share

The major difference is that with correlation we are only interested in an association between variables, but with regression we assign one variable to be the independent variable (x; the predictor) and the other to be the dependent variable (y; the criterion)

Note: While we typically are concerned with predictive relationships when using regression, the certainty with which we can say that two variables are causally related depends upon the design (true experiments are best!...why?) and quality of the study

Comparing Groups

If you have an experimental design (manipulation of IV), you should use the following analyses to compare groups (non-continuous variables).

In some cases, you will want to compare groups even when you don’t manipulate anything. Comparing male versus female on

performance. Examining differences in opinion of various

political or religious groups.

Independent samples (between-groups) t-test Used to determine dependent variable mean

differences between two different groups

Used with categorical or dichotomous variables (e.g., gender, democrat or republican, yes/no responses)

Only one independent variable with only two levels/groups

Example research hypothesis: do women have higher levels of depression on average than do men?

Analysis of Variance (ANOVA) Like t-test, used to determine dependent

variable mean differences between groups/levels, but can use when there are more than two groups (which is the same as more than two levels of your IV)

Remember, though, you’re still only using one IV, just there are more than two groups/levels (e.g., you might have the IV of political party, with the groups Democrat, Republican, and Independent)

Analysis of Variance (ANOVA) Can use for your final project if you have just

one IV with more than two groups (i.e., one-way ANOVA), but can be used for more than one IV (i.e. two-way ANOVA, where say you have political party [3 groups] that may differ by gender [2 groups] for a 2x3 two-way ANOVA)

Independent variable(s) have two or more levels; if you only have one IV with two levels, though, use a t-test

Example research question: Do psychology graduate students have higher IQs than medical students, physics graduate students, and English graduate students?

Repeated measures (within-groups) t-test or ANOVA Used to test mean differences in the DV for the

group measured at two or more times (e.g., the same group of clients is exposed to every type of counseling [the IV] and symptoms [the DV] are measured)

Remember, the t-test involves an independent variable with two levels; ANOVA involves independent variable(s) with more than two levels.

Population parameters and sample statistics the same as between-group t-test and ANOVA above.

Repeated measures (within-groups) t-test or ANOVA Example research hypothesis for repeated-

measures t-test: Do people like the taste of Pepsi more than Coke (same people would taste both types of pop)?

Example research hypothesis for repeated-measures ANOVA: Does exposure to death metal music lead to better cognitive performance than exposure to jazz, classical, or country music (same people would listen to each type of music [IV] and perform a cognitive task [DV])?

THREATS TO VALIDITY

Threats to Internal & External Validity Keep in mind that in science it is always

important to balance multiple concerns. Ethics: Harm versus Benefit Sampling: Representativeness versus Practicality

When choosing a research design, it is important to address whether: You can address or eliminate alternative

explanations for your results (Internal Validity). You can generalize your results (External

Validity).

Research Designs & Validity

In general, experiments with random assignment are high on internal validity. Why? (What does random assignment do?)

Correlational studies are often higher than experiments on external validity but lower on internal validity.

Ideally, you would want studies to both be high internal and external validities. No single method does this. Multiple methods important.

An example

You are interested in studying why individuals help each other. You believe that sadness leads to helping because it makes individuals feel better.

Let’s say you conduct your study on September 12, 2001. How do you think this would influence your results?

To the extent that other explanations (outside of sadness) can account for the associations that you find, your internal validity is threatened.

Internal Validity Threats

Given that the IV is related to the DV, is it plausibly causal from one operational variable to another? History – is there something about the experience & history

of one group that could lead to differences? Maturation – would natural development influence the

associations? Testing – does actual participation in the study change

behavior or responses? Instrumentation – what happens if the instruments you use

change during the course of a study or if different groups of participants interpret your questions differently?

Regression towards the mean – if you start with extreme groups, they are likely to be less extreme at another time point.

Selection – are participants in your study somehow different than those who are not?

Mortality – those who drop out of your study could be qualitatively different than those who remain.

Internal Validity Threats

Given that the IV is related to the DV, is it plausibly causal from one operational variable to another? Diffusion or imitation – what would happen if

participants in the different conditions communicated? Interaction with Selection – what if any of the above

happen differentially for one group? Direction of causality – can the association happen in

the opposite direction than what you predicted? Compensatory equalization of treatments – what if

experimenters know what conditions participants are in? Compensatory rivalry – what would happen if

participants in the different conditions started competing?

Resentful demoralization – is it possible that participants in one condition give up?

External Validity Threats

Given that there is probably a causal relationship from construct A to construct B, how generalizable is this relationship across persons, settings, and times? Interaction of selection and treatment

(sample to population) Interaction of setting and treatment Interaction of history and treatment (time)

For your paper

You will need to discuss the internal and external threats to validity in the discussion of your paper.

Keep in mind that no study is perfect. However, when you bring up threats to

validity, you MUST explain: How it is a threat to validity How your results would change if the threat were

addressed What you could do to address the threat

In other words, DO NOT simply state the threat.

Homework 9

For each of the three research hypotheses described below, list what type of statistical test you would use to test it (independent (between-groups) t-test, between-groups ANOVA, repeated measures t-test, repeated measures ANOVA, correlation, regression), and whether external or internal validity is most threatened and why (6 points; 2 for each hypothesis).

Research Hypotheses: Women who experience a great deal of stress during

pregnancy are more likely to give birth to low birth weight or premature babies than pregnant women who do not experience a great deal of stress.

The degree to which someone believes in God relates to his/her psychological well-being. Further, the more someone believes in God the higher his/her well-being will be.

Homework 9

Research Hypotheses: A new manualized (standardized) treatment for

depression will decrease depressive symptoms in a group of people seeking help at a mental health clinic. They will be assessed when they come into the clinic, immediately after the treatment is completed, and 6 months after treatment is complete.

Lastly, describe your hypothesis, what design you will use for your project (from previous homework), what statistical analysis you plan to conduct, and identify 2 threats to validity (any combination of external or internal) and how they could influence the interpretation of your results. (4 points)