Quantitative Analysis of Survey Data and Other Assessments for Non-Experts: How to do SoTL without a statistician on retainer June 2, 2011 Gintaras Duda Creighton University
Dec 17, 2015
Quantitative Analysis of Survey Data and Other Assessments for Non-Experts:
How to do SoTL without a statistician on retainer
June 2, 2011
Gintaras Duda
Creighton University
My Background
• I am a theoretical particle physicist
• Came to SoTL (PER) as a junior faculty member– New faculty workshop experience
• Huge roadblocks: no experience with1.How educational research is conducted
2.Quantitative or qualitative analysis
3.Weak background in statistics
Areas of SoTL interest
• Attitude of introductory physics students– Particularly how it affects learning
• Online discussion behavior• Realism in physics• Problem-Based Learning in upper division
courses• Student note taking• How students use the internet to learn
physics
Workshop Purpose
• As SoTL matures, publication requires more and more rigorous measures and evidence– Sophisticated statistical tests
– Careful survey design and analysis
– Mixed method research
– Evidence, evidence, evidence!
• But, many of us are untrained in these things
Who are you?
• Please share with the group:
1.Name, institution, and discipline
2.Why you picked this workshop
3.What you hope to gain
Workshop Purpose continued …
• Leave you with some simple tools to analyze:1. Likert scale surveys
2. Effects of instruction
3. Survey reliability and validity
• No stats class or methods courses required
Part I. What to do with Likert Scales
• Likert scale instruments seem ubiquitous in SoTL work
• Particularly useful in measuring students’ attitudes, feelings, opinions, dispositions, etc.– Can use pre-post scheme to see changes and
growth/deterioration– Of interest in Jesuit Pedagogy (another
workshop)
Example from physics
• Attitudinal surveys– Measure students’ changes in attitude towards
physics due to instruction– Instruments: VASS, MPEX, C-LASS, Attitude
II, and others
• These instruments all show a similar trend– Students have more negative attitudes towards
physics after instruction
Example Questions from Attitude II Instrument
1. Physics is irrelevant to my life
2. I can use physics in my everyday life
3. I will did find it difficult to understand how physics applies in the real-world
4. I see and understand physics in technology and the world around me
• 5 point Likert scale: Strongly agree, weakly agree, neutral, weakly disagree, strongly disagree
What do I do with Likert Scale Data?
• Two camps on analyzing Likert scale data
1.Interval Approach
2.Ordinal Approach
• Methods for data analysis differ between the two methods
Interval Data
• Basic philosophy: differences between responses are all equal– i.e. Difference to a student between strongly
disagree and weakly disagree is the same as the difference between a neutral response and weakly agree
• Basic technique: Sum the data and do some statistics
Ordinal Data
• Basic philosophy: Differences between responses are not equal– i.e. Students tend not to distinguish highly
between strongly and weakly statements– 3 pt Likert scale more appropriate?
• Basic technique: Examine statistically the number of students who agreed or disagreed
Controversy over neutral response
• Good debate in the literature about the neutral/neither agree nor disagree response
• Some claim it’s crucial
• Some claim you should get rid of it
• Not going to discuss it here
Analyzing Ordinal Data
• One method is to reduce the problem to a “binomial analysis”– Lump all disagrees together, all agrees together,
and don’t worry about neutral responses
• Visual method: Agree-disagree (Redish) plots
Redish, J. Saul, and R. Steinberg, “Student expectations in introductory physics,” Am. J. Phys. 66, 212–224 1998.
Agree-Disagree Plots
• Introduced by Redish et al. in their MPEX paper - called “Redish” plots
Change from pre to post must be > 2σ to be considered significant
(at 5% probability level)
New Agree PercentageNew Disagree Percentage
Standard Deviation Redish, J. Saul, and R. Steinberg, Am. J. Phys. 66, 212–224 1998.
40%
60%
80%
100%
0% 20% 40% 60%
Agree (Unfavorable)
Dis
ag
ree
(F
av
ora
ble
)
Blog non-participants pre
Blog non-participants post
Blog participants pre
Blog participants post
Figure 3: A-D plot for Semester II (spring 2006) for question 7: "I will (or did) find it difficult to understand how physics applies in the real world."
Example of an Agree-Disagree Plot
Duda, G., Garrett, K., Am. J. Phys. 76, 1054 (2008).
40%
60%
80%
100%
0% 20% 40% 60%
Unfavorable Responses
Fa
vo
rab
le R
es
po
ns
es
Blog non-participants pre
Blog non-participants post
Blog participants pre
Blog participants post
Figure 5a: A-D plot for Semester II (spring 2006) for an average of the "reality link" questions.
Duda, G., Garrett, K., Am. J. Phys. 76, 1054 (2008).
Analyzing Interval Data
• Basic idea here is assign a numerical value to each response
• Strong Disagree = -2 (or 0)• Weakly Disagree = -1 (or 1)• Neither Agree/Nor Disagree = 0 (or 2)• Weakly Agree = 1 (or 3)• Strong Agree = 2 (or 4)
• Sum the responses then analyze using standard statistical techniques
Simple (student) t-test
• The t-test is a simple (but robust) statistical test
• Tests a hypothesis: Is there a difference between two sets of data?– Are differences statistically significant?– 95% confidence level, i.e. only a 5%
probability the difference is due to statistical fluctuations
Example: The “Gender Gap” in Intro Physics
Students Normalized Gain Statistically Significant?
Control Group
Males (n=82) 0.67
Females (n=63) 0.50
Experimental Group
Males (n=50) 0.63
Females (n=96) 0.56
Is there a difference between male and female students?
Which image is random?
Sometimes our eyes can deceive us!And sometimes we think things are true because we’d like them to be true …
The “Gender Gap”: FMCE Gains
Students Normalized Gain P-value
Control Group
Males (n=82) 0.67 < 0.05
Females (n=63) 0.50 (significant)
Experimental Group
Males (n=50) 0.63 0.165
Females (n=96) 0.56 (not significant)
In the experimental group, there is no statistically significant difference between the two genders.
Student’s t-test
• Assumptions:– Each data set follows a normal distribution
• Parameters:– One-tailed vs. two-tailed– Types: paired, two-sample equal variance,
and a two-sample unequal variance test
• Can have different # of data points if conducting an unpaired test
Two Sample t-test
Group A Group B1.02 0.560.65 0.151.01 0.321.27 0.940.59 0.350.75 0.530.62 0.360.65 0.310.66 0.35
t-test output Group A Group BMean 0.801 0.428Variance 0.057 0.051Observations 9.000 9.000
Pooled Variance 0.054
Hypothesized Mean Difference 0.000df 16.000t Stat 3.404
P(T<=t) one-tail 0.002
t Critical one-tail 2.583
P(T<=t) two-tail 0.004
t Critical two-tail 2.921
Here p < 0.05, so the null hypothesis is falsified – statistical difference between Group A and Group B
Measuring Effects of Instruction
• Suppose you apply some educational innovation– Control group and experimental group– Or pre-test and post-test
• How do you know if it’s effective?
• Say you give some sort of standard assessment– How big do the changes need to be to be
statistically significant?
Method #1: Use a t-test
• You can always use a t-test
• Compare scores of control vs. experimental group
or
• Compare pre vs. post tests– More difficult due to other variables
Method #2: Effect Size
• Effect Size (ES) is a method to quantify how effective an educational intervention has been relative to a control group
• Extremely useful when there is no familiar scale to judge outcomes
A thought experiment
• Suppose we do a study to see if children learn better in the morning or afternoon
• Morning trial: 15.2 average on assessment• Afternoon trial: 17.9 average on assessment
• Is this a big difference? It depends on overlap!
Robert Coe: “What is an Effect Size: A guide for users”
Two distributions
If the distributions of scores looked like this, you would think the result is quite significant
Robert Coe: “What is an Effect Size: A guide for users”
Two distributions
But if the distributions of scores looked like this you wouldn’t be so impressed
Robert Coe: “What is an Effect Size: A guide for users”
Effect Size Continued
• The Effect Size– Compares the difference between groups in
light of the variance of scores within each group
ES= (mean of experimental group) – (mean of control group)
Standard Deviation
Actually quite simple to calculate
Robert Coe has great information online about ES
How to Interpret Effect Size
Effect Size % of control group below
mean of experimental
group
Probability you could guess
which group an individual belongs to
based on their score
0.0 50% 0.50
0.2 58% 0.54
0.4 66% 0.58
0.6 73% 0.62
0.8 79% 0.66
1.0 84% 0.69
2.0 98% 0.84
Robert Coe: “What is an Effect Size: A guide for users”
How to Interpret Effect Size
Effect Size How Large? Equivalent to the height
differences between
0.2 Small 15 and 16 yr old girls
0.5 Medium 14 and 18 yr old girls
0.8 Large 13 and 18 yr old girls
IQ differences between typical freshmen and Ph.D.s corresponds to an effect size of 0.8
Robert Coe: “What is an Effect Size: A guide for users”
Making a better survey
• In my experience surveys and assessment instruments are difficult to write
• How do you know your instrument is1. Reliable
2. Valid
• Are there alternatives to writing your own instruments?
Reliability: Cronbach Alpha
• Cronbach Alpha: measure of how closely items in a group are related
• Cronbach Alpha is often used for instruments which are not marked right or wrong– Think Likert Scale
• Measures if students responses are the same for similar types of questions
How to Cronbach Alpha
• You could calculate it by hand
• or you buy SPSS and figure out how to use it
• or you could download an excel spreadsheet which is programmed to do this: http://www.gifted.uconn.edu/siegle/research/Instrument Reliability and validity/reliabilitycalculator2.xls
Cronbach Alpha Values
• Typically a Cronbach Alpha (α) > 0.8 is considered good– At this level survey is “reliable”
• However, there are exceptions:– Different types of surveys/instruments may
have different natural levels of reliability– Experimental instruments may be still useful
even if α~0.6
Warning! Common Mistakes with Cronbach Alpha
• Paper: “Calculating, Interpreting, and Reporting Cronbach’s Alpha Reliability Coefficient for Likert-Type Scales” by Joseph A. Gliem and Rosemary R. Gliem
• Lesson– Use Cronbach Alpha for Likert scale surveys– Draw conclusions based on clusters of items– Single item reliability is generally very low
Instrument Validity
• Validity is never universal– “Valid” for a certain population and for a
specific purpose
• Three general categories of validity:1.Content validity
2.Predictive validity
3.Concurrent validity
Ideas for Establishing Validity
1. Establish content or face validity– Correlate with other independent measures
such as exam scores, course grades, other assessment instruments
2. Predictive validity– Longitudinal studies and student tracking are
needed here
3. Concurrent validity– Compare with other assessment instruments or
calibrate with the proper groups
Survey/Assessment Creation Tips
• Build in measures to show reliability– e.g. multiple questions within a survey on the
same topic (both positive and negative)– Questions that establish that students are
taking the survey seriously
• For content driven assessments, research student difficulties
• Beta-version: open ended questions
• Correlations can help show validity
Buros Institute of Mental Measurement
• “By providing professional assistance, expertise, and information to users of commercially published tests, the Institute promotes meaningful and appropriate test selection, utilization, and practice.”
http://www.unl.edu/buros/bimm
Conclusion
• Some simple statistical tests can provide rigorous evidence of– Student learning– Instructional effectiveness– Improvements in attitude
• All of these methods are extremely effective when coupled with qualitative methods
• Stats involved can be done with little or no training
My SoTL advice
• Plan a throw-away semester in any SoTL study– “trial” period to tinker with your study design– Flexibility to alter your study design when you
find it doesn’t work
• Involving students in SoTL work can be very effective
• Try to publish in discipline specific journals• When in doubt, ask your students!
Good References
• Analysis of Likert Scales (and attitudinal data in general) – CLASS survey– http://www.colorado.edu/sei/class/
• Effect Size:– “What is an Effect Size: A guide for users” by
Robert Coe (easily found by google)– Coe also has an excel spreadsheet online to
calculate effect size
Good references
• Reliability and Validity:– http://www.gifted.uconn.edu/siegle/research/I
nstrument%20Reliability%20and%20Validity/Reliability.htm
– http://www.gifted.uconn.edu/siegle/research/Instrument%20Reliability%20and%20Validity/validity.htm
• T-test– Step by step video on excel:
http://www.youtube.com/watch?v=JlfLnx8sh-o
Good References
• The FLAG: Field-Tested Learning Assessment Guide– www.flaguide.org– Contains “broadly applicable, self-contained
modula classroom assessment techniques (CATs) and discipline-specific tools for STEM instructors”