A-team Spring Session #2 Questionnaire Design February 28, 2007
Jan 05, 2016
A-team Spring Session #2Questionnaire Design
February 28, 2007
Questionnaire or Survey?
• Questionnaire is an actual instrument: I.e. web questionnaire (Perseus)
• Survey is actually a verb/method: “to study or examine comprehensively”
• Your questionnaire is actually “surveying”
• Often used interchangeably
Rea & Parker (1997)
Yields Different Types of Data:
• Descriptive– Socioeconomic parameters to better understand the
larger population represented by the sample. (e.g., income, age, college/school, major, class standing)
• Behavioral– Patterns of use, recreation, entertainment, personal
behavior. (e.g., # UGA bus rides per day/week)
• Preferential– Opinions & preferences about socio-political issues.
(e.g., opinion about new parking plan)
Rea & Parker (1997); Schuh & Upcraft (2001)
Types of Responses/Measurement Scales
• Nominal Scales – Used to categorize objects; “name” them– Object is in a category or it is not– No order implied along any dimension– Response sets that are nominal:
• Yes/No (dichotomous)• Can be “Choose one,” “Choose all that apply,”
“Choose one and specify,” or “Choose all and specify” from listing of characteristics
Schuh & Upcraft (2001)
Examples of Nominal Scales
• Immediately after aerobic exercising I generally
feel: ___Exhausted ___Invigorated ___Thirsty
___Sweaty ___Overheated ___Nauseated
[Note: make sure (+) and (-) options offered in listing]
• Indicate your sex: ___Male ___Female
• Have you ever resided in Brumby Hall?
___Yes ___No
Schuh & Upcraft (2001)
Measurement Scales (Cont’d)
• Ordinal Scales (a.k.a. rank, order, rank-order)
– Used to rank objects according to amount of characteristic the object possesses
– Order reflects varying amounts or levels– Rank reflects range from high to low amounts– Ranking has no absolute zero– Intervals from one rank to next not the same– Likert scales are ordinal but sometimes treated as
interval scales (judgment call)
Schuh & Upcraft (2001)
Examples of Ordinal Scales
• Order of finish in a horse race
• Rank in class (e.g., achievement)
• Highest degree earned
• Order of preference
• A higher number indicates a higher rank, e.g., “more” of characteristic possessed
• Watch for (reverse) coding
Schuh & Upcraft (2001)
Examples of Ordinal Scales
• Rank-order your on-campus living preferences for the next academic year, with 1 = first priority, 2 = second priority, and so on:
___ Single Room ___One-bedroom Apartment ___ Double Room ___ Multiple-bedroom Apartment ___ Suite (2 double rooms separated by a bathroom)
• Rank-order your reasons for attending this workshop, with 1=most influential reason, 2=second greatest influence, and so on:
___ Surveys are my life ___My boss sent me ___ To get the handouts ___ My thirst for knowledge ___ To earn CEUs ___ To get the free gift
Schuh & Upcraft (2001)
Measurement Scales (Cont’d)
• Interval Scales– A ranking/rating using interval score values– The difference between intervals is equal– The difference between 1 & 2 is the same as the
difference between 4 & 5– Still a focus on the amount of a characteristic an
object possesses– Likert-type (pronounced Lick-ürt) scales often treated
like interval scales (although considered ordinal):• 5=strongly agree, 4=agree, 3=no opinion, 2=disagree,
1=strongly disagree
Schuh & Upcraft (2001)
Examples of Interval Scales
• Likert Scale Example:Parking on campus should be free.__Strongly Agree __Agree __Neither Agree nor
Disagree __Disagree __Strongly Disagree
• Non-Likert Scale Example– When driving a UGA van, the safest following
distance under ideal conditions (in seconds) is__1.5 __3 __4 __8 __10 __25
Schuh & Upcraft (2001)
Examples of Interval Scales
• Non-Likert Scale Example (cont’d): Please rate your satisfaction with the following
student activities on a scale of 1 to 5, with 1=very dissatisfied, 2=dissatisfied, 3=neither dissatisfied nor satisfied, 4=satisfied and 5=very satisfied. [could add not applicable or did not attend option].
__Welcome Week __Dawgs After Dark __Movie-O-Rama __Concert on the Quad __ Halloween “I Vant to Drink Your BlooooooDrive”
Rea & Parker (1997); Schuh & Upcraft (2001)
Measurement Scales (Cont’d)
• Ratio Scales– Empirically meaningful zero/absolute zero; true
absence of characteristic; (e.g., height, weight; more common in physical/biological sciences)
– Have all characteristics of nominal, ordinal, and interval scales
– Can be converted to ordinal scales– Can be converted to categories– Education examples: income, age, # years of
education, # meetings with academic advisor
Schuh & Upcraft (2001)
Examples of Ratio Scales
• Indicate the number of times you accessed the University Health Center in the last 30 days: ___
• Indicate your age: ___
DeVellis (1991, p. 68-70))
Likert Scales
• Present question/item stem in a declarative sentence (one statement under consideration).
• Response options represent varying degrees of agreement or endorsement of one statement.
• Response options should be worded to represent approximately equal intervals; use equal # positive and negative possibilities.
• The question stem doesn’t have to span the range of the construct (as in Thurstone or Guttman); response options infer levels of phenomena.
DeVellis (1991, pp. 68-70)
Likert Scales
• Often 5, 7, or 9 response-options sets • A 6 response-options set is also common
– Strongly disagree– Moderately disagree– Mildly disagree– Mildly agree– Moderately agree– Strongly agree
DeVellis (1991, pp. 68-70)
Likert Scales
• Midpoint often used but optional• What does midpoint wording imply?
– Neither agree nor disagree: Apathy?– Agree and disagree equally: Strong paradox?
• Common midpoint wording– Neither agree nor disagree– Agree and disagree equally– Neutral
DeVellis (1991, pp. 68-70)
Likert Scales
• Most used in surveys of opinions, beliefs, attitudes
• Useful if statements are fairly strong (but not extremely)
• Everyone can agree, have no opinion, or have little opinion about a mild statement
• Write clear statements that reflect true differences of opinion
DeVellis (1991, p. 70)
Likert Scale Examples
• Exercise is an essential component of a healthy life-style.1=Strongly Disagree, 2=Moderately Disagree, 3= Mildly
Disagree, 4=Mildly Agree, 5=Moderately Agree, and 6=Strongly Agree
• Combating drug abuse should be a top national priority.1=Completely True, 2=Mostly True, 3=Equally True and
Untrue, 4=Mostly Untrue, and 5=Completely Untrue
DeVellis (1991, p. 70-71)
Semantic Differential Scales
• Response options consist of one but usually several adjective pairs
• One adjective is negative, the other positive; each serves as the (-) or (+) end of a continuum that characterizes the stimulus
DeVellis (1991, pp. 70-71)
Semantic Differential Scales
• Individual lines/points are placed between the two extremes (adjectives)
• 7 or 9 lines/points are common• Respondents check/select lines or points closest
to the adjectives if they hold extreme views• Respondents check/select lines or points toward
the middle of the continuum if they hold more moderate views
DeVellis (1991, pp. 70-71)
Semantic Differential Examples
• Automobile Salesmen• Honest __ __ __ __ __ __ __ Dishonest • Quiet __ __ __ __ __ __ __ Noisy• Friendly __ __ __ __ __ __ __ Not Friendly• Fair __ __ __ __ __ __ __ Unfair• Trustworthy __ __ __ __ __ __ __ Untrustworthy
Length of a Survey
• Sufficient to capture needed data• Short enough to hold participants’ attention• Type of survey affects length• Types of questions affect length• Quantitative/Qualitative/Mixed approach affects
length• Participant characteristics affect length
Measurement Scales: More Tips
• Avoid providing categories/options that overlap; difficult or impossible to analyze
• Frequently happens with age, income, class hours, years of service, hours worked, etc.
• Example:– Select the category that best describes your
annual, gross income: __$0-$10,000 __$10,000 – $30,000 __$30,000 - $60,000
DeVellis (1991); Schuh & Upcraft (2001)
Measurement Scales (Cont’d)
• Be thoughtful with Use of “other” or “does not apply” or “not applicable” in listing of characteristics/options
– Positive: Obtain option you may not have considered– Positive: Prevents forced responses – Negative: Can give response already listed or
spurious data– Potential Negative: Adds to analysis time
AnalysisType of Data Measure of
Central Tendency
Measure of Variability
Examine Responses of 1 Group to 1
Question
Interval
Ratio
Scale
Mean or Median(if skewed)
Variance or
Standard
Deviation
Frequency Distribution Bar or Line Graphs
Ordinal
Rank
Rank-Order
Median Frequency Distribution
Bar Graph
Line Graph
Freq. Dist.
Bar or Line Graphs
Nominal
Categorical
Dichotomous
Mode Frequency Distribution
Bar Graph
Freq. Dist.
Bar Graphs
Schuh & Upcraft (2001); Upcraft & Schuh (1996)
Analysis (Cont’d)Type of Data Compare
responses of 2 Grps to 1 Q
Compare responses of 1 Grp to 2 Qs
Degree of Relshp Bet. Response to
2 Qs
Degree of Relshp. Among responses to 3
or > Qs
Interval
Ratio
Scale
Paired Frq.
Distribution or Graphs
Scattergram Pearson’s
Product Moment Correlation
Multiple Corr. Partial Correl.
Ordinal
Rank
Rank-Order
Paired Frq.
Distribution
or Graphs
Cross-tab
Table
Spearman’s Rank-Order Correlation Coeff.
Kendall’s Partial Rank Correl.
Nominal
Categorical
Dichotomous
Paired Frq.
Distribution or Graphs
Cross-Tab
Table
Cramer’s Index of Contingency
References• DeVellis, R. F. (1991). Scale development: Theory and applications. Newbury Park,
CA: Sage.
• Miller, T. K., (1999). CAS: The book of professional standards for higher education. Washington, DC: Council for the Advancement of Standards in Higher Education.
• Payne, D. (1992). Measuring and evaluating educational outcomes. New York: Macmillan.
• Rea, L. M. & Parker, R. A. (1997) Designing and conducting survey research (2nd Ed.). San Francisco: Jossey-Bass.
• Schuh, J. H. & Upcraft, M. L. (2001). Assessment practice in student affairs: An applications manual. San Francisco: Jossey-Bass.
• Upcraft, M. L. & Schuh, J. H. (1996). Assessment in student affairs: A guide for practitioners. San Francisco: Jossey-Bass.