EQUIPE Plus Survey and questionnaire Mary Claire Halvorson Director of Professional Development Goldsmiths University of London Paris 17 November 2006
Dec 25, 2015
EQUIPE Plus Survey and questionnaire
Mary Claire Halvorson
Director of Professional Development Goldsmiths University of London
Paris 17 November 2006
Before you send it out Decide the purpose Develop appropriate questions Think about how you will analyse your
data Pilot the questionnaire
Informally: on colleagues / experts Formally (ideally): try it out on a sample of the
target audience Answer the questions and comment on them:
are they clear and unambiguous?
Deciding the purpose Hypothesis generating
Gaining a better understanding of issues without / with few preconceptions
Estimating population parameters E.g. how many people involved other colleagues
internally? Hypothesis testing
Were women more likely than men to work collaboratively?
Test development (e.g. creating a measurement of extraversion)
Test blueprints (1) If you need to cover all aspects of
something (e.g. a motivation construct)
Test blueprints (2) E.g.
Content areas might be subscales relevant to the construct you are measuring (e.g. extraversion)
Manifestations might be behavioural (e.g. what role you are willing to do and how you network informally at project dinners), cognitive (what you think about certain EU countries), affective (how you feel about working with certain people)
Number of questions How many will you respondents be
willing to answer? Ensure you have all the information
you need, including demographic Weighting: do you want to ask more
about some areas than others?
Response formats Closed questions/responses
Designed to get a response from a limited range of options, e.g.
• Yes/no• Do you work as an adult educator?
• Multiple choice• Rating scales (e.g. rate yourself from 1 to 10)
Open questions/responses Unpredictable or wide range of answers, e.g.
• How did you spend last year on the project?• What is your occupation? (vague)
Mixed Closed question plus space for elaboration
Closed responses Advantages:
Easy to analyse: can simply count the responses
Prevents vague/ambiguous responses Usually easier for the respondent to fill in
Disadvantages Forces response (may lose richness) Requires knowledge of possible responses More susceptible to response biases, e.g.
• Aquiescence• Response affected by options available
Open responses Richer information; may give new insights You do not need to know all the possible
answers Harder to analyse
But you can devise categories after seeing the responses
People more likely to interpret the question in differing ways In questionnaires, you cannot normally get
follow-up or clarification
Wording of questions (1) Even closed questions might not be as
simple as you think, e.g. Are you: (a) unemployed, (b) a student, (c)
employed (d) retired• How will someone respond if they are retired and a
student? On disability benefit?• Think about the unusual members of your population• Aim for exhaustive and mutually exclusive categories
(include ‘other’ and ‘don’t know’?)• or ask “Which of the following best describes you?” or
“… is most applicable to you” • or “Tick all that apply” (remember to think about how
you will analyse/present the outcome)
Wording of questions (2) Avoid vague or ambiguous questions
E.g. 1: ‘I am quite happy’• ‘Quite’ is subject to subjective interpretation• If someone disagrees, is it because they are unhappy
or because they are very happy? E.g. 2: How did you spend last Christmas?
• Some people may answer about Christmas Day, some about Christmas Day and Boxing Day, some about a longer period
• Different kinds of answers• ‘with my family’• ‘watching television’• ‘in Paris!’
Wording of questions (3) Avoid ill-defined terms
E.g. my Rector champions the Equipe Plus mission
Avoid jargon, technical terms and acronyms
(hard to do in EU working!)
Wording of questions (3) Avoid leading questions
E.g. “Do you agree that the University benefited from involvement?”
Instead: “The University benefited from involvement: agree/disagree”
Do not combine questions E.g. “The war in Iraq was unjustified and my institution
is interested in grundvig” Beware of hidden assumptions
E.g. “I have done something useful s coordinator for partners in the last week”
• Assumes everyone has been a coordinator rather than just a partner
Wording of questions (4) As simple and short as possible Self explanatory Grammatically correct Unambiguous Avoid double negatives
E.g. My Rector is not unclear Beware of offensiveness, and consider the
full range of people who may answer your survey
Wording of questions (5) If possible, use questions that have
already been shown to work Also facilitates comparisons with
previous studies
Some other issues about questions Response bias
E.g. acquiescence – a tendency to answer ‘yes’ Social desirability
E.g. “I plagiarise things”: you will not get a true response rate!
Order of questions may affect response Best to put more general questions first
Type of scale may affect response Keep balanced, e.g. not “very
happy/happy/not happy”
Questionnaire: explanations You may want to include:
Purpose of the study• If possible, what’s in it for the respondent
Who is responsible for the study Why and how participant was selected Confidentiality/anonymity How much time it will take
Questionnaire layout: instructions Clear and unambiguous How to choose a response and how to
indicate it on the form Do they need to respond to all items? What do do if unclear?
E.g. leave blank, tick the response that most applies to them, explain the problem
Confidentiality and anonymity
Questionnaire layout: questions Spacing (at least 1.5 lines
recommended) Consider guides from items to
responses, (e.g. … … …) Number each item
Questionnaire layout: general Aim for a logical order of questions Consider the appearance of the
finished product: does it look as simple as possible, not daunting?
“Thank you” at the end
Validity of tests These may not all apply if your
questionnaire is not a test. Face validity (are items meaningful to respondents?) Content validity (are items meaningful to experts in the
field?) Convergent validity (does the test correlate with other
tests of the same construct?) Divergent validity (does test differ from scores on other
construct?) Criterion validity (does the test correlate with actual
behaviour?) Construct validity (has the construct you are measuring
been shown to exist?)
Missing data
Are your respondents typical of the population? (selection bias, response bias)
If respondents did not answer all the questions, why? Does this suggest that the people who did answer the questions are not typical of the whole population?
Screening data
Always check you have entered your data correctly
If you have a lot of data, screen it: E.g. if the possible responses are 1-5, are
there any in your data outside that range? If you used (say) 0 or 9 for ‘no response’
make sure that you don’t include it in any averages!
Presenting the outcome (1) Simple descriptive statistics (e.g. 7%
of respondents felt the project had no benefit for their University)
Cross-tabulations, e.g.
Spent ChristmasAlone With others
Males 10 90Females 2 93
Presenting the outcome (2) Inferential statistics, e.g. chi-squared
tests Do think about the issue of multiple
comparisons, especially if you are deciding post hoc what to compare
Presenting the outcome (3) Graphs Bar charts (e.g. one I did earlier)
0% 20% 40% 60% 80% 100%
Shared ha ll/house etc
Clubs & Socs
Research Group
Department
RCS
Imperial College
Agree Tend to agree no answ er etc Tend to disag ree Disagree
Figure C2: Who respondents were interested in keeping in touch with
(those who have been postgraduates or staff members; 89 respondents)
Presenting the outcome (4) Pie charts, e.g.
Favourite supermarket of respondents
Waitrose
Tesco
Morrison
Sainsbury
Other
Rudyard Kipling used a set of questions to help trigger ideas and solve problems and immortalised them in a poem.
These questions can be used as stimuli to get thinking going in many situations.
I have six honest serving menThey taught me all I knewI call them What and Where and WhenAnd How and Why and Who
Further reading Questionnaires (and many other topics)
Sommer and Sommer (1991): a practical guide to behavioural research (3rd ed).
Statistics Coolican (2004): Research methods and
statistics (4th ed.)