EQUIPE Plus Survey and questionnaire Mary Claire Halvorson Director of Professional Development Goldsmiths University of London Paris 17 November 2006.

EQUIPE Plus Survey and questionnaire

Mary Claire Halvorson

Director of Professional Development Goldsmiths University of London

Paris 17 November 2006

Before you send it out Decide the purpose Develop appropriate questions Think about how you will analyse your

data Pilot the questionnaire

Informally: on colleagues / experts Formally (ideally): try it out on a sample of the

target audience Answer the questions and comment on them:

are they clear and unambiguous?

Deciding the purpose Hypothesis generating

Gaining a better understanding of issues without / with few preconceptions

Estimating population parameters E.g. how many people involved other colleagues

internally? Hypothesis testing

Were women more likely than men to work collaboratively?

Test development (e.g. creating a measurement of extraversion)

Test blueprints (1) If you need to cover all aspects of

something (e.g. a motivation construct)

Test blueprints (2) E.g.

Content areas might be subscales relevant to the construct you are measuring (e.g. extraversion)

Manifestations might be behavioural (e.g. what role you are willing to do and how you network informally at project dinners), cognitive (what you think about certain EU countries), affective (how you feel about working with certain people)

Number of questions How many will you respondents be

willing to answer? Ensure you have all the information

you need, including demographic Weighting: do you want to ask more

about some areas than others?

Response formats Closed questions/responses

Designed to get a response from a limited range of options, e.g.

• Yes/no• Do you work as an adult educator?

• Multiple choice• Rating scales (e.g. rate yourself from 1 to 10)

Open questions/responses Unpredictable or wide range of answers, e.g.

• How did you spend last year on the project?• What is your occupation? (vague)

Mixed Closed question plus space for elaboration

Closed responses Advantages:

Easy to analyse: can simply count the responses

Prevents vague/ambiguous responses Usually easier for the respondent to fill in

Disadvantages Forces response (may lose richness) Requires knowledge of possible responses More susceptible to response biases, e.g.

• Aquiescence• Response affected by options available

Open responses Richer information; may give new insights You do not need to know all the possible

answers Harder to analyse

But you can devise categories after seeing the responses

People more likely to interpret the question in differing ways In questionnaires, you cannot normally get

follow-up or clarification

Wording of questions (1) Even closed questions might not be as

simple as you think, e.g. Are you: (a) unemployed, (b) a student, (c)

employed (d) retired• How will someone respond if they are retired and a

student? On disability benefit?• Think about the unusual members of your population• Aim for exhaustive and mutually exclusive categories

(include ‘other’ and ‘don’t know’?)• or ask “Which of the following best describes you?” or

“… is most applicable to you” • or “Tick all that apply” (remember to think about how

you will analyse/present the outcome)

Wording of questions (2) Avoid vague or ambiguous questions

E.g. 1: ‘I am quite happy’• ‘Quite’ is subject to subjective interpretation• If someone disagrees, is it because they are unhappy

or because they are very happy? E.g. 2: How did you spend last Christmas?

• Some people may answer about Christmas Day, some about Christmas Day and Boxing Day, some about a longer period

• Different kinds of answers• ‘with my family’• ‘watching television’• ‘in Paris!’

Wording of questions (3) Avoid ill-defined terms

E.g. my Rector champions the Equipe Plus mission

Avoid jargon, technical terms and acronyms

(hard to do in EU working!)

Wording of questions (3) Avoid leading questions

E.g. “Do you agree that the University benefited from involvement?”

Instead: “The University benefited from involvement: agree/disagree”

Do not combine questions E.g. “The war in Iraq was unjustified and my institution

is interested in grundvig” Beware of hidden assumptions

E.g. “I have done something useful s coordinator for partners in the last week”

• Assumes everyone has been a coordinator rather than just a partner

Wording of questions (4) As simple and short as possible Self explanatory Grammatically correct Unambiguous Avoid double negatives

E.g. My Rector is not unclear Beware of offensiveness, and consider the

full range of people who may answer your survey

Wording of questions (5) If possible, use questions that have

already been shown to work Also facilitates comparisons with

previous studies

Some other issues about questions Response bias

E.g. acquiescence – a tendency to answer ‘yes’ Social desirability

E.g. “I plagiarise things”: you will not get a true response rate!

Order of questions may affect response Best to put more general questions first

Type of scale may affect response Keep balanced, e.g. not “very

happy/happy/not happy”

Questionnaire: explanations You may want to include:

Purpose of the study• If possible, what’s in it for the respondent

Who is responsible for the study Why and how participant was selected Confidentiality/anonymity How much time it will take

Questionnaire layout: instructions Clear and unambiguous How to choose a response and how to

indicate it on the form Do they need to respond to all items? What do do if unclear?

E.g. leave blank, tick the response that most applies to them, explain the problem

Confidentiality and anonymity

Questionnaire layout: questions Spacing (at least 1.5 lines

recommended) Consider guides from items to

responses, (e.g. … … …) Number each item

Questionnaire layout: general Aim for a logical order of questions Consider the appearance of the

finished product: does it look as simple as possible, not daunting?

“Thank you” at the end

Validity of tests These may not all apply if your

questionnaire is not a test. Face validity (are items meaningful to respondents?) Content validity (are items meaningful to experts in the

field?) Convergent validity (does the test correlate with other

tests of the same construct?) Divergent validity (does test differ from scores on other

construct?) Criterion validity (does the test correlate with actual

behaviour?) Construct validity (has the construct you are measuring

been shown to exist?)

Missing data

Are your respondents typical of the population? (selection bias, response bias)

If respondents did not answer all the questions, why? Does this suggest that the people who did answer the questions are not typical of the whole population?

Screening data

Always check you have entered your data correctly

If you have a lot of data, screen it: E.g. if the possible responses are 1-5, are

there any in your data outside that range? If you used (say) 0 or 9 for ‘no response’

make sure that you don’t include it in any averages!

Presenting the outcome (1) Simple descriptive statistics (e.g. 7%

of respondents felt the project had no benefit for their University)

Cross-tabulations, e.g.

Spent ChristmasAlone With others

Males 10 90Females 2 93

Presenting the outcome (2) Inferential statistics, e.g. chi-squared

tests Do think about the issue of multiple

comparisons, especially if you are deciding post hoc what to compare

Presenting the outcome (3) Graphs Bar charts (e.g. one I did earlier)

0% 20% 40% 60% 80% 100%

Shared ha ll/house etc

Clubs & Socs

Research Group

Department

RCS

Imperial College

Agree Tend to agree no answ er etc Tend to disag ree Disagree

Figure C2: Who respondents were interested in keeping in touch with

(those who have been postgraduates or staff members; 89 respondents)

Presenting the outcome (4) Pie charts, e.g.

Favourite supermarket of respondents

Waitrose

Tesco

Morrison

Sainsbury

Other

Rudyard Kipling used a set of questions to help trigger ideas and solve problems and immortalised them in a poem.

These questions can be used as stimuli to get thinking going in many situations.

I have six honest serving menThey taught me all I knewI call them What and Where and WhenAnd How and Why and Who

Further reading Questionnaires (and many other topics)

Sommer and Sommer (1991): a practical guide to behavioural research (3rd ed).

Statistics Coolican (2004): Research methods and

statistics (4th ed.)