Questionnaire Design 2012

8/10/2019 Questionnaire Design 2012

1/37

Research Ski l ls One

Oh, no!Please, no!

Not statistics!!


2/37

A ims o f the two f i rst year Research Ski lls

courses:

Basic gro und ing in research sk i l ls -

designing q uest ionnai res and exper iments

data analysis us ing stat ist ics, Excel and SPSS

wri t ing up stud ies using s tandard presentat ion convent ions


3/37


4/37


5/37


6/37

Comparison of RM1 exam marks for 2004-2008

0

5

10

15

20

25

30

35

40

0-10 11-20 21-30 31-40 41-50 51-60 61-70 71-80 81-90 91-100

exam mark

frequency(%)

2004 (N = 249)

2005 (N = 246)2006 (N = 243)

2007 (N = 214)

2008 (N = 234)


7/37


8/37

Outl ine of this c ou rse (Research Ski l ls One):

Issues in designing questionnaires and experiments.

Descriptive statistics - effective summarising of data.

Statistical tests (correlation, Chi-Square, nonparametric tests for

comparing groups or conditions).

Using SPSS("Statistical Package for the Social Sciences") for data

analysis and Excelfor producing graphs and tables.


9/37

Research Ski l ls Two:

More statistical theory.

More statistical tests.

Practice at running experiments and writing them up.

Ethics.


10/37

Organisat ion for th is cou rse:

One lecture and one practical session every week.

(Lectures: Thursdays at 1pm in Chichester Lecture Theatre

until week 12December 6th).

Practical sessions - details on Sussex Direct (you are split into

lots of smaller groups).


11/37

Course requi rements:

Assessment is by a mixture of coursework and

exam:

Coursework

SONA research participation (4 hours this term): 10%

Lab-report 1 10%

Lab-report 2 20%

Exam -

Unseen exam (in winter assessment period): 60%

Overal lpass mark (exam and coursework combined): 40%


12/37

This term's requi rements:

1. Write a lab-report based on a questionnaire from the STARS fast

food project (10%).

2. Write a lab-report based on a questionnaire on maths learning

experiences (20%).

3. Tackle the weekly statistics problem sheets (not marked, but

essential practice).

4. Four hours of research participation (10%).

5. Unseen 2-hour exam at the end of the course

statistical concepts (multiple choice questions);

deciding which statistical test is most appropriate;

interpreting statistical output from SPSS.


13/37

Recommended textbook s:

Field and Hole (2003). Sage publications.

Palgrave, 2006

Sage, 2006

BUY THIS ONE, I NEED THE MONEY !!!!!
http://www.amazon.co.uk/gp/product/images/0230362729/ref=dp_image_z_0?ie=UTF8&n=266239&s=bookshttp://www.amazon.co.uk/gp/product/images/0230362729/ref=dp_image_z_0?ie=UTF8&n=266239&s=books


14/37


15/37

Perm it ted calculators:

Casio FX82, 83, 85, 115, 570 or 991, all with any suffix.

No oth er type of calculator is acceptable.


16/37

Course documentat ion, hando uts, lecture sl ides:

Go to Sussex Direct - this course is

Research Skills One (C8511)

OR

Type Graham Hole into Google


17/37

Click on

this link


18/37

Everything

you need to

know for this

course


19/37

Who to con tact :

In the first instance, the course tutors running your practical

session.

Graham Hole ([email protected])


20/37

Quest ionnaire design :

Perhaps now you'll fill in

my questionnaire?


21/37

How does the way you collect data affect the datacollected?

How does question wording affect peoples answers?


22/37

Stages in quest ionnaire design :

5. Adm inister the quest ion naire, after i t has b een revisedin th e l igh t o f (4).

1. Formulate the research quest ion(s) clearly.

2. Ident i fy the popu lation and samp le.

3. Design the quest ionnai re: th ink about quest ion

wo rding ; quest ionn aire formatt ing; mode ofadm inistrat ion ; data analysis.

4. Pre-test the questionnaire.


23/37

Samples and popu lat ions:

Sample: a subset f rom a

po pu lat ion (e.g. f irs t-year

psy cho logy students).

Populat ion:a complete set

of th ing s (e.g. al l of

humanity).

For val id inferences to be made abou t a populat ion 's

characterist ics, a sample mu st be representat ive of i ts

paren t pop ulat ion (e.g. sim ilar in age, SES, IQ, etc.)


24/37

Methods of obtain ing quest ionnaire data:

Postal

questionnaire

Personal

interview

Phone

interview

Internet

questionnaire

Cost Low High Moderate Low

Data quality:

Respo nse r ate

Respondent

mot ivat ion

Interviewer bias

Low

Low

None

High

High

Moderate

Moderate

High

Low

Low

Low

None

Sample quality: Low, unless

high responserate

High Moderate to

high,especially

with random

digit dialling

Moderate, but

improving asInternet access

widens


25/37

Postal

Questionnaire

Personal

interview

Phone

interview

Internet

questionnairePossible interview

length:

Short Very long Long Short

Ability to clarify

and probe:

None High High None

Anonymity: High Low Low Low

Dependence on

respondents

literacy:

High None None High

Control of context

and questionorder:

None High High Depends


26/37


27/37

Quest ionnaire wording :

1. Should be exact.

2. Should be simple.

3. Avoid biased or emotive words.

Schuman and Presser (1981): subtle changes of wording

may influence responses.

e.g. Should the Governmentallow publ ic speeches by a

Communis t? produced 25% fewer pro-free-speech

responses when allowwas replaced with forb id .


28/37

4. Make all alternatives clear.

e.g. Payne (1951): "Do you th ink most manufactur ing

com panies that lay of f wo rkers dur ing slack per iods cou ldarrange thing s to avoid layof fs and give steady work

througho ut the year?"

63% - companies could avoid layoffs.

22% - couldnt avoid layoffs.

15% - no opinion.

Same question plus phrase "or do you think layof fs are

unavoidable?"

35% - companies could avoid layoffs.

41% - couldnt avoid layoffs.

24% - no opinion.


29/37

5. Avoid the format: "Some people say x: do y ou agree ordisagree?"

6. Avoid unwarranted assumptions. e.g. "What is youroccupation?" assumes person has a job.

7. Avoid double-barrelled questions. e.g. "Should immigrants

be repatriated and their possessions confiscated?" is twoquestions.

8. Avoid double negatives. e.g. "Are you against a ban onsmoking?"

9 C id h l i i f d


30/37

9. Consider the relative merits of open-endedand

closed-endedquestions.

Open-ended:allow unconstrained responses.

e.g. "How do you travel to the Univers i ty?".

May produce richly detailed responses, but hard and tediousto score.

Closed-ended:require choice from a limited range ofalternatives.

e.g. "Do you travel to the Univers i ty by

(a) bu s, (b) car, or (c) un icy cle (t ick one)".

Easy to code, but prone to bias.


31/37

Closed-ended questions must have

(a) a balanced response scale;

(b) mutually exclusive categories;

(c) facilities for handling "don't know" and "other"

responses.

R ti l


32/37

Rating scales:

The Likert Scale:

"Criminals should be flogged".Strongly agree Agree Neither agree

nor disagree

Disagree Strongly disagree

1 2 3 4 5

Can be 5-. 7- or 9-point scale (doesn't make muchdifference).

Visual Analogue Scale:

Strongly agree Strongly disagree


33/37

Prob lems w i th quest ionnai res abou tatt i tudes:

1. May not have an attitude - "doorstep opinions".

2. Attitudes may be complex and multi-dimensional.

3. Attitudes vary in intensity.

4. Expressed attitudes may depend on question wording,

sequence and interviewer effects.

Prob lems w i th quest ionnaires abou t


34/37

Prob lems w i th quest ionnaires abou tbehaviour:

1. Memory l imi tat ions-

e.g. Chapman and Underwood's (2000) study of drivers' memory for accidents

and near-misses.

Can be counteracted by

(a) asking specific questions;

(b) asking for birth date rather than age;

(c) using a chronological format;

(d) re-interviewing.

2. Response biasesdue to social desirability or suspicion,

especially for illegal or anti-social activities.

Can be counteracted by ensuring anonymity.

What are the prob lems w i th these quest ions?


35/37

5. Do your children prefer to eat in KFC or Macdonalds?

1. Do you visit fast food emporia regularly?

Over-complex wording. Exactly what does "regularly" mean?

2. How many burgers do you eat per month?

Assumes you eat burgers.

3. Some people suggest that fast food is leading to increased tooth decay and an

increase in obesity amongst teenagers in many parts of the U.K. Do you agree?

Over-long. Includes two separate questions. Implies you should agree with the viewsexpressed.

4. Which of the following methods do you use to travel to your fast-

food outlet? (a) Bus (b) Car (c) Bicycle

Does the questioner want you to choose only one option, or can you

choose more? No option for responding "other" .

What are the prob lems w i th these quest ions?

Assumes you have children.


36/37

Conclus ions:

It's hard to design questionnaires properly!

Always be sceptical of survey results -ask yourself

Who were they collected by?

Who were they collected from?

How were the questions worded, exactly?

Remember - 8 out of 10 cats prefer Whiskas" has

become "In tests, 8 out of 10 cats who expressed apreference, preferred Whiskas".

Useful references:


37/37

Useful references:

Burgess, T.F. (2001). A general int roduct ion to the design of q uest ionnaires

for su rvey research.

http://iss.leeds.ac.uk/info/312/surveys/217/guide_to_the_design_of_questionnaires

Taylor-Powell, E. (1998). Quest ionnaire design: asking quest ions w ith a

purpose.http://learningstore.uwex.edu/pdf/G3658-02.pdf

Questionnaire Design 2012

Documents