Questionnaires

Questionnaires

Bike with no hands - Helena Nelson

One look at you and I knewyou’d be able to ride a bike with no hands I tried it, of course, but could never do it.It was written all over your face that you would have practiced, bare legs, bloody knees,in the summer evenings, hours at a timewhen no-one was watching the mishaps, until

casually, coolly, at infinite ease,you’d ride, no handed, surveying the streets,as if you’d been born on a circus bike. I wish - but then, we are what we are.I drive with two hands, walk with both feetfirmly planted on sensible ground. AndI’ve got you. You can ride with no hands.

Today• A check on the way we think about the world• Quantitative vs Qualitative Research• The Carbon Footprint questionnaire• Questionnaires • OAP questionnaire and preparation• Tutorial on lit review assignment

Five assumptions

1. The Ontological Assumption

What is the nature of reality? What is real and knowable?

Reality is like the hardware of a

computer programme

What is real and knowable depends on

the software

Assumption Question

Ontological What is the nature Assumption of reality?

Quantitative Qualitative

Reality is objective Reality is subjective and singular, and multiple,

apart from the as seen by participants researcher in a study

Qualitative vs Quantitative

2. Epistemological assumption

What is the relationship of the researcher to that researched?

What we must do in order to know?

The outstanding researcher is all powerful

In order to know I must train myself to win “I want

to be a Millionaire”

Assumption Question

Epistemological What is the relationshipassumption of the researcher to that

researched?

Quantitative Qualitative

Researcher is Researcher interactsindependent from that with that being being researched researched

3. Axiological Assumption

Survival values – nature red in tooth and clawSocial Values – fitting in with societyTranscendent values – beyond self, for the greater good

Our values determine what we want to research

What is the role of values?

What is worth knowing and why?

Assumption Question

Axiological Assumption What is the role of values?

Quantitative Qualitative

Value-free and Value-laden and unbiased biased

Rhetorical AssumptionWhat is the language of research? How best to share our research with others.

Dramatic Appealing to the personalStretching the truthLarge group of believers present

Assumption QuestionRhetorical What is the language assumption of research?

Quantitative Qualitative

Formal InformalBased on set definitions Evolving decisionsImpersonal voice Personal voiceUse of accepted Use of accepted quantitative words qualitative words

Method-ological

assumptionWhat is the process of research?

What techniques will lead to

fruitful inquiries?

Research is complexThere is no single approachBefore you use any particular method you have to understand the assumptions behind it.Methodology deals with the assumptions behind specific methods of research

Assumption Question Methodological What is the process ofassumption research?

Quantitative QualitativeDeductive process Inductive processCause and effect Mutual simultaneous

shaping of factorsStatic design, categories Emerging design, categoriesidentified before study isolated during research Context-free Context-boundGeneralisations leading to Patterns, theories prediction and explanation developed for understandingAccurate and reliable Accurate and reliablethrough validity and through verification reliability

Qualitative Methods descriptive, theory building, meaning making

• Focus Groups• Interviews• Case Studies• Ethnographic Inquiry• Collaborative Inquiry• Grounded Theory• Qualitative analysis of documents, video,

observation, etcWhat are they?

Quantitative Methods measuring, theory confirmation, defining

• Experiments• Questionnaires and surveys• Psychometric Scaling and Tests• Quantitative analysis of

documents, video, observation, etc• Audits

What are they?

Mixed Methods• Research that uses both qualitative and quantitative

methods – e.g. Interviews and questionnaires. Gets information from two very different perspectives. If the data from both sources is in agreement, the research is more convincing.

• Multi-method: research that uses two quantitative or qualitative methods – e.g. interviews and focus groups OR structured observations and questionnaires. Multi Methods has many of the same benefits as Mixed Methods

Positivism

Metaphysical (nature of reality) assumptions • Nature is orderly and regular (measurable); • We can know nature. (Some theorists suggest

that there exists a limit to such knowledge. Up to now, such a limit has not been defined.)

• All natural phenomena have natural causes (Determinism).

• Nothing is self-evident

PositivismEpistemic (nature of knowledge) assumptions • Knowledge should only be derived from

experience. (Empiricism) • The meaning of a proposition consists in how

it is verified by experience. (verifiability). • The application of logical analysis will reach

the goal of unified science. (Logicism).• Sciences should all be unified syntactically

and semantically.

Interpretivism

Interpretivism rests upon idealism. Idealism holds the view that the world is the creation of mind; the world is interpreted through the mind; e.g., classificatory schemes. Given this, we cannot know the ‘true’ nature of the object world, separate from our perception of it. Research methods focus on phenomenological enquiry – interviews and focus groups that discover the lived experience of the participant.

Constructivism Constructivists maintain that scientific

knowledge is constructed by scientists and not discovered from the world. Constructivism believes that there is no single valid methodology and that qualitative research is a valid methodology for social science.

Constructivism criticizes objectivism (objects and facts exist independently of the mind), and holds that the only reality we can know is that which is represented by human thought

Structuralism & PoststructuralismIn Structuralism the meaning of any one word is determined

by its relationship with other words. Any approach that sees the meaning of something as subordinate to its place within a system is structuralism. It promotes the objectification of structures and systems. Systems analysis is a structuralist approach.

Poststructuralists reject structuralism and maintain that meanings and intellectual categories are shifting and unstable. Deconstruction and Narrative enquiry are poststructural research methods

Postpositivism

Acknowledges the constructivist, Interpretivist and post structural critiques of positivism and seeks to incorporate qualitative approaches into research methodology. It retains a place for positivist methodology and analysis. It mediates critical space between positivism and constructivism based upon the quest for a stable truth as a regulative ideal within a fallible scientific epistemology.

Transformative/EmancipatoryHas a focus on social justice, the experience of

oppression, the differentials of power, and the cultural, political, economic and historical perceptions of “reality”. It builds on Foucauldian ideas of ethics and asks for a constant effort to move taken-for-granted knowledge to conscious examination. It does accept that using quantitative approaches can be consistent with the central premises of the T-E paradigm

Pragmatism• The current meaning of an expression is to be determined

by practical consequences of belief in or use of the expression in the world

• We should consider what practical effects we conceive the object of our conception to have. Then our conception of these effects is the whole of our conception of the object

• Places high regard for the reality of and influence of the inner world of human experience in action.

• Knowledge is viewed as being both constructed and based on the reality of the world we experience and live in.

Questionnaires and Surveys1) Surveys are Structured interviews. Face to

face, or telephone/video/internet. A prescribed set of questions, mostly tick-box/numerical, some open ended questions, some opportunity for asking additional questions – Deaf Epidemiological Survey

2) Questionnaires. Mail out, internet or drop offs, or groups of people. Often anonymous. Tick/ numerical, some open questions – Carbon Footprint Questionnaire

3) Audits: These are the analysis of data already in the system. No new questions are created – Analysis of Mental Health Data

General Health Questionnaire-12

1. Able to concentrate2. Lost much sleep3. Playing a useful part4. Capable of making decisions5. Under stress6. Could not overcome difficulties7. Enjoy normal activities8. Can face up to problems9. Feeling unhappy and depressed10. Losing confidence11. Thinking of self as worthless12. Feeling reasonably happy

INSTRUCTIONS

We should like to know if you have any medical complaints, and how your health has been in general, over the past few weeks…..

We want to know about present and recent complaints, not those you had in the past.

Principles of Psychometric Tests• Three important concepts:

– reliability, validity and standardisation are essential criteria for a good psychometric test

• Test standardisation – ensures that the conditions are as similar as possible for all individuals who are given the test.

• Standardisation also ensures that no matter who gives the test and scores it, the results should be the same – i.e. the test is reliable

Test Reliability• Test Reliability – a test must measure the same thing

in the same way every time someone takes it• There are two types of test reliability

– Internal consistency reliability – all the parts of your test questionnaire are reliable throughout (Split half tests using Cronbach’s alpha ())

– Test–retest reliability – the test remains reliable over time (e.g. gives the same result under the same conditions). This is tested by correlation (r)

• A test can be reliable, but not valid

Test Validity• There are four types of test validity:

– Face validity: does your test appear to measure what it purports to measure. Would experts agree that people who score above 3 on the GHQ would will have MI signs

– Concurrent validity: does the GHQ correlate with existing standardised tests of MI?

– Predictive validity: do the results of your test predict future behaviour – good and poor MH outcomes

– Construct validity: if all our hypotheses about the test variable (construct) are supported then we have a high degree of construct validity

Carbon footprint questionnaire

• Do the questionnaire• Tick the answers closest to

your understanding of the way you live

• Add up your score by adding up the numbers in each box you ticked

• Discuss the outcome• Critique the questionnaire

Scores Earths

required

32-62 1.2-1.6

62-93 1.6-2.3

93-124 2.3-5.8

124-155 5.8-15.3

Question Issues• What types of

questions can be asked and how?

• Demographic vs topic

• How complex will the questions be?

• How long will the questionnaire/ survey be?

Demographic questionsThe big three:

age, gender, ethnicity/culture.

Socio-economic status:• Income (tax brackets/income support levels), • Occupation

1=factory, unskilled;2=clerical, fishing, farming, trades; 3=administrator, professional, technician

Demographic questions - 2

Socio-economic status (continued):• Employment (full-time, part-time, student,

not working, on the benefit),• Education (SC, UE/bursary, tertiary cert,

diploma, degree)• Family: partners, dependants,

responsibilities• Health: disabling conditionsBeliefs: religion, political affiliation, sexual

orientation

Developing your questions - 1 Managing bias. • The researchers – how easy is it to disprove

your hypothesis, theory, bias?• The participants – will the subject and the

questions put off the participant?Opening questions.• Simple • Key open ended questions to get

uninfluenced “off the top of my head” answers.

Developing your questions - 2

7. Would you like to help with any of the following activities? yes no

being involved in support group for your community setting up a support group for your community being a driver on field trips

helping with education activities (e.g.. cooking, sewing, computer skills) helping Shakti with policy development

being involved in activities aimed at changing government policy

Response option lists. (forced choice)

Developing your questions - 3Rating questions (likert scales)Did your mental health change because of the

help you got from the hospital or the service(s)? 1=much better; 2=better; 3=didn’t change/not

sure; 4=worse; 5=much worse• What to do about don’t know/not sure/did not

answer responses• Rating option lists – putting in an “other”

option space• Question Matrices

Developing your questions - 3The “other” option

3 Community health 3c. Where did the mental 3e. Did your mental health professionals/services health consultation(s) mainly improve because of the (other than overnight take place? help you got?services). place =

clinicsurgery,office, orrooms

1 GP or Doctor If>0 If>02 Visiting nurse (Community, District, Plunket) If>0 If>03 Community mental health service If>04 Community drug and alcohol (D&A) service If>05 Psychiatrist* If>06 Psychologist* If>07 Counsellor (include psychotherapists)* If>08 Social worker If>0 If>09 Deaf Association Service Coordinator If>0 If>010 Employment or occupational service (eg. If>0 If>0

work experience, Workbridge, NZES)11 Accommodation or housing service (eg. If>0 If>0

Housing New Zealand, rest homes) 12 Maori health service (eg. marae health If>0 If>0

service, Tohunga, spiritual healer)13 Alternative health service (eg. use of herbal If>0 If>0

and natural medicines, acupuncturist, chiropractor, spiritual healer)

14 Interpreter or communicator on their own for If>0 If>0

advice or discussion of problems (do not include social or friendship meetings)

15 Other health professional (eg. audiologist If>0 If>0

physiotherapist, optician, obstetrician,ear, nose and throat specialist)

(please circle which professionals you have been to or write their professions down if they are not in the list____________________________________________)

*Differences between a psychiatrist, a psychologist and a counsellor are 1) psychiatrists prescribe medication for mental health problems, the others do not; 2) Sessions with psychologists and counsellors are

usually longer than with psychiatrists (one hour compared with less than half an hour). 3) Psychologists use more cognitive and behavioural techniques to help you control your thoughts and behaviours

occasio

nally

Ho

spita

l or A

& E

In y

ou

r h

om

e

half t

he t

ime

alw

ays

mostly

did

no

t ch

an

ge

wo

rse

mu

ch w

ors

e

mu

ch b

ett

er

be

tte

r

If you have seen a mental health or D&A professional while using a community mental health or D&A service, do not count them again under their specific profession (eg nurse, social worker, psychiatrist).

never

3d. Was there an interpreter or communicator present?

3b. When you saw a health professional, were there times when you were feeling these things? (Show illness list) How many times?

3a. Over the past 12 months how many times have you consulted with the following people or services about your health?

At

the

ir p

lace

At

an

oth

er's

pe

rso

n's

pla

ce

Developing your questions - 4• “Why” questions sparingly used after key

options questions • Mopping up questions. Solutions, answers,

ideas, “anything else you want to say”• Reliability checks.

• Check response patterns for rigidity• Vary direction of rating responses• Set up correlating questions• Set up some very unlikely options

Strengths of Questionnaires• Good for measuring attitudes and eliciting other content

from research participants • Inexpensive (especially mail questionnaires and group-

administered questionnaires) • Can be administered to large probability samples • Quick turnaround from data collection to report• Can be administered to groups • Perceived anonymity by respondents is high • Moderately high measurement validity for well-

constructed and well-tested questionnaires • Low dross rate for closed-ended questionnaires • Ease of data analysis for closed-ended items

• May need validation and may have poor reliability • Must be kept short • Often has missing data, particularly to open ended

questions • Possible reactive effects (e.g. response sets, social

desirability, dislike of questionnaires) • Failure to reach certain groups – low SES, low literacy• Response rates generally low for mail questionnaires • Open-ended answers are vague or reflect differences in

verbal ability, obscuring the issues of interest • Data analysis sometimes time-consuming for open-

ended items

Weaknesses of Questionnaires

Sampling

Population issues• Can the population be counted? What data is

available• Are response rates likely to be a problem?

– Is the population literate? – Are there language issues? – Will the population cooperate? – What are the geographical restrictions?

• Generalisation.– How small is the effect size we are seeking?– What sub-groups do we want to compare?– How do we represent the people who won’t

answer?

Probability sampling• Based on statistics which can describe the

similarity of a sample to the population that it is supposed represent.

• The simplest form of random sampling is called simple random sampling. In this we select participants from a given population such that each person in the population has an equal chance of being selected.

Stratified Random Sampling Involves dividing a population into homogeneous

subgroups based on one factor and then taking a simple random sample in each subgroup (e.g. culture). Participants are randomly selected within each group.

Subgroups might be based several factors (e.g. culture (two), gender and age (two groups)) which would give 8 (2x2x2) equal subgroups within which participants are randomly selected.

Quota Sample

The entire population is first divided into homogeneous sub-groups with respect to the given characteristic such as culture. A specified number of people are recruited from each strata as you come across them rather than selecting them through random procedure. The resulting samples are called quota samples.

Purposive sampling

• Purposive sampling starts with a purpose in mind and the sample is thus selected to include people of interest and exclude those who do not suit the purpose. Subjects are selected on the basis of some characteristic/s. Purposive sampling is popular in qualitative research.

Examples of Purposive Sampling• Extreme or Deviant Case - Learning from highly

unusual manifestations of the area of interest• Intensity - Information-rich cases that demonstrate

the area of interest very clearly. Critical incident case, Politically Important Case

• Maximum Variation - Purposefully picking a wide range of variation within a group

• Homogeneous – Reducing variation to get a clear group or sub-group perspective

• Typical Case - Illustrates or highlights what is typical, normal or average. (e.g. the average student). Criterion based or Theory based selection

Examples of Purposive Sampling• Random Purposeful – random selection form a pool

of participants.• Purposive Quota Sampling Interviewers are given

quota of particular groups of people to interview and the quota are organised so that final sample is representative of population. If we want our sample to represent the age of our population and 20% are between 20 and 30, and sample is to be 20 then 4 of the sample (20%) must between 20 and 30. Complex quotas can be developed so that several characteristics (e.g. age, sex, marital status) are used simultaneously.

Waitakere City 2006 Census Data (People 16 years and over)Occupation

N% of pop-

ulationparticipants

in each category

rounded participants

Managers 12,645 9% 2.84 3Professionals 16,167 11% 3.64 4Technicians and Trades Workers 13,035 9% 2.93 3Community and Personal Service Workers

6,798 5% 1.53 2

Clerical and Administrative Workers 12,570 9% 2.83 3Sales Workers 9,258 7% 2.08 2Machinery Operators and Drivers 5,622 4% 1.26 1Labourers 7,362 5% 1.66 2Emloyed Not Elsewhere Included 4,689 3% 1.05 1Unemployed 5,349 4% 1.20 1Full-time and part-time Tertiary Students 13987 10% 3.15 3Not in the Labour Force (house persons, retired, disability)

27494 19% 6.18 6

Status Unidentifiable 7,311 5% 1.64 2Total 142,287 100% 32 33

Purposive Quota sample

Examples of Purposive Sampling• Snowball or Chain - Initially contact a few

potential respondents and then ask them whether they know of anybody with the same characteristics that you are looking for in your research.

• Opportunistic –Following new leads during fieldwork, taking advantage of the unexpected.

• Convenience - The sample comprises subjects who are available in a convenient way to the researcher.

The importance of enhancing whānaungatanga (the feeling of close connection as between kin)

between students at Unitec Waitakere