Design Into Practice These lectures tie into Terre Blanche chapter 4 and 5 Now you have a design – how do you run the study? Many practical issues involved.

Design Into Practice

• These lectures tie into Terre Blanche chapter 4 and 5

• Now you have a design – how do you run the study?• Many practical issues involved in converting a

design into a study to run

Conceptualisation

• We want to speak of abstract things• “Intelligence”, “ability to cope”, “life

satisfaction”

• We cannot research these things until we know exactly what they are• Conceptualisation os the process of defining

terms before research

• Once a thing has been conceptualised it is a “Construct”

Making conceptual definitions

• Begin with the lay understanding of the definition• This will be understood by the subjects

• Then consult the experts (literature)• Can be confusing, contradictory

• Create a preliminary definition

• “Test it” hypothetically• Use thought experiments

The danger of reification

• You must not try to make constructs out of things that don’t exist – reification

• Careful grounding of the construct in extablished theory will prevent this

• Eg. Is Homophobia a construct? (is it not just prejudice)?

• Using reified constructs leads to empty, disconnected research

Operationalising variables

• Your design specifies the variables

• How do you measure the variables?• How do you put a number to “intelligence”?

• How do you put a number to “capacity to cope”?

• We need to convert abstract variables into things which we can measure in the real world - operationalisation

Operationalisation (2)• Turn your variable into a directly

measureable thing• Eg: How would you operationalise “success at

university?”

• Often there are developed scales available

• If you operationalise badly, you end up not studying what you want

• Eg. Operationalising “success in career” by looking at the paycheque only

Measuring variables• The operationalisation implies what to

measure variable – how do you do it?• If at all possible, use an established scale• If no scale exists, construct one

• Scales must be valid and reliable• The more of each of these properties, the better

the scale• Validity and reliability need to be sorted out

before you run your study

Reliability in scales

• Reliability: stability of a measure over time• If I measure you now and then in half an hour,

do I get the same reading?

• Max reliability depends on the construct• Some construct are unstable (eg. heart rate)

• Low reliability implies that other variables (“noise variables”) are being measured also• Speaks of the “accuracy” of the scale

Ensuring reliability

• Reliability suffers when subjects have to interpret• Everyone’s interpretation is slightly different

• Objective scales are always more reliable• Allow little interpretation

• Using a fixed response format helps• Eg. Multiple choice, Likert type• Researcher does not have to interpret what the

subject meant

Examples of response types

• Open ended item:Briefly describe your most frightening

experience

• MCQ:The most frightening for me isA) DogsB) SnakesC) SpidersD) None of the above

More examples

• Likert type:

Circle the one which best describes your experience.

I find dogs to be1 2 3 4 5 6 7

Not frightening at all

Terrifyingly frightening

Validity in scales

• Validity: the degree to which a scales measures what it is supposed to

• Validity is subdivided into many types• We will look at 2 most important

• Criterion Realted Validity

• Construct Validity

Criterion Related Validity

• The degree to which this scales matches other established scales

• By comparing to a scale known to be valid, you can be sure yours is valid

• Why make a new scale if one already exists?• Maybe yours is quicker to do

• Maybe the established is not for group testing

How to check for criterion related validity

• This is done through a set of studies• Run a sub-study in which you give the subjects

your scale and the established one

• Run a correlation between the two scales

• If the correlation is statistically significant, your scale compares well to the established one.

• It is better to run several of these validity studies rather than just one.

Example: intelligence test• An accepted test is the WAIS-R

• Very long to run (3 hours)

• You need something quicker (20 minutes), create the QIQ

• Create a test, select a group of subjects• Make them take the WAIS-R and then the QIQ

• Compare the results (correlation)

• If they correlate well, your test is measureing intelligence

Construct Validity

• Construct validity: Does the scale actually measure the construct?• Eg: measuring cranial circumference to

measure intelligence

• Closely tied into the theory of the construct

• Most difficult to achieve, most important

• Measures lacking in construct validity are almost useless

How to check for construct validity

Think abou it for a minute:How can you show that a scale truly measures what it claims to?

How would you show that your depression scale has construct validity?

Hint: Compare it not to scales of the same thing, but to similar and dissimilar things

The strategy

• Similar procedure to criterion related validity:• Before your actual study, run a set of sub-

studies to check your measure

• You will need 2 sets of studies• Concurrent construct Validity

• Discriminant construct validity

Quick aside: direction of correlations

• Correlation: the degree of relationship between two variables, A and B

• Positive correlation: when A has a high value, B has a high value. When A has a low value, B has a low value

• Negative correlation: when A has a high value, B has a low value. When A has a low value, B has a high value

Correlations example

• Positive correlation: the relations ship between amount smoked and probability of heart disease

• Negative correlation: the relationship between amount of daily exercise and probability of heart disease

• No correlation: the relationship between whether you drink tea or coffee and the probability of heart disease

Concurrent validity

• Show that your scale relates positively to related concepts• People who do are depressed will have many

sad thoughts (mood conguency effect)

• Establish concurrent validity against several other constructs

Discriminant validity

• Show that your scale relates negatively to opposite concepts• People who are depressed will have very low

energy

• Establish discriminant validity against several other constructs

Ensuring construct validity

• Best way: be an expert on that construct• Theory should tell you what things to include

• BUT: only if the theory is well-established!

• Second way: consult the experts/literature closely

• Stay with the uncontroversial aspects of that construct

Validity & reliability summary

• Aim: make sure that your variables are correctly operationalised

• Reliability: scale is stable over time/place

• Validity: scale is truly measuring the construct not something else

Validity & Reliability summary (2)

• Ensuring reliability: require verly little interpretation / increase objectivity

• Ensuring validity: base the measure closely on current understanding of construct

• Measuring validity: positive correlations with related scales, negative correlations with opposite scales