Methodology Matters: Doing Research in The Behavioral and Social Sciences Student:Way Chang Cai Scott Lippert.

Methodology Matters:Methodology Matters:Doing ResearchDoing Research

in The Behavioralin The Behavioraland Social Sciencesand Social Sciences

Student:Student: Way Chang CaiWay Chang Cai

Scott LippertScott Lippert

Introduction Introduction

►Doing Research simply means the Doing Research simply means the systematic use of some set of systematic use of some set of theoretical and empirical tools to try to theoretical and empirical tools to try to increasing our understanding of some increasing our understanding of some set of phenomena or events.set of phenomena or events.

Some Basic Features of The Some Basic Features of The Research ProcessResearch Process

►Some content that is of interestSome content that is of interest

►Some ideas that give meaning to that Some ideas that give meaning to that contentcontent

►Some techniques or procedures by means Some techniques or procedures by means of which those ideas and contents can be of which those ideas and contents can be studied.studied.

Some Basic Features of The Some Basic Features of The Research ProcessResearch Process

► Substantive domain, which we draw contents Substantive domain, which we draw contents that seem worthy of our study and attention.that seem worthy of our study and attention. Understand both “actor” and “context” here refer to Understand both “actor” and “context” here refer to

human system human system ► Conceptual domain, which we draw ideas that Conceptual domain, which we draw ideas that

seem likely to give meaning to our results.seem likely to give meaning to our results. The idea give the meaning that we study The idea give the meaning that we study

► Methodological domain, which we draw Methodological domain, which we draw techniques that seem useful in conducting that techniques that seem useful in conducting that research.research. Methods are the tools – the instruments, techniques Methods are the tools – the instruments, techniques

and procedures – by which a science gathers and and procedures – by which a science gathers and analyzes informationanalyzes information

Methods are the toolsMethods are the toolsThe instruments, techniques The instruments, techniques and procedures.and procedures.

Ex. If you want to cut a apple, Ex. If you want to cut a apple, a hammer don’t help much. a hammer don’t help much. For that you need a saw.For that you need a saw.

All research method help you All research method help you gain knowledge.gain knowledge.

All method used to gather and All method used to gather and analyze evidenceanalyze evidence

However However

►All methods have inherent flaws, All methods have inherent flaws, through each has certain potential through each has certain potential advantages. You can’t avoid these advantages. You can’t avoid these flaws, but you can bring more than flaws, but you can bring more than one method to bear on each of aspect one method to bear on each of aspect of a problem.of a problem.

Research Strategies: Choosing a Research Strategies: Choosing a Setting for a StudySetting for a Study

►Research evidence in social and Research evidence in social and behavioral science always involves behavioral science always involves somebody doing something in some somebody doing something in some situation, which are who, what and situation, which are who, what and where. where.

► Three criteriaThree criteria Generalizability of evidence over the population Generalizability of evidence over the population

of Actors.of Actors. Precision of measurement of the behaviors that Precision of measurement of the behaviors that

are being studied.are being studied. Realism of situation or context within which the Realism of situation or context within which the

evidence is gathered, in relation to the contexts evidence is gathered, in relation to the contexts to which you want evidence to apply.to which you want evidence to apply.

► You may always want to maximize these You may always want to maximize these criteria, but increasing one of these three criteria, but increasing one of these three features reduces one or both of the other features reduces one or both of the other two.two.

Sample

Survey

Field

Study

Judgment Study

Experimental Simulation

Laboratory Experiment

Computer

Simulation

Formal

Theory

Field Experiment

Quadrant III

Respondent

Strategies

Quadrant II

Experimental

Strategies

Quadrant I

Field

Strategies

Quadrant IV

Theoretical

Strategies

Maxima for each criterion:

A = Generalizability

B = Precision

C = Realism

Obtrusive Unobtrusive

Abstract

Concrete

Quadrant IQuadrant I

► Field StudyField Study The researcher sets out to make direct The researcher sets out to make direct

observations of “natural”, ongoing systems, observations of “natural”, ongoing systems, while intruding on and disturbing those systems while intruding on and disturbing those systems as little as possible. (case studies)as little as possible. (case studies)

► Field experimentField experiment Researcher gives up some of the Researcher gives up some of the

unobtrusiveness of the plain field study, in unobtrusiveness of the plain field study, in interest of gaining more precision in the interest of gaining more precision in the information resulting from the study.information resulting from the study.

Quadrant IIQuadrant II

► Laboratory experimentLaboratory experiment Researcher is able to study the behaviors of Researcher is able to study the behaviors of

interest with considerable precision, and to do interest with considerable precision, and to do so under conditions where many extraneous so under conditions where many extraneous factors have been eliminated or brought under factors have been eliminated or brought under experimental control.experimental control.

► Experimental simulationExperimental simulation The researcher attempts to achieve much of the The researcher attempts to achieve much of the

precision and control of the laboratory precision and control of the laboratory experiment but to gain some of the realism experiment but to gain some of the realism ( apparent realism ) of field studies.( apparent realism ) of field studies.

Quadrant IIIQuadrant III

► Sample surveySample survey The investigator tries to obtain evidence that will The investigator tries to obtain evidence that will

permit him or her to estimate the distribution of permit him or her to estimate the distribution of some variables, and/or some relationships some variables, and/or some relationships among them, within a specified population.among them, within a specified population.

► Judgment StudyJudgment Study Researcher concentrates on obtaining Researcher concentrates on obtaining

information about the properties of a certain set information about the properties of a certain set of stimulus materials, usually arranged so that of stimulus materials, usually arranged so that they systematically reflect the properties of they systematically reflect the properties of some broad stimulus domain.some broad stimulus domain.

Quadrant IVQuadrant IV

►Formal theoryFormal theory The researcher focuses on formulating The researcher focuses on formulating

general relations among a number of general relations among a number of variables of interest.variables of interest.

►Computer SimulaComputer Simula Complete and closed system that models Complete and closed system that models

the operation of the concrete system the operation of the concrete system without any behavior by any system without any behavior by any system participants.participants.

Strategic IssuesStrategic Issues

BaseratesBaserates

►Provide a basis for comparisonProvide a basis for comparison Eg: “3 out of 10 smokers develop lung Eg: “3 out of 10 smokers develop lung

cancer”cancer” Is this high, low or normal?Is this high, low or normal? A baserate will give this informationA baserate will give this information

CorrelationCorrelation

►Does X covary with Y?Does X covary with Y? E.g. “Does happiness vary with age?”E.g. “Does happiness vary with age?”

►Covariance can have high positive or negative Covariance can have high positive or negative valuevalue

Covariance can be linear or nonlinearCovariance can be linear or nonlinear►Nonlinear is often overlookedNonlinear is often overlooked

Covariance shows that X and Y have a Covariance shows that X and Y have a relationship, but not necessarily a causal relationship, but not necessarily a causal oneone

The Difference QuestionThe Difference Question

►Does the presence of X alter the state Does the presence of X alter the state or magnitude of Y?or magnitude of Y? If so, to what degree?If so, to what degree?

Randomization in Randomization in ExperimentsExperiments

►Attempt to minimize the effect of Attempt to minimize the effect of extraneous factorsextraneous factors

►Randomization doesn’t guarantee Randomization doesn’t guarantee equal distribution of extraneous equal distribution of extraneous factorsfactors Does make highly unequal distribution Does make highly unequal distribution

unlikelyunlikely

Sampling, AllocationSampling, Allocation

►A large sample minimizes chance A large sample minimizes chance resultsresults

►Probability of chance causing results Probability of chance causing results can be estimatedcan be estimated Accuracy limited by knowledge of Accuracy limited by knowledge of

extraneous factorsextraneous factors►Even if results are likely not caused by Even if results are likely not caused by

chance, results don’t prove X caused Ychance, results don’t prove X caused Y Could be that ignored variable caused YCould be that ignored variable caused Y This can be prevented with careful designThis can be prevented with careful design

Validity of FindingsValidity of Findings

► Internal ValidityInternal Validity Degree the results allow causal relations Degree the results allow causal relations

to be determinedto be determined

►Construct ValidityConstruct Validity How well defined is the theory behind the How well defined is the theory behind the

study study

►External ValidityExternal Validity Concerned with the limits under which Concerned with the limits under which

the results are validthe results are valid

Classes of Measures and Classes of Measures and ManipulationManipulation

Potential Classes of MeasuresPotential Classes of Measures

► Self reportSelf report Questionnaire, diaryQuestionnaire, diary

► ObservationsObservations Visible or InvisibleVisible or Invisible

► Refers to whether subject knows he is being observedRefers to whether subject knows he is being observed

► Archival recordsArchival records Records usually made for non-research purposesRecords usually made for non-research purposes

► Trace measuresTrace measures

Self ReportsSelf Reports

►StrengthsStrengths Inexpensive, Easy to make, low rate of Inexpensive, Easy to make, low rate of

information discardedinformation discarded

►WeaknessesWeaknesses ReactivityReactivity

►This means that people respond as they think This means that people respond as they think they should, rather than how they normally they should, rather than how they normally wouldwould

ObservationsObservations

►StrengthsStrengths Large amounts of information createdLarge amounts of information created

►WeaknessesWeaknesses Large amounts of information created, Large amounts of information created,

reactivity, high cost, ethics issues reactivity, high cost, ethics issues

Trace MeasuresTrace Measures

►StrengthsStrengths Unobtrusive, non-reactiveUnobtrusive, non-reactive

►WeaknessesWeaknesses Information lacks context, information is Information lacks context, information is

not specific enoughnot specific enough

ArchivalArchival

►StrengthsStrengths Sometimes only option, inexpensiveSometimes only option, inexpensive

►WeaknessesWeaknesses Reactive, loose link between the Reactive, loose link between the

information desired and that archivedinformation desired and that archived

Techniques for Manipulating Techniques for Manipulating VariablesVariables

►SelectionSelection►Direct interventionDirect intervention► InductionInduction

SelectionSelection

►Group based on one criteriaGroup based on one criteria E.g. Age, sexE.g. Age, sex

►Lose random distribution of the Lose random distribution of the variable you are manipulatingvariable you are manipulating

Direct InterventionDirect Intervention

►Set up the desired situationSet up the desired situation E.g. creating juries of 12, 6E.g. creating juries of 12, 6 Inexpensive, gives reliable informationInexpensive, gives reliable information Can still distribute randomlyCan still distribute randomly

►Only works for tangible, superficial Only works for tangible, superficial variablesvariables

►Some reactivity existsSome reactivity exists

InductionInduction

►Different typesDifferent types Misleading instructionsMisleading instructions

►Lie to subjects about what is being measuredLie to subjects about what is being measured

False feedbackFalse feedback►Give feedback based on the needs of the Give feedback based on the needs of the

experiments, not the results obtained by experiments, not the results obtained by subjectsubject

Experimenters pretend to be subjectsExperimenters pretend to be subjects►Carry out pre-determined activities to Carry out pre-determined activities to

measure resultsmeasure results