METHODOLOGY MATTERS: DOING RESEARCH IN THE BEHAVIORAL and SOCIAL SCIENCES Joseph E. McGrath Psychology, University of Illinois, Urbana October 1994 Presented.

METHODOLOGY MATTERS:DOING RESEARCH

IN THE BEHAVIORALand SOCIAL SCIENCES

Joseph E. McGrathPsychology, University of Illinois, Urbana

October 1994

Presented By Presented By

Shadi and JingjingShadi and Jingjing

OverviewOverview

This work is about some of the tools with which researchers in the social and behavioral sciences go about doing research. It raises some issues about strategy, tactics and operations. It points out some of the inherent limits, as well as the potential strengths, of various features of the research process by which behavioral and social scientists do research.

What does “doing What does “doing research” mean?research” mean?

it means the systematic use of some set of theoretical and empirical tools to try to increase our understanding of some set of phenomena or events.

research evidence, in any area of science, is inherently tied to the means or methods by which that evidence was obtained.

understanding empirical evidence, its meaning, and its limitations, requires understanding the concepts and techniques on which that evidence is based on.

What does research What does research involve?involve?

It involves bringing together three domains:

(a) The Substantive domain, from which we draw contents that seem worthy of our study and attention.

(b) The Conceptual domain, from which we draw ideas that seem likely to give meaning to our results or to our content.

(c) The Methodological domain, from which we draw techniques or procedures that seem useful in studying those ideas and contents, and conducting that research.

i.e.i.e.

The content: might include the behavior of the users of the UCI library’s website.

The ideas: might include the hypothesis that humanities students perform search better than students in social sciences.

The techniques might include a questionnaire to evaluate the usability issues of the website.

Different levels of the Different levels of the three Domains. three Domains.

Research always deals with several levels of a phenomena: With relations between units or elements within a context or embedding system.

These levels(elements, relations, and embedding systems) have different forms in each of the three domains.

QuickTime™ and a decompressor

are needed to see this picture.

Substantive DomainSubstantive Domain Elements: are the phenomena itself. Elements: are the phenomena itself.

Relations: the patterns of the phenomena. Relations: the patterns of the phenomena.

The phenomena of interest involve the states and actions of some human systems - individuals, groups, organizations, communities, and the like - and the conditions and processes that give rise to and follow from those states and actions.

In this domain "actors behaving toward objects in context” are to be studied.

For example, study of a user navigating and performing tasks using the UCI library’s webpage.

Conceptual DomainConceptual Domain Elements: are the properties of the states and actions of those human systems.

Relations: any of a variety of possible ways in which two or more elements can be connected.

Causal relations or connections. Logical relations. Chronological relations.

For example, two elements can be equal or unequal, they can be related linearly or non-linearly, one can be a necessary or sufficient cause of the other, one can include the other, the relation between them can be one way or reciprocal, and many more.

Materials from the conceptual domain -properties, and relations among those properties - are the "ideas" that can give meaning to the phenomena and patterns to be studied in this domain.

Methodological DomainMethodological Domain

Elements: are the methods or Modes of Treatment of properties of phenomena. Methods are the tools - the instruments, techniques and procedures - by which a science gathers and analyzes information. Methods should be regarded as bounded opportunities to gain knowledge about some set of phenomena in substantive domain.

Relations: are the application of various comparison techniques.

Different Methods (Modes of Treatment): Techniques for measuring Techniques for manipulating Techniques for controlling

Methods (Modes of Methods (Modes of Treatments)Treatments)

Techniques for measuring i.e. a questionnaire, a rating scale, a personality test, instruments for observing and recording communications, techniques for assessing the quality of some products resulting from individual or group task performance, and the like.

Techniques for manipulating making that feature have one particular predetermined value or level for certain "cases" to be studied and another specific preordained value or level for certain other "cases," so that the effect of differences in that property can be assessed by comparing those two sets of "cases."

(a) giving instruction to participants (b) imposing constraints on features of the environment (c) selecting materials for use (d) giving feedback about prior performances (e) using experimental confederates

Modes of treatment (con’t)Modes of treatment (con’t)

Techniques for controlling the impact of various features.

Techniques for experimental control, by which you make certain features take the same predetermined value for all cases in the study (e.g., study only 6-year-olds to control on techniques for statistical control by which you try to nullify the effects of variations in a given property within a study by "removing" those variations by statistical means.

Techniques for distributing the impact of a number of features of the system and its context-without directly manipulating or controlling anyone of them- so that such impact can be taken into account in interpretation of results. The most prominent means for distributing impact of a number of features is called randomization, and refers to procedures for the allocation of "cases" among various conditions within the study.

Relations (Comparison Relations (Comparison Techniques) Techniques)

Comparison Techniques. These are methods or techniques by means of which the researcher can assess relations among the values of two or more features of the human system under study.

Three sets of features of the systems under study:

(a) the features that have been measured, and that are regarded as measures of the phenomena of interest (these are sometimes called "dependent variables")

(b) the features that have been measured or manipulated, and that are regarded as potential covariates of, or antecedents to, the phenomena of interest (these are sometimes called "independent variables')

(c) all of the other features of the system that are relevant to the relations of interest (between dependent and independent variables), and that you have (or have failed to) control, or whose impact you have (or have failed to) distribute or otherwise take into account. (i.e., other relevant features that were not studied directly but that nevertheless are a part of the meaning of results).

Research Methods Research Methods (opportunities and (opportunities and

limitations)limitations) Each method should be regarded as offering potential opportunities not

available by other means, but also as having inherent limitations.

i.e. the widespread use of questionnaires and other forms of self-report.

On the one hand, self-report measures (questionnaires, interviews, rating scales, and the like) are a direct way, and sometimes the only apparent way, to get evidence about certain kinds of variables that are worthy of study: attitudes, feelings, memories, perceptions, anticipations, goals, values, and the like.

On the other hand, such self-report measures have some serious flaws. For example: Respondents may try to appear competent to be consistent, to answer in socially desirable ways, to please (or frustrate) the researcher. Sometimes respondents are reactive on such self-report measures without even being aware of it.

SolutionSolution

bring more than one approach, more than one method, to bear on each aspect of a problem.

If you only use one method, there is no way to separate out the part that is the "true” measure of the concept in question from the part that reflects mainly the method itself.

If you use multiple methods, carefully picked to have different strengths and weaknesses, the methods can add strength to one another by offsetting each other's weaknesses.

If the outcomes of use of different methods are consistent, this way of proceeding can add credibility to the resulting evidence. If the outcomes differ across different methods, then you can avoid misinterpretation of the resulting evidence by properly qualifying your conclusions.

In SummaryIn Summary

(a) Methods enable but also limit evidence. (b) All methods are valuable, but all have weaknesses

or limitations. (c) You can offset the different weaknesses of various

methods by using multiple methods. (d) You can choose such multiple methods so that they

have patterned diversity; that is, so that strengths of some methods offset weaknesses of others.

Research Strategies: Research Strategies: Choosing a setting for a Choosing a setting for a

studystudy Research evidence involves somebody doing

something, in some situation.We can always ask about three facets:Who[which “actors”], what [which “behaviors”] and when and where [which “contexts”].

Actor refers to those human systems, at whatever level of aggregation (e.g., individuals, groups, organizations, communities) whose behavior is to be studied.

Behavior refers to all aspects of the states and actions of those human systems that might be of interest for such study.

Context refers to all the relevant temporal, locational and situational features of the "surround" within which those human systems are embedded.

Three Research Criteria:Three Research Criteria:

When you gather a batch of research evidence, you are always trying to maximize three desireable features or criteria:

A. Generalizability of the evidence over the populations of Actors.

B. Precision of measurement of the behaviors that are being studied (and precision of control over extraneous factors that are not being studied).

C. Realism of the situation or Context within which the evidence is gathered, in relation to the contexts to which you want your evidence to apply.

Dilemma of the Research Dilemma of the Research process:process:

Impossible to maximize all three of these criteria: Generalizability(A), Precision(B), Realism (C).

Increasing one of these three features reduces one or both of the other two.

i.e. conducting a carefully controlled laboratory experiment(B) will intrude upon the situation and reduce its "naturalness" or realism (C). It will also reduce the range of actors(A) to whom the findings can be generalized.

i.e. for example, conducting a field study in a natural situation(C) will reduce both the range of populations to which your results can be applied (A) and the precision of the information you generate (B).



The four quadrants:The four quadrants: Quadrant I contains research strategies that involve observation of ongoing

behavior systems under conditions as natural as possible. Quadrant II contains research strategies that are carried out in settings

concocted for the purpose of the research. Quadrant III contains research strategies that involve gathering responses

of participants under condition in which the setting is muted or made moot. Quadrant IV contains research strategies that are theoretical, rather than

empirical, in character.



When is each criteria When is each criteria maximized?maximized?

Criterion A, generalizability with respect to the population of Actors, is potentially maximized in the sample survey and in formal theory.

Criterion B, precision with respect to measurement and control of behaviors, is potentially at its maximum in the laboratory experiment and in judgment studies.

Criterion C, realism of context, is potentially at its maximum in the field study.

Quadrant I: The Field Quadrant I: The Field StratgiesStratgies

Field Study: the researcher sets out to make direct observations of

"natural", ongoing systems. Much of the ethnographic work in cultural anthropology would exemplify this strategy, as would many field studies in sociology and many "case studies" of organizations.

Field Experiment: the researcher gives up some of the unobtrusiveness of the

plain field study, in the interest of gaining more precision in the information resulting from the study. Typically, a field experiment intrudes the system by manipulating one major feature of that system and study the behaviors of the it

Distinctions of QI:Distinctions of QI:

The behavior system under study is "natural", in the sense that it would occur whether or not the researcher were there and whether or not it were being observed as part of a study.

The two strategies of QI differ in that: field study remains as unobtrusive as it can be (although

no study is ever completely unobtrusive), ability to make strong interpretations of resulting evidence.

field experiment attempts to gain the ability to make stronger interpretations of some of the results by moving towards obtrusive.

For example, that a behavior difference associated with the experimental manipulation may have been caused by the variables involved in that manipulation, but does so at a cost in obtrusiveness, hence in the naturalness or realism of the context.

Quadrant II: The Quadrant II: The Experimental StrategiesExperimental Strategies

Laboratory Experiments:

The investigator deliberately cococts a situation or behavior setting or context, defines the rules for its operation, and then induces some individuals or groups to enter the concocted system and engage in the behaviors called for by its rules and circumstances. The researcher is able to study the behaviors of interest with considerable precision under conditions where many factors have been eliminated or controlled.

The potential gain in precision in the measurement and control of behavior, which is the lure of the laboratory experiment, is paid for by increased obtrusiveness (high on criterion B), hence reduced realism of context (low on criterion C), and by a narrowing of the range of potential generalizability of results (low on criterion A).

Experimental Simulations:

The researcher attempts to achieve much of the precision and control of the laboratory experiment but to gain some of the realism (higher on C) of field studies. This is done by concocting a situation or behavior setting or context, as in the laboratory experiment, but making it as much like some class of actual behavior setting as possible. i.e. flight simulators.

Distinction between QI Distinction between QI and QIIand QII

Both QI and QII are dealing with real situations. But the distinction has to do with whether the situation exists prior to and independent of the investigator, versus having been concocted by the researcher, and therefore whether the participants are taking part in it as an ongoing part of their lives or a part of a research endeavor.

The issue is not one of reality, rather, the issue is one of motivation: Who has what stake in the behavior system under study.

Quadrant III: the Quadrant III: the Respondent Strategy Respondent Strategy

Sample survey:

The investigator tries to obtain evidence that will permit the researcher to estimate the distribution of some variables, and some relationships among them, within a specified population. This is done, typically, by careful sampling of actors from that population (high on criterion A), There is little opportunity for manipulation and/or control of variables and for precision of measurement. (low on criterion B). Since the responses are gathered under conditions that make the behavior setting irrelevant, the question of realism of context is made moot (hence, this strategy is low on criterion C).

Judgment study:

The researcher concentrates on obtaining information about the properties of a certain set of stimulus materials, usually arranged so that they systematically reflect the properties of some broad stimulus domain. The focus of study is the set of properties of the stimulus materials, rather than some attributes of the respondents (nullifying the context of behavior). They are high on precision/control of both the stimulus materials and the responses (high on criterion B) but (low on criterion A). Attempt to reduce or eliminate any properties of the behavior setting that might affect the judgments (low on criterion C).

i.e. psychophysics: These study the systematic relations between properties of the physical stimulus world and the psychological perception of those stimuli.

Relation/Distinction Relation/Distinction bet. Judgment Study and bet. Judgment Study and

Sample SurveySample Survey Relation: Both emphasizing the behavior of some respondents in reaction to

some stimulus materials, and deemphasizing the context within which those responses occur.

Distinction: has to do with two of their features: (a) whether the context is nullified by experimental controls or

transcended by the nature of the responses elicited; (b) whether the response of an individual to a stimulus is regarded as

information about the stimulus (hence, a judgment study) or information about that respondent (hence, a sample survey).

Distinction of Q III:Distinction of Q III:

Strategies of QIII concentrates on the systematic gathering of responses of the participants to questions or stimuli formulated by the experimenter, in contrast to the observation of behaviors of the participants within an ongoing behavior system.

They focus is on observing behavior under conditions where the behavior setting is made irrelevant to the response (neutralizing the context of the behavior.)

Quadrant IV: The Quadrant IV: The Theoretical StrategiesTheoretical Strategies

Formal theory:

does not involve the gathering of any empirical observations. focuses on formulating general relations among a number of variables of interest. these relations -propositions, or hypotheses, are intended to hold over some relatively broad

range of populations (high on criterion A). The formulation of theory in and of itself does not involve the operation of any concrete system (low on criterion C), nor does it involve the observation of any ongoing behavior (very low on criterion B)

i.e. any of the various general theories in behavioral and social sciences.

Computer Simulation:

it is an attempt to model some particular kind of real-world system (high on criterion C) in a complete and closed system that models the operation of the concrete system.

often done on the basis of evidence from prior empirical research. the "behavioral outcomes” have the form of predictions from the theory that the researcher

built into the model. (very low on criterion B). designed to model some particular class of system; have little generality over populations of

actors or situations (low on criterion A).

Quadrant IV: The Quadrant IV: The Theoretical StrategiesTheoretical Strategies

The two strategies of Quadrant IV are different in kind from the other six (non-empirical)

It is valuable for at least two reasons:

First, the two theoretical strategies are related to the empirical strategies in several ways. Second, the inclusion of these two strategies reminds us of the importance of the theoretical

side of the research process.

Inclusion of these two strategies also gives us the opportunity to note that one of the more powerful general strategies for research, and one that involves the use of multiple strategies on the same problem, is the simultaneous use of one of the theoretical strategies (say, the formulation of a general theory) and one of the empirical strategies (i.e., a laboratory experiment)

OutlineOutlineStudy design, comparison techniques & validity

•Introduction•Comparison techniques Baserates The correlational question The difference question•Randomization•Random sampling•Validity of findings Internal validity construct validity external validity

OutlineOutlineClasses of measures & manipulation techniquesClasses of measures & manipulation techniques

Potential classes of measuresPotential classes of measures Self-reportsSelf-reports ObservationsObservations Archival recordsArchival records Trace measuresTrace measures Techniques for manipulating variablesTechniques for manipulating variables SelectionSelection Direct interventionDirect intervention InductionsInductions Concluding remarksConcluding remarks



IntroductionIntroduction

In every empirical study, after we gather In every empirical study, after we gather observations and aggregate data, what is the observations and aggregate data, what is the next step?next step?

-- comparison, which is the heart -- comparison, which is the heart of our researchof our research

We need to draw conclusions based on our observations!We need to draw conclusions based on our observations!



Comparison techniquesComparison techniques

All research questions can be boiled down to All research questions can be boiled down to variations of three basic forms:variations of three basic forms:

BaseratesBaserates

Correlations Correlations

DifferencesDifferences

Comparison techniques -- Comparison techniques -- BaseratesBaserates

BaseratesBaserates

the baserate question asks: the baserate question asks: how often does Y how often does Y occur? occur?

If we do not know how often Y occurs in the If we do not know how often Y occurs in the general case, then we cannot determine general case, then we cannot determine whether the rate of Y in some particular case is whether the rate of Y in some particular case is or is not notably high or low.or is not notably high or low.

Comparison techniques -- Comparison techniques -- correlationcorrelation

The correlational question asks:The correlational question asks: Do the values of X covary with the values of Do the values of X covary with the values of

Y?Y? Positive correlation means when X occurs at a Positive correlation means when X occurs at a

high value, Y is also likely to be at a high high value, Y is also likely to be at a high value. value.

Negative correlation means when X occurs at Negative correlation means when X occurs at a high value, Y is likely to be at a low value. a high value, Y is likely to be at a low value.


An example: An example: does happiness vary with age?does happiness vary with age?

X: age Y: happinessX: age Y: happiness

Positive correlation Negative correlation


How to compute correlation?How to compute correlation?The correlation ρThe correlation ρX,X, YY between two between two random variablesrandom variables XX and and YY with with

expected valuesexpected values μ μXX and μ and μYY and and standard deviationsstandard deviations σ σXX and σ and σYY is is

defined as:defined as:

ρX, Y >0 positive correlationρX, Y < 0 negative correlation

ρX, Y = 0 no correlation

One should bear in mind that correlation cannot help us decide whether X is a cause of Y, or vice verse, or both, or neither.

http://www-306.ibm.com/software/awdtools/reqpro/



Comparison techniques -- Comparison techniques -- differencesdifferences

The difference question asks: whether Y is The difference question asks: whether Y is present (absent) under conditions where X is present (absent) under conditions where X is present (absent) ?present (absent) ?

How: How: dividing all samples into two groups, then…dividing all samples into two groups, then…

group2

X is absent

group1

X is present

Y is present?

Y is present?

Comparison techniques -- Comparison techniques -- differencesdifferences

The problems isThe problems is

1.1. the group 1 and group 2 should be comparablethe group 1 and group 2 should be comparable

2.2. other extraneous factors should be eliminatedother extraneous factors should be eliminated Otherwise, no conclusion can be drawnOtherwise, no conclusion can be drawn To strengthen the reliability of our finding, the To strengthen the reliability of our finding, the

thing we need to do is thing we need to do is RANDOMIZATION RANDOMIZATION



RandomizationRandomization

Randomization means using a random Randomization means using a random assignment procedure to allocate “cases” to assignment procedure to allocate “cases” to “conditions”“conditions”

samples

randomization

1 2 3

Studies with some procedures for random allocation of cases to conditions are called “true experiments” (Campbell & Stanley, 1966)

-- the key idea is removing artifacts and observations that happen by chance.


•Introduction•Comparison techniques Baserates The correlational question The difference question•Randomization•Random sampling•Validity of findings Internal validity construct validity external validity Threats to validity

Random samplingRandom sampling

How to choose cases that are to be included in How to choose cases that are to be included in our study? – it has a substantial effect on the our study? – it has a substantial effect on the validity of our finding.validity of our finding.

The cases in our study should be a “random The cases in our study should be a “random sample” of the populationsample” of the population

So our results can apply to the population of So our results can apply to the population of which our cases constitute a random sample.which our cases constitute a random sample.

Random samplingRandom sampling An exampleAn exampleTo improve the usability of UCI library website, we To improve the usability of UCI library website, we

need to interview users to make recommendations; so need to interview users to make recommendations; so the interviewees should be randomly selected from the interviewees should be randomly selected from the user population; otherwise our findings may not the user population; otherwise our findings may not apply to the whole population (only to specific apply to the whole population (only to specific groups).groups).

The size of samplesThe size of samplesThe larger the size, the more the distribution of samples The larger the size, the more the distribution of samples

will approach to the real distribution of the will approach to the real distribution of the population – population – the law of big numbersthe law of big numbers


•Introduction•Comparison techniques Baserates The correlational question The difference question•Randomization•Random sampling•Validity of findings Internal validity Construct validity External validity

Validity of findingsValidity of findings Internal validityInternal validityInternal validity has to do with the degree to which our findings permit us to Internal validity has to do with the degree to which our findings permit us to

make strong inferences about causal relations.make strong inferences about causal relations.ex. a difference in Y associated with a difference in X does not necessarily ex. a difference in Y associated with a difference in X does not necessarily

imply a causal role for X – why?imply a causal role for X – why?Rival hypotheses:Rival hypotheses:1.1. By chanceBy chance2.2. Other factors may have been covary with X and they, rather than X, Other factors may have been covary with X and they, rather than X,

might have produced the change in Y.might have produced the change in Y. The internal validity measures how well we can rule out all of the The internal validity measures how well we can rule out all of the

plausible rival hypotheses.plausible rival hypotheses.

Validity of findingsValidity of findings Structure validityStructure validityHow well defined are the theoretical ideas in our study?How well defined are the theoretical ideas in our study?How clearly understood are the conceptual relations How clearly understood are the conceptual relations

being explored?being explored? External validityExternal validityIt has to do with the generalizablity of our findings.It has to do with the generalizablity of our findings. Can our findings hold true upon replication?Can our findings hold true upon replication? Make predictions ? Make predictions ?



Potential classes of Potential classes of measuresmeasures Self-reports (the most popular approach)Self-reports (the most popular approach)

Self-reports of participants are always done under the conditions in which the Self-reports of participants are always done under the conditions in which the respondents know that their behavior is being recorded for research purpose.respondents know that their behavior is being recorded for research purpose.

Includes questionnaire response, interview protocols, rating scales, paper and Includes questionnaire response, interview protocols, rating scales, paper and pencil tests.pencil tests.

Pros:Pros:1.1. Versatile to potential contents and to the population to which they would Versatile to potential contents and to the population to which they would

apply toapply to2.2. Low cost in time and resourcesLow cost in time and resources3.3. Low “dross-rates” –little of the information that is gathered gets discardedLow “dross-rates” –little of the information that is gathered gets discardedCons:Cons: Potentially reactive, potentially flawedPotentially reactive, potentially flawedSince participant are aware that their response will be recorded, that may Since participant are aware that their response will be recorded, that may

influence how they response: try to make good impression, to give socially influence how they response: try to make good impression, to give socially desirable answers or to help the investigator get the results being sought …desirable answers or to help the investigator get the results being sought …

Potential classes of Potential classes of measuresmeasures

ObservationsObservationsThis term refers to records of behavior made directly by the investigator.This term refers to records of behavior made directly by the investigator.

Observations by a Visible Observer (OVO)

Observations by a Hidden Observer (OHO)

The presence of observer is known to participants?

Problems in OVO

1.Potentially reactive

2.Vulnerable to observer errors

3.Costly in time and resources

Problems in OHO

1.Vulnerable to observer errors

2.Costly in time and resources

3.Raise ethical concerns


Archival recordsArchival recordsA third way to get records of behavior is to analyze materials in existing A third way to get records of behavior is to analyze materials in existing

documents. The documents here are not collected by researchers, but by documents. The documents here are not collected by researchers, but by some third party, external to the research activity.some third party, external to the research activity.

Ex. Census data, production records, diaries,…Ex. Census data, production records, diaries,…

Pros:Pros:Less costly since someone else has already gathered themLess costly since someone else has already gathered them

Cons:Cons:Low versatility to content and populationLow versatility to content and populationHigh dross ratesHigh dross rates……


Trace measuresTrace measuresPhysical evidence of behavior left behind as unintended residue or Physical evidence of behavior left behind as unintended residue or

outcroppings of past behavior—participants are presumably not aware that outcroppings of past behavior—participants are presumably not aware that there will be a record of their behavior that would be used as for research there will be a record of their behavior that would be used as for research purpose.purpose.

Ex. the number and types of liquor bottles in the garbage of a community Ex. the number and types of liquor bottles in the garbage of a community could be an indicator of drinking habits of the residents.could be an indicator of drinking habits of the residents.

Pros:Pros:NonreactiveNonreactive

Cons:Cons:Not so versatile to content and populationNot so versatile to content and populationCostly in time and resourcesCostly in time and resources……



Techniques for manipulating Techniques for manipulating variablesvariables

Manipulating variablesManipulating variables

-- -- to carry out an experimental manipulation of features of a to carry out an experimental manipulation of features of a situation.situation.

An exampleAn example

Is lung cancer caused by smoking?Is lung cancer caused by smoking?

X=1

smokers

X=0

Non-smokers

•X is a variable in our experiment

•Based on manipulating variables, we can compare two sets of cases

Techniques for manipulating variables is dependent on our research purpose


SelectionSelectionSelection is the most convenient means to make sure that all Selection is the most convenient means to make sure that all

cases of a given conditions are alike on a certain variablecases of a given conditions are alike on a certain variable

Ex: Is lung cancer caused by smoking?Ex: Is lung cancer caused by smoking?X=1 => smokers; X=0 => non-smokersX=1 => smokers; X=0 => non-smokers

we select smokers and non-smokers from the whole population.we select smokers and non-smokers from the whole population.

the problem is with selection, we assign cases so as to differ the problem is with selection, we assign cases so as to differ systematically on X, and they will definitely differ systematically on X, and they will definitely differ systematically on other factors, going along with X.systematically on other factors, going along with X.

Thus , the conclusion drawn is more or less unreliable unless Thus , the conclusion drawn is more or less unreliable unless all other potential factors are removed.all other potential factors are removed.


Direct interventionDirect interventionExample:Example:If we want to compare 6-person group vs 12-person group…If we want to compare 6-person group vs 12-person group…Direct intervention requires that any participant has an Direct intervention requires that any participant has an equal equal

chancechance of being in the 6-person group or 12-person group of being in the 6-person group or 12-person group

The advantage is we not only manipulate the specific variable The advantage is we not only manipulate the specific variable we have in mind, but at the same time we can distribute the we have in mind, but at the same time we can distribute the impact of other factors that we are not studying.impact of other factors that we are not studying.


InductionsInductionsManipulations by less direct interventions are called experimental Manipulations by less direct interventions are called experimental

inductionsinductions Three major formsThree major forms1.1. Use of misleading instruction to the participants Use of misleading instruction to the participants 2.2. Use of false feed backUse of false feed back3.3. Use of experimental confederatesUse of experimental confederates ProsProsIt can potentially produce the desired conditions for the It can potentially produce the desired conditions for the

appropriate cases without raising reactivity problemsappropriate cases without raising reactivity problems ConsConsinvolves some ethical issuesinvolves some ethical issuesIf detected by participants, then they backfireIf detected by participants, then they backfire



Concluding remarksConcluding remarks

Results depend on methods. All methods have Results depend on methods. All methods have limitations. Hence, any set of results is limited.limitations. Hence, any set of results is limited.

It is not possible to maximize all desirable It is not possible to maximize all desirable features of method in any one study; tradeoffs features of method in any one study; tradeoffs and dilemmas are involved.and dilemmas are involved.

Each study must be interpreted in relation to Each study must be interpreted in relation to other evidence bearing on the same questions.other evidence bearing on the same questions.

Works CitedWorks Cited R. M. Baecker, J. Grudin, W. A. S. Buxton, S. Greenberg: Readings in Human-Computer Interaction: Toward the Year 2000. Morgan Kaufman, (p.152-169), 1995

R. Mack, J. Nielsen: Usability Inspection Methods: Report on a workshop held at CHI’92, Monterey, CA, May 3-4, 1992, cited in Readings in Human-Computer Interaction: Toward the Year 2000. Morgan Kaufman, (p.170-182), 1995

METHODOLOGY MATTERS: DOING RESEARCH IN THE BEHAVIORAL and SOCIAL SCIENCES Joseph E. McGrath Psychology, University of Illinois, Urbana October 1994 Presented.

Documents

research evidence

research process

research mean

causal relations

chronological relations

set of phenomena

conceptual domain properties

domain actors