Scientific Research Robert O. Briggs Delft University of Technology University of Arizona [email protected] Tucson, AZ 85721.

Scientific Research

Robert O. BriggsDelft University of Technology

University of [email protected]

Tucson, AZ 85721

Today’s program

Introductions Epistemology The Philosophy of Science The Scientific Approach

Which is Bob’sExciting Secret Identity?

Bob Briggs Facilitated the surrender of Napoleon

at Waterloo Invented the Internet Sang with Elvis Presley in concert

Photographic Evidence

Elvis

Bob

You ain’ nuttin’ but a hound dog!



Three Ways to Think About Academia

The Philosophical The Pragmatic The Publishable

EpistemologyThe Philosophical View

The study of the nature of knowledge Presuppositions Foundations Extent Validity

Epistemology:The Philosophical View

A way of knowing A way of creating knowledge

Prevailing Epistemologies

Interpretivism Criticalism Positivism

Interpretivism Creating knowledge about… The inferences people draw from

and the meanings people ascribe to the words and actions of others.

Key Assumption “There is no objective reality”

Positivism (Science)

Creating knowledge about… Cause-and-effect Key Assumption

“There is an objective reality”

Criticalism

Creating knowledge about… The nature of and resolution

of deep social ills Key Assumption

Deep social ills exist

Epistemology Myths

Positivism and Interpretivism are mutually exclusive world views

Objective Reality vs. No Objective Reality?

What is Reality?

Epistemology Myths

Positivists skew studies to find the result they want

Interpretivists don’t believe in gravity

Epistemology Myths

Interpretivism is qualitative Positivism is quantitative

Epistemology Myths

An epistemology is something you “are” I’m an interpretivist I’m a positivist

Pragmatic Epistemology

A set of mental disciplines To keep us from drawing (and then

publishing) bone-headed conclusions

Pragmatic Interpretivism

A set of mental disciplines To keep us from drawing bone-

headed conclusions About the inferences and

meanings people ascribe to the words and actions of others

Pragmatic Positivism

A set of mental disciplines To keep us from drawing bone-

headed conclusions About patterns of cause-and-effect

Publishable Positivism

Report of a study on the causes of a phenomenon-of-interest that…

Provides convincing arguments that…

The conclusions may not be bone-headed

The Philosophy of Science

Positivist Assumptions Regular patterns of causation Independent from human mind “Knowable”

The Boundaries of Science

If it’s not about cause-and-effect… It’s not science Period.

Goals of Science Create causal models for

phenomena of interest Test the usefulness of those

models Use those models to increase the

likelihood people will survive and thrive.

Why should you care about

Positivist Science?

Why Should You care?

Good science will Make it more likely that people will survive and

thrive Make you work smart Get you published in the good journals

Bad Science will Harm others Waste effort, time, and money Embarrass you for years

The Phenomenon of Interest

In the world of cause-and-effect… The phenomenon-of-interest is the

EFFECT The EFFECT is what you seek to explain The EFFECT is what you seek to

improve The EFFECT is the outcome you

measure

The three most exciting words in science are, “Gee, that’s funny…”

- Issac Azimov

The Positivist Disciplines

Phenomenon-of-Interest Who Cares? Theory Hypotheses Research Methods Analysis

The First Discipline

Explicitly Define The Phenomenon of Interest

The First DisciplineDefine The Phenomenon of Interest

Explicitly In writing Refine the definition as your

understanding deepens Challenge your definition

continuously

Explicitly Define thePhenomenon of Interest

Satisfaction First definition:

The degree to which needs are fulfilled Measures

I am satisfied My needs are fulfilled I feel satisfied

Better definition An affective arousal with a positive valance in

response to the judgment that needs have been or will be satisfied

Measures I feel satisfied with… …gave me a feeling of satisfaction I feel good about…

Phenomenon Vs Domain

The phenomenon-of-interest is the OUTCOME you hope to improve measurably Productivity Creativity

The domain is the setting in which the outcome manifests Requirements Engineering Project Management

Phenomenon vs. DomainThe Pragmatic View

You study the phenomenon of interest …Don’t ever forget it

You sell the domain To funding agencies To reviewers To readers

The Second Discipline

Who Cares?!?

Who Cares?!?

Why is this phenomenon-of-interest is worthy of study?

Philosophical “Who cares?”

Science must increase the likelihood that people will survive and thrive

Society provides the scarce resources for scientific enquiry. You must be able to justify your use of them.

Pragmatic “Who Cares”

Your reviewer just had a “much better” paper rejected by the same journal

Publishable “Who Cares?”

1. The phenomenon of interest is worth studying1.1 People are more likely to survive

and thrive if we understand the cause of this phenomenon

1.2 The existing literature does not fully explain the causes of this phenomenon

Publishable “Who cares?”

You must define explicitly the phenomenon of interest in the first or second paragraph

It’s your anchor for all that follows

Good “Who Cares?” 1.1 Organizations exist to create value for

stakeholders Organizations operate under risk Mitigate risk, the organization may survive Internal risk assessments can mitigate risk Risk assessments must be run by groups If we can make risk assessment groups more

productive, we may increase that people will survive and thrive!

Productivity is…. This study examines the use of GSS to make

risk assessment groups more productive.

Bad “Who cares” 1.1 Organizations do risk assessments

frequently We studied collaborative risk

assessment workshops

Ugly “Who Cares” 1.1

We collected some data about risk assessment workshops

Good “Who Cares 1.2” Connolly et al (1992) showed that productivity

of brainstorming teams could be improved by making them anonymous.

However, Johnson and Stephens (2003) showed better productivity when brainstorming teams were identified

A causal theory of productivity might be useful for explaining these seemingly disparate results, and might allow the development of even better brainstorming techniques.

This paper offers such a theory

Bad “Who Cares” 1.2 Jones (1983) said nothing has been

done about productivity Smith (1978) called for more

research on productivity Johnson (1981) studied productivity

among factory workers I studied productivity among

brainstorming groups

Ugly “Who Cares” 1.2

I searched 3 on-line databases and browsed 6 web search engines and only found 2 articles on this topic.

Little is known about this topic Nobody has studied this topic yet.

Publishable PositivismThe Opening Argument

Section 1. Who Cares?!?Argument: This phenomenon is worth studying.

1.1 People will be better off if we understand this phenomenon

1.2 Current literature does not yet fully explain it

The Third Discipline

Theory

Theory

A causal model of the phenomenon-of-interest

Drives all subsequent activity Hypotheses Experimental design Measures Analysis Conclusions

Data have no meaning except with respect to

the theory from which they spring

Today’s Message:Today’s Message:

Goals of Science Create causal models for

phenomena of interest (Theory) Test the usefulness of the models

(Experiment) Use those models to increase the

likelihood people will survive and thrive. (Application)

Anything Missing?

TruthTruthTruthTruth

Anything Missing?

Positivist Perspective

Science = Useful Science = Useful

Science <> TrueScience <> True

A useful model is better than Truth

Useful Is Better Than True

Useful Is Better Than True

Name the PhenomenonName the Phenomenon

BobeziteBlock

Describe the PhenomenonDescribe the Phenomenon

BobeziteBobeziteBlockBlock

A

B

Explore the PhenomenonExplore the

Phenomenon


A

B


Explore the PhenomenonExplore the

Phenomenon

BobeziteBobeziteBlockBlockBobeziteBobezite

BlockBlockBobeziteBobeziteBlockBlockBobeziteBobezite

BlockBlock

A

B


A

B

Describe the dynamics of the phenomenon

Describe the dynamics of the phenomenon

A Useful ModelA Useful Model

One Gear

TruthTruth

One Thousand Gears

When does the Model Become Useful?

When does the Model Become Useful?

When you want todo something newWhen you want todo something new

Therefore

For matters of cause-and-effectA useful model (Theory)

is better than Truth

An experiment, without a Theory is

meaningless

What is a Theory?

An excuse to not do anything meaningful?

Pie-in-the-sky disconnect from reality?

There is nothingmore useful

than a good theory

What is a theory?

Causal Model Internally Consistent Explains and/or predicts Proposes mechanisms of causation Testable

Structure of a Theory

Axioms Propositions

Axioms

Assumptions about the fundamental nature of the universe

Axioms are “received”

Example Axioms

Axiom 1: Human attention is limited Axiom 2: All action is purposeful for goal attainment

Axioms Are Received

Source is irrelevant Feynman’s Inspiration

Propositions

Functional Statements of cause-and-effect that must be logically true if the axioms are true

Propositions are...

Causal Composed of constructs Without empirical content

Useful Propositions

Proposition 1: Productivity is a function of effortProposition 2: Effort is a function of goal congruenceProposition 3: Effort is an inverse function of distraction

ProductivityEffort

Distraction

Goal Congruence

+

-

+12

3

Mathematical Propositions

P = (E)Where

P = Productivity

G = Goal Congruence

E = -(D)Where

E = EffortD = Distraction

Problematic Propositions

WORK DESIGN & EXECUTION OUTCOMES

IMPLICIT INCENTIVES

EXPLICITINCENTIVES

SOCIAL ENVIRONMENT

TECHNICAL ENVIRONMENT

RESOURCE ENVIRONMENT

ORGANIZATIONAL STRUCTURE

ENVIRONMENT

DISTRIBUTED WORK

ARRANGEMENT

ORGANIZATIONAL LEVEL

GROUP LEVEL

INDIVIDUAL LEVEL

INCENTIVE STRATEGY

(e.g. Reward & Compensation)

WORK DESIGN & EXECUTION OUTCOMES

IMPLICIT INCENTIVES

EXPLICITINCENTIVES

SOCIAL ENVIRONMENT

TECHNICAL ENVIRONMENT

RESOURCE ENVIRONMENT

ORGANIZATIONAL STRUCTURE

ENVIRONMENT

DISTRIBUTED WORK

ARRANGEMENT

ORGANIZATIONAL LEVEL

GROUP LEVEL

INDIVIDUAL LEVEL

INCENTIVE STRATEGY

(e.g. Reward & Compensation)

Qualities of a Good Theory

Parsimony Explanation/Prediction Boundaries

Pragmatic Theory

You usually start with propositions and work backward to axioms

You usually start badly and get better

You use someone else’s theory whenever you can

Your technology is probably not in your theory

Pragmatic Theory

A good theory will get you to the moon and back safely on the first try

Good theory will do more to save you from drawing bone-headed conclusions than any other discipline of positivism

Publishable PositivismAlternative Wordings for Propositions

Y is a function of Z Z causes Y Z determine Y The more Z you do, the more Y you

get Z has a positive influence on Y


Section 2. TheoryArgument: I understand what causes Z

If we assume X to be the case, then it must be that:

Proposition 1: Y is a function of Z.

The Fourth Discipline

Hypotheses

The Fourth DisciplineHypotheses Comparative statements

Some explicitly stated measurable outcome Compared across at least two treatments

Logically derived from propositions Tests the proposition Empirical content An answer to a research question

Example Hypothesis

H1: Brainstorming teams with access to an automated social-comparison-feedback graph will produce more unique ideas than teams with no automated graph

Example Hypothesis

H2: During brainstorming, the more we pound randomly on the walls, the fewer ideas a team will produce.

Problematic Hypotheses

H3: Groups using richer media will exhibit higher levels of cohesion initially

Problematic Hypotheses

H4: On negotiation tasks, face-to-face groups will outperform computer mediated groups, will experience less process difficulty, than computer-mediated groups, and will have more favorable reactions to their group task performance, interaction process, and communication medium


Section 3. HypothesesArgument: This theory is testable

If, as Propositon 1 posits, Y is a function of Z, then it must be that:

H1. People using Technology-1 will score higher on the Y-test than do people using Technology-2.

The Fifth Discipline

Research Methods

An experiment without a theory is meaningless

Today’s Message:Today’s Message:

Experiment

Compare outcomes Different treatments Control other possible causes

Experimental InquiryExperimental Inquiry

TreatmentTreatment11




IdenticalSubjectPools


ResultsResultsResultsResults


}} CompareCompare

Investigative InquiryInvestigative Inquiry

PopulationPopulation11






}} CompareCompareOneTreat-ment

OneTreat-ment

Positive Results mean...

Manipulation caused difference Hypothesis has support Theory has support

Negative Results Mean

Experiment Flawed? Hypothesis Flawed? Propositions Flawed? Axioms Broken?

The Only Scientific Truth

The Model is No Good


Section 4. MethodsArgument: I found a reasonable way to test the hypotheses

4.1 My DV instantiates the phenomenon of interest4.2 My IV instantiates a causal construct4.3 My approach would reveal a difference if there were one4.4 There are few alternative explanations for any difference discovered

An Experiment without a theory is

meaningless

Phenomena: Phenomena: Large, Odd-Smelling BoxesLarge, Odd-Smelling Boxes

Scientific Instrument: Scientific Instrument: DrillDrill

Collecting Data without A TheoryCollecting Data without A Theory

Collecting Data Without A TheoryCollecting Data Without A Theory

Collecting Data Without A TheoryCollecting Data Without A Theory

Collecting Data With a TheoryCollecting Data With a Theory





A Physicist Uses the A Physicist Uses the Elephant TheoryElephant Theory

+

=Fission!

A Farmer Uses the Elephant A Farmer Uses the Elephant TheoryTheory

A Farmer Uses the TheoryA Farmer Uses the Theory

There is nothingmore useful

than A Good Theory

An Experiment without a theory is

meaningless

Data have no meaning except in reference to the

theory from which they spring.

Kinds of Causal Theories

Descriptive Predictive Explanatory

Descriptive Model:Descriptive Model:

•What factors impact the length of pins?•Pin-length factors:

- Social Tone (Parties)- Bob-Presence- ?

A


B


A

B

Predictive Causal Model:Predictive Causal Model:

How can we predict the length of B?The length of B is directlyproportional to the length of A

Predictive Causal Model:Predictive Causal Model:

What about the Hacksaw Experiments?


A

B

A

B

Explanatory Model:Explanatory Model:

Why is the length of B proportional to the length of A?

A and B are linked by a gear.

Another View of Theory

Three out of four kinds of theories are dangerous

Levels of Theory

A - Fully Axiomatized B - Building or Broken C - Construct Theory D - Descriptive Theory

A - Level Theory

All Axioms in place Many propositions expressed Extensive, unequivocal empirical

support

A-Level Theory F = M * A

A good theory gets you to the moon

on your first try.

B-Level Theory

Some axioms in place Some propositions Little empirical support Danger - some unknown effects

C-Level: Construct Theory

Assert that a construct exists Find a way to measure it Danger: You always will find a way

to measure it

C-Level Studies

Communication Apprehension Instrument

Measure different groups Compare to other constructs

C-Level Studies

Locus of Control $3,000,000 study Disastrous result

D-Level: Descriptive Theory

Describe Characteristics Taxonomy, Framework Dangers:

Over Aggregation Infinite regression

TeamTeam

TaskTask

TechnologyTechnology

ContextContext

ProcessProcess OutcomeOutcome

An Input-Process-Output Model ofGroup Outcomes from GSS Use

From Nunamaker, et al. (1991)

Endlessly Divisible Constructs

Characteristics of the team Structure

Leadership Style Power differences Norms Intra group process

History Cohesiveness Heterogeneity Etc. Etc.

Infinite Regression

Conclusion: The phenomenon can’t be studied

Better Conclusion: I’m asking the wrong question

The Experiment

Points to Ponder You don’t have to measure cause, you only

have to manipulate it. You must measure every effect You must have a theoretical explanation for

every effect

Experimental ModelExperimental Model









}} CompareCompare








}} CompareCompareOneTreat-ment

OneTreat-ment


PTAPTAMembersMembers


Non-PTANon-PTAMembersMembersNon-PTANon-PTAMembersMembers

Loved Loved ItIt

Loved Loved ItIt

So-SoSo-SoSo-SoSo-So

}} CompareCompareEatAt

Joe’s

EatAt

Joe’s


Non-PTANon-PTAMembersMembersNon-PTANon-PTAMembersMembers



So-SoSo-SoSo-SoSo-So

LovedLovedItIt

LovedLovedItIt

}} CompareCompareEatAt

Joe’s

EatAt

Joe’s

•PTA Causes Change in Taste?•Joe was charismatic principal.

Experimental Logic

If every thing else is the same, the difference MUST be caused by my treatments.

Science and Technology

You do not study technology You study the effects to which

technology can be applied

Science and Technology

Every PRESCRIPTION implies an underlying model of cause-and-effect

The Dangers of “Match” and “Fit” Theories

“The quality of the building depends on the fit between the plan and the purpose”

The Dangers of Match and Fit Theories

Every Match or Fit theory implies one or more underlying models of cause-and-effect

But does not bother to articulate them

Experimental Design

Construct Validity Statistical Validity Internal Validity External Validity

Construct Validity Am I measuring the construct I think I’m

Measuring? Thermometer to measure time? Theory drives measures

Statistical Validity

Are statistics interpreted meaningfully

Theory Drives Statistics

Internal Validity

Did my treatment really cause the difference I observed?

Threats to Internal Validity

Unfavorable Comparison Group receiving “Poor” treatment

stops trying


Between-group competition:Group receiving the “poor”

treatment makes extra effort to excel


The Hawthorne Effect:Paying attention to people affects

their performance.

Control for Hawthorne Effect

Group1Group1

ControlControlGroupGroup

Treat-Treat-MentMent

Pay Attention to Both GroupsPay Attention to Both Groups


Novelty Effect: New situations stimulate

performance. Control: Longitudinal Study


Maturation: Perhaps the effect occurred simply

because the subjects got older.

Control for Maturation

Group1Group1



Measure HereMeasure Here


History:Something happens during the

experiment that causes the effect

Control for History

Group1Group1




Threats to Internal Validity Reactive Measures:Somehow the initial measuring

process causes the effect

Control for Reactive Measures

Group1Group1





Calibration:differences caused by shifts in

instrument calibration over the course of the study.

Control for Calibration

Group1Group1




Classic Books

Campbell & Stanley Cook & Campbell

External Validity

To what population do my results apply?

Generalizability

Theory Drives:

Hypothesis Measures Treatments Statistics

Scientific Method Discover Phenomenon Theorize Hypothesize Fastest Falsifications Experiment Conclude Apply

Selling Your Science: Getting Published Introduction: Who cares? Theory: Says Who? Hypotheses: Prove it! Design: Are you sure? Results: Did you get it? Discussion: So What? Conclusions: Theory Good?

Truth

Powerful theory will outperform powerful statistics every time!

Truth There is No Perfect Study You must pilot your study

Truth No Theory is made or broken by a single study

Remember

Experiments without theories are meaningless

Remember

Data Have No Meaning except in reference to the

theory from which they spring

Scientific Research Robert O. Briggs Delft University of Technology University of Arizona [email protected] Tucson, AZ 85721.

Documents