Causality, Reasoning in Research, and Why Science is Hard Sources: D. Jensen. “Research Methods for Empirical Computer Science.” William M.K. Trochim.

Causality, Reasoning in Research, and Why Science is Hard

Sources: D. Jensen. “Research Methods for Empirical Computer Science.”William M.K. Trochim. “Research Methods Knowledgebase”

More on CausalityWhat is causality?

What’s Important About Causality?Explanation

◦Association provides prediction, but not explanation

◦ Identifying causal mechanisms may uncover underlying reasons for relationships

Control◦Understanding causality allows us to

predict the effects of actions without performing them

◦Allows more efficient exploration of the space of possible solutions

Conditions for Causal Inference

Problems with Association

Are Feathers Associated with Flight?

Do they have a casual relationship with the ability to fly?

Related FallaciesCommon (Questionable) Cause Fallacy

◦ This fallacy has the following general structure: 1. A and B are regularly associated (but no third,

common cause is looked for). 2. Therefore A is the cause of B.

◦ Called “Confusing Cause and Effect” fallacy, if in fact, there is not common cause for A and B

Post Hoc Fallacy◦ A Post Hoc is a fallacy with the following form:

1. A occurs before B. 2. Therefore A is the cause of B.

Eliminating Common Causes

Control

Randomization

Modeling

Reasoning Methodologies in Research

Types of Reasoning in Research

Deductive vs. Inductive MethodologiesDeductive

Inductive

What is Abduction?

Examples of Abductive ReasoningA Medical Diagnosis

◦Given a specific set of symptoms, what is the diagnosis that would best explain most of them?

Jury Deliberations in a Criminal Case◦ Jurors must consider whether the prosecution

or the defense has the best explanation to cover all of the evidence

◦No certainty about the verdict, since there may exist additional evidence that was not admitted in the case

◦ Jurors make the best guess based on what they know

“… when you have eliminated the impossible, whatever remains, however improbable, must be the truth.”

- Sherlock Holmes

Abductive Reasoning in ScienceAbduction selects from among the

hypotheses being considered, the one that best explains the evidence◦Note that this requires that we consider

multiple alternative hypotheses Abductive Reasoning is closely

related to the statistical method of Maximum Likelihood Estimation

Possible threats to validity◦Small hypothesis spaces◦Small amounts of evidence to explain

Challenges in Abductive ReasoningCreating hypothesis spaces likely to

contain the “true” hypothesis◦Approach: create large hypothesis spaces

Knowing when more valid hypotheses are missing from the hypothesis space◦Approach: constantly evaluate and revise the

hypothesis space (multiple working hypotheses)

Creating good sets of evidence to explain◦Approach: seek diverse and independent

evidence with which to evaluate hypotheses

Why use multiple working hypotheses?Objectivity: Helps to separate you from your

hypotheses; shift from personal investment in hypotheses to testing the hypotheses

Focus: Reinforces a focus in falsification rather than confirmation

Efficiency: Allows experiments to be designed to distinguish among competing hypotheses rather than evaluating a single one

Harmony: Limits the potential for professional conflict and rejection because all hypotheses are considered and evaluated

“Strong Inference” John R. Platt, Science, October 1964

◦ “Strong Inference - Certain systematic methods of scientific thinking may produce much more rapid progress than others.”

Not all science/research is created equalDon’t confuse research activity with effective research

◦ Activity: building systems; proving theorems; conducting surveys; writing and publishing articles; giving talks; obtaining grants

◦ Research: improved predictions; better understanding of relationships; improved control of computational artifacts

◦ Many researchers are active; only a subset do effective research

Initial Questions for “Strong Inference”

Arguments and FallaciesAside from general reasoning methodologies, one

must ensure the validity of all arguments used in any research endeavor

An argument ◦ Consists of one or more premises and a conclusion ◦ A premise is a statement (a sentence that is either true or

false) that is offered in support of the claim being made, which is the conclusion (also a sentence that is either true or false)

◦ Modus Ponens (and Modus Tollens)A fallacy

◦ Generally, an error in reasoning (differs from a factual error),

◦ An "argument" in which a logically invalid inference is made (deductive) or the premises given for the conclusion do not provide the needed degree of support. (inductive)

Common Fallacies Ad Hominem Appeal to Authority Appeal to Belief Appeal to Common Practice Appeal to Popularity Begging the Question Biased Sample Hasty Generalization Ignoring A Common Cause Burden of Proof Straw Man

See:◦ http://en.wikipedia.org/wiki/Fallacy◦ http://www.nizkor.org/features/fallacies/

Why is Science Hard?

[Notes]