Using and understanding numbers in health news and research

11

Using and understanding Using and understanding numbers in health news numbers in health news

and researchand research

Heejung Bang, PhDHeejung Bang, PhD

Department of Public HealthDepartment of Public Health

Weill Medical College of Cornell UniversityWeill Medical College of Cornell University

22

A rationale for today’s talkA rationale for today’s talk Coffee is bad yesterday, but good today Coffee is bad yesterday, but good today

and bad again tomorrow. and bad again tomorrow. ““It's the cure of the week or the killer of the It's the cure of the week or the killer of the

week, the danger of the week.”week, the danger of the week.” says B. says B. Kramer.Kramer.

““I've seen so many contradictory studies I've seen so many contradictory studies with coffee that I've come to ignore them with coffee that I've come to ignore them all.”all.” says D. Berry. says D. Berry.

What to believe? For a while, you may just What to believe? For a while, you may just keep drinking coffee.keep drinking coffee.

33

Hardly a day goes by without a new Hardly a day goes by without a new headline about the supposed health risks headline about the supposed health risks or benefits of some thing…or benefits of some thing…

Are these headlines justified? Are these headlines justified?

Often, the answer is Often, the answer is NONO..

44

R. Peto phrases the nature of the conflict R. Peto phrases the nature of the conflict this way: “this way: “Epidemiology is so beautiful and Epidemiology is so beautiful and provides such an important perspective on provides such an important perspective on human life and death, but an incredible human life and death, but an incredible amount of rubbish is publishedamount of rubbish is published,” ,”

by which he means the results of by which he means the results of observational studies that appear daily in observational studies that appear daily in the news media and often become the the news media and often become the basis of public-health recommendations basis of public-health recommendations about what we should or should not do.about what we should or should not do.

55

3 major reasons for coffee-like situations3 major reasons for coffee-like situations

ConfoundingConfounding

Multiple testingMultiple testing

Faulty design/sample selectionFaulty design/sample selection

66

Topics to be covered todayTopics to be covered today1.1. Numbers in press releaseNumbers in press release2.2. Lies, Damn Lies & StatisticsLies, Damn Lies & Statistics3.3. Association vs. CausationAssociation vs. Causation4.4. Experiment (e.g., RCT) vs. Observational study Experiment (e.g., RCT) vs. Observational study 5.5. Replicate or Perish Replicate or Perish 6.6. Hierarchy of evidence and study designHierarchy of evidence and study design7.7. Meta-analysisMeta-analysis8.8. Multiple testingMultiple testing9.9. Same words, different meanings?Same words, different meanings?10.10. Data sharingData sharing11.11. Other Take-Home messagesOther Take-Home messages

77

1. Numbers in press release1. Numbers in press release

No p-value, no odds or hazards ratio in No p-value, no odds or hazards ratio in press release!press release!

-- Ask people on the street “what is p--- Ask people on the street “what is p-value?”value?”

-- Only we may laugh if I make a statistical -- Only we may laugh if I make a statistical joke using 0.05, 1.96 and 95%, etc.joke using 0.05, 1.96 and 95%, etc.

88

What is P-value?What is P-value? In statistical In statistical hypothesis testinghypothesis testing, the , the p-valuep-value is the is the

probabilityprobability of obtaining a result at least as extreme as of obtaining a result at least as extreme as a given data point, under the a given data point, under the null hypothesisnull hypothesis. .

-- If there is no hypothesis, there is no test and no p--- If there is no hypothesis, there is no test and no p-value.value.

Current statistical training and practice, statistical Current statistical training and practice, statistical testing/p-value are overly emphasized.testing/p-value are overly emphasized.

However, p-value (1 number, 0-1) can be useful to However, p-value (1 number, 0-1) can be useful to decision making. decision making.

-- you cannot say “it depends” all the times although it -- you cannot say “it depends” all the times although it can be true.can be true.

99

Numerator & denominatorNumerator & denominator

Always try to check numerator and Always try to check numerator and denominator (and when, how long)denominator (and when, how long)

Try to read footnotes under *Try to read footnotes under *

-- 100% increase can be 1 -- 100% increase can be 1 →→ 2 cases 2 cases

-- 20% event rate can be 1 out of 5 samples-- 20% event rate can be 1 out of 5 samples

1010

Large Number mythsLarge Number myths With large N, one will more likely find a difference With large N, one will more likely find a difference

when a difference truly exists – notion of statistical when a difference truly exists – notion of statistical power.power.

However, many fundamental problems (e.g., bias, However, many fundamental problems (e.g., bias, confounding and wrong sample selection) confounding and wrong sample selection) CANNOT be cured by large N. (more later)CANNOT be cured by large N. (more later)

Combining multiple incorrect stories can create Combining multiple incorrect stories can create more serious problems than reporting a single more serious problems than reporting a single incorrect story. (more later in meta) incorrect story. (more later in meta)

N>200,000 needed to detect 20% reduction in N>200,000 needed to detect 20% reduction in mortality (Mann, Science 1990)mortality (Mann, Science 1990)

Means (and t-test) can be very dangerous b/c with Means (and t-test) can be very dangerous b/c with large N, everything is significantlarge N, everything is significant

-- Perhaps, for DNA and race, Watson should see -- Perhaps, for DNA and race, Watson should see the entire distribution or SD!the entire distribution or SD!

1111

2. Lies, damned lies & statistics2. Lies, damned lies & statistics

There are three kinds of lies --There are three kinds of lies --B Disraeli & M Twain B Disraeli & M Twain

--- Title speaks for itself--- Title speaks for itself ““J Robins makes statistics tell the truth: Numbers in J Robins makes statistics tell the truth: Numbers in

the service of health” (Harvard Gazette interview)the service of health” (Harvard Gazette interview) If numbers/statistics are properly generated and If numbers/statistics are properly generated and

used, they can be the used, they can be the best piece of empirical best piece of empirical evidenceevidence..

--- some empirical evidence is almost always good to --- some empirical evidence is almost always good to havehave

--- it is hard to fight with numbers (and age)!--- it is hard to fight with numbers (and age)!

1212

Some AdviceSome Advice

No statistics is better than bad statistics.No statistics is better than bad statistics. Just present your data (e.g., N=3) when Just present your data (e.g., N=3) when

statistics are not necessary.statistics are not necessary. Descriptive statistics vs. inferential statisticsDescriptive statistics vs. inferential statistics

If you use wrong stats, you can be on the news. If you use wrong stats, you can be on the news.

See ‘Statistical flaw trips up study of bad stats’. See ‘Statistical flaw trips up study of bad stats’. Nature 2006Nature 2006

1313

3. Association vs. Causation3. Association vs. Causation #1 error in health news, Association=Causation#1 error in health news, Association=Causation In 1748, D. Hume stated ‘we may define a In 1748, D. Hume stated ‘we may define a

cause to be an object followed by another… cause to be an object followed by another… where, if the first object had not been, the where, if the first object had not been, the second never had existed.’second never had existed.’

---this is a true cause! ---this is a true cause!

A more profound quote from Hume is A more profound quote from Hume is ‘ ‘All arguments concerning existence are All arguments concerning existence are

founded on the relation of cause and effectfounded on the relation of cause and effect.’.’

1414

Misuses and abuses of “causes”Misuses and abuses of “causes”

You may avoid the words ‘cause’, ‘responsible’, You may avoid the words ‘cause’, ‘responsible’, ‘influence’, ‘impact’ or ‘effect’ in your paper or ‘influence’, ‘impact’ or ‘effect’ in your paper or press release (esp., title), if results are obtained press release (esp., title), if results are obtained from observational studies. Instead you may use from observational studies. Instead you may use ‘association’ or ‘correlation’.‘association’ or ‘correlation’.

Often, “may/might” not enough.Often, “may/might” not enough. Media misuses and public misunderstands this Media misuses and public misunderstands this

severely severely

--- Every morning, we hear new causes of some --- Every morning, we hear new causes of some disease are found.disease are found.

1515

50% risk reduction, 20% risk reduction, and so on. 50% risk reduction, 20% risk reduction, and so on. If you add up, by now all causes of cancer (& many If you add up, by now all causes of cancer (& many other diseases) should have been identified.other diseases) should have been identified.

Almost all are association, not causation.Almost all are association, not causation.

-- there are an exceedingly large number of -- there are an exceedingly large number of associated and correlated factors, compared to true associated and correlated factors, compared to true causes. causes.

-- a survey of 246 suggested coronary risk factors. -- a survey of 246 suggested coronary risk factors. Hopkins & Williams (1981)Hopkins & Williams (1981)

-- I believe cancer >1000 risk factors.-- I believe cancer >1000 risk factors.

‘ ‘Too many don’t do’ is no better than ‘do anything’.Too many don’t do’ is no better than ‘do anything’.

1616

Why Association Why Association ≠≠ Causation? Causation? ConfoundersConfounders

aka, third variable(s)aka, third variable(s) Biggest threat to any observational studies.Biggest threat to any observational studies. Definition of ‘confound’:Definition of ‘confound’:

vt. Throw (things) into disorder; mix up; vt. Throw (things) into disorder; mix up; confuse. confuse. (Oxford Dictionary) (Oxford Dictionary)

However, confounders CANNOT be defined However, confounders CANNOT be defined in terms of statistical notions alone (Pearl)in terms of statistical notions alone (Pearl)

1717

Confounder samplersConfounder samplers Grey Hair vs. heart attack Grey Hair vs. heart attack Stork vs. birth rateStork vs. birth rate Rock & Roll vs. HIV Rock & Roll vs. HIV Eating late & weight gain? Eating late & weight gain? Drinking (or match-carrying) & lung cancer Drinking (or match-carrying) & lung cancer No father’s name & infant mortalityNo father’s name & infant mortality Long leg & skin cancerLong leg & skin cancer Vitamins/HRT, too?Vitamins/HRT, too?

Any remedy? Any remedy? -- first thing to do is ‘Use common sense’. Think about any -- first thing to do is ‘Use common sense’. Think about any

other (hidden) factor or alternative explanation’.other (hidden) factor or alternative explanation’.

1818

Common sense & serendipityCommon sense & serendipity

Common senseCommon sense is the basis for most of the is the basis for most of the ideas for designing scientific ideas for designing scientific investigations.investigations.   --- M Davidian   --- M Davidian

although we should not ignore the although we should not ignore the importance of importance of serendipityserendipity in science in science

1919

By the way, why ‘causes’ are so By the way, why ‘causes’ are so important?important?

If causes can be removed, susceptibility ceases to If causes can be removed, susceptibility ceases to matter (Rose 1985) and the outcome will not occur. matter (Rose 1985) and the outcome will not occur.

Neither associated nor correlated factors have this Neither associated nor correlated factors have this power. power.

Gladly, some efforts have been made:Gladly, some efforts have been made: ‘ ‘Distinguishing Association from Causation:Distinguishing Association from Causation: A Backgrounder for Journalists’ from A Backgrounder for Journalists’ from American Council on Science and HealthAmerican Council on Science and Health

2020

Greenland’s Dictum (Science 1995)Greenland’s Dictum (Science 1995)

There is nothing sinful about going out and getting There is nothing sinful about going out and getting evidence, like asking people how much do you evidence, like asking people how much do you drink and checking breast cancer records.drink and checking breast cancer records.

There’s nothing sinful about seeing if that evidence There’s nothing sinful about seeing if that evidence correlatescorrelates..

There’s nothing sinful about checking for There’s nothing sinful about checking for confounding confounding variables.variables.

The sin comes in believing a The sin comes in believing a causal causal hypothesis is hypothesis is true because your study came up with a positive true because your study came up with a positive result, or believing the opposite because your result, or believing the opposite because your study was negative.study was negative.

2121

Association to causation?Association to causation?In 1965, Hill proposed a set of the following causal criteria:In 1965, Hill proposed a set of the following causal criteria:1.1. Strength Strength 2.2. ConsistencyConsistency3.3. SpecificitySpecificity4.4. Temporality (i.e., cause before effect)Temporality (i.e., cause before effect)5.5. Biological gradientBiological gradient6.6. PlausibilityPlausibility7.7. CoherenceCoherence8.8. ExperimentExperiment9.9. AnalogyAnalogy

However, Hill also said “None of my nine viewpoints can However, Hill also said “None of my nine viewpoints can bring indisputable evidence for or against the cause-bring indisputable evidence for or against the cause-and-effect hypothesis and none can be required as a and-effect hypothesis and none can be required as a sine qua nonsine qua non’. ’.

2222

Another big problem: Another big problem: bias and faulty design/samples bias and faulty design/samples

Selection bias: the distortion of a statistical analysis, due to Selection bias: the distortion of a statistical analysis, due to the method of collecting samples. the method of collecting samples.

The easiest way to cheat (intentionally or unintentionally)The easiest way to cheat (intentionally or unintentionally)-- Make group1: group2 = healthy people: sick people.-- Make group1: group2 = healthy people: sick people.-- Oftentimes, treatment is bad in observational studies, why?-- Oftentimes, treatment is bad in observational studies, why?-- Do a survey among your friends only-- Do a survey among your friends only-- People are different from the beginning?? (e.g., vegetarians -- People are different from the beginning?? (e.g., vegetarians

vs. meat-lover, HRT users vs. non-users)vs. meat-lover, HRT users vs. non-users) Case-control study & matching: easy to say but hard to do Case-control study & matching: easy to say but hard to do

correctly. correctly. -- Vitamin C and cancer-- Vitamin C and cancer For any comparison: FAIRNESS is most important!For any comparison: FAIRNESS is most important!-- Numerous other biases exist-- Numerous other biases exist

2323

Would you believe these p-values? Would you believe these p-values? (Cameron and Pauling, 1976)(Cameron and Pauling, 1976)

This famous study has failed to replicate 16 or so times! Pauling received two Nobel.This famous study has failed to replicate 16 or so times! Pauling received two Nobel.

2424

4. Experiment vs. 4. Experiment vs. Observational studyObservational study

Although the arguing from Although the arguing from experiments experiments and and observations observations by induction be no demonstration of by induction be no demonstration of general conclusions, yet it is the best way of general conclusions, yet it is the best way of arguing which the nature of things admits of.arguing which the nature of things admits of. --- I --- I NewtonNewton

Newton’s "experimental philosophy" of science: Newton’s "experimental philosophy" of science: Science should not, as Descartes argued, be based Science should not, as Descartes argued, be based on fundamental principles discovered by reason, on fundamental principles discovered by reason, but based on fundamental axioms shown to be true but based on fundamental axioms shown to be true by experiments. by experiments.

2525

Why clinical trials are Why clinical trials are important?important?

Randomized Controlled Trial (RCT) is the most Randomized Controlled Trial (RCT) is the most common form of experiment on humans.common form of experiment on humans.

‘‘Average causal effects’ can be estimated from Average causal effects’ can be estimated from experiment.experiment.

-- To know the true effect of treatment within person, -- To know the true effect of treatment within person, one should be treated and untreated at the same one should be treated and untreated at the same time. time.

Experimentation trumps observation. (power of Experimentation trumps observation. (power of coin-flip! Confounders disappear.)coin-flip! Confounders disappear.)

Very difficult to cheat in RCTs (due to Very difficult to cheat in RCTs (due to randomization and protocol).randomization and protocol).

““CausalityCausality: : God knows but humans need a time God knows but humans need a time machine. When God is busy and no time machine machine. When God is busy and no time machine is available, a RCT would dois available, a RCT would do.”.”

2626

Problems/issues of RCTsProblems/issues of RCTs

Restrictive settingsRestrictive settings Human subjects under experimentsHuman subjects under experiments Can be unethical or infeasibleCan be unethical or infeasible Short termsShort terms 1-2 treatments, 1-2 doses only1-2 treatments, 1-2 doses only Limited generalizabilityLimited generalizability Other issues: blinding, drop-up, complianceOther issues: blinding, drop-up, compliance

2727

Problems/issues of observational Problems/issues of observational studiesstudies

Bias & confoundingBias & confounding Post-hoc arguments about biological plausibility Post-hoc arguments about biological plausibility

must be viewed with some skepticism since the must be viewed with some skepticism since the human imagination seems capable of developing a human imagination seems capable of developing a rationale for most findings, however unanticipated rationale for most findings, however unanticipated (Ware 2003). (Ware 2003).

i.e., retrospective rationalization.i.e., retrospective rationalization. We are tempted to ‘Pick & Choose’!We are tempted to ‘Pick & Choose’! Data-dredging, Fishing expedition, Significance-Data-dredging, Fishing expedition, Significance-

chasing (p<0.05) chasing (p<0.05) Observational studies can overcomes some Observational studies can overcomes some

limitations of RCTs.limitations of RCTs.

2828

Ideal attitudesIdeal attitudes RCTs and observational studies should be RCTs and observational studies should be

complementary each other, rather than complementary each other, rather than competing.competing.

--because real life stories can be complicated.--because real life stories can be complicated. When RCTs and observational studies When RCTs and observational studies

conflict, generally (not always) go with RCTs. conflict, generally (not always) go with RCTs. Even if you conduct a observational study, try Even if you conduct a observational study, try

to think in a RCT way. (e.g., to think in a RCT way. (e.g., a prioria priori 1-2 1-2 hypothesis, protocol, data analysis plan, ask hypothesis, protocol, data analysis plan, ask yourself ‘Is this result likely to replicate in yourself ‘Is this result likely to replicate in RCT?’)RCT?’)

2929

Quotes for observational studiesQuotes for observational studies The data are still no more than observational, no The data are still no more than observational, no

matter how sophisticated the analytic matter how sophisticated the analytic methodology – anonymous reviewermethodology – anonymous reviewer

Observational studies are not a substitute for Observational studies are not a substitute for clinical trials no matter how sophisticated the clinical trials no matter how sophisticated the statistical adjustment may seem – D. Freedmanstatistical adjustment may seem – D. Freedman

No fancy statistical analysis is better than the No fancy statistical analysis is better than the quality of the data. Garbage in, garbage out, as quality of the data. Garbage in, garbage out, as they say. So whether the data is good enough to they say. So whether the data is good enough to need this level of improvement, only time will tell. need this level of improvement, only time will tell. – J. Robins– J. Robins

Remark: However, advanced statistical technique, Remark: However, advanced statistical technique, causal inference, may help.causal inference, may help.

3030

Some studies are difficultSome studies are difficult Diet/alcohol: Type/amount, How to measure? Diet/alcohol: Type/amount, How to measure?

Do you remember what we ate last week?Do you remember what we ate last week? Exercise/physical activity/SES: Can we Exercise/physical activity/SES: Can we

measure? Do you tell the truth? measure? Do you tell the truth? -- people tend to say ‘yes’, ‘moderately’-- people tend to say ‘yes’, ‘moderately’ Long term cumulative effectsLong term cumulative effects Positive thinking and spirituality?Positive thinking and spirituality? Quality and value of life: How to define and Quality and value of life: How to define and

measure measure -- priceless?-- priceless?

3131

5. Replicate or perish5. Replicate or perish Publish or perish: Old eraPublish or perish: Old eravs. Replicate or perish: New eravs. Replicate or perish: New era

Replicability of the scientific findings can never be Replicability of the scientific findings can never be overemphasized. Results being ‘significant’ or overemphasized. Results being ‘significant’ or ‘predictive’ without being replicable misinform the ‘predictive’ without being replicable misinform the public and needlessly expend time and resources, public and needlessly expend time and resources, and they are no service to investigators and and they are no service to investigators and science –S. Youngscience –S. Young

Given that we currently have too many findings, Given that we currently have too many findings, often with low credibility, replication and rigorous often with low credibility, replication and rigorous evaluation become as important as or even more evaluation become as important as or even more important than discovery - J. Ioannidis (2006) important than discovery - J. Ioannidis (2006)

-- Pay more attention to 2-- Pay more attention to 2ndnd study! study!

3232

Examples of highly cited heart-disease Examples of highly cited heart-disease studies that were later contradicted studies that were later contradicted

(Ioannidis 2005)(Ioannidis 2005)

-- -- The Nurses Health Study, showing a 44% relative risk The Nurses Health Study, showing a 44% relative risk reduction in coronary disease in women receiving reduction in coronary disease in women receiving hormone therapyhormone therapy. Later refuted by Women's Health . Later refuted by Women's Health Initiative, which found that hormone treatment Initiative, which found that hormone treatment significantly increases the risk of coronary events.significantly increases the risk of coronary events.

-- Two large cohort studies, the Health Professionals -- Two large cohort studies, the Health Professionals Follow-Up Study and the Nurses Health Study, and a Follow-Up Study and the Nurses Health Study, and a RCT all found that RCT all found that vitamin Evitamin E was associated with a was associated with a significantly reduced risk of coronary artery disease. significantly reduced risk of coronary artery disease. But larger randomized trials subsequently showed no But larger randomized trials subsequently showed no benefit of vitamin E on coronary diseasebenefit of vitamin E on coronary disease

3333

More IoannidisMore Ioannidis

Ioannidis (2005) serves as a reminder of the Ioannidis (2005) serves as a reminder of the perils of small trials, nonrandomized trials, and perils of small trials, nonrandomized trials, and those using surrogate markers.those using surrogate markers.

He concludes "He concludes "Evidence from recent trials, no Evidence from recent trials, no matter how impressive, should be interpreted matter how impressive, should be interpreted with caution when only one trial is availablewith caution when only one trial is available. It is . It is important to know whether other similar or larger important to know whether other similar or larger trials are still ongoing or being planned. trials are still ongoing or being planned. Therefore, transparent and thorough trial Therefore, transparent and thorough trial registration is of paramount importance to limit registration is of paramount importance to limit premature claims [of] efficacypremature claims [of] efficacy."."

3434

More FreedmanMore Freedman

Modeling, the search for significance, the Modeling, the search for significance, the preference for novelty, and lack of interest preference for novelty, and lack of interest in assumptions --- these norms are likely in assumptions --- these norms are likely to generate a flood of nonreproducible to generate a flood of nonreproducible results results

3535

6. Hierarchy of evidence 6. Hierarchy of evidence study designstudy design

3636

What goes on top?What goes on top?

ANSWER is ANSWER is total evidencetotal evidence..

RCT can provide strong evidence for a RCT can provide strong evidence for a causal effect, especially if its findings are causal effect, especially if its findings are replicated by other studiesreplicated by other studies

3737

When you read the article, you may When you read the article, you may check the study designcheck the study design

Cross-sectional study: which is first? what is Cross-sectional study: which is first? what is cause and what is effect?cause and what is effect?

e.g., depression vs. obesitye.g., depression vs. obesity Prospective cohort studies: much better but Prospective cohort studies: much better but

still still not causalnot causal Prospective is generally better than Prospective is generally better than

retrospectiveretrospective RCT is better than non-RCTRCT is better than non-RCT

3838

7. Meta-analysis7. Meta-analysis Statistical technique for systematic literature reviewStatistical technique for systematic literature review There are 3 things you should not watch being made: There are 3 things you should not watch being made:

law, sausage & meta-analysislaw, sausage & meta-analysis No data collection but Nothing is free. No data collection but Nothing is free. Can you find all studies in the universe including ones Can you find all studies in the universe including ones

in researchers’ file drawers? Or at least unbiased in researchers’ file drawers? Or at least unbiased subsample? Google or pubmed can do? subsample? Google or pubmed can do? NO!NO!

Publication bias (favoring positive studies) and Publication bias (favoring positive studies) and language bias, etc.language bias, etc.

Much bigger problem in obs studies than RCTs.Much bigger problem in obs studies than RCTs. Combining multiple incorrect stories is worse than one Combining multiple incorrect stories is worse than one

incorrect story. incorrect story.

3939

Funny (real) titles of papers about Funny (real) titles of papers about meta-analysismeta-analysis

Meta-analysis: apples and oranges, or fruitless Meta-analysis: apples and oranges, or fruitless Apples and oranges (and pears, oh my!): the Apples and oranges (and pears, oh my!): the

search for moderators in meta-analysissearch for moderators in meta-analysis Of apples and oranges, file drawers and garbage: Of apples and oranges, file drawers and garbage:

why validity issues in meta-analysis will not go why validity issues in meta-analysis will not go awayaway

Meta analysis/shmeta-analysis Meta analysis/shmeta-analysis Meta-analysis of clinical trials: a consumer's Meta-analysis of clinical trials: a consumer's

guide.guide. Publication bias in situ Publication bias in situ

4040

8. Multiple testing8. Multiple testing Multiple testing/comparisons refers to the testing Multiple testing/comparisons refers to the testing

of more than one hypothesis at a time. of more than one hypothesis at a time. When many hypotheses are tested, and each test When many hypotheses are tested, and each test

has a specified Type I error probability (has a specified Type I error probability (αα), the ), the probability that at least 1 Type I error is committed probability that at least 1 Type I error is committed increases with the number of hypotheses. increases with the number of hypotheses.

Bonferroni method: Bonferroni method: αα=0.05/# of tests=0.05/# of tests Many researchers’ thorny issue.Many researchers’ thorny issue.

-- Bonferroni might be the most hated statistician in -- Bonferroni might be the most hated statistician in history.history.

-- ‘Escaping the Bonferroni iron claw in ecological -- ‘Escaping the Bonferroni iron claw in ecological studies’ by Garcı´a et al. (2004)studies’ by Garcı´a et al. (2004)

4141

Two errorsTwo errors Type I (false positive: rejecting HType I (false positive: rejecting H00 when it is true) when it is true)

vs. Type II (false negative: accepting Hvs. Type II (false negative: accepting H00 when it is when it is false) false)

-- Controlling Type I is more important in stat and -- Controlling Type I is more important in stat and court. (e.g., innocent court. (e.g., innocent →→ guilty: disaster!) guilty: disaster!)

-- In other fields, Type 2 can be more important.-- In other fields, Type 2 can be more important. αα=p=0.05 – is this the law in science? Only 5% =p=0.05 – is this the law in science? Only 5%

error do you commit in your life?error do you commit in your life? αα==5% seems reasonable to one research 5% seems reasonable to one research

question/publication.question/publication.

4242

4343

Multiple testing in different formsMultiple testing in different forms Subgroup analysesSubgroup analyses -- -- You should always do subgroup analyses but You should always do subgroup analyses but

never believe themnever believe them. – R. Peto. – R. Peto-- -- Multiple testing adjustment and cross-validation Multiple testing adjustment and cross-validation

may be solutions.may be solutions.

Trying different cutpoints (e.g., tertiles, quintiles, Trying different cutpoints (e.g., tertiles, quintiles, etc.)etc.)

-- A priori chosen cutpoints or multiple testing -- A priori chosen cutpoints or multiple testing adjustment can be solutions.adjustment can be solutions.

Nothing is free. To look more, you have to pay.Nothing is free. To look more, you have to pay.

4444

Multiple testing Multiple testing (underlying mechanism)(underlying mechanism)

Lottery tickets should not be free. In random and Lottery tickets should not be free. In random and independent events as the lottery, the probability of independent events as the lottery, the probability of having a winning number depends on the N of tickets having a winning number depends on the N of tickets you have purchased. When one evaluates the you have purchased. When one evaluates the outcome of a scientific work, attention must be given outcome of a scientific work, attention must be given not only to the potential interest of the ‘significant’ not only to the potential interest of the ‘significant’ outcomes but also to the N of ‘lottery tickets’ the outcomes but also to the N of ‘lottery tickets’ the authors have ‘bought’. Those having many have a authors have ‘bought’. Those having many have a much higher chance of ‘winning a lottery prize’ than of much higher chance of ‘winning a lottery prize’ than of getting a meaningful scientific result. It would be unfair getting a meaningful scientific result. It would be unfair not to distinguish between significant results of well-not to distinguish between significant results of well-planned, powerful, sharply focused studies, and those planned, powerful, sharply focused studies, and those from ‘fishing expeditions’, with a much higher from ‘fishing expeditions’, with a much higher probability of catching an old truck tyre than of a really probability of catching an old truck tyre than of a really big fish. --- Garcı´a et al. (2004)big fish. --- Garcı´a et al. (2004)

4545

Multiple testing disaster IMultiple testing disaster I

In the 1970s, every disease was reported to In the 1970s, every disease was reported to be associated with an HLA allele be associated with an HLA allele (schizophrenia, hypertension.... you name (schizophrenia, hypertension.... you name it!). Researchers did case control studies it!). Researchers did case control studies with 40 antigens, so there was a very high with 40 antigens, so there was a very high probability of at least one was significant probability of at least one was significant result This result was reported without any result This result was reported without any mention of the fact that it was the most mention of the fact that it was the most significant of 40 tests --- R. Elston significant of 40 tests --- R. Elston

4646

Multiple testing disaster IIMultiple testing disaster II

Association between reserpine (then a popular Association between reserpine (then a popular antihypertensive) and breast cancer. Shapiro antihypertensive) and breast cancer. Shapiro (2004) gave the history. His team published (2004) gave the history. His team published initial results that were extensively covered by initial results that were extensively covered by media with a huge impact on research media with a huge impact on research community. When the results did not replicate, community. When the results did not replicate, he confessed that the initial findings were he confessed that the initial findings were chance due to thousands of comparisons chance due to thousands of comparisons involving hundreds of outcomes and hundreds of involving hundreds of outcomes and hundreds of exposures. He hopes that we learned for the exposures. He hopes that we learned for the future from his mistake. future from his mistake.

4747

Multiple testing disaster IIIMultiple testing disaster III You are what your mother eats (Mathews et al. You are what your mother eats (Mathews et al.

2008).2008). All over the places on the news and internet. Over All over the places on the news and internet. Over

50,000 Google hits for 150,000 Google hits for 1stst week.   week.   Numerous comparisons were conductedNumerous comparisons were conducted Sodium, calcium, potassium, etc. were significant Sodium, calcium, potassium, etc. were significant

(p<0.05), but sodium was dismissed claiming it is (p<0.05), but sodium was dismissed claiming it is hard to measure accurately.hard to measure accurately.

--possible ‘pick and choose’!--possible ‘pick and choose’! Other problems: lack of biological credibility, Other problems: lack of biological credibility,

difficulty in dietary data. difficulty in dietary data.

4848

Leaving no trace (Shaffer 2007)Leaving no trace (Shaffer 2007)

Usually these attempts through which the Usually these attempts through which the experimenter passed don't leave any experimenter passed don't leave any traces; the public will only know the result traces; the public will only know the result that has been found worth pointing out; that has been found worth pointing out; and as a consequence, someone and as a consequence, someone unfamiliar with the attempts which have unfamiliar with the attempts which have led to this result completely lacks a clear led to this result completely lacks a clear rule for deciding whether the result can or rule for deciding whether the result can or can not be attributed to chance.can not be attributed to chance.

4949

If you keep testing without If you keep testing without controlling controlling αα

Everything is Dangerous – S. YoungEverything is Dangerous – S. Young It is fairly easy to find risk factors for premature It is fairly easy to find risk factors for premature

morbidity or mortality. Indeed, given a large morbidity or mortality. Indeed, given a large enough study and enough measured factors and enough study and enough measured factors and outcomes, almost any potentially interesting outcomes, almost any potentially interesting variable will be linked to some health outcome – variable will be linked to some health outcome – Christenfeld et al. 2004.Christenfeld et al. 2004.

Even checking 1000 correlation can be a sin– S. Even checking 1000 correlation can be a sin– S. YoungYoung

The only thing to fear is fear itself……………………. The only thing to fear is fear itself……………………. …..………………………………and everything else …..………………………………and everything else

5050

Multiple testing adjustmentMultiple testing adjustment In RCTs: mandatory (by FDA)In RCTs: mandatory (by FDA) -- If not, more (interim) looks would lead what you -- If not, more (interim) looks would lead what you

wantwant In genetic/genomic studies: almost mandatoryIn genetic/genomic studies: almost mandatory -- think about # of genes!-- think about # of genes! In observational studies: almost infeasibleIn observational studies: almost infeasible

Realistic strategies can be:Realistic strategies can be:1.1. αα=5% for one hypothesis. Adjust multiple testing or =5% for one hypothesis. Adjust multiple testing or

state clearly how many tests/comparisons you state clearly how many tests/comparisons you conducted.conducted.

2.2. Think and act in RCT ways.Think and act in RCT ways.

5151

Replication, againReplication, again

is universal solution for multiplicity and is universal solution for multiplicity and subgroup analyses (Vandenbroucke 2008)subgroup analyses (Vandenbroucke 2008)

In genome-wide analyses, it is a prerequisite In genome-wide analyses, it is a prerequisite for publication (Khoury et al. 2007)for publication (Khoury et al. 2007)

-- However, replication is for someone else! The data -- However, replication is for someone else! The data analysis strategy of splitting the data into two parts, analysis strategy of splitting the data into two parts, testing and verification, can be considered.testing and verification, can be considered.

5252

9. Same words, different meanings?9. Same words, different meanings? Professionals vs. lay terms (or even among Professionals vs. lay terms (or even among

scientific disciplines): not always the samescientific disciplines): not always the same

e.g., risk, hazard, odds, likelihood, rate, e.g., risk, hazard, odds, likelihood, rate, prevalence, incidence, valid, unbiased, prevalence, incidence, valid, unbiased, consistent, cost-effective (consistent, cost-effective (≠≠cheap), cheap), efficient, SD vs. SEefficient, SD vs. SE

People on the street may not distinguish People on the street may not distinguish RCT from observational study.RCT from observational study.

As easy and intuitive as possible but should As easy and intuitive as possible but should be correctbe correct

5353

10. Data sharing10. Data sharing Although … is tax-supported, its data are not Although … is tax-supported, its data are not

available to us. ….Policies governing data available to us. ….Policies governing data dissemination need to be reconsidered, although dissemination need to be reconsidered, although due regard must be paid to patient confidentiality. due regard must be paid to patient confidentiality. Only by thorough scrutiny can error be avoided. Only by thorough scrutiny can error be avoided. Transparency is the Transparency is the best assurance of scientific best assurance of scientific quality. (Freedman & Pettiti 2005)quality. (Freedman & Pettiti 2005)

Open the data to public (after sufficient de-Open the data to public (after sufficient de-identification). identification).

““Alas, we don’t have the process”Alas, we don’t have the process” –D. Kenney –D. Kenney

responding to ‘File-drawer problem, revisited’ by responding to ‘File-drawer problem, revisited’ by Young & Bang (2004)Young & Bang (2004)

5454

So, who are responsible?So, who are responsible? Authors/scientistsAuthors/scientists: lack of integrity, pressure to get : lack of integrity, pressure to get

grant, pub, CV, want to be famous or on the news, grant, pub, CV, want to be famous or on the news, publish to publishpublish to publish

EditorsEditors: too many journals, reviewers are busy, 2 : too many journals, reviewers are busy, 2 wks review time, shortage of stat reviewerswks review time, shortage of stat reviewers

MediaMedia: don’t think critically, to surprise or shock : don’t think critically, to surprise or shock peoplepeople

Lay personsLay persons: like more shocking news, may not : like more shocking news, may not use common senseuse common sense

We are all responsible for allWe are all responsible for all ---Dostoevsky ---Dostoevsky (Rose’s Epi book)(Rose’s Epi book)

5555

Try to remember as scientistsTry to remember as scientists If false positive and false negative results If false positive and false negative results

continued to be produced with disturbing continued to be produced with disturbing frequency, it might be so true that ‘we are fast frequency, it might be so true that ‘we are fast becoming a nuisance to society….people don’t becoming a nuisance to society….people don’t take us seriously anymore, and when they do… we take us seriously anymore, and when they do… we may unintentionally do more harm than good’may unintentionally do more harm than good’ --- ---Trichopoulos. Trichopoulos.

Remember it is extremely difficult to un-shock the Remember it is extremely difficult to un-shock the shocked.shocked.

Any researchers who use observational studies Any researchers who use observational studies may want to remind them of one question when may want to remind them of one question when they do research ‘is this result likely reproducible in they do research ‘is this result likely reproducible in RCT (if it will ever happen)?’RCT (if it will ever happen)?’

Transparency!!!! Ultimately, data to be shared.Transparency!!!! Ultimately, data to be shared.

5656

Try to remember as readersTry to remember as readers RCT vs. Obs studies (remember hierarchy)RCT vs. Obs studies (remember hierarchy) RCT is the best available but not perfect RCT is the best available but not perfect Use your common sense, while don’t forget Use your common sense, while don’t forget

serendipity (Start from the null. Ask ‘Why’, rather serendipity (Start from the null. Ask ‘Why’, rather than ‘Why Not’?) than ‘Why Not’?)

Check N (denom & num, large N does not fix bias)Check N (denom & num, large N does not fix bias) Think about Third variables (i.e., confounders)Think about Third variables (i.e., confounders) Be careful about meta-analysis Be careful about meta-analysis Do not worship p-value (<0.05)Do not worship p-value (<0.05) Perhaps, by Chance? Multiple testing. How many Perhaps, by Chance? Multiple testing. How many

questions/analyses? Anything hidden?questions/analyses? Anything hidden?

5757

Today’s QuotesToday’s Quotes

In God we trust; all others must bring dataIn God we trust; all others must bring data

(protocol and SAS output). --- W. Deming ((protocol and SAS output). --- W. Deming (& & K. Griffin)K. Griffin)

All models are wrong, but some are useful. All models are wrong, but some are useful. --- G. Box --- G. Box

Do whatever you what. But you should be Do whatever you what. But you should be responsible for what you do. responsible for what you do.

5858

Useful reading (research articles)Useful reading (research articles) Bang, H. (2009) Introduction to Observational Studies. Bang, H. (2009) Introduction to Observational Studies. Young, SS. and Bang, H. (2004) The file-drawer problem, Young, SS. and Bang, H. (2004) The file-drawer problem,

revisited. Science. revisited. Science. Taubes, G. (1995) Epidemiology faces its limits. Science.Taubes, G. (1995) Epidemiology faces its limits. Science. Freedman, DA. (2008) Oasis or Mirage. Chance.Freedman, DA. (2008) Oasis or Mirage. Chance. Shapiro, S. (2004) Looking to the 21Shapiro, S. (2004) Looking to the 21stst century: have we century: have we

learned from our mistakes, or are we doomed to compound learned from our mistakes, or are we doomed to compound them?them?

Ioannidis, JPA. (2006) Evolution and translation of research Ioannidis, JPA. (2006) Evolution and translation of research findings: From bench to where? PLOS.findings: From bench to where? PLOS.

Breslow, NE. (2003) Are statistical contributions to medicine Breslow, NE. (2003) Are statistical contributions to medicine undervalued? Biometrics. undervalued? Biometrics.

Austin, PC. (2006) Testing multiple statistical hypotheses Austin, PC. (2006) Testing multiple statistical hypotheses resulted in spurious associations: a study of astrological resulted in spurious associations: a study of astrological signs and health. signs and health.

Begg, C. (2001) The search for cancer risk factors: when Begg, C. (2001) The search for cancer risk factors: when can we stop looking? can we stop looking?

5959

Useful reading (newspaper articles)Useful reading (newspaper articles)

Do we really know what makes us healthy? --- NYT Do we really know what makes us healthy? --- NYT 20072007

Scientists do the numbers: Coffee is good for you --Scientists do the numbers: Coffee is good for you --no, it's bad. Epidemiological studies can come up no, it's bad. Epidemiological studies can come up with some crazy results, causing some critics to with some crazy results, causing some critics to wonder if they're really worthwhile. --- LAT 2007wonder if they're really worthwhile. --- LAT 2007

Women's Health Studies leave questions in place of Women's Health Studies leave questions in place of certainty --- NYT 2006certainty --- NYT 2006

Why so much medical research is rot --- The Why so much medical research is rot --- The Economist 2007.Economist 2007.

6060

Do we really know Do we really know what makes us healthy? what makes us healthy?

‘‘We know exactly why certain people We know exactly why certain people commit suicide. We don’t know, within the commit suicide. We don’t know, within the ordinary concepts of causality, why certain ordinary concepts of causality, why certain others don't commit suicide. …. We know others don't commit suicide. …. We know a great deal more about the causes of a great deal more about the causes of physical disease than we do about the physical disease than we do about the causes of physical health.’ causes of physical health.’ --- Scott Peck, --- Scott Peck, MD, in the book ‘The Road Less MD, in the book ‘The Road Less Travelled’.Travelled’.

6161

Modified from “Entering Hillsville”, by Dana Fradon 1977 New Yorker

Founded 1804 Population 3700 Mean 1978.66 Altitude 432 SD 1640.98 Total 5936