Objectives To enable you to recognize Forms of bias Forms of bias Threats to validity, internal and external Threats to validity, internal and external.

ObjectivesObjectives

To enable you to recognizeTo enable you to recognize Forms of biasForms of bias Threats to validity, internal and externalThreats to validity, internal and external Sampling errorsSampling errors

The Scientific MethodThe Scientific Method

Research**Research**

Systematic process to answer questions Systematic process to answer questions that generate knowledgethat generate knowledge

Formal standards and conditions that guide Formal standards and conditions that guide the procedurethe procedure

Can be applied to many situations rather Can be applied to many situations rather than one situationthan one situation

Seeks to confirm knowledge or discover Seeks to confirm knowledge or discover new knowledgenew knowledge

Reproducible.Reproducible.

The scientific method has three steps***The scientific method has three steps***..

Observation and description of a Observation and description of a phenomenon or group of phenomena. phenomenon or group of phenomena.

Formulation of an hypothesis to explain Formulation of an hypothesis to explain the phenomena. the phenomena.

Experimental tests to test predictions by Experimental tests to test predictions by several independent experimenters and several independent experimenters and properly performed experiments.properly performed experiments.

Scientific Method - Rules of Scientific Method - Rules of Testing**Testing**

Operational DefinitionOperational Definition

GeneralityGenerality

Controlled observationControlled observation

ConfirmationConfirmation

ConsistencyConsistency

Operational Definition**Operational Definition**

Descriptive statements made in a research Descriptive statements made in a research study should be carefully defined study should be carefully defined

These relate to either observations or These relate to either observations or measurements mademeasurements made

Forces the researcher to define concepts Forces the researcher to define concepts so they can be tested and retested. so they can be tested and retested.

Generality**Generality**

Findings must be able to explain more Findings must be able to explain more than the specific items/subjects being than the specific items/subjects being studied.studied.

Controlled Observation**Controlled Observation**

Change in variable A produces a change Change in variable A produces a change in variable Bin variable B

Only if all variables other than A can be Only if all variables other than A can be discounted, can you show A as the causediscounted, can you show A as the cause

Controls are established to account for Controls are established to account for other factors that may produce change.other factors that may produce change.

..

CAUSE & EFFECT CAUSE & EFFECT RELATIONSHIPS**RELATIONSHIPS**

Koch’s Postulates: developed in 19th Koch’s Postulates: developed in 19th century by Robert Koch. Conditions that century by Robert Koch. Conditions that needed to be fulfilled before needed to be fulfilled before microorganisms could be considered the microorganisms could be considered the cause of a disease.cause of a disease.

Koch’s Postulates**Koch’s Postulates**

If the cause is present the effect is present. If the cause is present the effect is present. The cause is sufficient to produce the effect.The cause is sufficient to produce the effect.

If the cause is absent, the effect is absent. If the cause is absent, the effect is absent. The cause is necessary to produce the effect.The cause is necessary to produce the effect.

Useful in studying conditions with a single Useful in studying conditions with a single cause.cause.

Contributory cause, less rigid, useful forContributory cause, less rigid, useful for multiple causes or chronic conditions.multiple causes or chronic conditions.

Repeated observation***Repeated observation***

A single observation does not guarantee A single observation does not guarantee something is true, so repeated something is true, so repeated observations are necessary.observations are necessary.

Confirmation**Confirmation**

Once an explanatory statement is made, Once an explanatory statement is made, other events can also be explainedother events can also be explained

One unsuccessful prediction and the One unsuccessful prediction and the statement is disproved.statement is disproved.

Consistency**Consistency**

If two explanatory statements are If two explanatory statements are contradictory, then one must be falsecontradictory, then one must be false

““The great tragedy of science, The great tragedy of science, the slaying of a beautiful the slaying of a beautiful

hypothesis by an ugly fact” hypothesis by an ugly fact” Thomas Henry HuxleyThomas Henry Huxley

PURPOSE OFPURPOSE OF Experimental Experimental RESEARCH DESIGN**RESEARCH DESIGN**

* To help the researcher answer the To help the researcher answer the research question.research question.

* ““To control for possible rival hypotheses To control for possible rival hypotheses or extraneous variables that might or extraneous variables that might compete with the independent variable compete with the independent variable as an explanation for the cause-effect as an explanation for the cause-effect relationship.”relationship.”

*Goal of Experimental ResearchGoal of Experimental Research

* All experimental research should attempt to All experimental research should attempt to control ALL the threats to internal validitycontrol ALL the threats to internal validity

* Research should try to control for asResearch should try to control for as many many threats to external validity as is possiblethreats to external validity as is possible

* ““The best way to insure the validity of an The best way to insure the validity of an experiment is for the researcher (or another experiment is for the researcher (or another researcher) to replicate the experiment.researcher) to replicate the experiment.

Hypothesis***Hypothesis***

Research hypothesis: a general prediction Research hypothesis: a general prediction of resultsof results

Null hypothesis: a difference does not Null hypothesis: a difference does not exist between experimental groupsexist between experimental groups

Alternative hypothesis: a difference does Alternative hypothesis: a difference does exist between experimental groupsexist between experimental groups

Rival hypothesis: Other explanation for Rival hypothesis: Other explanation for outcome of studyoutcome of study

Errors in testing the null Errors in testing the null hypothesis***hypothesis***

Type I error: Rejecting the Ho when it is Type I error: Rejecting the Ho when it is true (based on statistical tests) or claiming true (based on statistical tests) or claiming the effect exists when it does notthe effect exists when it does not

Type II error: Accepting the Ho when it is Type II error: Accepting the Ho when it is not true (based on statistical tests) or not true (based on statistical tests) or failing to detect the effect that existsfailing to detect the effect that exists

Subjects/SamplingSubjects/Sampling

*Sampling process includes two Sampling process includes two stepssteps

*Choosing the subjects to be Choosing the subjects to be included in the studyincluded in the study

*Determining which subjects receive Determining which subjects receive treatmenttreatment

Sample**Sample**

* Must define the POPULATION. The more Must define the POPULATION. The more a definition is limited, the less applicability a definition is limited, the less applicability to the general population.to the general population.

* A SAMPLE is a portion of the population A SAMPLE is a portion of the population investigated to draw conclusions about the investigated to draw conclusions about the entire populationentire population* Sampling is used because it is not practical to Sampling is used because it is not practical to

use the entire populationuse the entire population

Sampling Bias**Sampling Bias**

* Occurs when one of the two (or three) Occurs when one of the two (or three) study groups differ in one or more study groups differ in one or more variables that would affect the outcome of variables that would affect the outcome of the study.the study.

* Choosing a sample that is not reflective of Choosing a sample that is not reflective of the target population of the treatmentthe target population of the treatment

““Reporting of age data in clinical trials of Reporting of age data in clinical trials of arthritis. Deficiencies and solutions”arthritis. Deficiencies and solutions” Arch Intern Med 1993:153:243-8***Arch Intern Med 1993:153:243-8***

Review of 73 studiesReview of 73 studies

9664 patients9664 patients

2.1% of sample of was over age of 652.1% of sample of was over age of 65

62% of NSAID are consumed by population over 62% of NSAID are consumed by population over 65.65.

Older people more likely to have adverse reactionsOlder people more likely to have adverse reactions

In order to keep side effects minimal, older people In order to keep side effects minimal, older people excluded.excluded.

Types of Samples **Types of Samples **

* ConvenienceConvenience

* Random Random * Definition: each member of the population has Definition: each member of the population has

an equal opportunity to be included in the an equal opportunity to be included in the studystudy

Types of Random Samples***Types of Random Samples***

* Simple random samplingSimple random sampling

* Stratified random samplingStratified random sampling

* Cluster samplingCluster sampling

Simple random sampling***Simple random sampling***

* Purest form of sampling, but not Purest form of sampling, but not necessarily the best. After defining the necessarily the best. After defining the population each individual is randomly population each individual is randomly assigned a group. assigned a group.

Stratified random sampling***Stratified random sampling***

* Used when certain characteristics of a Used when certain characteristics of a population exhibit established proportions.population exhibit established proportions.

Cluster sampling**Cluster sampling**

* Gives everyone within a population an Gives everyone within a population an equal chance of being chosen for the equal chance of being chosen for the studystudy

* But subject to sampling error at each But subject to sampling error at each stage of clusteringstage of clustering

Random Sampling Methods*Random Sampling Methods*

* Table of random numbersTable of random numbers

* Computer generated numbersComputer generated numbers

* Draw straws, marbles, etc.Draw straws, marbles, etc.

Sample size***Sample size***

The larger the sample the easier it is to measure The larger the sample the easier it is to measure small but significant effectssmall but significant effects

If the effects of the study are to be great, a If the effects of the study are to be great, a smaller sample size could be usedsmaller sample size could be used

If using parametric tests you need at least 30 If using parametric tests you need at least 30 subjects in each subgroupsubjects in each subgroup

Bias*Bias*

* Webster’s Dictionary: “a one-sided inclination of Webster’s Dictionary: “a one-sided inclination of the mind”the mind”

* In research: “the systematic disposition of In research: “the systematic disposition of certain trial designs to produce results certain trial designs to produce results consistently better or worse than other trial consistently better or worse than other trial designs”designs”

Areas where bias can be Areas where bias can be introduced**introduced**

Selection biasSelection bias

Reporting qualityReporting quality

BlindingBlinding

DuplicationDuplication

GeographyGeography

Size of sampleSize of sample

StatisticsStatistics

LanguageLanguage

PublicationPublication

Importance of Randomization***Importance of Randomization***

Bandolier, “Does TENS Work”, Mar Bandolier, “Does TENS Work”, Mar 1997;37-31997;37-3 In randomized studies TENS found to be In randomized studies TENS found to be

effective in 2, ineffective in 15effective in 2, ineffective in 15 In inadequately or not randomized studies In inadequately or not randomized studies

TENS was found to be effective in 17 and TENS was found to be effective in 17 and ineffective in 2ineffective in 2

Non or poorly randomized trials increase Non or poorly randomized trials increase effect 30 to 41% (effect 30 to 41% (JAMA, JAMA, 1995, 273:408-12)1995, 273:408-12)

Importance of Blinding***Importance of Blinding***

* 17 % increase in effect (17 % increase in effect (JAMA,JAMA,1995, 273:408-1995, 273:408-12)12)

* Completely different result in blind and non-blind Completely different result in blind and non-blind studies (studies (Arch Int Med Arch Int Med 1998, 158:2235-2241)1998, 158:2235-2241)

Importance of Quality Reporting***Importance of Quality Reporting***

* Overall quality which includes randomizing, Overall quality which includes randomizing, blinding, dropout rate, threats to validity, etcblinding, dropout rate, threats to validity, etc

* Increases efficacy 25% (Increases efficacy 25% (Arch Int Med 1996, Arch Int Med 1996, 156:661-6 and 156:661-6 and Lancet 1998, Lancet 1998, 352:609-13)352:609-13)

Importance of Duplication Importance of Duplication (Covert)**(Covert)**

* Results of some trial are reported more than Results of some trial are reported more than onceonce

* Effect on meta-analysis increases efficacy 20% Effect on meta-analysis increases efficacy 20% ((BMJ 1997, BMJ 1997, 315:635-40315:635-40

Importance of Geography***Importance of Geography***

* Of particular importance to alternative therapiesOf particular importance to alternative therapies* Acupuncture almost universally positive when Acupuncture almost universally positive when

conducted in Asia but only positive 50% of the conducted in Asia but only positive 50% of the time when conducted in western countriestime when conducted in western countries

* Therapies other than acupuncture are Therapies other than acupuncture are overwhelmingly positive when conducted in overwhelmingly positive when conducted in China, Taiwan, Japan, or Russia., much more so China, Taiwan, Japan, or Russia., much more so than in other parts of the worldthan in other parts of the world

* Control Clin Trials 1998, Control Clin Trials 1998, 19:159-16619:159-166

Importance of size of Sample***Importance of size of Sample***

* Small trials may overestimate treatment effects Small trials may overestimate treatment effects by 30% by 30% (BMJ 1998, (BMJ 1998, 316:33-8 and 316:33-8 and Pain 1998 Pain 1998 78:217-220)78:217-220)

* Some researchers feel trials with less than 10 Some researchers feel trials with less than 10 subjects should be ignoredsubjects should be ignored

Importance of Statistics**Importance of Statistics**

* Statistical mistakesStatistical mistakes* Data presented as statistically significant when it is Data presented as statistically significant when it is

notnot* ““Fishing” or “data trawling, where a single statistical Fishing” or “data trawling, where a single statistical

significance is obtained and a paper is written significance is obtained and a paper is written around itaround it

* Power of words: even when there is no statistical Power of words: even when there is no statistical significance, words can make the test sound as if it significance, words can make the test sound as if it was successful. Especially apparent in abstractswas successful. Especially apparent in abstracts

* Data manipulationData manipulation

Importance of Language and Importance of Language and Publication Bias***Publication Bias***

* Often search strategies limit themselves to the Often search strategies limit themselves to the English language. Positive findings are more English language. Positive findings are more likely to appear in English language journals and likely to appear in English language journals and negative findings in non-English language negative findings in non-English language journals (journals (Lancet 1997,Lancet 1997,350:326-29) 350:326-29)

* There is a greater likelihood for positive trials There is a greater likelihood for positive trials than negative trials to be publishedthan negative trials to be published

TESTTEST

Pick a NumberPick a Number

1 2 3 41 2 3 4

All wild and wicked party All wild and wicked party people choose 3people choose 3

Threats to ValidityThreats to Validity

Statistical Conclusion ThreatsStatistical Conclusion ThreatsThreats to Construct ValidityThreats to Construct Validity InternalInternalExternalExternal

Statistical Conclusion Threats***Statistical Conclusion Threats***

Inadequacies of the power of the Inadequacies of the power of the statistical test used Parametric Vs. statistical test used Parametric Vs. NonparametricNonparametric

FishingFishing Use of unreliable measuresUse of unreliable measures Unreliable implementation of treatmentUnreliable implementation of treatment

Threats to Construct Validity***Threats to Construct Validity***

One must clearly define independent and One must clearly define independent and dependent variabledependent variable

Without clear definitions study cannot be Without clear definitions study cannot be generalizedgeneralized

*Internal Vs External Validity***Internal Vs External Validity***

* Internal validity-- refers to the causal Internal validity-- refers to the causal relationship. The effect between the relationship. The effect between the independent and dependent variablesindependent and dependent variables

* External validity—refers to how representative External validity—refers to how representative were the subjects in the study and can one were the subjects in the study and can one generalize the findings to other populations, generalize the findings to other populations, settings, treatments, etc. settings, treatments, etc.

*Threats to Internal Validity**Threats to Internal Validity**

* MaturationMaturation* TestingTesting* InstrumentationInstrumentation* MortalityMortality* SelectionSelection

* Compensatory rivalryCompensatory rivalry

Threats to Internal Validity (1)**Threats to Internal Validity (1)**

* MaturationMaturation* Not only do events (history) around the Not only do events (history) around the

subject change during the course of a subject change during the course of a research study, but the subject changes.research study, but the subject changes.* Biological growth and/or changesBiological growth and/or changes* Psychological changesPsychological changes


* TestingTesting* The effect of the repeated testing process The effect of the repeated testing process

may have an effectmay have an effect* E.g. repeated IQ or attitude testingE.g. repeated IQ or attitude testing


* InstrumentationInstrumentation* Mechanical devices wear, lose calibration & Mechanical devices wear, lose calibration &

otherwise become degraded with use.otherwise become degraded with use.* Human evaluators or judges may also have Human evaluators or judges may also have

varying degrees of experience and skill that varying degrees of experience and skill that impacts subject evaluation.impacts subject evaluation.


* Statistical RegressionStatistical Regression* Shift of the mean based on extreme scores Shift of the mean based on extreme scores

from study subjects. from study subjects. * Caused by imperfections in the test Caused by imperfections in the test

instruments (e.g. IQ tests, personality instruments (e.g. IQ tests, personality inventories)inventories)

*Threats to Internal Validity (5)Threats to Internal Validity (5)

* MortalityMortality* Subjects/patients dropping out during the Subjects/patients dropping out during the

study may not be that same as those who study may not be that same as those who remain.remain.


* Compensatory rivalryCompensatory rivalry* One groups realizes it is in the less effective One groups realizes it is in the less effective

treatment and either becomes resentful or treatment and either becomes resentful or becomes more motivated to improve becomes more motivated to improve

EXTERNAL VALIDITYEXTERNAL VALIDITY

11 Threats11 Threats

Threats to External Validity (1)***Threats to External Validity (1)***

* Experimental population vs. target Experimental population vs. target population…population…* Control: Random selection & random Control: Random selection & random

assignmentassignment

Threats to External Validity (2)**Threats to External Validity (2)**

* Interaction between the treatment and Interaction between the treatment and subject characteristics.subject characteristics.* Controls: Insure that characteristics of the Controls: Insure that characteristics of the

population/sample studied are not unique. population/sample studied are not unique.


* Describing the independent variableDescribing the independent variable* ““If the researcher is vague in describing the If the researcher is vague in describing the

independent variable, the reader would have independent variable, the reader would have difficulty in determining specifically what is to difficulty in determining specifically what is to be generalized”be generalized”* Control: Adequate detailed description of the Control: Adequate detailed description of the

independent variableindependent variable


* Describing the dependent variableDescribing the dependent variable* Were the test instruments or measurements Were the test instruments or measurements

reliable?reliable?* Were the appropriate measurement Were the appropriate measurement

instruments used?instruments used?* What was the degree of reliability of the What was the degree of reliability of the

examiners or judges?examiners or judges?


* Multiple treatment interferenceMultiple treatment interference* 2 or more treatments may be applied to the 2 or more treatments may be applied to the

same subjectsame subject* Researcher can not separate the effects of Researcher can not separate the effects of

one treatment from the otherone treatment from the other


* Interaction of time of treatment and the Interaction of time of treatment and the treatment effecttreatment effect* Usually the treatment effect is measured right Usually the treatment effect is measured right

after the treatmentafter the treatment* But it may last only a very short timeBut it may last only a very short time* Or any effect from treatment might be delayed Or any effect from treatment might be delayed

fore some time and not be measurable fore some time and not be measurable immediately after the treatmentimmediately after the treatment


* Interaction of history & treatment effectInteraction of history & treatment effect* Some events events such as political, social, Some events events such as political, social,

religious, sports, etc. may impact favorable or religious, sports, etc. may impact favorable or unfavorably one individual or group unfavorably one individual or group

* These events may impact the treatment effectThese events may impact the treatment effect


* Pretest and posttest sensitizationPretest and posttest sensitization* The researchers focus and attention to The researchers focus and attention to

measuring the dependent variable and the measuring the dependent variable and the subject becomes more conscious of the subject becomes more conscious of the variablevariable

* Especially true with personality, opinion or Especially true with personality, opinion or attitude studiesattitude studies


* Hawthorne EffectHawthorne Effect* Subjects knowledge that they are in an experiment Subjects knowledge that they are in an experiment

may influence their behaviormay influence their behavior* Demand characteristics- “refers to all the cues in the Demand characteristics- “refers to all the cues in the

treatment situation which might convey the purpose of treatment situation which might convey the purpose of the study…”the study…”

* Evaluation apprehension-some studies create anxiety & Evaluation apprehension-some studies create anxiety & affects the subjectaffects the subject

* Social desirability-patient wants to do the ‘right thing’Social desirability-patient wants to do the ‘right thing’

* Placebo effect-subject tends to believe in the Placebo effect-subject tends to believe in the effectiveness of the treatment because it is given in a effectiveness of the treatment because it is given in a scientific setting scientific setting

Threats to External Validity Threats to External Validity (10)**(10)**

* Experimenter or Rosenthal EffectExperimenter or Rosenthal Effect* In some studies it is possible for the In some studies it is possible for the

researcher to unintentionally modify the researcher to unintentionally modify the subject’s behavior through active or passive subject’s behavior through active or passive cues. cues.

Threats to External Validity Threats to External Validity (11)**(11)**

* Interaction of setting and treatmentInteraction of setting and treatment* Would a control group in Georgia exhibit the Would a control group in Georgia exhibit the

same characteristics as a experimental group same characteristics as a experimental group in Vermontin Vermont

Objectives To enable you to recognize Forms of bias Forms of bias Threats to validity, internal and external Threats to validity, internal and external.

Documents

Objectives To enable you to recognize Forms of bias Forms of bias Threats to validity, internal and external Threats to validity, internal and external.