Research Systematic process to answer questions that generate knowledge Formal standards and conditions that guide the procedure Can be applied to many.

Research

• Systematic process to answer questions that generate knowledge

• Formal standards and conditions that guide the procedure

• Can be applied to many situations rather than one situation

• Seeks to confirm knowledge or discover new knowledge

• Reproducible.

The Scientific Method

• Observation and description of a phenomenon or group of phenomena.

• Formulation of an hypothesis to explain the phenomena.

• Experimental tests to test predictions by several independent experimenters and properly performed experiments.

Scientific Method - Rules of Testing

• Operational Definition

• Generality

• Controlled observation

• Confirmation

• Consistency.

Operational Definition

• Descriptive statements made in a research study should be carefully defined

• These relate to either observations or measurements made

• Forces the researcher to define concepts so they can be tested and retested.

Generality

• Findings must be able to explain more than the specific items/subjects being studied.

Controlled Observation

• Change in variable A produces a change in variable B

• Only if all variables other than A can be discounted, can you show A as the cause

• Controls are established to account for other factors that may produce change.

• .

CAUSE & EFFECT RELATIONSHIPS

• Koch’s Postulates: developed in 19th century by Robert Koch. Conditions that needed to be fulfilled before microorganisms could be considered the cause of a disease.

Koch’s Postulates

• If the cause is present the effect is present. The cause is sufficient to produce the effect.

• If the cause is absent, the effect is absent. The cause is necessary to produce the effect.

• Useful in studying conditions with a single cause.

• Contributory cause, less rigid, useful for multiple causes or chronic conditions.

Repeated observation

• A single observation does not guarantee something is true, so repeated observations are necessary.

Confirmation

• Once an explanatory statement is made, other events can also be explained

• One unsuccessful prediction and the statement is disproved.

Consistency

• If two explanatory statements are contradictory, then one must be false

“The great tragedy of science , the slaying of a beautiful

hypothesis by an ugly fact” Thomas Henry Huxley

*Objectives

* To gain an understanding of threats to internal and external validity

* To gain an understanding of and effect of bias

PURPOSE OF Experimental RESEARCH DESIGN

* To help the researcher answer the research question.

* “To control for possible rival hypotheses or extraneous variables that might compete with the independent variable as an explanation for the cause-effect relationship.”

*Goal of Experimental Research

* All experimental research should attempt to control ALL the threats to internal validity

* Research should try to control for as many threats to external validity as is possible

* “The best way to insure the validity of an experiment is for the researcher (or another researcher) to replicate the experiment.

Hypothesis

• Research hypothesis: a general prediction of results

• Null hypothesis: a difference does not exist between experimental groups

• Alternative hypothesis: a difference does exist between experimental groups

• Rival hypothesis: Other explanation for outcome of study

Errors in testing the null hypothesis

• Type I error: Rejecting the Ho when it is true (based on statistical tests) or claiming the effect exists when it does not

• Type II error: Accepting the Ho when it is not true (based on statistical tests) or failing to detect the effect that exists

Terminology

• Bias: Distortion introduced into research data by experimenter, instrumentation, participants or procedures

• Blinding: Experimenter in contact with subjects ignorant of participants category and/or participant is ignorant of category

• Cohort: Group of individuals moving through some process as a group.

Terminology

• Control group: comparison group to experimental group

• Dependent variable: data measures in research study

• Independent variable: phenomenon of interest in a research study (treatment)

• Intra-rater reliability: Consistency of observations of the same observer at different times

Terminology

• Inter-rater reliability: Consistency of observations of between observers in a research study

• Likert scale: Format for questionnaires in which subjects indicate degree of agreement or disagreement.

• Manipulation: Changes in the level of the independent variable.

Terminology

• Practice effects: The effect taking a test more than one time has on the outcome of subsequent tests.

• Randomization: Choosing groups in such a manner that the participant has an equal chance of being in either group

• Reliability: Stability or trustworthiness of a measure

Terminology

• Test-retest reliability: The stability of a test determined by administering the test, waiting an interval and readministering the test and correlating test results.

• Validity: Accuracy of a study or data collection instrument.

Bias

* Webster’s Dictionary: “a one-sided inclination of the mind”

* In research: “the systematic disposition of certain trial designs to produce results consistently better or worse than other trial designs”

Areas where bias can be introduced

• Selection bias• Reporting quality• Blinding• Duplication• Geography

• Size of sample• Statistics• Language• Publication

Importance of Randomization

• Bandolier, “Does TENS Work”, Mar 1997;37-3– In randomized studies TENS found to be effective

in 2, ineffective in 15– In inadequately or not randomized studies TENS

was found to be effective in 17 and ineffective in 2

• Non or poorly randomized trials increase effect 30 to 41% (JAMA, 1995, 273:408-12)

Importance of Blinding

* 17 % increase in effect (JAMA,1995, 273:408-12)* Completely different result in blind and non-blind

studies (Arch Int Med 1998, 158:2235-2241)

Importance of Quality Reporting

* Overall quality which includes randomizing, blinding, dropout rate, threats to validity, etc

* Increases efficacy 25% (Arch Int Med 1996, 156:661-6 and Lancet 1998, 352:609-13)

Importance of Duplication (Covert)

* Results of some trial are reported more than once* Effect on meta-analysis increases efficacy 20%

(BMJ 1997, 315:635-40

Importance of Geography

* Of particular importance to alternative therapies* Acupuncture almost universally positive when

conducted in Asia but only positive 50% of the time when conducted in western countries

* Therapies other than acupuncture are overwhelmingly positive when conducted in China, Taiwan, Japan, or Russia., much more so than in other parts of the world

* Control Clin Trials 1998, 19:159-166

Importance of size of Sample

* Small trials may overestimate treatment effects by 30% (BMJ 1998, 316:33-8 and Pain 1998 78:217-220)

* Some researchers feel trials with less than 10 subjects should be ignored

Importance of Statistics

* Statistical mistakes* Data presented as statistically significant when it is not

* “Fishing” or “data trawling, where a single statistical significance is obtained and a paper is written around it

* Power of words: even when there is no statistical significance, words can make the test sound as if it was successful. Especially apparent in abstracts

* Data manipulation

Importance of Language and Publication Bias

* Often search strategies limit themselves to the English language. Positive findings are more likely to appear in English language journals and negative findings in non-English language journals (Lancet 1997,350:326-29)

* There is a greater likelihood for positive trials than negative trials to be published

TEST

Pick a Number

1 2 3 4

Threats to Validity

Statistical Conclusion ThreatsThreats to Construct ValidityInternalExternal

Statistical Conclusion Threats

Inadequacies of the power of the statistical test usedParametric Vs. Nonparametric

FishingUse of unreliable measuresUnreliable implementation of treatment

Threats to Construct Validity

One must clearly define independent and dependent variable

Without clear definitions study cannot be generalized

*Internal Vs External Validity

* Internal validity-- refers to the causal relationship. The effect between the independent and dependent variables

* External validity—refers to how representative were the subjects in the study and can one generalize the findings to other populations, settings, treatments, etc.

*Threats to Internal Validity

* History* Maturation* Testing* Instrumentation* Mortality* Selection

* Compensatory equalization of treatments

* Compensatory rivalry

*Threats to Internal Validity (1)

* History* What else happens to the subject during the

course of the research?* The longer the time between observations or

reevaluations, the more likely that history might have an influence on the subject.


* Maturation* Not only do events (history) around the subject

change during the course of a research study, but the subject changes.

* Biological growth and/or changes

* Psychological changes


* Testing* The effect of the repeated testing process may

have an effect* E.g. repeated IQ or attitude testing


* Instrumentation* Mechanical devices wear, lose calibration &

otherwise become degraded with use.* Human evaluators or judges may also have

varying degrees of experience and skill that impacts subject evaluation.


* Statistical Regression* Shift of the mean based on extreme scores from

study subjects. * Caused by imperfections in the test instruments

(e.g. IQ tests, personality inventories)


* Mortality* Subjects/patients dropping out during the study

may not be that same as those who remain.


* Compensatory equalization of treatments* Researchers offer one group more than the

other due to guilt associated withholding effective treatment


* Compensatory rivalry* One groups realizes it is in the less effective

treatment and either becomes resentful or becomes more motivated to improve

*EXTERNAL VALIDITY

*11 Threats

*Threats to External Validity (1)

* Experimental population vs. target population…* Control: Random selection & random

assignment


* Interaction between the treatment and subject characteristics.* Controls: Insure that characteristics of the

population/sample studied are not unique.


* Describing the independent variable* “If the researcher is vague in describing the

independent variable, the reader would have difficulty in determining specifically what is to be generalized”

* Control: Adequate detailed description of the independent variable


* Describing the dependent variable* Were the test instruments or measurements

reliable?* Were the appropriate measurement instruments

used?* What was the degree of reliability of the

examiners or judges?


* Multiple treatment interference* 2 or more treatments may be applied to the

same subject* Researcher can not separate the effects of one

treatment from the other


* Interaction of time of treatment and the treatment effect* Usually the treatment effect is measured right

after the treatment* But it may last only a very short time* Or any effect from treatment might be delayed

fore some time and not be measurable immediately after the treatment


* Interaction of history & treatment effect* Some events events such as political, social,

religious, sports, etc. may impact favorable or unfavorably one individual or group

* These events may impact the treatment effect


* Pretest and posttest sensitization* The researchers focus and attention to

measuring the dependent variable and the subject becomes more conscious of the variable

* Especially true with personality, opinion or attitude studies


* Hawthorne Effect* Subjects knowledge that they are in an experiment may

influence their behavior* Demand characteristics- “refers to all the cues in the treatment

situation which might convey the purpose of the study…”

* Evaluation apprehension-some studies create anxiety & affects the subject

* Social desirability-patient wants to do the ‘right thing’

* Placebo effect-subject tends to believe in the effectiveness of the treatment because it is given in a scientific setting


* Experimenter or Rosenthal Effect* In some studies it is possible for the researcher

to unintentionally modify the subject’s behavior through active or passive cues.


* Interaction of setting and treatment* Would a control group in Georgia exhibit the

same characteristics as a experimental group in Vermont

Research Systematic process to answer questions that generate knowledge Formal standards and conditions that guide the procedure Can be applied to many.

Documents

false slide

effect of bias slide

hypothesis research

causeeffect relationship

single cause

contributory cause

cause controls

goal of experimental