Unit 4: Poor Experimental Design, Community of Scientists, Ethics, Critical Thinking Core 270 Spring 2008 Dr. Sharon Fredericks.

Unit 4: Poor Experimental Design, Community of Scientists,

Ethics, Critical ThinkingCore 270

Spring 2008

Dr. Sharon Fredericks

Experimental Design

• Flaws in the design of an experiment will cause the results to be _______________ and lead to ________________conclusions.– i.e., not taking into account confounding

variables, poor sample selection, inappropriate statistical analysis

– The unethical scientist will cover up errors and continue to publish, causing mistrust in his work and in the field itself.

Sampling Bias

• Differences between a sample and the population it represents should result only from random chance.

• When selecting the group to be studied, it must be done with no bias.– ___________________________in selecting all subjects

– __________________________________dividing the subjects into the control and experimental groups

• Otherwise, one is favoring the outcome. Sampling bias can be minimized with a ______________ group.

http://edweb.sdsu.edu/Courses/Ed690DR/Class04/samplingbiasbackground.html

Sources of sampling bias• Failure to adhere to the random sampling procedures. • Omission of specific subgroups of the population from the

sampling frame and therefore from the sample. • Faulty measuring devices

– this may be in terms of the specific questions used in a questionnaire

– may also arise in a survey that involves taking physical measurements, when the measuring device is incorrect, e.g., using a tape measure that has been stretched, so that all measurements are too small

• Violations of equal probability of selection principles because of duplicate listings in the sampling frame, or other causes.

• Non-response to a survey by specific subgroups of the population that are relevant to the measures of concern in the survey.

http://npts.ornl.gov/npts/1995/courseware/Useable_Nav3_19_27.html

Examples of Sampling Bias• Example given in text1: Diet pills were

found to be more effective because the experimental group had a higher percentage of healthier, more active, and more motivated people.

• A web-site poll or survey is inherently biased and inaccurate because– Only a ______________________________,

motivated people, will log on– A person may be able to vote multiple times

Example in the News

• The Union of Concerned Scientists conducted a scientific survey of more than 1,600 federal climate scientists.

• This was presented as evidence that the Bush administration was engaged in “wide-ranging political interference in research related to global warming.”

http://www.foxnews.com/printer_friendly_story/0,3566,251119,00.html

Poor Sample

• Low response rate of 17 percent. – “This means we don't know the views and experiences

of the other 83 percent...”• ______________________________of the group• “Many peer-reviewed academic journals will not

accept papers relying on samples smaller than 50 percent.”

• The survey also may be infected with a “selection bias”.– Scientists most upset about perceived interference filled

out the questionnaire.

Faulty Measuring Device (survey)

• Lumped into the same category scientists who said they actually experienced the alleged tampering and scientists who simply “perceived” that it happened to someone else.– 43% of the respondents reported they had perceived or personally

experienced changes or edits during review of their work that changed the meaning of their scientific findings

– In the study's appendix, and only 15 percent of the respondents said that they had actually experienced such interference.

• Similarly, 43% perceived or experienced “fear of retaliation for openly expressing concerns about climate change outside my agency.”– The question didn't ask how many actually experienced retaliation,

instead of just fearing it. Actually, only 14% personally harbored such a fear. The other 29% perceived it in others.

Faulty Measuring Device (survey)

• “The researchers assume that all these responses refer to officials' efforts to alter certain kinds of findings about global warming. But that is __________________ in the questionnaire.”– For example, a scientist could find a change in

working conditions without it having anything to do with global warming.

Misinformation reported by the Media

• NBC's Andrea Mitchell reported that “nearly half [the scientists] were pressured to eliminate the words ‘climate change’ or ‘global warming’…”

• The New York Times reported that 60% of the scientists “personally experienced” interference.

• ABC's Jake Tapper said, “scientists say their work on global warming has been watered down and twisted by the White House...”

Lack of Knowledge Regarding Subjects

• Determine if changes in the subject is due to the independent variable being tested or if it is a by product of the ____________________________

• Examples in text1: – Study on the physiological effect of LSD on an

elephant. Researchers based the dosage on weight and not on metabolism, causing permanent damage to the animal and resulting in it being euthanized.

– Wild prairie flower could only thrive on grazed land due to competition with grasses.

Another Example

• Passive sample collectors such as Minnow traps and Breder traps are simple, inexpensive and easily replaceable.

• They are used to collect fish for experiments.• Species specific characteristics of the fish such as

behavior and size will affect what is caught in minnow traps.

• So, the contents of a trap may not reflect the population of fish or a _________________sample.

http://www.yale.edu/eeb/layman/pdf/Layman%20and%20Smith%202001.pdf#search='sampling%20bias'

Lack of Statistical Significance

• The _______________and more ____________ the group, the more significant is the statistical analysis.

• Larger groups are expensive, so usually small pilot studies are done first. However, in drug studies, not all negative side effects may be encountered.

• Need to be sure that statistical, not anecdotal evidence, supports findings.– Anecdotal evidence is the experience of a single person

or a few people.

Statistical Analysis

• Scientists use more than one method and the most appropriate should be used, not the one that supports the hypothesis the best.

• Need to be sure the statistical analysis is done correctly.

• All analytical methods used should be discussed in the research paper.

Example• A study collected data from 51,603 women.• It reportedly showed that the 1,007 women who

increased their consumption of regular soft drinks over a period of four years from less than one per week to one or more per day gained an average of 10.3 pounds.

• Among the approximately 16,600 women who consumed more than one soft drink per day, the researchers reported 83 percent more cases of type 2 diabetes.

http://www.foxnews.com/story/0,2933,130263,00.html

Example Continued• When the researchers ___________________

adjusted their results for bodyweight (a risk factor for diabetes) and for caloric intake (a proxy measure for consumption of sweetened foods other than soda), the 83 percent increase dropped to a 32 percent increase.

• That result is of the same magnitude as the study’s reported 21 percent increase in diabetes among consumers of more than one diet soft drink per day. – Diet drinks do not contain any sugar at all.

Failure to Control Variables

• Results of a study should be due to changes in the variable of interest, and not due to a confounding variable.

• ________________________________is critical.• Example in text: Study of the effect of personal

interaction, bright colors and mobiles on infant development was confounded by the night cleaning staff playing and interacting with both the experimental and control groups of infants.

Another Example• Early studies of the effects of alcohol on heart disease were

_____________________________. • Alcohol actually prevents heart disease; while smoking

causes heart disease. • Unfortunately for investigators, smoking and alcohol

exposure are positively correlated (smokers drink more than non-smokers).

• Early studies appeared to show that alcohol caused heart disease; when actually the small protective effect of alcohol was masked by the large causative effect of smoking.

• Once smoking behavior was taken into account, the apparent negative effect (heart disease) of alcohol disappeared.

http://hta.uvic.ca/3lecturesample.html#ethics

Final Example• In August of 2001, an article was published in the

Journal of the American Medical Association (JAMA) stating that soy-based infant formula was not harmful and as good as cow milk-based formula

• Critics say that the study:– Was poorly designed– Did not ask the right questions – failure to control

variables– Did not use all data equally– Lacked statistical significance

http://www.mercola.com/2001/sep/15/soy_formula.htm

Poor Design

• Telephone interviews with 282 adults fed soy formula and 563 adults fed milk formula during controlled feeding studies at the University of Iowa between 1965-1978.

• "Data derived from telephone interviews, particularly interviews that ask a lot of embarrassing questions, cannot be used to draw any meaningful conclusions," said Dr. Naomi Baumslag, Professor of Pediatrics at Georgetown University.

Failure to Control Variables & did not use data equally

• The questions were geared to assess reproductive disorders and age of maturation.

• The study provided no information on dose length or quantity, nor on the ages at which ingestion ended, all vital in a study on toxicity.

• Dr. Mary Enig, President of the Maryland Nutritionists Association, stated, "The research team glossed over negative findings and ________ them from the Abstract and Conclusions..."

Lacked Statistical Significance & Financial Bias

• Many of the negative findings for the soy-fed group were not "statistically significant." (according to the study)

• But critics point out that the group of 282 soy-fed individuals was _________________________ for statistical significance to be achieved.

• Dr. Samuel Fomon, the main researcher of the paper, played an important role in the development of soy infant formula.

Disregard for the Condition of Subjects

• In this age, it is very difficult to do experiments on animals. It’s a long process to get permission. There are strict guidelines on care, treatment, etc.– Rules for housing

– Veterinarian oversees care

• Scientists must treat their subjects humanely in order to determine whether the results are due to the variable studied and not due to stress, poor health, and poor living conditions.

Should Some Experiments ever be done?

• With the present economic environment, funding is tight. Research groups must give strong support for their studies.

• In addition to the practical aspect, e.g., How will the results help mankind or produce a product?, there are the ethical questions:– Should we be trying to do this? E.g., cloning humans

– Is the study not putting unnecessary harm to the subjects? E.g., clinical drug trials

Community of Scientists2,3

• Scientists rarely work alone.• Scientists, being human beings, are subject to

biological and psychological factors.3 – Well-respected scientist can withhold invitations to

present papers at conferences or reject papers in the peer review process

• Balance between _____________ and sharing of knowledge and ____________ for fame and prizes.

• Science is a “social institution”.2 Social aspects influence the actual practice of science. – E.g., allocation of resources is strongly influenced by

politics3

Community of Scientists3

• Scientists may work in 3 areas– – –

• Scientists live within a society composed of many cultures and create their own cultures within their institutions.

• Within these communities, ideas progress from initial proposals to paradigms and may undergo a scientific revolution.

Scientific Norms2

• Also called “Ethos of science”– By Merton and Ziman

• Rules and foundations on how the community of scientists are expected to __________________.

• CUDOS– Communism– Universalism– Disinterestedness– Originality– Skepticism

Communism2

• Also called scientific communalism• Scientific knowledge _______________________.• Scientists are obligated to publish their results and

share with the community. • Reward is recognition and promotion.• Exceptions

– Military research– Commercial research– Computer security research

Universalism and Disinterestedness2

• Work of scientist should be judged on competence and merit and not on any ____________________________________

• Nationality, race, gender, religion, political persuasion

• Work should not be judged on the past accomplishments of the scientist

• Disinterestedness is not the same as uninterestedness.

• Scientists should conduct work without __________ and not for the purpose of gaining fame and fortune or to advance a political cause.

• Scientists’ purpose should be to advance knowledge.

Originality & Skepticism2

• Work should be original: new research, new ideas, new insight on old problems, new techniques

• Skepticism in the form of careful, __________________________________ of one’s work and the work of others is required.

Becoming a Scientist2

• Children are natural scientists because they are curious and want to understand what they observe.

• If this interest is maintained, then they may decide to become a scientist.

• Undergraduate degree, usually in a scientific field, where _______________________________________is learned. – Bachelor of Science

• Graduate degree in a specialty of a natural science– – Ph.D. (doctor of philosophy) earned

• Post-doctoral position– Additional skills in a related field– Usually still under the supervision of a more experienced scientist

• Job in of the 3 areas mentioned earlier.

Questionable Practices of Scientist2

• Honorary authorship• Inadequately supervising subordinates as

Ph.D. advisor, post-doc advisor, department and laboratory heads

• Interfering in another scientist’s research• Self-plagiarism• Misrepresentation of one’s own record• Conflict of interest

Reward System2

• – Higher salary or company sponsored “chairs”– Profits from patents – Cash awards (e.g., Nobel prize)

• _________________________________recognition– Eponymy (having something named after you – element, unit of

measure, period of time/age, law)– Nomination into a society– Citation in other people’s papers

• – Being hired at a famous university or lab– Being published in a highly selective journal– Receiving grant money from a prestigious agency

Nobel Prizes1

• Alfred Nobel (born 1833) invented blasting powder and dynamite. Because of the military applications of dynamite, he became a millionaire

• He left $9 million in a trust to establish the Nobel prizes. Winners’ works benefit mankind.– Established in 1900; first prizes given out in 1901.– 5 categories:

– Announced in the fall every year by the Royal Swedish Academy of Sciences

– 34 women: 11 in Literature, 12 for Peace, 7 in physiology/medicine, 3 for chemistry, 2 in physics with Marie Curie with 2 prizes (chemistry and physics)

Sources: www.nobelprize.org and http://nobelprize.org/nobel_prizes/lists/women.html

Nobel Prize• Laureates: 777 individuals and 20 organizations • The youngest winner was Lawrence Bragg (25 years old,

physics, 1915) and the oldest was Leonid Hurwicz (90 years old, economics, 2007).

• Linus Pauling is the only person to receive 2 unshared Nobel prizes (chemistry, peace). 3 other individuals (J. Bardeen, physics; Marie Curie, physics & chemistry; F. Sanger, chemistry) and 2 organizations (Red Cross & UNHCR) have received multiple Nobel prizes.

• 2 people have refused the Nobel prize– Jean-Paul Sartre (1964, Literature) had consistently declined all

official honors.– Le Duc Tho (1973, Peace) jointly with US Secretary of State Henry

Kissinger. He state that he was not in the position to accept due to the situation in Vietnam.

http://nobelprize.org/nobel_prizes/nobelprize_facts.html

2007 Nobel Prize Winners• Physics – Albert Fert (France) and Peter

Grünberg (Germany) ‘for the discovery of Giant Magnetoresistance’. – Created technology to read data on hard disks of

computers– Allowed for smaller hard disks by using

nanotechnology and creating more sensitive read-out heads

2007 Nobel Prize Winners• Chemistry– Gerhard Ertl (Germany) ‘for his

studies of chemical processes on solid surfaces’– Developed techniques for surface chemistry– Important for industrial processes

• Production of artificial fertilizers

• Cleaning up exhaust emissions in cars

• Determine how fuel cells function

• Use of catalysts on surfaces to speed up reactions

2007 Nobel Prize Winners• Physiology/medicine – Mario R. Capecchi (U. of

Utah), Sir Martin J. Evans and (United Kingdom), and Oliver Smithies (UNC, Chapel Hill) ‘for their discoveries of principles for introducing specific gene modifications in mice by the use of embryonic stem cells’– Led to the method called gene targeting in mice which

has been used in basic research and therapies– Basically, a single gene can be knocked out and made

inactive to determine the role of that gene in health and disease

– Use of homologous gene recombination to repair defective genes

– Combination of the 2 techniques

2007 Nobel Prize Winners

• Literature – Doris Lessing (United Kingdom) ‘that epicist of the female experience, who with scepticism, fire and visionary power has subjected a divided civilisation to scrutiny’

• Peace – Intergovernmental Panel on Climate Change (IPCC) (Switzerland) and Al Gore (USA) ‘for their efforts to build up and disseminate greater knowledge about man-made climate change, and to lay the foundations for the measures that are needed to counteract such change’

The Bank of Sweden Prize in Economic Sciences in Memory

of Alfred Nobel– Not originally in the Nobel’s will– Called by most the Nobel prize in Economics– 2007 Winner: Leonid Hurwicz (U. of MN),

Eric S. Maskin (Inst. For Advanced Studies, Princeton), Roger B. Myerson (U. of Chicago) ‘for having laid the foundations of mechanism design theory’

Scientific Commandments2

• Be honest• Never manipulate data• Be precise• Be fair with regard to priority and ideas• Be without bias with regard to the data and

ideas of your rivals• Do not make compromises in trying to

solve a problem

A Matter of Trust• Most science progresses by building and

expanding on previous work• Researchers must trust that the work is valid

– –

• If the work can not be independently reproduced, using a different technique if possible, doubt is cast on all that research group’s work.

• May spread to the whole field, causing funding to dry up.

• May tarnish image of all scientists

Misconduct in Science2

• Fabrication :• Falsification: _____________________

data or results, like deleting data that does not support hypothesis

• Plagiarism: use of someone else’s words or ideas without giving proper credit;

• Self-plagiarism: publishing the same work in more than one journal

Falsifying Documentation

• This is best caught through __________________. – Scientific work is formally reviewed by other experts in

the field before being published.– Same type of process is used for grant money

applications– Not perfect because of competition

• If a group does not want to openly share the evidence and results of a study, one should be suspicious.

• Another check:

Related Questionable Practices

• Failing to retain significant research data for a significant period of time

• Maintaining inadequate research records

• Refusing to give peers reasonable access to unique research materials or data that support papers

Reasons for Bias

• _____________is usually the reason for bias.– Also fame, prestige, awards

• A research group may want continued or more funding, so the results need to be favorable or progress needs to be demonstrated.

• A company sponsors a study in order for the results to favor the sale of a product.

Research with Human and Animal Subjects

• Limitations on what type of research is allowed• Projects must undergo scrutiny and pass review

boards• Must weigh potential knowledge versus pain and

suffering on the part of the subjects• Strict rules on the care of animal subjects to avoid

______________due to stress in living conditions• Examples of unethical studies:

– Tuskegee syphilis study in 1932

Informed Consent

• Most ________________________________ ethical aspect with human subjects

• Subjects are adequately informed of– Aims, methods, anticipated benefits, and potential

hazards of experiment

– Option to not participate or stop participating at any time

• Subjects should agree without coercion• Scientists need to document consent in writing

Informed Consent continued

• Deception in consent is only allowed if – Risk to the subjects is minimal or non-existent– Rights & welfare of the subjects are not

affected– It is necessary for the experiment to be

conducted without ____________________– True nature of the study is explained afterwards

Example #1• In February 2000, a promising young physicist named

Jan Hendrik Schön published some startling experimental results.

• Schön and his partners had started with molecules that don't ordinarily conduct electricity, and claimed they had succeeded in making them behave like semiconductors, the circuits that make computers work.

• In a field where publishing 2-3 articles a year makes you productive, Schön was the lead author on dozens of articles, most of them appearing in the industry-leading journals.

http://www.salon.com/tech/feature/2002/09/16/physics/print.html

Example #1 continued

• A small group of researchers at Bell Labs contacted Princeton physics professor Lydia Sohn and whispered that all was not right with Schön's data.

• She and Cornell University's Paul McEuen found some disturbing coincidences in Schön's results: The same graphs were being used to illustrate the outcomes of completely different experiments.

• "You would expect differences," she said, "but the figures were identical. It was a smoking gun."

Example #1 continued

• The duplicated graphs are not the only smoking gun. There's also the serious problem that despite numerous attempts, no other physicist has _______________________________Schön's results. Physicists around the world have spent _____________________________________

• If no one else can repeat the results of an experiment, both experiment and experimenter come under suspicion. "It is part of the process of science," says investigative committee head Beasley, "that things get winnowed out because they don't work."

Example #2• 2 years after it was published in Science, a highly

prestigious journal, an article is retracted.• It purported to show that the recreational drug

Ecstasy (methylenedioxymethamphetamine, or MDMA) caused severe damage to dopaminergic neurons, predisposing takers to Parkinson disease

• The retraction came about because George Ricaurte of Johns Hopkins University, lead author of the paper in question, discovered that certain reagents had been mislabeled after ____________ ________________________________________

http://www.biomedcentral.com/news/20030916/04

Example #2 continued

• "It's an outrageous scandal," said Leslie Iversen, a prominent pharmacologist who holds professorships at King's College London and Oxford University.

• "It's another example of a certain breed of scientist who appear to do research on illegal drugs mainly to show what the governments want them to show. They extract ______________________________ from the government to do this sort of _________ work… I hope the present retraction and embarrassment to the people involved will be some sort of lesson to them.“

Example #3

• Woo Suk Hwang of Seoul National University and his team reported that they had removed DNA from a human egg, replaced it with DNA from a mature cell of the same young woman, and then grown the altered egg into a cluster of cells. – Published in March 12, 2004 issue of Science, no

human cloning had been previously reported in a scientific journal

– DNA testing showed that the cloned cell had DNA of mature cell and not that of egg cell, removing the possibility that the cloned cell was an egg cell that retained its own DNA

Example #3 continued

– “This is reality,” says stem cell researcher John Gearhart of Johns Hopkins University. “Here is a bona fide, refereed journal saying that a human embryo has been cloned and a cell line derived from it.”[Science News (SN) 2/14/04]

• May 2005: Improve technique leading to 11 cloned embryonic stem cell lines, tailor-made to individuals (patients were women and men, ages 2-56)– Published in June 17, 2005 issue of Science

• The following August, they announced that they cloned the first dog

Example #3: Red Flags

• Retraction of paper request by Hwang and Schatten in December 2005– Gerald Schatten of the University of Pittsburgh, asked

that Science remove his name from the paper, citing “substantial doubts about the paper's accuracy.” He also referred to allegations “from someone involved in the experiments that certain elements of the report may be ______________________”[USA Today, 12/15/2005]

– Study coauthor Roh Sung Il of MizMedi Hospital in Seoul claimed in the Korean media that Hwang had told him that some of the stem cell lines described in the paper had been replaced by _______. [SN: 12/24/2005]

Example #3 continued

• Hwang resigned from heading a new embryonic stem cell clearinghouse last month, acknowledging ethics lapses. Two researchers in his lab had donated eggs for research and $1,445 payments were made to other egg donors, contrary to __________________________________ made by the 2004 study authors.

• In a letter released Tuesday by Science, eight prominent stem cell researchers -- including Ian Wilmut, who cloned Dolly the sheep in 1997 -- called for an ____________ _________________of the Hwang lab's cell-cloning feats.

• Questions have arisen about the validity of DNA fingerprinting patterns that were a key component of the 2004 study.

[USA Today, 12/15/2005]

Example #3: Ramifications

• “Either way, the fiasco is a setback for stem cell research, other scientists say “

• “ ‘I think this is a shame,’ says Leonard I. Zon, a stem cell researcher at Harvard Medical School in Boston. The controversy won't be cleared up until _______________________________________, he says. Meanwhile, Zon says that he hopes that stem cell research will proceed.”

[Science News 12/24/2005]

Whistleblowing

• The reporting of unethical behavior.• The person who does this is called the

whistleblower.• Unfortunately, this is very difficult due to the

consequences.– Whistleblowers undergo scrutiny, professional attacks,

harassment, professional losses

• Should be done cautiously and not to cause the downfall of competitors

Critical Thinking2

• Richard Paul developed a list of 35 strategies of critical thinking– Affective strategies involve attitudes and behavior– Cognitive strategies involve the basic skills

• My top 5– Affective: 1) thinking independently; 2) developing

intellectual perseverance– Cognitive: 3) comparing analogous situations; 4)

analyzing or evaluating arguments, interpretations, beliefs, or theories; 5) reading critically

References

1. Kelinsteuber, et al., Natural Science 5th edition, King’s College, PA, 2004.

2. Lee, J.A., The Scientific Endeavor, Addison Wesley Longman, CA, 2000.

3. Ben-Ari, Just a Theory: Exploring the Nature of Science, Prometheus Books, Amherst, NY, 2005.

4. References individually cited.