Practical Problems

Practical ProblemsWeek 6 Lecture

H615 - Advanced Evaluation & Research DesignBy: Krissi Hewitt and Tassnym Sinky

The Ethics of Experimentation

Assign groups of 4-5 for activity:

• Ethical Codes• Informed Consent• IRBs• Legal Issues

Ethics of Experimentation in Practice

In regard to the human subjects in this study...★ Consider the following for your assigned topic as you watch the

video clip…• What is currently in place?• Violations?• Ways the process could be improved?

Ethics of Experimentation in Practice

• Discuss in your assigned groups • Come up with presentation points about your topic

5 minutes

Practical Problems: Recruitment• Participants affect construct and external validity• Can be difficult to locate some participants• Can get lower number than expected/desired for power• Volunteers vs. Non - different outcomes

Practical Problems: RecruitmentIf you can’t find your participants you can:

1. Conduct pre-surveys to locate2. Conduct pipeline studies3. Pilot tests of solicitations4. Hire trained outreach specialists5. Hire aggressive recruiters6. Study potential barriers to enrollment

Practical Problems: RecruitmentIf you don’t have enough participants you can:

1. Extend time frame2. Intensify outreach3. Alter eligibility (interactions may occur)4. Reduce prop. assigned to treatment5. Terminate experiment

Methods of Random Assignment• Simple RA• Restricted RA to force equal sample sizes• Restricted RA to force unequal sample sizes• Batch• Trickle process• Adaptive• RA from matches or strata

Develop a Random Assignment Method

• Case study on sheet of paper• As a group, come up with a random assignment method

for your case.• Present your case and method to the class

Questions to consider:What method(s) would you use? How does it work? What

drawbacks/benefits are there to this method?

Practical Problems: When Pretest Means Differ

Matching ★ before randomization★ match on a variable★ can match on pretest scores (preferred)

Stratifying★ more units than conditions ★ Ex. males and females are RA to treat/control

Matching and ANCOVA★ covariate - variable not part of the experiment but that influences

the outcome (DV)★ Added into regression model first to see effect on IV after.

• Reduces within-group error variance• Eliminates confounding variables• Limit # of covariates because high number inc. degrees of

freedom.

Matching and Stratifying• Both increase statistical power• Recommended for small sample sizes

Caveats:1. Matching on unrelated variable can dec. statistical

power without benefit 2. Should be done PRE RA

Successful Implementation of RA1. Plan to explain RA and its benefits2. Pilot procedures3. Develop procedures for implementing, controlling and monitoring

RA4. Hold negotiation meetings5. Have a backup plan in case RA fails6. Seize naturally occurring opportunities to facilitate RA7. Match RA design with experimental context

Chapter 10 - Treatment Implementation & Attrition

• The problems of treatment implementation and attrition threaten the very reason for doing an experiment: to get a good estimate of a treatment effect.

Treatment Implementation• Failure to get the full intervention

o Complianceo Assignment

• Crossing over to get a different treatment• Treatment diffusion

Inducing & Measuring Implementation

• Induction (implemented as intended)• Components of implementation

o Treatment deliveryo Treatment receipto Treatment adherence**

• Need to increase these three and measure each

Treatment Delivery• How to improve: Treatment manuals, training service

providers, giving verbal reminders to providers to include all treatment procedures, on-the-spot instructions to them during treatments, administering treatment by videotape/audiotape.• Complex, burdensome, long, inconvenience, expensive

treatments that require recipient to alter lifestyle will be delivered with less integrity• Measured with staff meetings, reviewing/scoring tapes,

assess differential delivery.

Treatment Receipt• How to improve: Written handouts, using established

communication strategies (i.e. repetition, making deliverer appear expert, question recipient about key treatment features to induce cognitive processing, have recipients keep logs of treatment-related activities)• Measured using manipulation checks, written tests of change

in recipients’ experience during treatment, monitoring physiological changes that the treatment should induce or asking recipient if they are confident in applying treatment skills.

Treatment Adherence• Threats: Lack time, forget to do it, unsure of correct treatment

procedures, disappointed by initial results, lack access to appropriate setting, lose motivation.

• How to improve: Assigning written homework, using family members to encourage adherence, physical aids, motivational cards, reinforcements

• Measured by interviewing recipients and other informants, biological assays

Overlooked Targets for Implementation Assessments

• Extra-study treatments that participants are getting while they are in an experiment.

• Those who are assigned to a no-treatment control condition.• Unplanned things that service providers do in treatment

(need capacity for discovery - Qualitative methods)

Assessing Program Models• Implementation involves inputs that the treatment requires,

contextual issues, funding.o Anticipate potential breakdowns in interventiono Descriptions of the context of implementation

• 2 methods to accomplish these goals:o Process modelo Good description of all these matters in study reports

Treatment Implementation in Efficacy and Effectiveness Studies

• Efficacy trials: treatments are often standardized and full implementation is the goal (usually prefered when treatment is first being studied)• Effectiveness trials: inclusion criteria loosened and recipient

compliance may be left variable so that researchers can gauge how well it will perform in less-than-ideal circumstances - yields an internally valid estimate of the effectiveness of that treatment-as-standardized-and-implemented.

Analyses Taking Implementation into Account

• When treatment implementation data are available, experiments may analyze them in three ways:o An intent-to-treat analysiso An analysis by amount of treatment actually receivedo By one of a variety of newly-developed analyses that try

to combine some of the benefits of the first two options.

POST-ASSIGNMENT ATTRITION• Post-assignment attrition: any loss of response from participants

that occurs after participants are randomly assigned to conditions.• Lowers statistical power• Treatment-correlated attrition threatens internal validity in

randomized experiment

Preventing Attrition• Attrition caused by treatment or by the research procedures

o Premature terminationo Noncompliance with medicationo Interpersonal conflict between research staff and participantso Solutions

(treatment) Manipulation - informing participants of the nature of the treatment and the expectations a client should have and tailoring treatments to more closely match client expectations.

(research process) Debriefing participants and asking dropouts why they failed to return

Retention & Tracking Strategies1. Gather complete location information at baseline from the

participants, friends, or relatives and any available records or agencies that may know their whereabouts

2. Establish formal and informal relationships with public and private agencies that may help find participants

3. Create a project identity 4. Emphasize the importance of tracking to project staff and

ensure they are well-supported and compensated

Retention and Tracking Strategies cont.5. Use the simplest and cheapest tracking methods first6. Make research involvement convenient and rewarding for

participants7. Expend the greatest amount of tracking effort at the initial

follow-up period when most attrition occurs8. Customize tracking efforts to the individual participant’s

situation and the study’s circumstance.

Preventing Treatment Attrition versus Measurement Attrition

• Measurement attrition: a failure to complete outcome measurement, whether or not treatment is completed• Treatment attrition: those research participants who do not

continue in treatment, whether or not they continue taking the measurement protocol.• Prevent measurement attrition even when you cannot

prevent treatment attrition

Minimizing Time and Obstacles Between Randomization & Treatment

• Attrition is lower when the time and obstacles between random assignment and treatment implementation are minimized• “Running-in” procedure• Methods to minimize attrition - increase selectivity,

reduce generalizability

Minimizing Treatment-Correlated Attrition• Differential attrition is more important than total attrition as a threat to internal validity.• Can result from various factors:

o Differential vigilanceo Desirability

• Solutionso Informed-consent procedure - agrees to accept assignment to any experimental

condition - reduces generalizabilityo Two-stage informed-consent procedure

Request participants cooperation with measurement - assignment to conditions is made from those who consent

Request agreement to the experimental conditions from those participants assigned to a treatment (not control unless there are ethical issues)

Those who refuse second consent are continued in the measurement protocol to which they already consented, reducing measurement attrition

Preventing Measurement Attrition

• Use of personal or telephone (versus) mail surveys• Use of incentives to answer • Providing prior notice of the questionnaire’s arrival• Using the foot-in-the-door method that gets the respondent to agree to a

smaller task first and a larger one later - ethics??• Personalizing letters and other forms of contact• Follow-up letters

• Appealing so social values/flattery were not effective

A flawed approach - replacing Dropouts

• Would only solve attrition as an internal validity threat if:o Both attrition and replacement are randomo Both former and replacement participants have the same latent

characteristics, especially as pertains to outcome

Analyses of AttritionGoal: how much it threatens the validity of a conclusion about

treatment effectiveness• Simple Descriptive Analyses• Identifying Different Patterns of Attrition• Accounting for Attrition when Estimating Effects

o Imputing Values for Missing Datao Estimating Effects in Data Sets with Attrition

Exercise• Same groups as before• Decide on a research topic of interest to you• Discuss how you would design your experiment to maximize valid

treatment implementation and minimize attrition.10 minutes