Pilot Studies Nuts and Bolts Deborah Grady, MD, MPH Professor of Medicine Associate Dean for Clinical and Translational Research UCSF.

Pilot StudiesPilot StudiesNuts and BoltsNuts and Bolts

Deborah Grady, MD, MPHDeborah Grady, MD, MPH

Professor of MedicineProfessor of Medicine

Associate Dean for Clinical and Associate Dean for Clinical and Translational ResearchTranslational Research

UCSFUCSF

What is a Pilot Study?What is a Pilot Study?

• Early study of a new idea designed to obtain Early study of a new idea designed to obtain preliminary data to support a larger study preliminary data to support a larger study –Usually smallUsually small–Goals often focus on feasibility and logisticsGoals often focus on feasibility and logistics–Outcomes often surrogates Outcomes often surrogates –Typically unfunded or funded intramurallyTypically unfunded or funded intramurally–Require a lot of work - protocol, IRB approval, Require a lot of work - protocol, IRB approval,

data forms, recruitment, data entry, analysis…data forms, recruitment, data entry, analysis…

• Distinguish from Distinguish from pre-testingpre-testing of methods, of methods, questionnaires, measuresquestionnaires, measures

Clear Reasons to Do a Pilot StudyClear Reasons to Do a Pilot Study

• Demonstrate feasibilityDemonstrate feasibility–Determine if you can do itDetermine if you can do it

• Improve logisticsImprove logistics–Determine the fastest, easiest, most cost-Determine the fastest, easiest, most cost-

effective way to do iteffective way to do it

• Estimate costEstimate cost

• Demonstrate to fundersDemonstrate to funders– that you can do it within budget!that you can do it within budget!

Possible Reasons to Do Possible Reasons to Do a Pilot Studya Pilot Study

• Decide whether to do larger studyDecide whether to do larger study

• Estimate parameters for designing Estimate parameters for designing larger studylarger study–Rate of outcomes in placebo groupRate of outcomes in placebo group–Benefit (effect size)Benefit (effect size)–Variability of outcome measureVariability of outcome measure

Yoga to Prevent DiabetesYoga to Prevent Diabetes

• Evidence suggests yoga decreases Evidence suggests yoga decreases stress, improves metabolic risksstress, improves metabolic risks

• Possible larger studyPossible larger study– randomized trial of yogarandomized trial of yoga vs. control in adults vs. control in adults with metabolic syndromewith metabolic syndrome– outcome - new diabetesoutcome - new diabetes

Yoga Pilot Study DesignYoga Pilot Study Design

• Randomized trial in persons 30-65 yo with Randomized trial in persons 30-65 yo with metabolic syndromemetabolic syndrome

• Intervention - 10 yoga poses taught in Intervention - 10 yoga poses taught in group classes x 10 weeks + home practicegroup classes x 10 weeks + home practice

• Outcomes - diabetes; fasting glucose, Outcomes - diabetes; fasting glucose, insulin, lipids; insulin sensitivity, heart rate insulin, lipids; insulin sensitivity, heart rate variability, measures of stress, BMI, etc.variability, measures of stress, BMI, etc.

FeasibilityFeasibility

• RecruitmentRecruitment– Sources of participantsSources of participants– N of participantsN of participants– % eligible% eligible– % willing to enroll% willing to enroll– Time and cost of recruitmentTime and cost of recruitment

Responded to Ads/fliersResponded to Ads/fliers388388

Telephone ScreenTelephone Screen255255

Eligible for Clinical Eligible for Clinical Screen 89Screen 89

Eligible 28Eligible 28

YogaYoga

1414

ControlControl

1212

FeasibilityFeasibility

• Interventions and measuresInterventions and measures– Can obese persons perform the postures? Can obese persons perform the postures? – Will there be side effects?Will there be side effects?– Can participants do insulin clamp, 2-h GTT, Can participants do insulin clamp, 2-h GTT,

other tests and questionnaires?other tests and questionnaires?

• Adherence and follow-upAdherence and follow-up– Will participants stay in the Will participants stay in the

study and practice at home?study and practice at home?

Logistics and CostLogistics and Cost

• What staff are needed?What staff are needed?

• How should the flow of visits be organized?How should the flow of visits be organized?

• Where will participants park? Where will participants park?

• How long do visits take?How long do visits take?

• Can participants answer questionnaires while Can participants answer questionnaires while completing GTT?completing GTT?

• How much will the How much will the trial cost?trial cost?

Yoga Pilot Study FindingsYoga Pilot Study Findings

• Trial is feasibleTrial is feasible– Able to recruit and randomizeAble to recruit and randomize– Participants able to do 9 Yoga posturesParticipants able to do 9 Yoga postures– Adherence excellent Adherence excellent

• 13.7 classes attended (goal 14)13.7 classes attended (goal 14)• 100 min practice/week (goal 90)100 min practice/week (goal 90)

• Logistics improvedLogistics improved– One posture (Downward Dog) droppedOne posture (Downward Dog) dropped– Added video for home practiceAdded video for home practice– Visit time reducedVisit time reduced

• Cost reasonableCost reasonable

Yoga Pilot Study ResultsYoga Pilot Study Results

VariableVariable DifferenceDifference P-valueP-value FavorsFavors

Weight, Weight, kgkg 2.3 2.3 0.360.36 YY

BMI, BMI, kg/mkg/m22 0.9 0.9 0.300.30 YY

Waist, Waist, cmcm 0.00.0 0.990.99 - -

SBP, SBP, mmHgmmHg 9.2 9.2 0.070.07 YY

DBP, DBP, mmHgmmHg 4.6 4.6 0.100.10 YY

Perceived Stress ScalePerceived Stress Scale 0.40.4 0.120.12 YY

Energy Level Energy Level -0.8-0.8 0.0080.008 YY

Psych Well-being Psych Well-being -0.3-0.3 0.060.06 YY

Depression (CES-D)Depression (CES-D) 0.20.2 0.450.45 YY

Self-rated healthSelf-rated health -0.3-0.3 0.350.35 YY

Insomnia (% none)Insomnia (% none) 30.030.0 0.010.01 YY

Yoga Pilot Study ResultsYoga Pilot Study Results

VariableVariable DifferenceDifference P-valueP-value FavorsFavors

Insulin Sens IndexInsulin Sens Index 0.30.3 0.560.56 CC

Fast. GlucoseFast. Glucose -2.1-2.1 0.530.53 CC

2-hour Glucose2-hour Glucose 0 0 0.990.99 --

HDL- cholesterolHDL- cholesterol 2.02.0 0.500.50 CC

TriglyceridesTriglycerides 14.214.2 0.550.55 YY

ALTALT -2.5 -2.5 0.580.58 CC

YIKES!YIKES!

Yoga Pilot Bottom LineYoga Pilot Bottom Line

• Full-scale Yoga study funded by Full-scale Yoga study funded by NCCAM and in progressNCCAM and in progress

Should Pilot Studies Include Should Pilot Studies Include Randomization?Randomization?

• Maybe not…Maybe not…–Nonrandomized studies can generally be Nonrandomized studies can generally be

smaller, faster and cheapersmaller, faster and cheaper–Obtain more data on intervention if all Obtain more data on intervention if all

participants receive itparticipants receive it

• Maybe…Maybe…– If need to know if randomization is possibleIf need to know if randomization is possible– If need to know anything about the control If need to know anything about the control

group (adherence, placebo effect, etc)group (adherence, placebo effect, etc)– If trying to estimate between group effectsIf trying to estimate between group effects

Do Pilots Need Control GroupsDo Pilots Need Control Groups??

• Phase I trials (evaluate short-term safety)Phase I trials (evaluate short-term safety)– generally uncontrolledgenerally uncontrolled

• Phase II trials (evaluate range of doses)Phase II trials (evaluate range of doses)– generally randomized and controlled generally randomized and controlled

• Feasibility, logistics and costFeasibility, logistics and cost– controls generally not needed controls generally not needed

• Effect and sample size estimatesEffect and sample size estimates– controls helpful, esp. if need to control for controls helpful, esp. if need to control for

placebo effectplacebo effect

How Do You Estimate Sample Size How Do You Estimate Sample Size for a Pilot Studyfor a Pilot Study

• Sample size often set by traditionSample size often set by tradition–Lab studies 1 - 3Lab studies 1 - 3–Many phase I trials start with 3, add 3Many phase I trials start with 3, add 3–Pilots for feasibility often 5-20Pilots for feasibility often 5-20–Pilots to estimate effect size can be largePilots to estimate effect size can be large

• Sample size often dictated by resources Sample size often dictated by resources and timeand time

Should You Publish Pilot Results?Should You Publish Pilot Results?

• Restorative yoga in adults with metabolic Restorative yoga in adults with metabolic syndrome: a randomized, controlled pilot syndrome: a randomized, controlled pilot trial. Cohen BE, Chang AA, Grady D, trial. Cohen BE, Chang AA, Grady D, Kanaya AM. Metab Syndr Relat Disord. Kanaya AM. Metab Syndr Relat Disord. 2008, 3:223-9.2008, 3:223-9.

• PubMed search for PubMed search for ““pilotpilot”” in title, last 12 in title, last 12 months, in Englishmonths, in English–2649 publications!2649 publications!

Funding for PilotsFunding for Pilots

• UCSFUCSF– Research Allocation Program (RAP)Research Allocation Program (RAP)– Training programs, mentorsTraining programs, mentors

• NIH - R03, R21, U34, K awardsNIH - R03, R21, U34, K awards

• OtherOther– Foundations, philanthropyFoundations, philanthropy– Pharmaceutical companiesPharmaceutical companies

• PI discretionary funds PI discretionary funds

Summary: Pilot StudiesSummary: Pilot Studies

• Crucial to the development and testing of Crucial to the development and testing of novel ideas and interventionsnovel ideas and interventions

• Necessarily small and under-poweredNecessarily small and under-powered

• Have different or additional goalsHave different or additional goals

• Are recognized as a specific study design Are recognized as a specific study design by NIH, local funders, journalsby NIH, local funders, journals

• But little attention to design, methods and But little attention to design, methods and interpretation of pilot studiesinterpretation of pilot studies

Nuts and BoltsNuts and Bolts

• ResourcesResources •• Contracts Contracts

• BudgetsBudgets •• Study team Study team

• SpaceSpace •• Recruitment Recruitment

• Start-upStart-up •• Forms Forms

• Adherence and follow-upAdherence and follow-up

• Money MattersMoney Matters •• Closeout Closeout

Funding SourcesFunding Sources

• Small trialsSmall trials–NIH - R03, R21, R01NIH - R03, R21, R01–Specialty organizations/foundationsSpecialty organizations/foundations

• AHA, ACS, ACOG, ADA, etcAHA, ACS, ACOG, ADA, etc• FoundationsFoundations

–Pharmaceutical, device companiesPharmaceutical, device companies

• Large trialsLarge trials–NIH - R01, P01, U01NIH - R01, P01, U01–VA, AHRQ, CDC, DODVA, AHRQ, CDC, DOD–Pharmaceutical, device companiesPharmaceutical, device companies

UCSF ResourcesUCSF Resources

• Senior colleagues in your disciplineSenior colleagues in your discipline

• Experienced trialists and trial staffExperienced trialists and trial staff

• CTSI Consultation ServiceCTSI Consultation Service–designdesign–ethicsethics–statistical issuesstatistical issues–database development and managementdatabase development and management

ContractsContracts

• Required if trial supported by industryRequired if trial supported by industry

• Must be approved by UCSF Industry Must be approved by UCSF Industry Contracts & companyContracts & company’’s legal divisions legal division– lots of puzzling legaleselots of puzzling legalese–you should understand and revise, esp. you should understand and revise, esp.

scope of workscope of work–best to start with UCSF template, revise best to start with UCSF template, revise

and submit to companyand submit to company

• Between UCSF Regents and companyBetween UCSF Regents and company–signed by PIsigned by PI

Elements of a ContractElements of a Contract

• Scope of workScope of work

• Timetable for deliverablesTimetable for deliverables

• Budget and timing of paymentsBudget and timing of payments–get a bolus up frontget a bolus up front–dondon’’t link payment to deliverables out of t link payment to deliverables out of

your controlyour control

• Ownership of data and publishing rightsOwnership of data and publishing rights

• Rules for breaking contractRules for breaking contract

• Confidentiality, indemnificationConfidentiality, indemnification

• Patents and inventionsPatents and inventions

BudgetsBudgets

• NIH-style (NIH, VA, AHRQ, CDC)NIH-style (NIH, VA, AHRQ, CDC)–prepared far in advanceprepared far in advance–governed by strict rulesgoverned by strict rules

• funds restricted to categoriesfunds restricted to categories• permission for carry-over requiredpermission for carry-over required• certain expenses not allowedcertain expenses not allowed• subject to government auditsubject to government audit

• Pharmaceutical company budgetsPharmaceutical company budgets–prepared just before trial startsprepared just before trial starts–negotiablenegotiable–much more flexiblemuch more flexible

Money MattersMoney Matters

• Pre-award managerPre-award manager–help prepare budget and help prepare budget and ““faceface”” pages pages–help with budget justificationhelp with budget justification–make sure you follow rules/timelinesmake sure you follow rules/timelines

• Post-award managerPost-award manager–pay salaries, buy equipment and suppliespay salaries, buy equipment and supplies–monthly report of expendituresmonthly report of expenditures–projections over life of trialprojections over life of trial

Who is the Study Team?Who is the Study Team?

• Principle investigatorPrinciple investigator

• Project director/clinic coordinatorProject director/clinic coordinator

• RecruiterRecruiter

• Data managerData manager

• Programmer/analystProgrammer/analyst

• StatisticianStatistician

• Quality control supervisorQuality control supervisor

• Administrative assistantAdministrative assistant

• Financial/personnel managerFinancial/personnel manager

How to Hire the Study TeamHow to Hire the Study Team

• Work with your department or Work with your department or unit personnel (HR) managerunit personnel (HR) manager–write a job descriptionwrite a job description–decide on job series, step, salarydecide on job series, step, salary–post the job at UCSF and advertisepost the job at UCSF and advertise– review resumesreview resumes– interviewinterview–selectselect

Where to Find the Study TeamWhere to Find the Study Team

• Other studiesOther studies

• UCSF employeesUCSF employees

• Recent graduates, studentsRecent graduates, students

• Friends and colleaguesFriends and colleagues

• CraigCraig’’s Lists List

• Chronicle, webChronicle, web

Training the Study TeamTraining the Study Team

• Describe, emphasize and demonstrate Describe, emphasize and demonstrate the importance of:the importance of:– following the protocol/operations manualfollowing the protocol/operations manual–meeting recruitment goalsmeeting recruitment goals–complete adherence and follow-upcomplete adherence and follow-up– full outcome ascertainmentfull outcome ascertainment–maintaining participant safety and maintaining participant safety and

confidentialityconfidentiality–ethical conduct of researchethical conduct of research

• Formal training and certification of staffFormal training and certification of staff

Operations Manual and Operations Manual and Training MeetingTraining Meeting

• Purpose and design of the trialPurpose and design of the trial

• Each measurement and visit procedureEach measurement and visit procedure– methodsmethods– qualified personnelqualified personnel– calibration and testing calibration and testing

• Completing and altering study formsCompleting and altering study forms

• Data entry or submissionData entry or submission

• Safety and ethical issuesSafety and ethical issues

• Certification and training of new staffCertification and training of new staff

Team LeadershipTeam Leadership

• Management training usefulManagement training useful

• You are responsible and the bossYou are responsible and the boss

• Keep your staff happyKeep your staff happy– involvement in the science involvement in the science –salary, working conditionssalary, working conditions– level of responsibilitylevel of responsibility

• Set an exampleSet an example

• Promote good staff interactionsPromote good staff interactions

• Some staff should be firedSome staff should be fired

SpaceSpace

• ConsiderationsConsiderations–accessibility, parkingaccessibility, parking–design and décordesign and décor–privacyprivacy–clinical needs, special testsclinical needs, special tests–safety measuressafety measures–costcost

• CTSI CRCsCTSI CRCs

• On-campus or medical spaceOn-campus or medical space

• Off campus rentalOff campus rental

RecruitmentRecruitment

• Hire an experienced Hire an experienced recruiterrecruiter

• Provide adequate Provide adequate time and moneytime and money

• Monitor resultsMonitor results

• Make changesMake changes– different approachesdifferent approaches– more centersmore centers– longer recruitmentlonger recruitment– change eligibilitychange eligibility

Plan Multiple Approaches to Plan Multiple Approaches to RecruitmentRecruitment

• Referral from providersReferral from providers• Targeted mailing (age, gender, disease)Targeted mailing (age, gender, disease)

• UCSF Integrated Data RepositoryUCSF Integrated Data Repository• Kaiser, DMV, HCFA, registriesKaiser, DMV, HCFA, registries

• AdvertiseAdvertise– hospital, clinics, special sites, churcheshospital, clinics, special sites, churches– radio, newspaper, TV, internet (Facebook)radio, newspaper, TV, internet (Facebook)– celebrities, leaderscelebrities, leaders

• CTSI Participant Recruitment ServiceCTSI Participant Recruitment Service

Follow HIPPA and confidentiality rulesFollow HIPPA and confidentiality rules

Start-upStart-up

• ProtocolProtocol

• Operations ManualOperations Manual

• FormsForms

• IRB approvalIRB approval

• HiringHiring

• TrainingTraining

• Database designDatabase design

• Database validationDatabase validation

Forms and Database DevelopmentForms and Database Development

• Define variables, create data dictionaryDefine variables, create data dictionary

• Determine method of data entryDetermine method of data entry– Humans Humans – Machine readable formsMachine readable forms

• Design database (CTSI Consultation)Design database (CTSI Consultation)

• Use prior forms - forms librariesUse prior forms - forms libraries

• Pretest, validatePretest, validate

Adherence and Follow-upAdherence and Follow-up

• Monitor ratesMonitor rates

• Obtain contact data at baselineObtain contact data at baseline

• Pleasant, professional staffPleasant, professional staff

• Develop personal relationshipsDevelop personal relationships

• Encourage resumption of Encourage resumption of intervention and return to follow-upintervention and return to follow-up

• Cards, small gifts, meetings, partiesCards, small gifts, meetings, parties

Quality ControlQuality Control

• Standard Operating ProceduresStandard Operating Procedures– protocolprotocol– operations manualoperations manual– statistical analysis planstatistical analysis plan– data and safety monitoring plandata and safety monitoring plan

• Training and certification of staffTraining and certification of staff– ethical conduct of researchethical conduct of research– measurements and other visit proceduresmeasurements and other visit procedures– data entrydata entry

• Performance reviewPerformance review

Quality ControlQuality Control

• Special procedures for quality controlSpecial procedures for quality control– study drugstudy drug– laboratory procedureslaboratory procedures– data qualitydata quality

• missing, out of range, illogicalmissing, out of range, illogical• variation by site, investigator, staff variation by site, investigator, staff

membermember

• Periodic checks of source dataPeriodic checks of source data

• Periodic reports (missing forms/data, Periodic reports (missing forms/data, queries, differences, drift)queries, differences, drift)

CloseoutCloseout

• Final visit and post-trial plansFinal visit and post-trial plans–make final measurementsmake final measurements–say goodbye and thank yousay goodbye and thank you–inform of test resultsinform of test results–?inform of treatment status?inform of treatment status–?inform of trial results?inform of trial results–?make clinical recommendation?make clinical recommendation

SummarySummary

• Trails are costly and complicatedTrails are costly and complicated–get advice from experienced colleaguesget advice from experienced colleagues–get materials used in prior trialsget materials used in prior trials–use UCSF servicesuse UCSF services

• financial, legal, IRB/CHRfinancial, legal, IRB/CHR• CTSI Clinical Research CenterCTSI Clinical Research Center

–Space, phlebotomy, study staff• CTSI Consultation ServicesCTSI Consultation Services

–Design, ethics, statistics, database Design, ethics, statistics, database managementmanagement