March 15, 2011: I. SimClinical Research 2.0 Epi – 206 Medical Informatics Clinical Research 2.0 Ida Sim, MD, PhD March 15, 2011 Division of General Internal.

March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics

Clinical Research 2.0

Ida Sim, MD, PhD

March 15, 2011

Division of General Internal Medicine, and Graduate Group in Biological and Medical Informatics

UCSF

Copyright Ida Sim, 2011. All federal and state rights reserved for all original material presented in this course through any medium, including lecture or print.


Questions About Homework?

Today

Virtual Patient

Transactions

Raw data

Medical knowledge

Clinical research

transactions

Raw research

data

Dec

isio

n su

ppor

t

Med

ical

logi

c

PATIENT CARE / WELLNES RESEARCH

Workflow modeling and support, usability, cognitive support, computer-supported cooperative work (CSCW), etc.

CRMSsEHRs


IRB Funding Agency

Study DB

Data analysis

Results reporting

Contract R

esearch O

rganization (C

RO

)

Protocol

Trial DesignSponsorsAcademic PIs

?Site 1 Site 2 Site 3

Site Management Organization (SMO)

Today

Clinic 2008

FrontDesk

Radiology

MedicalInformationBureau

Walgreens

Pharm BenefitManager

Benefits Check(RxHub)

HealthNet

B&T

UCare

Specialist

ReferralAuthorization

Internet Intranet Phone/Paper/Fax

Lab

UniLab

(HL-7)


Health Care Quality

• Doing the right thing– based on scientific evidence

• right – without error

• to the right people– e.g., blood pressure meds by ethnicity

• at the right time– beta-blockers at hospital discharge for heart attacks

Clinici

an View


General Drivers of Change

• A “grand convergence” of– maturation of the Internet as connective technology

– social computing tipping past critical mass

– ubiquity of microchips in computers, appliances, and sensors

– explosion of data from everywhere and everything (Big Data)

• For all fields, frontiers of research driven by– ability to do large-scale multi-disciplinary data analysis,

visualization, etc.


Biomedical Drivers of Change• Personalized medicine, geno-pheno correlations

– need genomic and phenotype data in computable form for large-scale small signal correlations

• predictors more likely to be rare vs common variants

• Genomic data will be a commodity– SNPs, whole genome analysis

• Large-scale phenotype is the bottleneck– incl. patient-centered phenotypes (symptoms, side effects)

• Requires tighter connection between research and care – huge volume, complex data that needs to be made sense of


Health Care Quality--Pt. View

• Stay healthy for as long as possible• Take as few medicines as possible

– they all work, with fewest side effects– at lowest cost– for as short a time as possible

• I know what makes me worse and what makes me better, and can do the right thing

• I have all my questions answered• I know what other people are thinking, and I feel

supported in my overall health status


Clinical Research/Care 2.0

• Methodological considerations of ICT• Open data/open science on epic scale fully integrated into

personalized person-centered care• “not your grandfather’s clinical research”

• What do we need to get there?– “building the research ‘cyberinfrastructure’ is the single

most important challenge confronting the nation’s science laboratories” (NSF)1

– communities of self-improving practice (evaluation and feedback)

– architecture of sustainability

1http://www.nsf.gov/news/special_reports/cyber/index.jsp


Outline

• Methodological Considerations of ICT • Towards Clinical Care/Research 2.0• Open Science• Class Summary


Methodological Considerations

• ICT system design – user interaction design, rapid prototyping

• Study design– beyond RCTs

• Participant recruitment and sample size– selection bias, sampling error

• Outcomes assessment– measurement error and bias


Intervention Design• User interaction design

– “form factor”: kiosk, desktop, laptop, notebook, tablet, phone… – platform (e.g., Mac), browser (e.g., Safari), operating system

(e.g., Android or iPhone) incompatibilities– use Flash, Java, etc requiring plug-ins or version compatibility – readiblity (font too small), need to scroll, confusing navigation,

bugs• What technology does respondent group use?

– check some browser and platform usage statistics sources • e.g., http://www.statowl.com/web_browser_market_share.php

– need to test and double-test for various configurations

• UCSF Usability Lab pending…consider paying for design services


Rapid Prototyping & Feedback• Traditional approach

– design a website or system; pilot it, improve it; run RCT; analyze data; publish (over 2-3 years)

• Rapid prototyping, participatory design, agile programming– Google launched 450 search engine improvements in 2007– each one tested rigorously, real-time feedback through

clicks, user logs, etc.• http://googleblog.blogspot.com/2008/08/search-experiments-lar

ge-and-small.html

– combine quantitative with qualitative user studies

• Why wait till end of RCT to find out that “the system” didn’t work or wasn’t used, and have little idea why?

http://googleblog.blogspot.com/2008/08/search-experiments-large-and-small.html

http://googleblog.blogspot.com/2008/08/search-experiments-large-and-small.html


Embedded RCTs

• Identify part of the system for testing (e.g., videos vs. pictures of inhaler use)

• Randomly assign each person landing on the screen to videos or pictures

• Test for comprehension with short survey• Compare % correct answers, set “stopping rule,” run

study until answer clear or test is over• Can run multiple embedded user interaction RCTs to

optimize design features in parallel








Current RCT Research Approach• RCTs are expensive, difficult to conduct, 30-40% of studies never

accrue enough patients– estimated 2 million pts needed/yr for US-based trials

– will be worse with personalized medicine• Studies take years to answer limited questions in limited

populations• Study designs and results are heterogenous, limiting ability to pool

findings or make summary interpretations• Research questions don’t address combination treatments (e.g.,

ACEI and amlodipine)• Research questions don’t track with front-line clinical needs

– no good data on mid- to long-term efficacy or effectiveness of antidepressants

• Overall lack of relevance, generalizability, and sustainabilityMoss, et al. NEJM 2011; 364(9):789-761Crowley, et al. JAMA 2004; 291(9):1120-6etc.


New Vogue

• Data (evidence) mining of aggregated EHR data– new “comparative effectiveness research”

– effectiveness, not efficacy

– subject to confounding by indication

– subject to errors and biases of data coding (e.g., IDR Cohort Selection Tool example)

– under-estimating GIGO (garbage in garbage out) problem


Mining vs. Farming

• Mining– traditional but troubled RCTs1, mining for gold

– in a clinical data warehouse “garbage pit”• may be biased, error-laden population-level effectiveness

estimates

• Sustainable evidence farming– “systematic effort by practitioners to collect and analyze data

on actual practice experience that is used to develop clinical evidence for practice” (R. Kravitz, UC Davis)

– i.e., to cultivate data for use in generating relevant evidence for sharpening therapeutic precision for individual patients

1DeMets, Califf. JAMA 2011; 305(7):713-714.


Promising Evidence Farms

• Practice-Based Research Networks, Accountable care organizations, etc.– PROS, AAFP, AHRQ, HMO Research Network

– Kaiser Permanente

– UCSF?

• Practitioners (and patients?) own the data and enjoy the harvest – Practice informs research and vice versa

• Practitioners help to select research questions

• Results can be used to craft guidelines, improve practice organization, develop locally relevant clinical evidence

– Joint symptom monitoring and/or N-of-1 trials• using EHR, PHR, mHealth apps

http://www.hmoresearchnetwork.org/


Hypothetical Case

• MR is a 38 year old man with depression• Condition characterized by depressed mood,

anhedonia, poor sleep, difficulty concentrating; all of moderate severity

• Treatment options are Zoloft and Effexor• Which treatment is best for MR?• Traditional EBM approach applies, in order

– best research evidence

– clinical experience

– patient valuesAcknowledgement for evidence farming slides: R. Kravitz, UC Davis


But…

• Latest research indicates an 8-week remission rate that is greater for Effexor than for Zoloft (68% vs. 45%)1

• Among the past 20 patients Dr. Jones has treated, 6 got Effexor and 14 got Zoloft– Among Effexor patients, 3 remitted (50%)– Among Zoloft patients, 12 remitted (86%)

• Should Dr. Jones get with the published evidence?

1Mehtonen OP, et al. Randomized, double-blind comparison of venlafaxine and sertraline in outpatients with major depressive disorder. Venlafaxine 631 Study Group. J Clin Psychiatry. 2000 Feb;61(2):95-100.


The Clinical Dilemma

• Guidelines (based on global, exogenous evidence) say one thing

• (Systematic) experience (based on local, endogenous experience) says something else

• Who to listen to?• Both!

– traditional evidence sets the prior based on population-level efficacy data (top down)

– evidence farming methods update the prior using systematic local/personalized observations (bottom up)


Why Bother with Local Evidence?

• Context matters: RCT evidence not fully generalizable– Patient values and preferences vary across

practices • psychotherapy vs. SSRIs

– Patient risk, responsiveness, and vulnerability vary across practices

• 2-drug therapy for hypertension in blacks

– Practitioners and organizations differ• carotid endarterectomy


Empirical Bayes Approach

• Empirical Bayes method balances between the two parcels of evidence (borrow from strength)

• Ti,EB = (1 – Bi) * TGLOBAL + Bi * Ti,LOCAL

Bi = sBETWEEN2 / (sBETWEEN

2 + sWITHIN,i2)

• The greater the between-group variance relative to the total variance, the more weight is placed on local evidence

• Local evidence accumulates over time


Rigorous Symptom Monitoring

• Depression symptom self-monitoring– triggered by GPS activity (staying in bed all day?) or

customized for patient (e.g., after elder care) – timing based on focus group input (e.g., end of day

preferred)– very simple, low overhead data input

• Feedback on Bayesian trends, discontinuities, etc. correlated with current medication (e.g., Zoloft) and dosage

• Or more formal N-of-1 studies1

– randomized and blinded to Effexor and Zoloft in within-subject crossover fashion

1Kravitz, et al. Contemp Clin Trials 2009; 30:436-445


Barriers to Implementation

• Electronic infrastructure• Culture• Resources & incentives








Subject Recruitment• Recruitment is biggest bottleneck of clinical research

– 30-40% of clinical trial costs – >80% of trials have recruitment delays– 1/20 recruited patients actually enroll

• Web-based recruitment can be international, cheap, fast– e.g., www.stopsmoking.ucsf.edu Dec 05 - Feb 07

• 350,000 hits, 60,000 entered data, 20,000 enrolled• 2/3 Spanish-speaking, 1/3 English• 131,517 visits from 121 countries Jan 12, 05 to April 5,

06

http://www.stopsmoking.ucsf.edu/


Visits0=>1=>100=>1,000=>10,000

Distribution of Visits to www.stopsmoking.ucsf.edu Jan 12, 2005 to April 5, 2006

(131,517 visits from 121 countries)

http://www.stopsmoking.ucsf.edu/


Reducing Sampling Error

• Social sciences and marketing are most advanced in population-level recruitment methodology– e.g., Joint Statistical Meetings of the American Statistical

Association

– http://www.knowledgenetworks.com/

• Two major methods– recruit a representative sample

– use a pre-assembled representative cohort

Disclosure: I have no relationship with KnowledgeNetworks

http://www.knowledgenetworks.com/knpanel/index.html


Recruit Representative Sample

• Random digit dialing (RDD) with web supplementation equally representative as (land-line) telephone RDD– RDD sampling (landline and cellular)

– if respondent agrees, provide them with free Internet access (via MSN TV, formerly WebTV) or other necessary hardware for duration of participation

– e.g.,http://knowledgenetworks.com/


Representative Cohorts

• Maintained by e.g., large survey and marketing firms– http://www.knowledgenetworks.com/knpanel/index.html

• KnowledgePanel is representative of US

• can target specific respondents, “survey response rates of 65-75%, abandonment rate <2%”

– www.surveysampling.com• ~6 m people from 72 countries, incl. health-specific panels

– http://www.experimentcentral.org/ • NSF-funded representative panel for social science survey

research

http://www.knowledgenetworks.com/knpanel/index.html

http://www.surveysampling.com/

http://www.experimentcentral.org/


Other Recruitment Methods

• With higher risk of sampling bias– search engines, with search engine

optimization (SEO) techniques (e.g., Google adwords)

– links from related pages– listservs, Facebook, etc.

• Can combine radio, TV, or print w/ – website (URL, Uniform Resource

Locator)

– text messaging Common Short Codes (http://www.usshortcodes.com/csc_whatDo.html)

– QR code (or matrix bar code) launches URL, vCard, displays text, etc.

http://www.usshortcodes.com/csc_whatDo.html

http://www.usshortcodes.com/csc_whatDo.html



Search Engine Ranking• Search engines have their own (secret) algorithm for ranking

pages– Google uses >100 factors, esp. how many pages link into a

page

• Google AdWords – put in your keywords, see cost-per-click

• https://adwords.google.com/select/KeywordToolExternal?defaultView=3

– pay only if someone clicks, more if the keyword historically brings more traffic

https://adwords.google.com/select/KeywordToolExternal?defaultView=3

https://adwords.google.com/select/KeywordToolExternal?defaultView=3


Note on Sample Size

• Estimating sample size– e.g., Google provides traffic history for various keywords

(adwords.google.com)

• Since incremental cost of web surveys/ICT interventions are often negligible, less pressure to minimize sample size– not unusal to get large samples (>10,000)

• But high sample size = high accuracy!– may be precise (p < 0.05) but inaccurate if sample is non-

representative








Measurement Error/Bias

• What you designed may not be what respondent sees

• Client’s browser/phone displays the intervention based on – platform, browser, monitor, screen/window size

– different users see different survey, e.g., • small screen/window size makes “Next” button not visible

• text doesn’t fit on small window, or requires scrolling for some respondents and not others

• colors, graphics (e.g., visual analog scales) may appear differently


Loss to Follow-up• Influenced by

– respondent familiarity with functions (e.g., using CSCs)– technical design of survey– disability accesibilty (Section 503 compliance)– bandwidth– convenience (can interrupt survey?)

• Can use mixed-mode surveys/interventions to increase completion rates/follow up, e.g., – combined web/voice, web/mail surveys

– combined desktop/laptop and tablet/mobile (I.e., use different form factors for initiating vs. sustaining intervention effect)


Outline


“In bounds” to “beyond bounds”

• Clinician-driven• Intermittent• Retrospective, delayed• Verbal report to clinician• Research and care

separated• Data is siloed• One patient at a time• Transactional support is

central

• Patient engaged• 24/7/everywhere• Real-time• Multi-modal• Research and care

integrated• Data is shared,

aggregated• “All” patients at a time• Evidence is central

Stovepipe Model• Almost 1/2 of

Americans have >1 chronic disease

• Enter data over and over again

Stovepiped Health 2.0

Plateau of Diminished Promise

Internet Hourglass Model

• Standardize and make open the “narrow waist”

• Reduces duplication, spurs community innovation, supports commercial and non-profit uses

Health 2.0 Hourglass• Key challenge is what

should be in the narrow waist– interoperation protocols

– privacy architecture

• Phase the introduction– start in mobile health

– tackle heavily siloed enterprise health later

http://openmhealth.org/• Orange waist

– standard connections to PHRs and EHRs

– ultrasecure personal data vault (“questions go to the data”)

• Green “middleware”– shared library of standardized

of outcomes, data variables, vocabularies

– widgets for symptom assessment, graphing, risk communication, surveying, randomization, etc.

– N-of-1 infrastructure


What Next at UCSF

• IDR/MyResearch– undergoing campus-wide review for future plans

• mHealth– working group, listserv, Facebook: email Ida to join

– mHealth methods class

– consultation services: design, etc.

– technical and legal services: either common open platform (e.g., openmhealth.org) or preferred contractors

• Planning for major initiative in informatics…TBD– Care Science Institute -- Cyberinfrastructure for Integrated

Care and Research?


Outline



Science in Era of “Big Data”

• “Data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time.”– e.g., the whole web, all EHR data, all human studies data

• Science is increasingly open– open source: NIH-funded software expected to be open source

– open data: sequencing data, GWAS, Alzheimers, all of NSF, large NHLBI studies, Science Commons, Community Health Data Initiative, etc.

– open publishing: PLoS, Biomed Central, PubMed Central

– open community curation of knowledge: e.g., CHDwiki, Wikigenes with authorship tracking (the long tail…)

http://sciencecommons.org/

http://homes.esat.kuleuven.be/~bioiuser/chdwiki/index.php/Main_Page

http://www.wikigenes.org/


Data Mining with “Big Data”• Text mining, data mining, model building across ALL

data– within and outside biomedicine

• using shopping data to track diet, or population salt intake

– supervised (e.g, neural net) and unsupervised (e.g., clustering) learning

• not your grandfather’s logistic regression…

• Puts premium on– natural language processing (e.g., Watson)– semantic interoperation– maintaining context

• Alb 1.7 in 20 y/o MVA burn patient, vs. homebound 88 y/o

– developing a culture of sharing, policies for attribution of credit


Towards Semantic Web

• Web 1.0– web of text

• Web 2.0– web of people

• Web 3.0 aka Semantic Web– web of data: data must be “tagged” with semantics, using

vocabularies, ontologies (e.g., Science Commons, linked open data repositories)

• Web 4.0– web of knowledge: nowhere near this (e.g., computable

statement that diabetes increases MI risk)




Clinical Care/Research 3.0

Administrative Clinical Care Research

Physical Networking

Standard Communications Protocols (e.g., HL-7)

PracticeManagement

Systems

EHRExecutionAnalysis

Medical BusinessData Model

Clinical CareData Model

Clinical StudyData Models

Open de-identified repositories

OWL Ontologies of Everything


Open Discussion

• How to balance standardization and comparability (e.g., of EHR notes, of research variables) with flexibility and innovation?

• Medicine and biomedical research is conservative– will all this web 2.0/3.0 stuff pass right by us?

• How will this change what you do/how you think, if at all?

• What would you like to see from academia/UCSF to help you stay as competitive in research as possible?

• ???


Outline



State of Health IT Use• EHR adoption still low

– barriers include finances, lack of organizational change expertise, fragmentation of health care system, misaligned incentives

• Recovery Act will spur EHR adoption, for good or ill• EHR and data warehouses can but don’t always help

research • Limited success of decision support systems• Fundamental tradeoff of coding effort vs. “smartness”

of system limits both EHR and CDSS return on investment


Computers Must Interoperate• In a networked world, data and actions must be

shared across people and computers– syntatic interoperation: a common grammar for machines

talking to each other in biomedicine (e.g., HL7)– semantic interoperation: predictable and meaningful

exchange of common meaning• requires standard vocabularies and standard data models

• SNOMED most comprehensive but use is unproven

• Other challenging things that need standardization in biomedicine– “common data elements” in research– a standard EHR data model so all EHRs “look” alike– standard protocol models for human studies, etc.


Summary• Informatics helps make sense of data and knowledge

– is necessary for better care and research

• Today’s technologies promise transactional support – major barriers are economic, policy, and workflow related

• Need brand new technologies for other 3/4 of Big Picture

• Disruptive change to clinical research seems quite possible – as mobile technologies break down time and space barriers– as social computing takes off– as open data becomes more common


Take-Home Message• IT focuses on storing, accessing, and exchanging

data • Informatics is use of computers to make sense of

data • The more “computable” the information, the more the

computer can do for us• ...not just individually, but together as a community of

care and science

• Course evaluation– http://psg-mac43.ucsf.edu/ticr/CourseEvaluations/ticreval.asp?id=433

http://psg-mac43.ucsf.edu/ticr/CourseEvaluations/ticreval.asp?id=433

March 15, 2011: I. SimClinical Research 2.0 Epi – 206 Medical Informatics Clinical Research 2.0 Ida Sim, MD, PhD March 15, 2011 Division of General Internal.

Documents

phenotype data

complex data

sensorsexplosion of

right peoplee

right thingi

research cyberinfrastructure

right timebetablockers

clinical careresearch