The Secrets to Implementing a Successful Metrics Program...• Ensure all trial deliverables are met according to timeline, budget, and resource requirements • Partner closely with

The Secrets to Implementing a

Successful Metrics Program

Ravin Warnakulasuriya

Sr. Director Vertex Pharmaceuticals

MCC Clinical Trial Risk and Performance

Management Summit

The Loews Philadelphia Sept 4-5th 2019

Agenda

• Finding your Data

• Data Bad Habits / Data Ailments• Data Indigestion

• Data Hoarding

• Data Dilution

• Data Control• Third Party Data

• Vendor Management

• Subscriptions and Data Dilution

• Data Culture (The Human Element)• The Technology Illusion

• The Human Technology Interdependency

• Data Control (who do you trust)

• The 4 Pillars to Establishing a Data Culture

The Data Wood for the Technology Trees

• Emerging Concepts

• Risk (Hype Curve)

• Early Adopter

• Increased responsibility

(adopter vs user)

• Closed Systems

• Confined Data

• Multiple vendors

• eClinical

• Best in Breed

• Business / Financial Driver

• Coalition pros and cons

• Tactical vs Partnership

• Product Roadmap

• Bottled Water (onboarding vs new service)

• Its YOUR data, you only need the data and

you’re entitled to it

• Be System Agnostic

• Be Vendor Agnostic

• Accessibility Right

• Accountability

Back to Basics: Reclaiming your Data

The Hype Cycle

Hype Cycle

Bad Habits

1. Consume your Data like you would consume your food!

• CDW = communal

kitchen refridgerator

• Who’s is it?

• How was it prepared?

• Is it still good?

• How old is it?

• What needs to be

thrown out?

• Does it smell funny?

2. Data Hoarding is Unhealthy

Data Indigestion

• Data Noise

• Data Purity / Data Relevence

• Muddying the waters (Averages, medians)

• Inconsistency in Governance over time

• Meshing and Mapping efforts

• Orphan data

• Public data and its assumptions

• Benchmarking and Big Data Dilution

• Swimming against the legacy current

• Data evolves with the company

• Be Picky with your Data

• Standardization efforts keep us honest

• Out of the box vs configuration

• Canonicals as standard onboarding

• Circle of Life (take what you need)

• Qualify it before ingesting it

• Data Integrity Score• System

• Process

• People

• Validation

• Cleanliness is next to Godliness.

• Data ‘Value Proposition’

Data Selection

Third party Data – a Hostage Situation

• Third Party data• CROs

• Technology vendors

• Subscriptions / Data brokers

• Bells and Whistles• Canned Reports are still canned

• Report Builders

• Dashboards

• APIs and ETLs• APIs are no longer shiny

• Expect it from your Partners

• The Onboarding Advantage• RFIs and RFPs

• Using Canonicals

• It’s YOUR Data!!!

Third Party Data

• Buying Data ( Data subscriptions)

• Public data• Broad inferences (Informed decision)

• Industry data• What’s your Commitment

• What’s your return

• Definitions

• Beneath the surface• The devil is in the details

• Volume for Validity

• Study designs getting more complex

• Are we truly all operating the same?

• The deeper you go, the narrower it

• Normalization and Vendor Agnosticism• Consider your vendor strategy / portfolio

• Aggregation vs Configuration

• Knowing when to Normalize your Data

• Using Canonicals across vendors

Question . . .

1. Is your Third Party Data managed / accessed differently from the

rest of your data?

2. Do you have APIs and data provision requirements as part of

your vendor selection criteria or RFI / RFP Process?

3. For those of you who have APIs set up, do you pay for this?

4. Do you mix your third party data with the rest of your data /

dashboards?

The man Element – Data Culture

The Technology Illusion

Purpose / Need

Provider /

Access

Interpreter

Analyzer

Keeper

Inferences

Decision

Options Action

“Even a driverless car

Still needs a passenger”

Calculations Context

Data Driven Emotional

The 4 Pillars of Establishing a Data-Driven Culture

ACCOUNTABILITY

TRANSPARENCY.

INTERPRETATION

ANALYTICS

Culture

1. Accountability, Responsibility and Support

ROBUST GOVERNANCE MODEL

• Champions (Executive Support)

• Data Owners (Functional Accountability)

• Sys .Owners (Tech Perf. & Enhancements)

• Stewards (User Input, Diagnostics, Champ)

• Process Owners (Formal Adherence)

• Trainers (Awareness)

INVESTMENTS

• Talent

• Bandwidth

• Financial

EMBEDDED BUSINESS MODEL

• Integrated IT/Business Operations

• Integrated IT/Business Strategy

• Cadence and Committees (Operational /

Steering)

DATA ACCOUNTAIBILITY

2. Transparency, Trust and Co-Dependency

• Trust

• Data Sharing

• Xfunctional Dependency

• Single Source of Truth culture

• Data Integrity

• Trusting the Source / Owner

• Data Origins / Definitions / Logic

Accessibility

• Appropriateness

• Ease of Access

• Integration / Formalization

• Formal Process

• Marketing

• Adoption and Expansion

DATA TRANSPARENCY

3. Analysis, Access and Autonomy

• Robust visualizations

• Interactive user autonomy

• Interoperability (meshed reports)

• Accessibility Appropriateness

• User Friendly (internet, exports)

• Up to date (nightly feeds)

• Diagnostics (alerts and reports)

• Intended Uses• What questions are being

asked?

• Problem vs Solution oriented

• Exploratory vs Purposeful• Data Developers

• Data Scientists

DATA ANALYTICS

4. Application, Understanding and Action

• Meaningful Metrics

• Actionable outcomes

• Measurable

• Data Driven Decisions

• Reaction and Context• Risk reporting (R/A/G signals)

• Metadata vs Text

• Emotional Quotient

• (Adoption / Application)• Exposure (reluctance)

• Paranoia (delays)

• Defensive (downplay)

• Chore vs Tool (deprioritize)

• Adopter (dependent)

DATA INTERPRETATION

Question . . .

1. Does your company have a robust Data Culture?

2. Do you have a formal Data Governance Model?

3. Do you trust the data that you don’t own?

4. Are you willing to surface your Data to Executive Management?

Summary

• Find your Data

• Be Data Healthy and Hygienic

• Free your Data

• Find your sweet spot• Depth vs Breadth

• Volume vs Validity

• Configuration vs Standardization

• Data Agnosticism• Use Canonicals

• Data Accountability

• Establish a Data driven Culture

Questions?

Secrets to Implementing Successful Metrics Programs

Mike Fitzpatrick04 Sep 2019

Agenda

Poll Question

My Analytics Journey – Challenges and Recommendations

Case Study - Treehouse Brewing

Key Takeaways

Clinical trial teams excel at

leveraging data in my company.

“The ability to take data – to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it’s going to be a hugely important skill in the next decades”

Google’s Chief Economist Hal Varian – 2009 Interview

Poll Question:

Strongly Agree

Agree 4

Neither 3

Disagree 2

Strongly Disagree

Traditional Clinical Trial Manager Skills &

Qualifications

do not optimize

Clinical Trial Execution

My Hypothesis….

Responsibilities (Trial Execution)

• Day to day management of assigned clinical trial to ensure deliverables are met in all phases of trial

• Communication to internal team members on deliverables and ensuring timelines are met

• Manage trial feasibility process

• Work closely with Rare Disease or Infectious Disease Clinical Operations Strategy Leader to implement Operations plan for Clinical Study Teams.

• Drafting site materials including training, manuals and support documentation

• Monitoring of clinical data to ensure timely entry and readiness for data review meetings

• Periodic review of clinical data

• Review of clinical monitoring reports to ensure timely completion and identification of issues

• Ensure trial samples are ready to be sent to vendors in order to meet trial deliverables

• Maintain trial metrics

• Ensure trial is conducted in accordance to company and ICH/GCP standards including multiple regions in global trials

• Support the CRO in testing clinical trial systems/databases (i.e. UAT)

• Develop and maintain strong, collaborative relationships with key stakeholders within company

• Lead strategic planning and management of all study and team activities, leveraging internal and development partner resources, expertise and knowledge, along with optimizing CRO resources, expertise and knowledge.

• Lead development of overall clinical strategic operations plan for assigned projects including timelines, internal and external costs, CRO/Vendor Oversight Plans, key deliverables and risk/mitigation strategy.

• Provide oversight of study timelines and trial execution.

• Ensure all trial deliverables are met according to timeline, budget, and resource requirements

• Partner closely with Clinical Research Organization (CRO) interactions, including sponsor oversight of relevant functional activities (e.g., study management, monitoring, site management, and project master files).

• Monitor oversight of study progress including: CRO and third party vendors, ensuring compliance with relevant regulations and guidelines by maintaining regular contact with external vendors, including CRAs, as applicable.

• Work with CRO to develop and/or revise scope of service agreements, budgets, plans and detailed timelines and ensures that performance expectations are met.

• Collaborate with internal medical director in development and authorship of study protocol

• Participate in Case Report Form design, user acceptance testing in partnership with Data Management.

• Execute CRO/Third Party Oversight Activities including but not limited to: Monitoring visit report review, periodic review of data quality, review of CRO/Vendor KPIs, performing Sponsor Oversight Visits, etc.

• Contribute to and/or author clinical sections of regulatory documents including Investigators’ Brochures, safety updates, and responses to Health Authority / Ethics Committee questions

• Identify potential study issues, proactively recommending and implementing solutions or corrective actions, as needed.

• Develop and actively manage Site/Investigator relationships; serve as a liaison and resource.

• Organize, schedule and attend internal & external team trial-specific meetings.

• Perform occasional travel to CRO/vendor facilities and/or investigative sites to train and assess study team and CRAs.

• Assist Senior Clinical Trial Manager in management of clinical programs, preparation and execution of sound clinical development strategies, and Clinical Operations Department Initiatives.

• Participate in the development, review and implementation of departmental SOPs and processes.

• Other Job-related duties as assigned.

Job Description #1 (Biotech ~700 employees): Job Description #2 (Pharma ~2500 employees):

Two randomly selected CTM job postings from Linkedin, Aug 2019