Stamatiadis PCT Nov 2015 final

!"

KPIs: Managing Performance and Risk in Clinical

Development

PCT Partnership in Clinical Trials November 17th, 2015 – Hamburg, Germany

Why do we measure performance?

# Because there is competition (internal and external)

# Because there are rules and regulations (compliance)

# Because resources are not infinite (cost)

# Because you have plans for the future (strategy)

#  In other words: because we need to make decisions

“When you can measure what you are speaking about, and express it into numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind…”

Lord Kelvin 1824-1904

Why worry about performance in clinical trials?

#  Clinical development is the main driver of the cost of new drugs

#  Clinical trials are expensive, lengthy and risky

#  In the future, pressure from payers on cost will increase

#  Good performance in clinical development is therefore key to the success and often to the very survival of a pharmaceutical company

Discussed topics

1.  The keys to the success in Clinical Trials

2.  Performance and risk

3.  Setting KPIs

4.  Using data to make decisions

5.  Small data / Big data

6.  Using KPIs to align the organization

Well, what is success?

Actionable Strategies Through Integrated Performance, Process, Project, and Risk Management Stephen S. Bonham, Artech House; 1 edition (May 31, 2008)

Strategy + Performance - Risk

Execute Reach result

Follow procedures

Respect budget

Plan

Understand Clinical Strategy “Only 14% of employees understand the company’s strategy and direction”

David Witt, May 21, 2012

#  In most companies there is a clearly defined strategy cascaded to the functions and periodically communicated

#  Still, a majority of employees declare ignoring the strategy of their company and even of their division

#  This is because there are no adequate tracking tools to follow the progression against the strategy and to help decision-making.

#  Most decisions are therefore based on intuition

#  This is not necessarily wrong but intuition can be strengthened by facts

Do you align with your clinical strategy?

# How does it link to the drug development strategy and beyond that to the company overall strategy?

# How does it combine the target results with investment and compliance requirements?

Your CTTS collects BI data

How much of this is reliable?

How much of this is used to support decision making?

How much does this all cost?

You may even perform risk-based monitoring*

$  What is the impact of identified deviations?

$  How do you measure the impact of actions taken?

$  Is the improvement worth the investment?

*Courtesy: Clinerion Quality Risk Radar

From information collection to Performance Management

#  Information is regularly collected during drug development and can be displayed on various dashboards and scorecards

# However, this is not sufficient to extract all the value of this information for performance management

#  Turning information into KPIs implies a weighting link to strategy and risk

KPIs and the one-eyed deaf driver

KPIs with a single dimension are incomplete

Only three-dimensional KPIs provide the necessary information to support decision-making

The three dimensions of an effective KPI

# Performance = Obtaining the right results, in the right way, at an optimal cost # Results = Anything we aim to

achieve, quantitative or qualitative # Know-how = Qualification,

experience and processes # Means = Time (duration), running

cost (salaries, fees) and fixed assets (machines)

Let’s apply the principle to clinical trials

Results: You must obtain positive results confirming the hypothesis that your drug has the expected effect and is safe to use in order to obtain marketing authorization

Know-how: You must follow GCP to the letter and be able to prove that you did so all along

Means: Any delay has high cost for company revenue and budgets are more and more restricted

Therefore, any KPI in clinical trials must address these three areas

Understanding risk as a driver for performance

# ISO definition of risk: “effect of uncertainty on objectives”

# Weight can be applied to strategic objectives using risk assessment

# Risk will define what to measure

# Risk will define actions related to objectives

# Risk frequency defines measure frequency

Risk = Hazard x Vulnerability

#  Risk: A potential deviation from objectives resulting from the correlation between a hazard and a vulnerability

#  Hazard: Originates in the environment. Can not be controlled but can be foreseen(this is the “uncertainty”)

#  Vulnerability: Inherent to the organization. Can be acted upon (Risk management)

#  Performance and Risk are two sides of the same coin

!"

Decision making

Intuition helps decision (Chinese proverb)

# Decision is often made by intuition but facts are necessary to support it

# Clinical trials performance data is abundant but rarely complete and accurate

# Selecting the way of using data is paramount

Are you rational or intuitive? An example from WWII

Too many planes returned badly damaged or were simply missing

Preserving aircrafts and pilots was critical to the outcome of the war

Reinforcing the entire plane, or even the cockpit would make the aircraft too heavy to fly

The solution proved to be somewhat counter-intuitive…

And data can help intuition

# You have enough data. %  Build on the existing infrastructure

# Reinforce intuition with facts % Calibrate indicators to agree with your intuitive knowledge

# Move from observation to decision %  Set actions based on thresholds

# Manage urgencies by prioritizing %  Everything is “urgent” but is it a priority? Use risk to find out

The trouble with data

#  Perfectly clean data require a lot of work (checks, queries, dry runs etc.)

#  Devoting the same amount of work to cleaning large amounts of performance data is not acceptable

#  Therefore the result of performance indicators is often imprecise and may be looked upon with suspicion

#  Should we limit the amount of performance data to what can be reasonably clean, or…

#  …can an imprecise measurement be correct to the degree needed for decision making?

Two ways of looking at data: big or small

# Small is beautiful (the traditional approach) % Carefully select the most meaningful indicators

and collect data as clean and consistent as possible

%  Set clear trigger values and action plan

# Big is powerful (the big data approach) % Collect as much information as possible %  Estimate gross data quality % Work with thresholds and trends to define

actions

Big data analytics: The new trend

Top Enterprise technology initiative

Way above data and document management combined

The earthquakes and the flu

Measuring the intensity of wireless communications right after an earthquake provides better insight to the location and intensity of the quake than specialized instruments

Detecting influenza epidemics using search engine query data, provides an alert on flu epidemics 2 days earlier than the flu surveillance network

Jeremy Ginsberg et al, Nature Vol 457, 19 February 2009,

What do you know about OBD-II?

A big data approach to performance management

% Build on existing business processes and infrastructure

% Use Big data analytics

% Observe thresholds and tendencies to making decisions

% Calibrate according to your intuitive knowledge

Big data analytics: Discover, Drive and Decide

28

Observation: Reporting (real-time, customized, ad-hoc), scores, dashboards

Performance visibility

Alarms Investigation:

SWOT / Action Plans

“What if” scenarios

Discover opportunities Uncover risks

Strategic alignment: Let’s work together

% Use performance metrics (KPIs) to: % Communicate goals

and achievements % Explain actions % Focus efforts

What does alignment mean to us?

#  Understand the reasons why the company is developing a particular drug (expertise and recognition in the clinical area, discovery opportunity, collaboration etc.)

#  Understand the structure of the clinical development plan (choice of studies, decision points etc.)

#  Understand the design of each study and the reason behind it (active comparator, placebo, choice of population etc.)

#  Understand the timing for submission and approval (market constrains, medical urgency etc.)

#  Work in the same direction and optimize the strategy deployment

A balanced approach to strategy

The keys to clinical trial success always link to the strategy

Staffing, training, experience, sourcing (means)

Processes (know-how): Good program and study design (hypothesis, endpoints, comparators, knowledge of patient population, site distribution)

Operational excellence (know-how): GCP compliance, timely site setup, timely patient enrollment, DBL, pharmacovigilance

Customer (results): Timely reporting, data quality, documentation quality

Budget discipline (means): Timely and accurate budget spending

You got information? Put it to work!

# Formalize strategic objectives along the four BSC perspectives: People, Processes, Customer, Finance

#  Identify strategic goals and use risk to weight them and define actions

#  Implement actions and measure their effects using 3D KPIs to detect deviations from targets either in results, know-how or cost

# Working with thresholds allows to combine qualitative information and numeric data at any scale. This way, all indicators can be combined to support decision-making at all levels up to the top of the corporate structure

Six things to remember

# Success = Strategy + Performance – Risk

# Performance = Results + Know-how + Means

# Risk = Hazard (external) x Vulnerability (internal)

# Decision is made by intuition but facts are necessary to support it

# Performance data is abundant but rarely complete and accurate so you need to choose your method

# Alignment across the organization is key to success

Dr. Dimitri Stamatiadis, PhD MBA MAIA Consulting Performance in Pharmaceutical R&D www.maiaconsulting.eu