Quality Metrics Journey to Excellence Steve Greer Corp QA External Engagement Leader Procter & Gamble
Quality Metrics
Journey to Excellence
Steve Greer
Corp QA External Engagement Leader
Procter & Gamble
Russ Wesdyke – FDA, CDER/OPQ/OS
Tara Gooen Bizjak – FDA, CDER/OPQ/OPPQ
Nelson Webb – P&G
ISPE Quality Metrics Core Team
PCPC Quality Assurance Committee
Tracy Kreuser – ASQ
Acknowledgements
Journey to …
FDA Quality Metrics
ISPE Quality Metrics Pilots
Metrics that Matter
PCPC Industry Support
Journey Mileposts
FDA Quality Metrics
Overview
Quality Metrics help FDA:
Prioritize inspectional resources
Recognize high performers
Reduce product shortages
Encourage operational excellence
Why?
Lot Acceptance Rate Operational Excellence
Invalidated OOS Rate Laboratory Excellence
Product Quality Complaint Rate Voice of Consumer
APR/PQR On Time Rate Quality System
Senior Management Engagement Quality Culture
CAPA Effectiveness Quality of Investigations
Process/Performance Capability Voice of the Process
FDA Quality Metrics
Start small
Don’t start with reporting of APIs or Optional Metrics
Voluntary during learning period
Report metrics by site
Allow grouping of formulae
Clarify definitions
CMOs report directly
Context
OTC Industry Comments
Draft Guidance comment period closed Nov ‘15
Revised Draft will be issued by end of 2016
Start of Reporting?
Q Metrics Guidance Timing
Start: Phased Approach, Voluntary
Simplified Approach
Revised Guidance Insights
ISPE Quality Metrics
ispe.orgConnecting Pharmaceutical Knowledge
2012 2013
Quality Metrics Timeline
ISPE Wave
2 Pilot
Industry Whitepapers
Launch Quality Metrics Wave 1 Pilot
July 2012 FDASIA signed into law
2014 2015
Launch Wave 2 Pilot
2016 Beyond
…
Brookings Institute FDA FRN/Draft Guidance
Wave 2 Pilot enrollment Data collection and submission
ISPE Quality Metrics Wave 2 Pilot Timeline
Individual reportsIndustry analysis
Jun 2015
Wave 2 pilot Report
Jun 2016
14ispe.orgConnecting Pharmaceutical Knowledge
Wave 2 Metricsblue = FDA guidance
metrics
SOURCE: ISPE Quality Metrics Initiative
Quantitative metrics▪ Total complaints rate
– Per million packs, incl. LOE
– Per million packs, excl. LOE
– Per ’000 attempted lots released,
incl. LOE
– Per ’000 attempted lots released,
excl. LOE
▪ Critical complaints rate
– Per million packs
– Per ’000 attempted lots released
▪ Total recall events per year
▪ Lot acceptance (%)
– Per finally dispositioned lots
– Per attempted lots
▪ Invalidated OOS rate
– Per ’000 lots tested
– Per ’000 tests performed
– Per total OOS per tests performed
▪ Right first time (%) per released lots
attempted
▪ Deviations rate
– Per ’000 finally dispositioned lots
– Per ’000 attempted lots
▪ Recurring deviations rate (%)
▪ Lots pending disposition more than 30
days (%) per lots attempted
▪ Culture survey scores (% top boxes)
– Total score
– Leadership score
– Integrity score
– Mindset score
– Governance score
– Capabilities score
▪ CAPAs with preventive actions (%)
▪ Planned maintenance rate (%)
▪ Employee turnover rate (%)
▪ Human error deviations (%)
▪ Deviations with no assigned root cause
(%)
▪ CAPA requiring retraining (%)
External Quality Outcomes Internal Quality Outcomes Culture Indicators
15ispe.orgConnecting Pharmaceutical Knowledge
Wave 2 Correlations Overview
SOURCE: ISPE Quality Metrics Initiative
FRN metrics
Total Complaints
(per pack)
CAPA with
Preventative
Actions
Planned
Maintenance
Rate
External
Quality
Outcomes
Internal
Quality
Outcomes
Culture
Indicators
Total Complaints
(per Released
Lots)
Correlations with p-value <0.05
16ispe.orgConnecting Pharmaceutical Knowledge
Planned maintenance rate to total complaints (n=15
solids sites)
Planned Maintenance Rate & Product Q Complaints Rate
Note: Major outliers excluded – more than 2 interquartile ranges away from sample median – 2 sites excl. on total complaints both with very high LOE rates
R2 measures to what extent metric Y (dependent variable) is explained by the variability of metric X (independent variable)
P- value is probability that correlation between X and Y is zero, value below 0.05 indicates statistically significant results.
SOURCE: ISPE Quality Metrics Initiative
▪ Planned maintenance rate explains
~45% of the variability in total
complaints per million packs
▪ This relationship may be related to the
common factors that influence both
planned maintenance rate and total
complaints – such as focus on
prevention, operational excellence,
quality improvement mindset
R² = 46%
P=0.00
-10
0
10
20
30
40
50
60
70
80
1000 4020 806010 30 50 70 90
Total complaints rate (including LOE) per million pack
Planned maintenance rate, %
17ispe.orgConnecting Pharmaceutical Knowledge
FDA Guidance Metrics Definitions
Product Quality Complaint Rate – proposed normalization by packs
released
– When normalized by released lots attempted has no connection to other
quality metrics
– When normalized by packs there is correlation to cultural metrics
SOURCE: ISPE Quality Metrics Initiative
18ispe.orgConnecting Pharmaceutical Knowledge
Available from ISPE
Metrics that Matter
Pursue the Red
Wide variability in the Quality of information
Only 50% of consumers have product at time of reporting
Financially motivated fraud artificially inflates the numbers
New product launch and product changes result in an increase in complaints
OTC Complaint Reality
Consumer Comment Journey
P&G Approach to FDA Q Metrics
Leadership role in Industry initiatives
ISPE Pilots: 2 sites
Developing IT solutions – $$$$
Internal FDA Quality metrics reporting program
Leadership reviews -> Quality Council integration
Internal FDA Q Metrics Reporting
Top-Level Learning
FDA Q Metrics have provided new insights
Product grouping approach
Initial challenges to align on definitions and adapt to supply chain differences
Manual reporting burden high at start – improves over time
Complaint rate normalized by released lots is not meaningful
27
ASQ Quality Culture Workshop
28
Cultureofquality.org/assessment
Quality Culture Assessment
P&G
Consumer Products
All Respondents Understanding CustomerNeeds
Leadership Commitment toExcellence
Adapting AcrossGeographies/Cultures
Investment in PerformanceImprovement
Employee Engagement
Performance Metrics andMeasurement
Pursuit of PerformanceExcellence
Quality Vision and Values
Risk Management Culture
Customer Engagement
Average Scores
Overall By Theme
30
ASQ Culture of Quality Roadmap
Define
Align
Manage
1. Evaluate Current
Performance 2. Identify How Culture
Plays a Role
3. Clarify Improvement
Vision
4. Define Values andExpected Behaviors
5. Clarify Strategies and Goals6. Engage
Team to define
SMART Goals
7. Clarify & Track Key Measures
8. Maintain a Management System for Priorities/
Goals
9. Manage Communicati
on Habits
10. Build Motivations Throughout
PCPC Industry Support
32
60%
40%
2016 Awareness of FDA Q Metrics Program
Yes
No
0
2
4
6
8
10
12
14
16
18
1 2 3 4 5
2016 Level of Readiness
44%56%
2016 Level of Interest Participating in
Benchmarking
Yes
No
Group Discussion
What can PCPC do to help your company prepare?
How can you leverage the FDA Quality Metrics program to accelerate your journey?
Where is Your Journey Headed
… Compliance or Excellence?