Quarterly update on the CDRH Case for Quality Voluntary Medical … · 2018-02-27 · • Quarterly, 1-2 hour digital meetings between organization and LA • Orgs provide progress

1

February 27, 2018

MDICx webinar seriesQuarterly update on the CDRH Case for Quality Voluntary Medical Device Manufacturing and Product Quality Pilot Program

George ZackTwo Harbors ConsultingKimberly KaplanCMMI InstituteJoe FriedrichBoston ScientificFrancisco VicentyFDA Center for Devices and Radiological Health

Medical DeviceDiscovery Appraisal Program

MDICxQuarterly update:CDRH Case for Quality Pilot Program February 27, 2018

• Case for Quality & MDDAP

• Program Success Indicators and Metrics

• What is CMMI

• Program Overview and Lessons Learned

• Additional Information

Agenda

Case for QualityMedical Device Manufacturing and Product Quality Program

2011: Launch of Case for Quality - Dr. Jeff Shuren Discusses Case for Quality

2014: MDIC joins the Case for Quality - Video Overview of the Case for Quality

2015: Deloitte White Paper - CMMI Selection

2016: Proof of Concept Pilots - Outcomes

2017: Launched Pilot Program

https://youtu.be/X4KAOCi1RuE

http://mdic.org/cfq

https://www.youtube.com/watch?v=5ne0cjCOwhs

http://mdic.org/wp-content/uploads/2015/06/MDIC-Deloitte_Maturity-Model-Research-Report_June-2015.pdf

http://www.cmmiinstitute.com/

http://mdic.org/wp-content/uploads/2015/11/MDIC-Maturity-Model-Pilot.MDIC_.Oct-26-Final.team-roster.pdf

What is this program?

Reduced defects / rework

Accelerated time to market

Increased Customer Satisfaction

A culture of quality - across the organization.

This pilot program leverages the CMMI framework as the standard maturity model by which medical device organizations may measure their capability to produce high quality devices and increase patient safety. FDA will adjust their engagement activities and submission requirements as a recognition of this independent assessment of quality maturity. The CDRH Voluntary Medical Device Manufacturing and Product Quality pilot was announced in the Federal Register on December 28, 2017.

Reduced costs

• Value to ParticipantsWhat value are stakeholders getting from program?“Program Effectiveness”

• Consistency & Scalability Is the program sustainable?“Program Adoption”

• Elevating IndustryThe long term “next steps”…

• What: Appraisal identifies opportunities for improvement, reduced regulatory burden, increased innovation, faster time to market How: Survey results, participant feedback, appraiser feedback, FDA feedback, lessons learned incorporated into program

• What: Program operations are performed consistently for growing # new participantsHow: Number of appraisals, wait time to appraisal, number of appraisers trained in the program, lessons learned incorporated into program

• What: Industry baseline for organizations to benchmark improvement journeyHow: trend participant results & quality performance metrics over time

What does success in this pilot program potentially look like?

Success Components Success Identification

• # of Enrollees13 Organizations | 17 Facilities

• # of Appraisals6 Complete | 3 Scheduled

• Time from enrollment to appraisal execution (days)Range = 52 - 144 | Mean = 98 | Target = 90

Program Adoption Metrics to Date

• Did the appraisal identify areas or processes that could improve how work is performed to increase product quality?

91% YES | 9% NO

• Did the appraisal practice areas conflict with any regulatory compliance assessment areas?

98% NO | 2% YES

• Did you find the appraisal to be of value? 100% YES | 0% NO

• Would you recommend this program?100% YES | 0% NO

Program Effectiveness Metrics to Date

• The Capability Maturity Model Integration (CMMI®) is a capability improvement model that can be adapted to solve any performance issue at any level of the organization in any industry.

• The Model provides guidelines and recommendations for helping your organization diagnose problems and improve performance.

• Used by over 10,000 organizations from more than 100 countries all over the world, CMMI helps you identify and achieve measurable business goals.

What is CMMI®?

60%DECREASE

100%MILESTONE

FIDELITY

50%DECREASE

on timecost of poor quality defects

Czech Republic

50%DECREASE

non-conformities on project deliverables

INCREASE

On Budget Performance

INCREASE

On Time Performance

Customer Satisfaction

INCREASE

39% 71% 53%

Productivity 25%INCREASE

Organization High Level Roles for Pilot

Pilot SteeringCommittee

Provides leadership, direction and pilot process input

FDA Provides regulatory modifications, verifies participants, reviews detailed findings and improvement, provides pilot process input

MDIC Coordinates working groups of enrollment, appraisal, metrics, communications and program oversight; provides pilot process input

Appraisers Execute appraisals, provide findings to participants, executes checkpoints, provides full datasets to coordinating center, provides pilot process input

ParticipatingDevice

Manufacturers

Receives appraisals, drives improvements within business, participates in checkpoints to report progress, provides pilot process input

CMMI® InstituteProgram

Management Office

Provides model, manages enrollment/de-enrollment, provides playbook for appraisers, provides appraiser training, connects appraisers to participants, adjusts appraisal scope as necessary, assures appropriate appraisal and appraiser consistency, collects appraisal data, trends data, provides deidentified data to participants / steering committee, manages appraisal issues, adjust approach based on feedback from steering committee and stakeholders

Pilot SteeringCommittee FDA MDIC Appraisers

Participating DeviceManufacturers

CMMI® InstitutePMO

These benefits reduce the burden and disruption of audits, accelerate the review and approval process for changes, and shift resources to innovation and improvement.



CMMI® InstitutePMO

Onboarding and Scoping

• Kick-Off Call with PMO

• Intake Call & Follow-up Scoping Calls

• Sampling method for devices

• Identifying Appraisal Participants

• Determining Appraisal Schedule

• Plan reviewed with Org and PMO

• Submission of Quality Metrics to FDA

Lessons Learned: Inspection benefit moved until after appraisal is scheduled, allows FDA time to implement & ensures org engagementProgram kick-off meetingTraining created to expand appraiser bench



CMMI® InstitutePMO

Appraisal Execution

• 5 days, 2+ appraisal team members

• CMMI and Medical Device Experience

• Includes tour of the facility for context

• Interview-based, no doc review

• Conversations with those who do work

• Validation sessions

• Final results belong to organization

• Org identifies weaknesses to focus on

Lessons Learned: Intake/scoping templates, standardized heatmap tool, post-appraisal survey, unique individual journey, mentorship



CMMI® InstitutePMO

QA, Trending of Data

• Data submitted to PMO for QA

• Data aggregated for submission to FDA

• Data de-identified for trending

---

• Submission of Quality Metrics to FDA

• Metrics dashboard for individual

improvement journey

• Identification of common metrics for

de-identified trending

Lessons Learned: collection of quality metrics shifted to FDA activity

What data is shared?To Appraised Organization and CMMI PMO

Full scorecard and specific results against model

What data is shared?To FDA

Process area results and overall score for an organization• Elevating Industry

The long term “next steps”…

• What: Industry baseline for organizations to benchmark improvement journeyHow: trend participant results & quality performance metrics over time

Not actual results

Future: How we can use these results



CMMI® InstitutePMO

Post-Appraisal Activities

• Quarterly, 1-2 hour digital meetings

between organization and LA

• Orgs provide progress report, LA

provides feedback on activities

• Conversations inform second appraisal

---

• Submission of Quality Metrics to FDA,

org can change after appraisal results

• Individual improvement journey,

understand why metrics have changed

Lessons Learned: standardized template for check point activity, 2nd appraisal, variable submission of quality performance metrics

• FAQs: http://cmmiinstitute.com/MedicalDeviceFAQ

• October 10th FDA Public Meeting Presentation: https://www.fda.gov/MedicalDevices/NewsEvents/WorkshopsConferences/ucm568069.htm

• November 15th MDIC Meeting Presentation: http://mdic.org/cfq/case-for-quality-public-forum-presentations/

Additional Information

http://cmmiinstitute.com/MedicalDeviceFAQ

https://www.fda.gov/MedicalDevices/NewsEvents/WorkshopsConferences/ucm568069.htm

http://mdic.org/cfq/case-for-quality-public-forum-presentations/

24 Boston Scientific Confidential -- For Internal Use Only. Do Not Copy, Display or Distribute Externally

CMMI MDDAP AppraisalFDA Case For Quality

Boston Scientific Experience

Corporate Sponsor: Conor Dolan, VP Quality Systems, BSCSite Sponsor: Joe Friedrich, Quality Director, Arden Hills, MN


Quality Policy

BSC Culture of Quality

Quality Organization MissionWe exceed expectations with customer-centric quality

solutions that transform patient lives


MDDAP Overview and BSC Status

BSC Arden Hills (since Jan 10)

BSC Clonmel(as of Feb 19)


Why did BSC decide to participate?

• Use a defined maturity model and practice capabilities rather than a compliance approach, to understand specific areas our site can work to improve.

• Operationalize significant regulatory benefits as it relates to PMA submissions, manufacturing site change submissions, and the 30 day regulatory submission process.

• Create a relationship with an external partner organization that is proactive and focused on continuous improvements in overall product quality and patient outcomes.

• Opportunity to be part of shift towards more patient-centered culture across med device industry and FDA.


Appraisal Preparation and Execution

Appraisal Planning

•Scoping: Three 2 hour meetings

•Defined Scope –Practice Areas

•Defined Duration

•Identified Interviewees

•Multi-site Approach: Two 1 hour meetings

•Site Coordinator: Lead Appraiser –Five 1 hour calls

•Locked Appraisal Plan and Logistics

Logistics

•Two Appraisal Teams

•Two conferences rooms

•1 Board Room•Internet and

Projection•On-site Lunch

Resources

•31 Participants•1 Site

Coordinator•8 Hours Active

Engagement / Interviews Over 3 Days

•4 Hours Verification

•2 Hours Results•7:30 to 5:30

Daily

Support

•No Backroom•No SOPs•No Records•No Laptops


Product Sampling

For the appraisal team• The Product Sampling is a means to show appropriate site coverage

Product Line Differences Process Differences Device Class Differences

For the appraisal site• The product sampling is used to identify a range of individuals who have

participated on given sample projects. • These projects are used as a frame of reference for participants to

provide examples that support practice area questions.


CMMI Appraisal vs. FDA Inspection

CMMI Appraisal FDA Inspection• Capability maturity model appraisal of

business practices and processes.

• Focused on eleven (11) CMMI Model practice areas.

• Appraisers conduct group interviews of ‘doers’ responsible for work products.

• Appraisers engage in discussions to truly understand how the business operates relative to best practices.

• Many participants outside of QA.

• Minimal disruption to site resources and no need for Back Room / Front Room

• Weaknesses are opportunities to improve business process.

• Inspection of quality system procedures and records.

• Focused on 21 CFR Part 820 requirements.

• Inspectors interrogate Quality leaders, process experts, and record owners.

• Inspectors review procedures, records, storyboards looking for evidence of noncompliance to regulations.

• Few participants outside of QA.

• Large support team with Back Room / Front Room, streams, scribes, etc.

• 483s require escalation to CAPA and formal agency response.


CMMI vs. FDA: Participant Guidance

CMMI Appraisal FDA Inspection• Be open in answering questions and it’s

ok if you’re not 100% sure.

• It’s ok to volunteer information that may be tangentially related to the question.

• Highlight improvements made over time, as well as future plans if appropriate.

• Explain other systems and tools used to do your work. For example, PLCP, Business Essentials, VIPs, CI Boards, etc.

• No need to prepare for questions. Just respond as best you know based on you experience and perspective.

• Do not bring your laptop, exhibits, or documentation to your interview. Is it just an open discussion.

• Only answer questions asked. If you don’t know the answer, don’t guess.

• Don’t volunteer information that doesn’t pertain to question asked.

• Don’t discuss improvement opportunities or future plans.

• Don’t deviate from what is documented the quality system, i.e. policies, procedures, records, etc.

• Be prepared to speak to any pertinent documentation. Facilitators will ensure the right people answer questions.

• Bring in procedures and records as appropriate. A laptop may be needed to view certain electronic files.


CMMI vs. FDA: Resource Allocation

CMMI Appraisal – 5 Days

Prep Appraisal TotalSite Coordinator

80 Hrs 20 Hrs 100 Hrs

Participants 60 Hrs 180 Hrs 240 Hrs

Total 140 Hrs 200 Hrs 340 Hrs

2 Day FDA Inspection 1370 Hrs


BSC Feedback on CMMI Appraisal

Positives

• Appraiser Accessibility• Appraiser Competency• Valuable Participant Briefing• Structured & Organized• Valuable use of Participants Time• Conversational, Engaging and Positive Group Interviews• High-Level Questions Drove Good Discussion• Valuable verification sessions ensured site ownership

Improvements

• Initial / immediate definition / understanding of approach would be helpful• Expand Practice Area briefing to better define what will be covered• High-Level questions seemed vague. More up-front education on the level of detail.• Terminology at times was difficult, but rephrasing helped.• Pilot metrics may be difficult to define and implement.


So what have we learned?

• Our practices lined up well with the CMMI model overall• “Weaknesses” in areas of project and resource management• Cross-functional groups are engaging in healthy conversations about our business practices and

improvement opportunities• Need to gain alignment with other business functions on value proposition to help drive change

from appraisal results• Many stakeholders remain skeptical of the pilot

– How will the CDRH benefits actually work and will they truly materialize?– Will pilot investment (dollars and time) be worth it?– What risks do we bring on by participating?– Will quarterly metric reviews be value added?– What will be expected of us down the road to maintain benefits?

• This is a pilot… the plane is being built as we’re flying it!

Case for Quality

Program PilotCisco Vicenty

Program Manager, Case for Quality

Office of Compliance, CDRH

February 27, 2018

35

www.fda.gov

For Discussion

What can it show?

What are we seeing?

Lessons Learned

Next Steps

Comments to the Docket

Engagement and Collaboration

36

www.fda.gov

What can it show?

• Shows performance of the organization beyond establishing compliant processes

• Quickly identifies strengths at an organization or industry level and allows pilot to shift focus

• Allows FDA to engage collaboratively where common opportunities for improvement exist

• Allows organizations to benchmark their execution, acknowledge success, and focus improvement

37

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%EST

PLAN

MC

CM

MPM

RDMPQA

GOV

II

PI

TS

Benchmark Performance

Manufacturer A Manufacturer B Manufacturer C Manufacturer D

This data is not actual data but a representative demonstration.Results scores were randomly generated.

www.fda.gov

What are we seeing?

38

• Early trends• Strengths• Opportunities

• Performance is not necessarily dependent on product classification

• Some participants are scoring highly across scoped practice areas.

• Collaboration and sharing among participants

• FDA is asking different questions

Opportunities• EST – Estimating

• Estimation of the size, effort, duration, and cost of the work and resources needed to develop, acquire, or deliver the solution

• II – Implementation Infrastructure• Ensure that the processes

important to an organization are persistently and habitually used and improved

Strengths

• TS - Technical Solution• Cost-effective design and

solution that meets customer requirements and reduces rework

• PI – Product Integration• Assemble and deliver the

solution that addresses functionality and quality characteristics

www.fda.gov

Lessons LearnedLessons

• Small manufacturers are experiencing cost concerns

• High-maturity in the medical device space for certain practices. Review of these areas may not be value-added

• Potential for manufacturers to de-enroll. Need to adjust the start of FDA Modifications

• FDA internal inspection planning process has long lead times. Needed more active mechanism for communication

• Metrics submission and discussion was confounding the appraisal

Actions• CMMI has been developing solutions to

manage costs. Participants are also providing additional options.

• Activity in-process to modify scope of appraisal and practice areas to provide value

• Unanticipated constraints impacted some enrollees. FDA is modifying start of modifications until after the scoping is confirmed by CMMI

• FDA has implemented an active notification system to communicate enrolled sites among the planning organizations

• FDA will work individually with participant during scoping phase to discuss the metrics and submission

39

www.fda.gov

Next Steps• Continue scheduled appraisals• Evaluate current learnings and adjust

pilot if necessary• First quarterly check-ins will start in

March• First modified submissions to begin

and rolled out to other participants• Expand in-scope practice areas for

increased application in 510(k) space• Evaluate metrics submitted and

correlations to progress

40

Reward the journey, not the destination

www.fda.gov

Comments on the program

• Docket Comments• Received comments from AdvaMed and

the BRIDGE Coalition• FDA is working to incorporate insights

from those comments into the pilot and into the next steps

• Current Efforts• Transparency• Increasing application to 510(k) • Common vocabulary

41

www.fda.gov

Engagement and Collaboration

42

• Voluntary Pilot was announced on December 28, 2017 and is actively enrolling participants

• Engagement and feedback are critical• For additional information, enrollment, or feedback

• http://mdic.org/cfq/• http://mdic.org/cfq/enroll/• [email protected]

• For participants• For any issues or concerns through out the pilot contact

[email protected] or [email protected].

http://mdic.org/cfq/

http://mdic.org/cfq/enroll/

mailto:[email protected]



www.fda.gov

Questions?

43

Thank You!

Questions and Discussion• Submit your questions via the chat box

• Recording and slides will be available on MDIC’s web site

Coming this springPlease plan to join MDIC, FDA and your industry colleagues at Xavier University on May 1st for a face-to-face workshop to discuss the preliminary outcome and future direction of the pilot study.

More information and registration information at: http://www.xavierhealth.org/case-for-quality-workshop.

http://www.xavierhealth.org/case-for-quality-workshop

Quarterly update on the CDRH Case for Quality Voluntary Medical … · 2018-02-27 · • Quarterly, 1-2 hour digital meetings between organization and LA • Orgs provide progress

Documents