2007 Annual Conference Measuring Regulatory Excellence: Key Performance Indicators Bruce G. Matthews, P.Eng. Professional Engineers Ontario
2007 Annual Conference
Measuring Regulatory Excellence: Key
Performance IndicatorsBruce G. Matthews, P.Eng.
Professional Engineers Ontario
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Overview• Project Definition and Scope
• What is a Key Performance Indicator?
• Project Process
• Challenges
• List of Potential KPIs
• Survey re: Relevance and Importance
• Recommended KPIs
• Aftermath
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Project Definition and Scope
• Project arose from a Strategic Plan Objective: “Increase the public’s confidence in PEO as a regulator”
• Associated Strategic Initiative: “Develop and implement key performance indicators and targets of regulatory effectiveness, benchmarked against other comparable regulatory bodies.”
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Project Definition and Scope con’t
• Literature review re: best practices
• Review of other regulators’ performance measurement practices
• Survey of stakeholders re: perceptions and expectations
• Assess availability of data for ongoing tracking / reporting of KPIs
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
What is a Key Performance Indicator?• A metric used to reflect the critical success
factors of an organization
• Can be financial or non-financial
• Nature of KPIs will differ depending upon the nature of an organization
• A KPI is a key part of a measurable objectives (which includes a direction, KPI, benchmark, target, timeframe)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Effective KPIs…• Should help define:
– The What – Are we doing the right things? What is our outcome in terms of impact and accomplishments?
– The How – Are we doing things right? Do we have the right inputs in terms of resources and processes?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Project Process• Literature Review
• Environmental Scan - Review performance measurement practices of other professional regulators
• Identify potential KPIs
• Survey stakeholders re: relevance and importance of potential KPIs
• Review pros and cons of potential KPIs
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Literature Review re: Best Practices
• Performance measurement best practices– Balanced scorecard– UK National Consumer Council– “The Regulatory Craft”– “Red Tape to Smart Tape” International
Conference (2002)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Balanced Scorecard• Developed by Harvard professors Kaplan
and Norton
• Four categories– Customers– Internal Business Processes– Financial– Organizational Capabilities
• Weakness re: definition of “customer”
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
UK National Consumer Council
• Three steps to credible regulation– Objectives– Content– Structure and Governance
• www.ncc.org.uk
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
“The Regulatory Craft”• Book by Malcolm K. Sparrow (Brookings
Institution Press, 2000)
• Controlling risks, solving problems and managing compliance
• Matrix of “Regulatory Competencies” and “Business Results”
• Acknowledges “internal” versus “external” measures
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Red Tape to Smart Tape Conference
• Sponsored by the Ontario Government in association with the Institute of Public Administration of Canada
• Core concept – Traditional business results measurements do not address public welfare issues and are not suitable for regulatory purposes
• www.smarttape.ca
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Environmental Scan• Other engineering regulators in Canada
• Other professional regulatory bodies in Ontario and across Canada
• Most report key transactional statistics (e.g., number of licence applications)
• Little evidence of outcome measurement
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Challenges• What does it mean to be an effective
regulator?
• Inability to benchmark due to lack of reporting by other “comparable regulatory bodies”
• Can’t use measures typical for professional associations
• KPIs must be evaluated in context
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
List of Potential KPIs• Competence programs
• Intrusions on regulatory mandate
• Complaints about licence holders or unlicensed individuals
• Licensing turnaround times
• Compliance action statistics
• Council evaluation
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
List of Potential KPIs con’t
• Fairness & dignity survey
• Complaints about PEO
• Employee engagement
• Percentage of licence holders with no reported violations
• Public confidence survey score
• Ratio of administrative overhead to program delivery cost
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Survey re: Relevance and Importance
• PEO Council, Senior Management and selected external stakeholders
• Relevance – the applicability of a KPI as a measure of PEO’s effectiveness as a regulator
• Importance – the priority that should be given to a KPI as a measure of PEO’s effectiveness as a regulator
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Survey re: Relevance and Importance
• Survey also included proposed short-term measures for each KPI, intended to cover the introductory transitional period
• Survey results were assessed in view of the best practices review– Need balance of internal and external
measures)– Need to ensure key regulatory competencies
are represented
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Recommended KPIs• Competence programs
• Complaints about licence holders or unlicensed individuals
• Compliance action statistics
• Employee engagement
• Public confidence survey score
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Competence Programs• Pros
– Competent members reflect a well regulated profession
– Competency is fundamental to professional regulation and easily understood by the public
• Cons– PEO currently has no competence programs
in place and none planned in the near term
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Complaints about Licence Holders etc.
• Pros– Easily measured (transaction statistics
already reported)
• Cons– Frivolous or vexatious complaints could skew
the data– Does complaint volume really speak to
regulatory effectiveness?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Compliance Action Statistics
• Pros– Clear statement of enforcement activity– Some statistics already reported
• Cons– Need to define what qualifies as a
“compliance action”– Of greater interest to licence holders than to
the public
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Employee Engagement• Pros
– Regular survey of employees already in place– Survey is widely recognized and used
• Cons– Is PEO willing to take action in deficient
areas?– Are the survey areas relevant to a
professional regulatory organization?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Public Confidence Survey Score
• Pros– Solid external measure of how PEO is
perceived– Survey would be conducted by an
independent research firm
• Cons– Frequency of measurement– Who is “the public”?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Implementation Issues• Currently no competence programs in
place at PEO
• Measurement data exists for only three of the recommended KPIs
• Public Confidence Survey still in development
• Issues of KPI reporting (timing, frequency, format) still to be determined
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Aftermath• Project report received and considered by
PEO Council in November 2006
• Concern that recommended KPIs were all “lagging indicators” – there were no “leading indicators”
• Concern that recommended KPIs would drive policy and program development
• Concern about lack of competency programs
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Aftermath con’t• PEO Council struck a task force to further
examine the question of KPIs for PEO
• Task force report (June 2007) recommends a new project with more research work re: critical success factors before defining KPIs
• Status as of September 2007: awaiting new project charter / terms of reference
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Speaker Contact Information
Bruce G. Matthews, P.Eng.Manager, Complaints & DisciplineProfessional Engineers Ontario25 Sheppard Avenue West, Suite 1000Toronto, Ontario M2N 6S9T: 416-840-1076F: 416-224-9974E: [email protected]
2007 Annual Conference
Measuring Regulatory Excellence – A Self Audit Tool Focused on Entry to Practice
Jan RobinsonRegistrar & CEO
College of Physiotherapists of OntarioSeptember 8, 2007
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Interesting times
• accountability
• transparency
• movement of internationally educated professionals
• public perspective
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Politics in Ontario
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Ontario Regulators for Access (ORA)
• Info Sharing
• Best practices
• Collaboration
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
“measures” ➪ government driven
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
report card to progress report
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Key Factors Included• # of internationally trained applicants
(annual comparisons)• top five source countries• types of license granted• project initiatives
Focus on “10”
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Thomson review
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Key Recommendations
• fair registration practice code
• information for applicants
• criteria for decisions
• reasonable costs
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
• alternative evidence for unattainable documents
• training
• third party review
• self audits with reporting
• appeal procedures
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
2002 – ISO 17024
General requirements for bodies operating certification schemes
for persons
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
ANSI
SCC
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
2006 Fair Access to Regulated Professions Act
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Key features
➪ access centre ➪ Fairness Commissioner ➪ audit
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Consider self audit tool
• gap analysis
• demonstrate commitment
• continuous improvement
• achieve best practice
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Focus on entry to practice
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Conversion to standards38 – registration10 - organization
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Examples• The College has policies and procedures
related to confidentiality and privacy
• The College has clear processes for reviewing, evaluating and making changes to registration requirements
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Examples• The College has uniform competence
requirements
• The College ensures that individuals with a conflict of interest are not involved in the decision-making process related to an applicant.
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Format
• standard definition
• evidence
• analysis
• recommendation
• ISO reference
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Key Findings
• identified no major gaps or concerns
• links between College and third party hard to evaluate
• Identified new areas for policy work activity
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
• identified areas where evidence weak
• tedious
• how often necessary?
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Key Outcomes
• confidence in fairness and transparency of process
• evidence of performance all included in one place
• celebration of excellence
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Next steps
• develop work plan to implement recommendations
• share with government and other regulators
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Speaker Contact Information
Jan Robinson
Registrar & CEO
College of Physiotherapists of Ontario
e-mail : jrobinson @collegept.org
www.collegept.org