Vendor Performance Evaluation

Vendor Performance Evaluation

27th CIPMM National Workshop June 7-8, 2016

1

Agenda

1)! Objectives and Project Background

2)! Evaluation Process

3)! Evaluation Templates and Scoring

4)! Use of Vendor Score

5)! Status To Date

2

Objectives & Background Process Templates &

Scoring Score Usage

Status to Date

Background !! In January 2015 the City launched an evaluation system

for construction contracts as a result of: •! Council Direction.

•! Auditor General recommendation in Construction Services Audit.

•! Best practices.

3


Scoring Score Usage

Status to Date

Objectives

The objectives of performance evaluation are to: •! Improve communication between vendors and the City.

•! Provide feedback with the goal of performance excellence.

•! Create and track standard Key Performance Indicators (KPI).

•! Support the Contract Administration process used to address non-performance.

4


Scoring Score Usage

Status to Date

Development Process

!! System designed through researching best practices and

industry consultations. !! Reviewed reports and evaluation systems in use:

5

FEDERAL: PWGSC, Defence Construction Canada

PROVINCIAL: MTO, Infrastructure Ontario

MUNICIPAL: Calgary, Toronto, Hamilton, Oakville, Mississauga


Scoring Score Usage

Status to Date

Development Process

!! Research identified 5 key important evaluation categories

that needed to be evaluated:

6


Scoring Score Usage

Status to Date

QUALITY CUSTOMER SERVICE

COST CONTROL TIMELINESS

SAFETY

Development Process

!! Consulted with numerous industry organizations:

!! Industry very supportive of an evaluation system.

7

CONTRACTORS: •! Ottawa Construction Association / Canadian Construction Association

•! General Contractors Association of Ottawa •! National Capital Heavy Construction

Association CONSULTANTS: •! Consulting Engineers of Ontario

•! Ontario Association of Architects / Ottawa Regional Society of Architects

•! Ontario Association of Landscape Architects


Scoring Score Usage

Status to Date

Process Design – Industry Feedback Detailed process was built with focus on consistency and fairness. Process included key industry feedback: 1.) Industry wanted better defined scoring criteria.

•! There are standard templates that describe each criteria and performance level.

•! A start-up meeting will further clarify the criteria as it relates to project specific expectations .

8


Scoring Score Usage

Status to Date

Process Design – Industry Feedback 2.) Industry wanted multiple controls to ensure

consistency and fairness. •! Project Managers (the evaluators) will receive technical

and process training.

•! All evaluations are reviewed and approved by higher level managers.

•! Evaluations based on supporting documentation. There are more rigorous reviews for very high scores and those below satisfactory.

•! Scoring trends will be monitored internally.

9


Scoring Score Usage

Status to Date

Process Design – Industry Feedback 3.) Industry wanted to give written feedback in the online

system prior to approving or appealing the evaluation.

•! The system enables open text vendor feedback.

4.) Industry wanted a chance to review and discuss the evaluation prior to it being entered into the system.

•! The performance evaluation will be a mandatory topic at monthly update meetings and project completion. Feedback will be ongoing.

10


Scoring Score Usage

Status to Date

Process Design – Industry Feedback 5.) Industry wanted the evaluation to reflect certain extra

costs and delays outside their control. •! Vendors will not be penalized for things outside their

control (requested scope changes etc.).

•! Regular communication and documentation will ensure these issues are addressed at the time of occurrence and reflected as such in the evaluation.

11


Scoring Score Usage

Status to Date

Process Design – Industry Feedback 6.) Industry wanted an appeals process.

•! Appeals incorporated into evaluation process.

•! Vendors given fair period to file an appeal.

•! Evaluation score suspended until final decision rendered.

!! General contractors, design consultants, and contract administrators all contribute to the success of a construction project. They are all subject to evaluation to help ensure consistency and fairness.

12


Scoring Score Usage

Status to Date

Development Challenges !! Previously there were only paper-based evaluations on

some projects. No formal rules and results not analyzed. !! Had to engage multiple user groups across the City and

develop standard processes and templates.

!! Had to configure a centralized electronic database with the City’s processes to host the evaluations.

!! Had to develop a “phase-in” strategy to speed up

implementation.

13


Scoring Score Usage

Status to Date

Evaluation Scope !! Notified up front on eligible contracts. ‘Construction’

includes: •! Engineering consultants (design, contract admin work etc.).

•! Construction (general) contractors.

!! Evaluations done for the prime contractor/consultant based on contract value. Evaluation Thresholds: Consultants: $15,000 Contractors: $100,000

14


Scoring Score Usage

Status to Date

“No-Surprises” Evaluation Process !! Operational departments go through the evaluation

process: •! Start-up meeting – to discuss expectations for the project.

•! Progress meetings – to discuss performance to date and identify areas for improvement.

•! Close-out – to discuss final evaluation rating. “No surprises” approach after regular communication.

•! Appeal – vendors have 15 calendar days to appeal a final rating.

15


Scoring Score Usage

Status to Date

“No-Surprises” Evaluation Process Close-out/Appeal continued: !! VPM performance is discussed throughout the project.

!! Vendors can request a debrief meeting to discuss the issued evaluation in detail.

!! Appeals resolved within 15 business days.

16


Scoring Score Usage

Status to Date

Templates – KPI Scoring !! 3 templates: general contractor, engineering consultant,

and contract administration. !! A rating guide describes each KPI with 5 performance

levels and a corresponding rating:

17

Performance Level (Highest to Lowest)

KPI Rating (% of total possible Points)

Outstanding 100% Commendable 85%

Satisfactory 70% Needs Improvement 50%

Not Satisfactory 25%


Scoring Score Usage

Status to Date

General Contractor (Construction) Template

!! 8 KPI for general contractors with the following point weightings:

18

Key Performance Indicators Points Overall Project Management 15

Supervision 10 Quality 15

Health and Safety 15 Cooperation and Client Relations 10

Cost Control 10 Site Management 10

Schedule Management 15 Total 100


Scoring Score Usage

Status to Date


!! Sample general contractor KPI:

19


Scoring Score Usage

Status to Date


!! Sample general contractor KPI:

20


Scoring Score Usage

Status to Date

Engineering Consultant Template !! 6 KPI for engineering consultants with the following point

weightings:

21


Schedule Management 15 Budget Management 20

Quality of Design 20 Issue and Risk Management 15

Communication and Cooperation 10

Total 100


Scoring Score Usage

Status to Date

Engineering Consultant Template !! Sample design consultant KPI:

22


Scoring Score Usage

Status to Date

Engineering Consultant Template !! Sample design consultant KPI:

23


Scoring Score Usage

Status to Date

Contract Administration Template !! 8 KPI for contract administrators with the following point

weightings:

24


Schedule Monitoring 15 Cost Control 15

Technical Support 10 Oversight of Contract Compliance 15

Issue and Risk Management 10 Communication and Cooperation 10

Records Management 10 Total 100


Scoring Score Usage

Status to Date

Contract Administration Template !! Sample contract admin KPI:

25


Scoring Score Usage

Status to Date

Contract Administration Template !! Sample contract admin KPI:

26


Scoring Score Usage

Status to Date

Scoring !! Individual evaluation scores and overall Vendor Scores

(OVS) are tracked. !! OVS is a weighted-average over the last 3 years and can

be template-specific:

!! Low scores will have supporting comments and

documentation.

27

(Year 3 refers to the most recent year, also used by MTO and IO)


Scoring Score Usage

Status to Date

Scoring !! Individual evaluations and OVS scores fall into 5

performance levels:

!! Those with an OVS below Satisfactory (in yellow) are monitored more closely.

28

Performance Level Score Range

Outstanding 90-100% Commendable 80-89%

Satisfactory 70-79% Needs Improvement 50-69%

Not Satisfactory <50%


Scoring Score Usage

Status to Date

Rationale for Using Past Performance !! The initial lowest priced bidder isn’t always best value or

even the lowest bid (cost overruns later).

!! City mandate is to get “Best Value” on purchases. Best value is the optimal balance of performance and cost.

!! Evaluations reflect the vendor’s ability to perform -

deliver good quality/service, meet deadlines, control costs etc.

29


Scoring Score Usage

Status to Date

Use of Vendor Score – Contractors !! Vendors scores will in the future be a criteria in tender

evaluations. •! May bypass the lowest bidder for a better performing

contractor. •! Reduces pressure to make trade-offs from competing on

just price alone.

Tender Evaluation Example:

30

Price /70 Vendor Score (OVS) /30

Total Evaluation Score /100


Scoring Score Usage

Status to Date

Use of Vendor Score – Consultants !! Vendor scores will in the future be an additional criteria in

Requests for Proposals for design and contract admin.

!! The weighting could be standardized or vary by RFP based on a sensitivity analysis.

!! Putting the vendor performance criteria after the initial pass/fail technical component is ideal in practice. •! Having it in the technical component leads to a loss of control

over the pass/fail on the technical. •! Technical rating is also project-specific first screen.

31


Scoring Score Usage

Status to Date

Program Status to Date At the end of Q1 2016 . . .

!! 176 completed final evaluations for 84/157 vendors with projects for evaluation.

!! 5/176 final evaluations below satisfactory (70%).

32


Scoring Score Usage

Status to Date

Projects Opened: 613

Engineering Consultant: 262 43% Contract Administration: 139 23% Construction Contractor: 212 34%

Program Status to Date At the end of Q1 2016 . . . !! 80.46 average vendor score (in commendable range).

!! Average scores by template (performance program):

33


Scoring Score Usage

Status to Date

Average Score:

Highest Average KPI: Lowest Average KPI:

Engineering Consultant:

83.05 85.22 – Overall Project Management

81.75 – Issue and Risk Management

Contract Administration:

79.63 82.50 – Schedule Monitoring

76.07 – Communications and Cooperation

Construction Contractor:

79.55 85.30 – Schedule Management

75.36 – Health and Safety

Program Status to Date At the end of Q1 2016 . . . !! No-surprises approach resulted in only 4 appeals.

!! More evaluations are required to allow for meaningful analysis and use of vendor score.

!! Conducted follow-up consultations to review the initial year.

34


Scoring Score Usage

Status to Date

Program Status to Date Feedback After Initial Year (2015). . . !! General feedback from all groups positive. Key points:

•! Continued training support and program monitoring essential.

•! Vendors want to know their relative performance (vs. average score, percentile).

•! Using scores can’t be rushed – needs thorough internal/external consultations and adequate data.

•! Performance feedback needs to be ongoing from project start to end.

35


Scoring Score Usage

Status to Date

Program Status to Date Lessons Learned So Far . . . !! Be wary of project scope. Development, learning and

change management take substantial resources.

!! Getting industry involved in development process promotes adoption and improves procedures.

!! Don’t rush software acceptance testing.

!! Expect there will need to be adjustments to business processes and software to improve.

36


Scoring Score Usage

Status to Date

37

THANK-YOU

QUESTIONS?

Vendor Performance Evaluation

Documents