RFP Evaluation Training

Request for Proposals (RFP) Evaluation Training

Aviation Procurement UnitMarch 26, 2009

2

AGENDA

When to Use an RFP Approach?

Overview of the Evaluation Process?

What are the Expectations for Evaluation Committee Members?

What is the Approach to the Evaluation of Proposals and Scoring Guidelines?

What is the Approach to Evaluating Technical Proposals?

What is the Approach to Evaluating Cost Proposals?

Overview and Steps of Consensus Scoring?

Best Practices for Evaluators

Next Steps

Questions and Answers

3

Best Value Should Always be the Purpose of the Evaluation Process?

A Best Value Award is one which optimizes quality, performance, cost, time, and efficiency in a fair manner to the contractor community

4

When is an RFP the Right Approach?

The Bid process is the default solicitation option.

When we are not sure of the best solution and multiple options exist.

When performance based incentive contracts are used.

When complex factors in addition to price are very important.

When there are complex requirements.

When it is a new program/ project that has not been tested.

When it is desirable to find out if there are alternative options to what is normally performed.

5

What is the Best Way to Handle Criteria for RFP Evaluations?

All evaluation criteria MUST be established prior to the issuance of RFP.

Criteria should be clearly linked to desirable outcomes.

Be clear and concise in language to avoid confusion

Evaluation information should be documented, dated, and secured throughout the evaluation process.

6

How Do We Develop an Evaluation Strategy?

Identify (eliminating) Minimum Qualifications

Define CriteriaDevelop SpecificationsDecide importance of Oral Interviews/ PresentationsIncorporate Reference Checks into Industry ExperienceDecide importance of Site Visits

How important is Cost; Cost to Technical Score Ratio?

Determine distribution of points among each criteria

7

What are Examples of Standard Criteria in a Technical Proposal?

Corporate Experience:What are the proponents’ past performance and experience in similar types of contracts? What is the experience level of the essential subcontractors?

Project Organization and Staffing:What is the staffing approach?What are the qualifications and past performance of the personnel?

Implementation Approach:What is the adequacy of their work breakdown structure/ project management?Is it Comprehensive? Reasonable?Proposed Deliverables?Preparation Plan?

8

What are Examples of Standard Criteria in a Technical Proposal?

Operation/ Management Approach:Plan for Regular/ Daily Operations?Approach to Quality Control and Assurance?What are the Commitments to Performance Standards?What are the Plans for Maintenance?What are the Contingency procedures in the event of an emergency?

Corporate and General Capacity:What is the overall capacity of the organization?Do they have a plan to sufficiently staff the contract project?How long has the company been operating?What is the turnover ratio?

9

Overview of the Evaluation Process

Rules and regulations that are applicable to the evaluation process should be followed throughout the entire process.

DOP makes a determination of responsiveness and responsibility based upon minimum qualifications.

APU facilitates and coordinates process between DOP and evaluators

Overview of the Evaluation Process

Evaluation committee members independently review proposals based upon the specific evaluation criteria

Technical and cost scores are assigned applying the designated evaluation criteria

Oral interviews are conducted, if necessary

Negotiations are conducted with the winning proponent

10

11

What are Standards of Conduct While Serving as an Evaluator?

No Conflicts of Interest

No solicitation or acceptance of gifts or anything of value

Integrity and Honesty during scoring and communications

Proper handling and discretion of all information; No Disclosure during process

Practice Ethical Behavior

12

What is Expected from Evaluation Committee Members?

The member has knowledge or experience regarding the RFP SubjectMatter

The member has sufficient time to dedicate in order to effectively evaluate the proposals.

Such time is dedicated to reading all proposal related materialsextensivelyAttending all Evaluation-related Meetings: Training, Evaluation Sessions, Oral Interviews, Debriefings, Criteria Planning

Conduct an independent review.Identify potential deficiencies in a proposal that could result in a determination of non-responsiveness or non-responsibility.Communicate any issues to the Contracting Officer immediately.

Limit communications regarding the RFP to those DOA and DOP staff who are directly assigned to the project.

13

What is the Purpose of Oral Interviews?

Allows proponents to clarify on areas that may be unclear in their proposals

Possibly reduces the likelihood of misunderstandings.

Evaluation committee members determine whether or not oral interviews will be necessary.•Oral interviews should follow group discussions•Oral interviews should ALWAYS precede scoring.

14

What is the Purpose of Reference Checks?

Reference Checks may be conducted with prior customers of the proponentReference Checks may also include references of key personnel.

Examples of Questions include:Were the projects/ deliverables completed on time?Were there any problems with the management team?Were the Project Management plans realistic?Were there any other significant problems?Did the Prime Contact stay the same or change over the duration of the project?Were the Personnel Knowledgeable and Competent?Would you do business with this vendor/ contractor again?How would you rate your level of satisfaction from 1-5?

Results of Reference Checks will be distributed to the evaluation team prior to scoring

Ensuring Security and Confidentiality

Keep proposals in a secure place.

Evaluators shall not discuss the contents of submitted proposalsor the procurement activities with any persons outside of the evaluation team or Procurement staff.

Evaluators shall not communicate the scoring outcomes or content of proposals and shall not disclose the status of any proposal.

15

16

How are Proposals Evaluated?

All Proposals shall be evaluated in accordance with the City’s Code of Ordinances and the criteria specified on the Percentage Evaluation Form and considering the information required to be submitted in each Proposal.

17

Evaluation Scoring Guidelines

Scoring will be conducted for each criterion on a scale of 1-10

Technical Criteria will be scored as such:

8-10:Oustanding: Proponent’s proposal exceeds expectation and demonstrates an excellent ability to reach the goals and objectives of the procurement. Proposal is consistently strong.

5-7: Highly Effective: Proponent’s proposal demonstrates a good ability to reach the goals and objectives of the procurement.

3-4: Effective: Proponent’s proposal demonstrates a fair ability to reach the goals and objectives of the procurement.

1-2: Marginally Effective: Proponent’s proposal demonstrates a minimal ability to reach the goals and objectives of the procurement.

18

Evaluation Scoring Guidelines

Scores will be adjusted to reflect the weight assigned to each criterion.

Each criterion must be reviewed according to the requested items of the RFP and scored based on the scoring definitions above.

Only the Criteria in the Evaluation Form will receive scores.

Any Proponents Proposal that does not meet the minimum requirements of the RFP should be identified as this may deem them non-responsive.

19

When is a Proposal Deemed Non-Responsive?

Only the Department of Procurement has authority to deem a proponent non-responsible and/or non-responsive.

Evaluators should notify the DOP Contracting Officer of deficiencies and/or clarifications needed.

All minimum requirements will be scored on a pass/fail basis.

Failure to pass all mandatory requirements may deem a proposal nonresponsive

e.g., number of similarly completed projects

Evaluators should promptly notify Procurement staff if there is a problem with the proposal.

What Approach Should We Use to Evaluate the Technical Proposal?

Minimum requirements should be evaluated on a pass/fail basis.

Review the established criteria and related weights.

Evaluate proposals strictly by DOA’s criteria as defined in the RFP for each proposal.

21

What Should Technical Scoring Look Like?

Weight Criteria Company 1 Company 2 Company 3

20% Organization/Resumes of Key Personnel

10 3 1

30% Overall Experience, Qualifications andPerformance ofPrevious Similar Projects

9 7 4

20% Strategic ApproachTo Increasing Federal Assistance to theCity through Government Agencies and Congress

8 6 3

Only the criteria indicated will receive scores. Scoring will be conducted on a scale of 1-10Technical Scoring should reflect the needs of the Proposal

22

What Approach Should We Use to Evaluate the Cost Proposal?

After technical aspects of proposal responses are initially reviewed, then cost proposals are reviewed.

Cost proposals may reflect price escalation, deceleration due toperformance, time, CPI, etc. may be applied.

Cost Evaluation Must be independent of the Technical Evaluation.

A Clear Formula should be used to Calculate Cost Scores.

Scores should be directly linked solely to costs proposed on a formulaic basis.

Cost proposals/ financial offers must be taken at face value when the cost proposal is evaluated.

Examples of Cost Proposals…

Performance Based Contract RFPs may read like this…..

Please propose Base Costs and Incentives associated with performance beyond mandatory requirements…

E.g., Beyond 90% Customer Satisfaction Scores consecutively for one year will lead to a 3% bonus (e.g.., janitorial)

Cost Savings Contract RFPsmay read like this…..

Please propose Base Costs and Incentives associated with cost savings beyond mandatory requirements…

E.g., Company X will receive 10% of all cost savings beyond performance guarantee (e.g.., workers’ compensation)

23

Let’s Take a Look at the Cost Proposal Scoring…

The lowest fee offer receives the maximum points possible (10) . In the event, that it is a revenue generating proposal, it would be the highest offer.

Points are distributed based upon percentile in which the fee offer is found in relationship to the lowest offeror.

Formula Used: Lowest Offer/Offer at hand= Percentage Relationship to Lowest Offer

24

EXAMPLE Cost Offer Points

Cost Proposal 1 $780k 10 points Lowest Offer

Cost Proposal 2 $1.3 million 6 points 780K/1.3 million= .6

Cost Proposal 3 1.8 million 4.3 points 780k/ 1.8 million= .43

25

Let’s Recap all Scoring Elements…

Percentage Weights Criteria

10 Corporate Experience

15 Project Organization and Staffing

15 Implementation Approach

20 Operation/ Management Approach

10 Corporate and General Capacity

10 Cost/ Financial Offer

15/0 OCC (Minority Participation)

5 Financial Capacity

100 TOTAL

26

How Does Consensus Scoring Work?

Consensus Scoring consists of the Entire Evaluation Committee participating in Joint Discussion, Assessment, and Scoring of Technical and Cost Criteria in order to develop one unified score for each criteria.

27

Team discussion leads to greater sharing of each other’s Expertise in cross- functional projects, which are quite common in DOA RFPs

Generates clarification where misunderstandings may Exist

Ensures that all evaluators are truly engaged in the Evaluation process

Let’s Review the Benefits of Consensus Evaluation and Scoring…

Let’s Review the Benefits of Consensus Evaluation and Scoring…

Encourages deeper probing, questioning, discussion, and analysis of proposals to the evaluation criteria

A unified understanding of responses to all proposals is reached

Eliminates discrepancies among scores

Overall, the result is a more comprehensive and accurate assessment of the proposals

28

29

How Should Individual Evaluators Prepare Prior to the Consensus Evaluation Session?

Review thoroughly and comprehensively all proposal responses, repeating review of Responses where necessary.

Keep in mind how the information relates specifically to the stated criteria in the RFP.

Identify all points of information in the responses that need clarification

Identify the pros and benefits of each proposal response.

Identify the drawbacks, cons, and concerns of each proposal Response.

Evaluators should not assign scores during independent review of proposal.

30

What are the Steps to the Consensus Scoring Evaluation Session?

The Evaluation team continues until all desired elements of discussion are exhausted by all evaluators.

Each Member indicates points of information where they need clarification and ask questions to fill such voids, when possible, with other evaluation team members.

Each Member identifies the Pros and Benefits of each proposal.

Each Member identifies the Cons, Costs, and Concerns of each proposal.

Each Member communicates their assessment of the Overall Quality of each Proposal Response based upon each criteria.

Only after all Proposals are discussed, then each proposal response is scored by the team based upon all criteria.

The Facilitator records the Final scores for Documentation and Reporting.

31

What is the Role of the Facilitator in the Consensus Evaluation Session?

Provide Structure and Ensure that all Evaluators Participate andContribute to the Process in a Balanced and Generous manner

Monitor and Ensure that an Objective Stance is upheld throughout the entire Evaluation Process.

Keep the Team Focused and Ensure that Each Criteria is Discussed by the Entire Team

Remind Evaluators to Focus Strictly on the Evaluation Criteria

Ensure that all Needed Clarification is Obtained

Calculate Final Scores and Rankings

Important Documentation Steps for Consensus RFP Evaluations…

Technical Evaluation matrix (one score for each company)

Cost Evaluation Scores (one score for each company)

Total Final Score Matrix

Evaluation Session Notes by Facilitator

IMPORTANT: The Facilitator/ Procurement Staff takes notes during the Collaborative Evaluation Session.

32

33

Best Practices in EvaluationsMake every effort to differentiate the quality among proposals reviewed.

RFP Evaluations for each contract are expected to be complete within 10 days.

Keep in mind that you are making a decision that has an impact on significant dollars, often millions- be dedicated and focused.

Cost proposals should be evaluated strictly based on a formula.

Technical proposals should be evaluated strictly by the already developed criteria.

Benchmark costs to ensure reasonableness of proposed costs.

Thoroughly read the evaluation criteria.

Attend every scheduled evaluation team meeting and contribute to the discussion.

34

Now the Recommendation has to be approved…

The Project Manager and respective AGM writes a supporting letter for the recommendation by the Evaluation Committee.

The General Manager sponsors the recommendation and sends to the Chief Procurement Officer.

City Council and mayor must approve the recommendation through legislation in order for the City to enter into the contract with the proponent.

The Contract must be signed by the GM, Law, the CPO, the Mayor and the Clerk.

35

Questions & Answers

Aviation Contracts AreaLepora Manigault, Procurement Officer – 404.530.2421

Justina Mann, Contracts Supervisor – 404.530.5408Diane Akinsipe, Contracting Officer – 404.209.2922Erin Jenkins, Contracting Officer – 404.209.3171