Evaluation Training Guide

HARTSFIELD-JACKSON ATLANTA INTERNATIONAL AIRPORT - CITY OF ATLANTA

Aviation Procurement Unit

RFP Evaluation Training Manual

A V I A T I O N P R O C U R E M E N T U N I T

RFP Evaluation Training Manual

© 2009 City of Atlanta. Aviation Procurement Unit 6000 North Terminal Parkway • Atrium Suite 4000

Atlanta, GA 30320 Phone 404.530.5408 • Fax 404.905.1589

Table of ContentIntroduction................................................................................ 1 Definition ................................................................................... 1 The RFP as a Solution .............................................................. 2

When to Use an RFP Approach?.................................................... 2 What is the Best Way to Handle Criteria for RFP Evaluations? ...... 3 How Do We Develop an Evaluation Strategy?................................ 3 What are Examples of Standard Criteria in a Technical Proposal?. 4

Overview of the Evaluation Process.......................................... 6 Standards of Conduct While Serving as an Evaluator..................... 6 What is Expected from Evaluation Committee Members? .............. 7 Oral Interviews ................................................................................ 7 Reference Checks .......................................................................... 8 Ensure Security and Confidentiality ................................................ 9

Approach to Evaluating Proposals .......................................... 10 Evaluation Scoring Guidelines ...................................................... 10 When is a Proposal Non-Responsive?.......................................... 11 Approach to Evaluating Technical Proposals................................ 11 Approach to Evaluating Cost Proposals........................................ 13

Consensus Evaluation and Scoring ........................................ 15 Benefits of Consensus Evaluation and Scoring ............................ 15 Preparing for the Consensus Evaluation Session ......................... 15 Steps to the Consensus Scoring Evaluation Session ................... 16 Role of the Facilitator .................................................................... 16 Documentation Steps for Consensus RFP Evaluations ................ 17

Helpful Hints and Best Practices ............................................. 18 Next Steps: The Recommendation ......................................... 18 Frequently Asked Questions ................................................... 19

T H E R F P A S A S O L U T I O N

1

Introduction A Best Value Award is one which optimizes quality, performance, cost, time, and efficiency in a fair manner to the contractor community within the contract.

ids and quotes cannot always address the needs of the Department of Aviation (DOA). Generic specifications may not be available or difficult or impossible to draft and conventional evaluation for award based on lowest cost bid may not get the product or service required. Many high tech

products and complex services cannot be obtained by conventional bidding. The Request for Proposal (RFP) is a solicitation used for situations like these.

Definition The RFP is a formal competitive sealed bid process. The RFP outlines the requirements of the Business Unit by describing the

purpose, scope, description, minimum requirements or expectations, qualifications or capability of the proposers, evaluation criteria, and other requirements. In the RFP response, the vendor offers a solution for the particular need described in the RFP. The RFP is evaluated according to predetermined weighted standards.

After completion of the RFP process negotiations may, in some circumstances, be utilized to secure more advantageous terms or reduced cost.

Volume

1

B

R F P E V A L U A T I O N T R A I N I N G M A N U A L

The RFP as a Solution When to Use an RFP Approach?

A successful Request for Proposals (RFP) requires much planning. No two RFP’s are alike and preparing one can be a difficult task. However, a well-written RFP can alleviate many problems. Business Units are encouraged to involve the Aviation Procurement Unit (APU) early in the development of the RFP. APU can provide

assistance and valuable advice.

The following are typical instances when an RFP is the best approach;

When the use of competitive sealed bidding is either not practible or not advantageous to the DOA.

When we are not sure of the best solution and multiple options exist.

When performance-based incentive contracts are used.

When complex factors in addition to price are very important.

When there are several complex requirements.

When it is a new program/ project that has not been tested.

When it is desirable to find out if there are alternative options to what is normally performed.

A RFP is one of several procurement tools. APU will work with the Business Unit Project Manager to determine whether it is the best tool for the goods or services that are needed. The Chief Procurement Officer (CPO), upon receiving recommendation from APU and the Contracting Officer (CO), will ultimately determine if the RFP process is appropriate.

NOTE:

The bid process is the default solicitation option.


3

What is the Best Way to Handle Criteria for RFP Evaluations? It is important to identify all evaluation criteria and their relative importance, including price, early in the RFP development stages. These criteria will be the only way to properly evaluate the proposals and assure that the awarded proposals meets all the requirements of the solicitation. Criteria not specified in the RFP cannot be used for evaluating the proposals.

Begin by making a detailed list of the most important aspects of the services or goods required, including cost. Each item on your list is a potential evaluation criterion. Arrange the list in the sequence of what is most important.

Next, assign a point factor to each criterion based on its relative importance. The most important items should be evaluated heavier and have more points available. Points assigned to each criterion are included in the RFP. Including points makes offerors aware of which items are relatively more important than others can influence an offeror in the preparation of their RFP response.

All Evaluation Criteria MUST be established prior to the issuance of the RFP.

Criteria should be clearly linked to desirable outcomes.

Be Clear and Concise in language to avoid confusion.

Evaluation Information MUST be documented, dated, and secured throughout the evaluation process.

How Do We Develop an Evaluation Strategy? The following should be considered in developing your evaluation strategy.

1. Identify (eliminating) Minimum Qualifications

These are requirements that a vendor must meet in order to accomplish the work outlined in the RFP. They may include such things as proper licensing or accreditation and special insurance or bonding. They are evaluated on a strictly pass-or-fail basis. Make a list of the things that will be absolutely required for a successful offeror to have in order to enter into a contract. Do not include “desired” things or items that could be obtained by an offeror at a later time prior to contract award. Generally, if an offeror fails on any portion of the mandatory requirements, their proposal will be rejected, so put careful thought to this step.

2. Define Criteria

Develop Statement of Intent or Purpose and Scope of Work - The Statement of Intent or Purpose is a description of the general type of


4

service or goods required. The Scope of Work is a general summary of the work to be performed by a contractor. Developing these brief descriptions first will assist you to begin organizing your thoughts and help you decide on proper evaluation factors.

Decide importance of Oral Interviews/ Presentations - To properly evaluate proposals, oral presentations may be scheduled to answer questions for evaluation committee members. After consultation with the CO, only those firms that are potentially acceptable are invited to participate in oral presentations. The offeror’s original proposal cannot be changed in any aspect at the oral presentation. The oral presentation is only to allow offerors to clarify portions of their proposal.

Incorporate Reference Checks into Industry Experience - The purpose of contacting references is to verify the corporate capabilities and prior performance of the proponent and the qualifications of proposed project personnel. Reference checks are made by telephone. The APU Contracting Officer designated to assist the Evaluation Team will contact the references identified by the proponent. The results of the reference checks are compiled and provided to each evaluator to assist in scoring specific criteria. All reference questions must be relevant to the scope. The same questions must be asked to all references for all proponents and should be structured to assist the evaluators as they review information from each reference and to support follow-up questioning on selected reference responses.

Decide importance of Site Visits –The purpose of the site visit is to acquaint the vendors with the conditions under which the work must be performed.

3. How important is Cost?: Cost to Technical Score Ratio;

4. Determine distribution of points among each criterion.

What are Examples of Standard Criteria in a Technical Proposal? The following are examples of standard criteria in a Technical Proposal.

I. Corporate Experience:

What are the proponents’ past performance and experience in similar types of contracts?

What is the experience level of the essential subcontractors?


5

II. Project Organization and Staffing:

What is the staffing approach?

What are the qualifications and past performance of the personnel?

III. Implementation Approach:

What is the adequacy of their work breakdown structure/ project management?

Is it Comprehensive? Reasonable?

Proposed Deliverables?

Preparation Plan?

IV. Operation/ Management Approach:

Plan for Regular/ Daily Operations?

Approach to Quality Control and Assurance?

What are the Commitments to Performance Standards?

What are the Plans for Maintenance?

What are the Contingency Procedures in the event of an emergency?

V. Corporate and General Capacity:

What is the overall capacity of the organization?

Do they have a plan to sufficiently staff the contract project?

How long has the company been operating?

What is the turnover ratio?


Overview of the Evaluation Process Evaluation of the RFP responses is best accomplished in a team effort. The evaluation committee is comprised of the evaluation team, the APU Contracting Officer and the Department of Procurement (DOP) Contracting Officer as the Facilitator.

NOTE:

APU facilitates and coordinates processes between the Department of Procurement and evaluators.

Rules and regulations that are applicable to the evaluation process must be followed throughout the entire process. The following is a brief overview of the evaluation process:

Evaluation committee members independently review proposals based upon the specific evaluation criteria.

Technical and cost scores are assigned applying the designated evaluation criteria.

Oral interviews are conducted, if necessary.

Negotiations are conducted with the winning proponent.

Standards of Conduct While Serving as an Evaluator The evaluation team should be made up of individuals with varied talents and expertise to assure impartiality. Team members need to be aware of the possibility of an extended time commitment before agreeing to be a participant.

The following standards of conduct must be adhered to while serving as an evaluator:

No Conflicts of Interest

No solicitation or acceptance of gifts or anything of value

Integrity and Honesty during scoring and communications

Proper handling and discretion of all information; No Disclosure during process

Practice Ethical Behavior


7

What is Expected from Evaluation Committee Members? The evaluation committee members play a significant role in the decision to select a contractor, which results in the award of a contract. Accordingly, it is imperative that everyone understands what makes a good evaluation committee member. The following are standards established by DOP and APU;

1. The member has knowledge or experience regarding the RFP subject matter.

2. The member has sufficient time to dedicate in order to effectively evaluate the proposals.

Such time is dedicated to reading all proposal related materials extensively.

Mandatory attendance to all evaluation-related meetings: trainings, evaluation sessions, oral interviews, debriefings, criteria planning.

3. Conduct an independent review.

Identify potential deficiencies in a proposal that could result in a determination of non-responsiveness or non-responsibility.

Limit communication regarding the RFP project to those DOA and DOP staff who are directly assigned to the project.

4. Refrain from any communications with potential vendors/ contractors.

5. Determine whether or not oral interviews will be necessary.

Oral interviews should follow group discussions

Oral interviews should ALWAYS precede scoring.

Oral Interviews In complicated services, it may be beneficial to require offerors to make an oral presentation. Discussions may be conducted with responsible and responsive offeror’s who submit proposals determined to be reasonably susceptible of being selected for award.

The purposes of Oral Interviews are as follows:

Allows proponents to clarify on areas that may be unclear in their proposals.

Possibly reduces the likelihood of misunderstandings.


8

The evaluation committee members determine whether or not oral interviews will be necessary. Oral interviews should follow group discussion, and should ALWAYS precede scoring.

Reference Checks Reference Checks will be conducted with prior customers of the proponent. Reference Checks may also include references of key personnel. A standard form is used to collect responses. However, additional subject matter questions may be included when requested. When additional reference questions are to be included as part of the

evaluation, the following information is required:

List of questions for references.

Instructions to be given to the reference including a numerical scale to be used in rating the offeror.

References should be contacted only once. The APU Contracting Officer will conduct the reference check, which may include a conference call with all members of the evaluation team present when there are subject matter specific questions. A Reference Analysis will be provided to the Evaluation Team prior to the collaborative scoring and evaluation session.

The Reference Verification Form includes factors/ratings for technical performance, management performance, and customer satisfaction. The references will be asked to rate their level of satisfaction from 1-5.

Examples of Questions include:

Were the major tasks/milestones/ deliverables on schedule?

Were they able to identify and solve problems expeditiously?

What was the effectiveness and reliability of key personnel?

Were they able to recruit and maintain qualified personnel?

Were they able to effectively manage essential subcontractors?

What was the overall performance in planning, scheduling and monitoring?

How would you rate their overall technical performance?

How would you rate their overall management performance?


9

How would you rate their ability to be cooperative, business-like and concerned with the interests of the customer?

Would you do business with this vendor/ contractor again?

How would you rate your level of satisfaction from 1-5?

Ensure Security and Confidentiality Throughout all phases of the evaluation, the confidentiality and security of proposals and the scoring process must be maintained. To ensure confidentiality and security, evaluation sessions will be closed to the public and staff who are not supporting the Evaluation Team.

The evaluators shall not discuss the contents of proposals or the procurement activities with any persons outside of the evaluation team or Procurement staff. All evaluators and all other staff involved in the evaluation effort must strictly adhere to the following requirements.

Evaluators must keep proposals in a secure place.

Evaluators are not permitted to discuss the procurement or evaluation process with any persons outside of the evaluation team or procurement staff.

Evaluators shall not communicate the scoring outcomes or content of proposals and shall not disclose the status of any proposal.

Cost components shall be evaluated upon the completion of the scoring of the technical components.


10

Approach to Evaluating Proposals All proposals shall be evaluated in accordance with the City of Atlanta’s Procurement Code and the criteria specified in the RFP on the Percentage Evaluation Form and considering the information required to be submitted in each Proposal.

The evaluators will independently read and assess compliance with the criteria identified in the RFP to assess the completeness, quality, and desirability of proponent responses. All members of the Evaluation Team will document their findings for evaluation criteria within each of these areas.

Each criterion is independently reviewed by evaluators, documenting the pros and cons of each. Review reflects individual, independent evaluations of a proposal and response to criteria. Criteria include questions or items for consideration for evaluators to use as a guide in discussion during the collaborative scoring and evaluation session.

The Evaluation Team should not score the criteria during their independent evaluation.

Evaluation Scoring Guidelines Each technical proposal will be evaluated against a set of pre-determined criteria to assess the degree to which it meets that criterion. Compliance with requirements will be assessed as a point score on a scale from 1 to 10. Technical criteria will be scored as shown below:

8-10: Outstanding: Proponent’s proposal exceeds expectation and demonstrates an excellent ability to reach the goals and objectives of the procurement.

5-7: Highly Effective: Proponent’s proposal demonstrates a good ability to reach the goals and objectives of the procurement.

3-4: Effective: Proponent’s proposal demonstrates a fair ability to reach the goals and objectives of the procurement.

1-2: Marginally Effective: Proponent’s proposal demonstrates a minimal ability to reach the goals and objectives of the procurement.

Scores will be adjusted to reflect the weight assigned to each criterion. Each criterion must be reviewed according to the requested items of the RFP and scored based on the scoring definitions above. Only the Criteria in the Percentage Evaluation Form will receive scores.

All notes taken regarding any proposal are subject to the Open Records Act.


11

Note

Any Proponents’ Proposal that does not meet the minimum requirements of the RFP should be identified as this may deem them non-responsive.

When is a Proposal Non-Responsive? Only the Department of Procurement has the authority to determine whether a proposal is non-responsive or non-responsible. A Proposal will be determined to be non-responsive when the proposer is not responsive to the mandatory requirements clearly established within the RFP.

If an evaluator at any time has a concern with the proponents’ proposal meeting the stated requirements, they should immediately notify the Facilitator and Procurement staff.

Approach to Evaluating Technical Proposals The purpose of this phase is to measure the individual merits of the technical components of the proposal against pre-established criteria. Members of the Evaluation Team will review all Technical Proposals that pass the Mandatory Technical Requirements.

Evaluators are instructed to read through the proposal Executive Summary (if present) before beginning to evaluate and score detailed criteria. The material in the proposal introduction should provide all evaluators with a broad understanding of the entire proposal. In assessing individual responses within a section, evaluators may elect to review related topics within other sections of the proposal. Evaluators are restricted to evaluating information contained within the proposal. Information not part of the proposal may not be considered.

Evaluators will proceed according to the following steps to evaluate responses to each criterion:

1. Review the appropriate section of the RFP.

2. Locate the section(s) of the proposal where the criterion is addressed.

3. For each criterion, note the RFP sections referenced in this document.

4. Review and evaluate section(s) of the proposal.

Evaluators should promptly notify the CO if there is a problem with the Proposal.


12

5. For each criterion, evaluate and assess the criterion based on the proponent’s overall response to the requirements indicated for the criterion (see step 8). Some of the criteria have additional questions that may be considered in the evaluation.

6. Evaluate how well the proponent’s responses in the referenced sections correlate with other pertinent sections of the RFP and the overall approach taken to address the technical components of the proposal.

7. Evaluate the criterion based on all information available that pertains to it directly or indirectly, including reference checks.

8. Develop a list of pros and cons for the criterion based on the evaluation of the proponent’s capability to meet that criterion.

Evaluators are encouraged to request technical support from the CO or from the Department of Procurement in preparation for criteria documentation. Training will be provided at the inception of each Evaluation Committee, and prior to proposal distribution.

What does Technical Scoring Look Like?

Weight Criteria Company 1 Company 2 Company 3

20% Organization/Resumes of Key Personnel

10 3 1

30% Overall Experience, Qualifications and Performance of Previous Similar Projects

9 7 4

20% Strategic Approach To Increasing Federal Assistance to the City through Government Agencies and Congress

8 6 3


13

Approach to Evaluating Cost Proposals After technical aspects of proposal responses are initially reviewed, then cost proposals are reviewed.

Cost proposals may reflect price escalation, deceleration due to performance, time, CPI, etc. may be applied.

Cost Evaluation must be independent of the Technical Evaluation.

A Clear Formula must be used to Calculate Cost Scores.

Scores should be directly linked solely to costs proposed on a formulaic basis.

The cost proposal/ financial offer must be taken at face value when the cost proposal is evaluated.

Note

Scores are documented only in the Evaluation and Collaborative Scoring Session by the Facilitator/Procurement Staff.

Traditionally, the Lowest Fee Offer receives the maximum points possible (10). In the event that we are soliciting a revenue generating proposal, the highest offer would receive the maximum points. Points are distributed based upon percentile in which the fee offer is found in relationship to the lowest offeror.

Example Cost Offer Points

Cost Proposal 1 $780k 10 Lowest Offer

Cost Proposal 2 $1.3 mil 6 780k/1.3mil = .6

Cost Proposal 3 $1.8 mil 4.3 780k/1.8 mil = .43

Formula Used: Lowest Cost Proposal Offer/ Cost Proposal Offer at Hand = Percentage Relationship to Lowest Offer


14

What Are Some Examples of Cost Proposals?

The following are examples of what the cost proposals may read like for the types of RFP’s and resulting contracts:

Type of Contract Suggested wording… Scenario Example

Performance Based Please propose Base Costs and Incentives associated with performance beyond mandatory requirements…

Beyond 90% Customer Satisfaction Scores consecutively for one year will lead to a 3% bonus (e.g.., janitorial)

Cost Savings Please propose Base Costs and Incentives associated with cost savings beyond mandatory requirements…

Company X will receive 10% of all cost savings beyond performance guarantee (e.g.., workers’ compensation)


15

Consensus Evaluation and Scoring Consensus Scoring consists of the Entire Evaluation Committee participating in Joint Discussion, Assessment, and Scoring of Technical and Cost Criteria in order to develop one unified score for each criterion.

The evaluators are allotted five (5) days to independently review the proponents’ proposals. Next, the evaluation committee will meet to evaluate the proposals, discussing the pros and cons of each proposal as well as clarifying their assessment.

Benefits of Consensus Evaluation and Scoring While the benefits of consensus evaluation and scoring are vast, the following portrays the immediate results;

Team Discussion leads to greater Sharing of Each Other’s Expertise in Cross- Functional Projects, which are quite common in DOA RFPs

Generates Clarification where Misunderstandings May Exist

Ensures that All evaluators are truly Engaged in the Evaluation process

Encourages deeper Probing, Questioning, Discussion, and Analysis of the content of Proposals in response to the Evaluation Criteria

A Unified Understanding of Responses to All Proposals is reached

Eliminates discrepancies among scores

Overall, the result is a more Comprehensive and Accurate Assessment of the proposals

Preparing for the Consensus Evaluation Session In preparing for the consensus evaluation session, evaluation team members should review thoroughly and comprehensively all proposal responses, repeating review of responses where necessary.

Keep in Mind How the Information Relates Specifically to the stated Criteria in the RFP.

Identify All Points of Information in the Responses that Need Clarification

Identify the Pros and Benefits of Each Proposal Response.


16

Identify the Drawbacks, Cons, and Concerns of Each Proposal Response.

IMPORTANT: Evaluators should not assign scores during independent review of proposal.

Steps to the Consensus Scoring Evaluation Session The Evaluation team continues until all desired elements of discussion are

exhausted by all evaluators.

Each Member indicates points of information where they need clarification and ask questions to fill such voids, when possible, with other evaluation team members.

Each Member identifies the Pros and Benefits of each proposal.

Each Member identifies the Cons, Costs, and Concerns of each proposal.

Each Member communicates their assessment of the Overall Quality of each Proposal Response based upon each criterion.

Only after all Proposals are discussed, then each proposal response is scored by the team based upon all criteria.

The Facilitator records the scores for Documentation and Reporting.

Role of the Facilitator The DOP CO serves as the Facilitator for the Evaluation Committee. The Facilitator provides structure and ensures that all evaluators participate and contribute to the process in a balanced and generous manner. The Facilitator will lead the consensus scoring and evaluation session with the Evaluation Team and remind evaluators to focus strictly on the evaluation criteria. The Facilitator will be responsible for ensuring that the evaluation plan is followed and that scoring decisions are sound and defensible.

The Facilitator will monitor and ensure that an objective stance is upheld throughout the entire evaluation process. Facilitators will work closely with the evaluation team to ensure that all needed clarification is obtained regarding the proposals under review, and will resolve any compliance issues, and performs the final ranking of the proposals.


17

Documentation Steps for Consensus RFP Evaluations Individuals selected as evaluators are responsible for the execution of the technical components evaluation as defined in this manual. Specifically, evaluators will apply the pre-established procedures and criteria to determine if each proponent’s technical components are responsive and

rate each of the evaluation categories. An RFP Evaluation Training is provided at the onset of each project by the Department of Procurement for evaluators, and again prior to the proposals being released to the evaluators.

Templates are provided for each of the evaluators to provide the following documentation for consensus RFP evaluations.

Technical Evaluation matrix (one score for each company)

Cost Evaluation Scores (one score for each company)

Total Final Score Matrix

Evaluation Session Notes by Facilitator

IMPORTANT: The facilitator/procurement staff will take notes during Collaborative Evaluation Session


18

Helpful Hints and Best Practices 1. Make every effort to differentiate the quality among

proposals reviewed.

2. RFP Evaluations for each contract are expected to be complete within 10 days.

3. Keep in mind that you are making a decision that has an impact on significant dollars, often millions- be dedicated and focused.

4. Cost proposals should be strictly evaluated based on a formula.

5. Technical proposals should be evaluated strictly by the already developed criteria.

6. Benchmark costs to ensure reasonableness of proposed costs.

7. Thoroughly read the evaluation criteria.

8. Attend every scheduled evaluation team meeting and contribute to the discussion. This is mandatory.

Next Steps: The Recommendation Once the evaluation process is complete and the most responsive and responsible proponent(s) has been identified, the following next steps can be anticipated to get to contract award:

1. The Project Manager and respective AGM write a supporting letter for the recommendation by the Evaluation Committee.

2. The General Manager sponsors the recommendation and sends to the Chief Procurement Officer.

3. City Council and Mayor must approve the recommendation in order for the City to enter into the contract with the proponent.

4. The Contract must be signed by the GM, Law, the CPO, the Mayor and the Clerk.


19

Frequently Asked Questions The following are Frequently Asked Questions that may be useful as you are involved in the RFP Evaluation process:

1. Can Evaluators bring their notes to the Consensus Meeting?

Yes. Evaluators may bring their notes to facilitate discussion during the Consensus Evaluation and Scoring Session.

2. If the evaluation team cannot reach a consensus, if averaging allowed?

No. Averaging is not done as a part of the collaborative scoring process. Evaluators must agree to a score based on discussions regarding the pros and cons of each proposal in order to reach a consensus score.

3. Are we allowed to take notes during our independent review of the proposals?

Yes.

4. Can you discuss the RFP with subject matter experts not on the evaluation panel?

No. Evaluators are prohibited from discussing the contents of RFP or any procurement activities with anyone outside of the evaluation committee.

5. Can you take the proposals home to review?

Yes. However, evaluators must ensure that the proposals remain secure as the content is confidential.

6. How many should comprise the evaluation team?

The evaluation team should have between three and five members.

7. When are reference checks conducted and by whom?

The APU Contracting Officer will conduct the reference check after the proposals are received. The results will be distributed to the evaluation team prior to the collaborative scoring session.

8. When are scores for OCC and Financial criteria provided?

The scores for OCC and financial criteria are provided by the DOP Contracting Officer after the conclusion of the collaborative scoring.


20

Evaluation Training Guide

Documents