2020 A&E Oversight PERFORMANCE MEASURES #20-01

2020 A&E OversightPERFORMANCE MEASURES #20-01 March 2021

Prepared By: Original Signature By

The’ Pham, Acting Program Manager Date A&E Oversight

Approved By: Original Signature By

Kamal Sah, Office Chief, Office of Date Guidance and Oversight

2020 A&E Oversight Performance Measures #20-01

Table of Contents 03 EXECUTIVE SUMMARY

06 BACKGROUND

08 PURPOSE AND OBJECTIVE

09 SCOPE AND METHODOLOGY

10 DATA ANALYSIS

14 OBSERVATIONS AND RECOMMENDATIONS

18 SUCCESSFUL PRACTICES

19 ADDRESSING 2019 ACTION PLAN

20 ACTION PLAN

21 APPENDIX A

22 APPENDIX B

Upward Direction

49%

'' ''

3 2020 A&E Oversight Performance Measures #20-01

EXECUTIVE SUMMARY

TheArchitecturalandEngineering(A&E) Oversight branch within the Office of Guidance & Oversight (OGO) performed an annual process review of A&E consultant contracts procured by local government agencies (LGAs). The review resulted in overall compliance of 86%. The review has 22 performance measures identified to assess LGAs’ compliance with 23 Code of Federal Regulations (CFR) part (§) 172, 2 CFR §200, and 48 CFR §31. The A&E Oversight branch upholds the federal requirements to ensure that LGAs when procuring A&E services promote full and open competition, the selection is qualifications-based, and that contract cost is fair and reasonable.

The A&E Oversight branch has the delegated authority to provide guidance on consultant contracts for LGA. The A&E Oversight’s strategic plan embraces a mission to empower and ensure success to the LGA. The team's vision is to achieve excellence through progress and innovation.

Since 2014, five annual process reviews have been conducted. Each following year has shown an upward trend of overall compliance increasing from 49% to 86%. Figure 1. LGA's Year-Over-Year Overall Compliance chart is presented below.

Figure 1. LGA's Year-Over-Year Overall Compliance

At a glance

86% Confirmed

385 LGAs Overall A&EContracts Compliance & Amendments

22 4 Performance Most Deficient

Measures AreasIdentified Action Plan

In 2018, the A&E Oversight branch established a target average overall compliance goal of 85%. This year’s review demonstrated an overall compliance rate of 86%. In conjunction with last year’s data verification and analysis, this year’s review confirms and demonstrates LGAs are consistently procuring consultant contracts at the targeted goal.

The objectives of the review were as follows:

• Determine ifLGAscomplywith 23CFR §172 – Procurement, Management, and Administration of EngineeringandDesignRelatedServices for contracts executed in FFY 2019-2020.

• Identify deficient areas in the LGAs’ procurement processes, recommend changes to Local Assistance Procedures Manual (LAPM) Chapter 10: Consultant Selection guidance, and create an action plan for increasing compliance.

Embraces a mission to empower and ensure success to the local partners

96%

100%

1¥1

100%

70%

91%

96%

100%

96%

100%


EXECUTIVE SUMMARY

The 22 performance measures are categorized under the five guiding principles: Procurement Planning, Full and Open Competition, Qualifications Based Selection, Cost is Fair and Reasonable, and Contract Administration. Performance measures that are ≥ 85% have demonstrated compliance, between 70% and 85% are categorized to be monitored, and below 70% are considered most deficient. There are 15performance measures that are at or above the A&E Oversight branch’s target compliance goal of 85%. There are 7 performance measures that are below the target goal. The four most deficient areas are highlighted red in Table 1: 2020 Performance Measures Compliance. These areas are specifically targeted for improvement in 2021. Table 6: Heat Map Compliance Rate - Annually provides a visual representation of the performance levels for each area.

The current process for reviewing A&E contracts and collecting data through the use of the Exhibit 10-C form continues to have a positive impact on the LGA's awareness of the federal requirements. In 2020, the Exhibit 10-C process was shifted to a web-based database that allowed for a much better data management practice and increased efficiency. Focused training and providing clear guidance continue to be the main goals for next year.

Resolutions for this report are to provide more guidance documents such as tools and sample forms. Also, clarify guidance in the LAPM Chapter 10: Consultant Selection, and provide statewide training focusing on the deficient measured areas.

The future goal of the A&E Oversight branch remains the same, which is to meet the 85% target goal. Recommendations are included in this report for reaching the targeted future goals for all performance measures.

Table 1. 2020 Performance Measures Compliance

Procurement Planning 69 Cost is Fair & Reasonable 78 A&E Policies & Procedures Method of Payment Conflict of Interest Price/Fee Negotiations CMSR Exhibit 10-H Cost Proposal Independent Cost Estimate Follow Audit Process

Full & Open Competition 87 Cost Analysis

Public Solicitation/Advertisement Contract Administration Records of Procurement/Submittal Exhibit 10-C A&E Checklist

DBE Contract Approval

Qualifications Based Selection 98 Contract Expiration Date

Score Sheets Signed & Dated Exhibit 10-R Provisions Secondary Score Sheets Retained Selection Using RFP/RFQ Criteria Original Score Sheets Retained Task Orders Compliance

98

---

05 Executive Summary

Heat Map of Performance Measures Table 6. Heat Map Compliance Rate - Annually

Legend PERFORMANCE

MEASURES 2014 2017 2018 2019 2020

A&E Policies & Procedures

Conflict of Interest

CMSR

Independent Cost Estimate

Public Solicitation/ Advertisement Records of Procurement & Submittals

DBE

Score Sheets Signed & Dated

Secondary Score Sheets Retained

Selection Criteria & Weights

Selection Made Using Criteria in RFP/RFQ Original Score Sheets Retained

Task Orders Compliance

Method of Payment

Price/Fee Negotiations

Exhibit 10-H Cost Proposal

Follow Audit Process

Cost Analysis

Exhibit 10-C A&E Checklist

Contract Approval/ Authorization Action

Contract Expiration Date

Exhibit 10-R Contract Provisions

Rate >=85%

70% < Rate < 85%

Rate =< 70%

2020 A&E Oversight Performance Measures #20-01 06

•

• •

• •

•

• • • •

BACKGROUND

23 CFR §172.1 requires State Transportation Agencies (STA)suchasCaltrans toensure thatsub-recipients comply with the requirements of 23 CFR §172 in addition to the 2 CFR §200 Uniform Administrative Requirements, Cost Principles and Audit Requirements for Federal Awards rule. The master agreement between Caltrans and LGAs requires all agencies to follow the LAPM Chapter 10 for consultant contracts.

Federal Highway Administration (FHWA) considered A&E consultant contracts as a high-risk area in 2014. A&E consultant contracts is still a high-risk area. The A&E Oversight branch was created to reduce this risk and has continuously improved the LAPM and training to increase compliancesince the first process review in 2014.

The process review performed in 2014 measured 15 areas of interest and set a baseline of compliance. The 2017 report measured the same 15 areas and included 3 more areas to establish a baseline for 18 areas for future comparisons. The 2018 process review added 4 additional areas for a total of 22 areas reviewed. In addition, the target goal of an 85% compliance rate was established. No other areas were added for the 2019 and 2020 reviews.

Table1:PerformanceMeasuresCompliance includes the full list of the 22 areas measured. The 4 areas that are being targeted for specific improvement in 2020 are highlighted in red. Table 2: Overall Compliance Yearly provides a comparison from 2014 to 2020.

LGAs are required to follow the LAPM Chapter 10: Consultant Selection when procuring contracts utilizing federal funds through the Division of Local Assistance (DLA). LAPM Chapter 10 is intended as a guide for LGAs and includes the State and Federal requirements listed in 23 CFR §172.

Since the 2017 report, various measures were taken to increase compliance such as:

Implemented a Consultant in a Management Support Role (CMSR) process Implemented the Exhibit 10-C review process Provided in-person training including a webinar for the Exhibit 10-C process Updated the LAPM Chapter 10 Provided focused training at nine Caltrans districts in 2019 Recorded a live A&E training webinar. Training is posted online at https://dot.ca.gov/programs/local-assistance/guidance-and-oversight/consultant-selection-procurement

Deficiencies below the target compliance rate of 85% were found in 7 of the 22 areas reviewed in 2020. The four most deficient areas are listed with their respective compliance rates:

IndependentCostEstimate –24% Cost Analysis –48% Price/Fee negotiations – 59% A&E Policies and Procedures – 57%

Most of these critical areas are directly related to the financial oversight of contracts. This creates a significant financial risk for LGAs.

The A&E Oversight branch will specifically focus on the four areas above. The areas that had compliance rates at or above 85% for five consecutive years will be considered for removal in the next review.

The data from this review will be used to report LGAs’ compliance, improve guidance documents, and create focused statewide training.

https://dot.ca.gov/programs/local-assistance/guidance-and-oversight/consultant-selection-procurement



•

•


PURPOSE AND OBJECTIVE

The purpose of this review is to assess compliance with Federal and State regulations for A&E consultant contracts procured by LGAs. The information and data will be used to improve guidance, training, and oversight provided to the Districts and LGAs.

The objectives of the review were as follows:

Determine if LGAs comply with 23 CFR §172 – Procurement,Management,andAdministrationof Engineering and Design Related Services for contracts executed in FFY 2019-2020.

Identify deficient areas in the LGAs’ procurement processes, recommend changes to Local Assistance Procedures Manual (LAPM) Chapter 10: Consultant Selection guidance, and create an action plan for increasing compliance.

size =

• • •

•


SCOPE AND METHODOLOGY

The A&E Oversight branch had collected Exhibit 10-C: Consultant Contract Reviewers Checklist for all A&E consultant contracts in FFY 2019-2020. A sampling design was established to represent the whole population of Exhibit 10-Cs (see Table 2: Overall ComplianceYearly). A representative sample for this report was designed to be random without bias from the population of interest, but still representative of the population as a whole. In FFY 2019-2020, there were a total of 385 Exhibit 10-C submittals, including project-specific and on-call contract type. From the 385 Exhibit 10-C submittals, there were 56 on-call contracts. A combination of stratified random sampling and systematicsamplingwas applied.The selection criteria of sample size included Federally funded projects, on-call contracts, and contracts with an amount equal to or greater than $250,000. The first contract from the list per LGA was selected and At least one on-call contract per LGA was chosen. Following these criteria, twenty-three (23) projects representing ten (10) districts and twenty-two (22) agencies were selected. LGAs from Districts 2 and 12 did not have on-call contracts issued in FFY 2019-2020.

Accordingly, the review was based on a random sample of 23 federally funded on-call contracts greater than $250,000 chosen throughout the State. The team reviewed supporting documentation for Exhibit 10-C for 22 Federal requirements chosen from 23 CFR §172 regulations.

Thesamplesize formulaandparametersusinga 95% confidence level is shown below:

Where: e is the desired margin of error at 15%. z is the z-score at 1.96 for a 95% confidence level p is the sample proportion of contracts in compliance. Assume p =0.5. N is the population size at 56 for on-call contracts received in FFY2019-2020.

The required sample size per the formula above is 24; however, as documentation was being requested, two contracts were found to be the exact same contract, although they were submitted with slightly different information. Therefore, due to this late finding, the sample size was reduced to 23.

Appendix A contains the list of selected projects and Appendix B contains the list of measured criteria.

A questionnaire checklist was created (Appendix B) that included relevant questions and the respective regulatory citation. Supporting documentation corresponding to the checklist questions were requested from the LGAs. The team then completed a review of the supporting documentation and analyzed the results.

:MEASURES

A&E Policies & Procedures

Conflict oflnterest

CMSR

independent Cost Estimate

Public Solicitation/ Advertisement Records of Procurement & Submittals

DBE

Score Sheets Signed & Dated Secondaty Score Sheets Retained Selection Criteria & Weights Selection Made Using Criteria Stated in RFP/RFQ Original Score Sheets Retained

Task Orders Complia1.1ce

Method of Payment

Price/Fee Negotiations

Exhibit 10-H Cost Proposal

Follow Audit Process

Cost Aualysis

Exhibit 10-C

Contract Approval

Contract Expiration Date

Exhibit 10-R Contract Provisions

20 14 201 7

38% 50%

13 % 89%

9% 64%

56% 92%

53%

39% 78%

40% 70%

56% 8 1%

68% 85%

72% 89%

76% 58%

13% 69%

48% 78%

80%

41% 92%

62% 92%

86% 92%

81 %

20 18 2019 2020 YEARLY TREND

60% 67% ••••• 95 % 83% -••·· 100% 83% 100% ••• 78% 43% -••·-95% 100% 100% ••••• 78% 86% 70% ■ Ill 90% 81% 91% Ill 95 % 95% 96% -·•·· 65% 100% 100% -···· 95 % 100% 100% ••••• 95% 95% 96% ••••• 90% 95% 100% ••••• 71% 67% 83% Ill 90% 95% 100% ••••• 38% 50% -••·· 90% 90% 100% -···· 78% 81% 83% 1111 24% 45% -·· 98% 100% 100% -···· 98% 95% 91 % ••••• 93% 95% 100% ••••• 90% 86% 100% ••••


Data Analysis

The data sets of performance measures for2014 and2017 through2020 are presented in Table 2: Overall Compliance Yearly. The four lowest performance areas are highlighted in red. The one performance measure highlighted in yellow is considered an anomaly and needs to be carefully monitored next year since it is the only trend in the monitored category that went down this year. Statistical tests were used to evaluate the performance measures of the LGAs’ A&E procurement. Tables 3(a) through 4(b) compares the 2020 statistical data to 2014 and to 2019.

Table 2: Overall Compliance Yearly


Data Analysis

Table3a:Performance Measures 2014 vs 2020 Table 3b: Statistical Test for Paired t-test, one tail

PERFORMANCE MEASURES

2014 2020 Trend

A&E Policies & Procedures 38% 57%

Conflict of Interest 13% 96%

Independent Cost Estimate 9% 24%

Public Solicitation/Advertisement

56% 100%

Score Sheets Signed & Dated 39% 96%

Secondary Score Sheets Retained

40% 100%

Selection Criteria & Weights

56% 100%

Selection Made Using Criteria Stated in RFP/RFQ

68% 100%

Original Score Sheets Retained

72% 100%

Method of Payment 76% 100%

Price/Fee Negotiations 13% 59%

Exhibit 10-H Cost Proposal 48% 100%

Exhibit 10-C A&E Checklist 41% 100%

Contract Approval/ Authorization Action

62% 91%

Contract Expiration Date 86% 100%

Average 48% 88%

STATISTICAL TEST

Paired Sample t-test Ho: µ2020 = µ2014 α=0.05

p-Value < α = 0.05 Reject Null Hypothesis; Perform Comparison

p-Value > α = 0.05 Fail to Reject Null Hypothesis

From the Excel spreadsheet, for a paired t-test with one tail, p=1.43E-06 < 0.05. As a result, the null hypothesis is rejected. A comparison can be made to compare the performance measure between the two data sets. When reviewing the data in Table 3a: Performance Measures 2020 vs. 2014, a distinct conclusion can be made that the performance levels measured in 2020 are at a much higher level of compliance than in 2014.


Data Analysis

Table4a:Performance Measures 2019 vs 2020 Table 4b: Statistical Test for Paired t-test, one tail


-

2019 2020 Delta



CMSR 83% 100%


Public Solicitation/ From the Excel spreadsheet, for a paired t-test with 100% 100%Advertisement one tail, p=0.221869 > 0.05. As a result, we fail to Records of Procurement &

86% 70% reject the null hypothesis. It can be concluded that Submittals the performance measures of 2019 and 2020 are not

statistically different. DBE 81% 91%

Score Sheets Signed & Dated 95% 96%

Secondary Score Sheets 100% 100%Retained

Selection Criteria & Weights 100% 100%

Selection Made Using Criteria 95% 96%Stated in RFP/RFQ

Original Score Sheets Retained 95% 100%

Task Orders Compliance 67% 83%

Method of Payment 95% 100%


Exhibit 10-H Cost Proposal 90% 100%

Follow Audit Process 81% 83%

Cost Analysis 45% 48%

Exhibit 10-C 100% 100%

Contract Approval 95% 91%

Contract Expiration Date 95% 100%

Exhibit 10-R Contract 86% 100%Provisions

Average 83% 86%

STATISTICAL TEST

Paired Sample t-test Ho: µ2020 = µ2019 α=0.05

p-Value < α = 0.05 Reject Null Hypothesis; Perform Comparison

p-Value > α = 0.05 Fail to Reject Null Hypothesis

.


Data Analysis The four most deficient areas in 2020 is compared to 2019 below.

• A&EPolicies&Procedures– A decrease in compliance from 67% to 57%. The A&E policies and procedures was prescribed in the LAPM in 2019 but the procedure was not clear on how to adopt LAPM Chapter 10. The compliance rate would increase once a procedure is formalized.

• Independent Cost Estimate – A decrease in compliance from 43% to 24%. Similar to 2019, the decrease in compliance can be attributed to several factors, namely, interpretation of regulations, experience of reviewer and feedback from regulatory agencies that could be clearer. An independent cost estimate spreadsheet tool was created in January 2021 to address the 2019 deficiency and would potentially increase the compliance rate.

• Price/Fee Negotiations – An increase in compliance from 48% to 59%. However, the achieved mark of 59% is below the target and will require further guidance and training. A price/fee determination spreadsheet tool was created in January 2021 to address the 2019 deficiency and would help LGAs to document negotiations. We are projecting an upward trend next year due to the new tool.

• CostAnalysis – This compliance rate did not change from last year. Similar to the independent cost estimate, Similar to 2019, the decrease in compliance can be attributed to several factors, namely, interpretation of regulations, experience of reviewer and feedback from regulatory agencies that could be clearer A cost analysis spreadsheet tool was created in January 2021 to address the 2019 deficiency and would potentially increase the compliance rate

.

Table 5: Monitoring Deficient Areas Identified in 2019 shows the compliance rate for the deficient areas in 2019 and 2020 and the direction of the compliance rate. The four deficient areas have been continuously out of compliance since 2014, when the first process review was performed. Although Records of Procurement & Submittals is not considered a high deficient area, it is considered an anomaly and needs to be carefully monitored next year since it is the only trend in the monitored category that went down this year. Refer to Table 2: Overall Compliance Yearly.

Table 5: Monitoring Deficient Areas Identified in 2019


-

2-Year 2019 2020 Trend



CMSR 83% 100%


Records of Procurement & Submittals 89% 70%

DBE 81% 91%

Task Orders Compliance 67% 83%


Followed Audit Process 81% 83%

Cost Analysis 48% 48%


OBSERVATIONS AND RECOMMENDATIONS Objective #1: Determine if local agencies were in compliance with 23 CFR§172– Procurement,Management,andAdministrationofEngineering andDesignRelated Services for contracts executed in FFY 2019-20.

Observation The LGAs have an overall compliance rate of 86% from the 22 areas measured. The average compliance rate in 2020 is slightly higher than 83% in 2019.

Condition The team reviewed 23 on-call contracts (Appendix A) out of 385 recorded Exhibit 10-C. The areas measured included 22 items considered significantly important for 2020 are listed in Appendix B.

Criteria The A&E Oversight Process Review Questionnaire contains Federal requirements for procuring A&E contracts found in 23 CFR§172. LGAs are required to adhere to all of 23 CFR 172, and 22 requirements were chosen for review.

Cause Thefederal regulationsaredetailedandcomprehensive.Although necessaryforensuringcompetitivequalifications-basedselection for A&E consultant contracts, LGAs are not currently knowledgeable of all the requirements.

Effect Caltrans DLA, FHWA, and the Independent Office of Audits and Investigations (IOAI) all consider A&E consultant contracts as high risk. IOAI conducts audits and consistently finds deficiencies in A&E contracts, resulting in funding loss.

Compliance Issue Several measured areas do not meet the target goal of 85% compliance rate. There are 13 areas at 95% or above, 2 areas between 85% and 94%, and 7 areas below 85%.

Recommendation Research and collaborate with FHWA, IOAI, stakeholders, and partners to clearly define and communicate expectations and regulations. Clearly

•


OBSERVATIONS AND RECOMMENDATIONS

define and communicate all requirements of the Federal regulations in the LAPM Chapter 10: Consultant Selection toserveasthestandardsfor expectationsandreviews.

Resolution Caltrans will continue improving the risk-based review procedures of Exhibit 10-C and continue conducting annual process reviews to monitor compliance.

Objective#2: Identifydeficientareas inLGAs’procurementprocesses and recommendchangestoLAPMChapter10:ConsultantSelection guidance and create an action plan for increasing compliance.

The full list of areas reviewed are in Appendix B. The 22 performance measures are categorized under the five guiding principles:

• Procurement Planning • Full and Open Competition • Qualifications-Based Selection • Cost is Fair and Reasonable

Based on the 2020 performance data, the two guiding principles Procurement Planning and Cost is Fair & Reasonable achieved rates of 69% and 78% respectively and were below the target compliance rate of 85%. Refer to Table 1: Performance Measures Compliance. These two guiding principles were below 85% target in 2019 as well.

Observation#2.1: Procurement and Planning is the lowest performingcategory at 69%.

Condition Only two performancemeasuresunder theProcurementand Planning guiding principle are under the target goal of 85%. These two are A&E Policies & Procedures and Independent Cost Estimate.

Criteria In 23 CFR §172.5 and §172.7 provides the following requirements for the contracting agency:

Adopt the written policies and procedures prescribed by the state agency.

•



Prepare a detailed independent cost estimate with an appropriate breakdown of the work or labor hours, types or classifications of labor required, other direct costs, and consultant's fixed fee for the defined scope of work. The independent estimate shall serve as the basis for negotiation.

Cause Similar to last year’s review findings, LGAs do not fully understand the requirements of an independent cost estimate and the requirement to adopt the LAPM Chapter 10 as the policy and procedures manual as required in the LAPM Chapter 10.

Effect Inadequate procurement planning will result in not meeting the 23 CFR §172 requirements. Two of the common procurement audit findings are the lack of A&E policies and procedures and an incompletecost estimate use for negotiations.

Compliance Issue A federal-aid A&E consultant contract not meeting the criteria section above is non-compliant with 23 CFR §172.

Recommendation Continue to simplify guidance and provide additional examples that meet the federal requirements. Create policy and procedure to adopt the LAPM Chapter 10 as well as updating the LAPM and providing more focused training on the deficient performance measures

Resolution An independent cost estimate spreadsheet tool was created in January 2021 to address the recommendation in the 2019 Performance Review and may potentially increase the compliance rate next year. Caltrans is in the process of creating policy and procedure to adopt the LAPM Chapter 10. Caltrans will update the LAPM and provide more focused training on the deficient performance measures.



Observation #2.2: Contract costs may be overpriced and not fair andreasonable.

Condition Similar to last year’s review findings, LGAs do not appear to perform negotiations and cost analysis

Criteria Federally funded contract is required to be fair and reasonable as specified in 23 CFR §172.

Cause LGAs do not fully understand the A&E procurement requirements and that contract cost is supported from negotiations and estimates.

Effect The lack of not performing or not documenting activities that relate to supporting the contract cost may result in a higher contract cost amount.

Compliance Issue A federal-aid A&E consultant contract not meeting the criteria section above is non-compliant with 23 CFR §172.

Recommendation Provide clear guidance and more focused training.

Resolution Two separate spreadsheet tools were created in January 2021 to address these issues: Price/Fee and cost analysis. Through better guidance and more focused training,the compliance rate may potentially increase next year. Caltrans will continue to provide better guidance resources and provide more focused training on the deficient performance measures.


SUCCESSFUL PRACTICES Training such as in-person training, short videos and live webinars continue to have made the most impact on LGAs’ compliance. The training varied based on an LGA’s level of experience. The team offered three distinct training: an overview of the procurement process, Exhibit 10-C specific training, and focused training including independent cost estimates, cost analysis, and DBE. In-person and live webinar training as well as short videos have had a clear impact on improving compliance.

The process review performed near the end of 2019 identified A&E Policies & Procedures, independent cost estimate, task orders compliance, price/fee negotiation, and cost analysis as the 5 most deficient items.

The process implemented in October 2017 requiring LGAs to submit Exhibit 10-C was critical for collecting pertinent data. Most deficiencies were found during the Exhibit 10-C process review that resulted in further changes to the DLA program. For example, the independent cost estimate, cost analysis, and Policies & Procedures adoption were consistently identified as deficiencies. The Exhibit 10-C process review continues to be the most beneficial and practical way to ensure data collection. The team has also been able to continuously assess the health of the A&E Oversight program.

LAPM Chapter 10: Consultant Selection provides detailed A&E consultant contract procurement process and information. This is continuously updated based on the Exhibit 10-C risk-based reviews, training and survey feedback, and the annual process reviews. The LAPM is the primary guidance for LGAs.

Guidance documents, such as the cost analysis worksheet and the A&E Consultant Procurement Checklist are provided to the LGAs as further guidance. Several spreadsheet tools such as independent cost estimate, cost analysis, and profit/fee determination have been created to address the low compliance in these areas. These tools were recently implemented in January 2021, so the effect on compliance will be assessed the end of 2021.


ADDRESSING 2019 ACTION PLAN 2019 Observation #1: The LGAs have an overall 83 % compliance rate amongst the 22 areas measured.

2019 Resolution:TheA&E Oversightbranchwill clarify themeasured areas needed for future reports, modify LAPM Exhibit 10-C accordingly, and streamline the data collection process to improve efficiency.

Completion Target: December 1, 2020.

Action Taken: A plan was devised to remove performance measured items that meet the target goal of 85% for 5 consecutive years from future reports. Based on Table 6. Heat Map Compliance Rate – Annually, the only performance measured item to meet this plan is Contract Expiration Date and would be slated for removal. This is a continuous effort to have all performance measured items meet the target goal of 85%.

2019 Observation2#: The LGAshad 9 areasof the22measured below the targeted goal of 85% compliance rate. The deficiencies found include:

• A&E Policies and Procedures • Conflict of Interest • CMSR • Independent Cost Estimate • Records of Procurement & Submittals • DBE • Task Orders Compliance • Price/fee Negotiation • Cost Analysis

2019 Resolution: The A&E Oversight branch will provide more guidance documentssuchasexamplesandsampleforms.Theteamwillconduct training throughout the State and focus on the deficient areas found in this review.

Completion Target: November 30, 2020

Action Taken: Several spreadsheet tools were created in January 2021 to address the major deficiencies in the 2019 process review. These tools are independent cost estimate, cost analysis, and profit/fee determination. Short videos were also developed to address some of the deficiencies. The tools and short videos will potentially increase the compliance rate. Training was scheduled for 2020 to provide more guidance to the LGAs; however, due to the Covid-19 pandemic, training was postponed.


ACTION PLAN Observation #1: The LGAs have an overall 86 % compliance rate amongst the 22 areas measured.

Resolution: The A&E Oversight branch will continue to streamline the data collection process to improve efficiency.

Completion Target: December 31, 2021.

Observation2#: The LGAshad 7 items of the22 measured below the targeted goal of 85% compliance rate. Refer to Table 7: Items Below 85% Target below.

Resolution: The A&E Oversight branch will provide more guidance documents such as examples, sample forms, and tools. The team will conduct training throughout the State and focus on the deficient areas found in this review and monitor their progress. Records of Procurement & Submittals will be closely monitored since it is considered an anomaly and needs to be carefully monitored next year since it is the only trend in the monitored category that went down this year. Refer to Table 2: Overall Compliance Yearly. More focused short videos and on demand training will be created. In-person training is a key part in improving the understanding of A&E procurement requirements resulting in improving compliance rate. A&E Oversight branch will continue in-person post Covid-19 pandemic.

Completion Target: November 30, 2021.

Table 7: Items Below 85% Target

2020

A&E Policies & Procedures 57%

Independent Cost Estimate 24%

Records of Procurement & Submittals 70%

Task Orders Compliance 83%

Price/Fee Negotiations 59%

Followed Audit Process 83%

Cost Analysis 48%



Meet the Team

Name Title Kamal Sah Office Chief of Office of Guidance and Oversight The’ Pham Acting Program Manager John Yang Senior Transportation Engineer Bing Luu Senior Transportation Engineer Quyen Ngo Transportation Engineer Hiep Nguyen Transportation Engineer

of Local Agency On-Call Contracts Selected District Agency Contract Amount Start Date End Date

1 Humboldt County $1,500,000 3/3/2020' 3/3/20'25 3 City o f Chico $2,700,000 1/6/2020 1/6/2025 3 City o f Elk Grove $2,000,000 8/28/2019 8/28/2022 3 County o f El Dorado, Departm ent o f Transp o rtat ion $300,000 1/7/ 2020 1/6/2023 3 Sacrament o County $2,500,000 7/1/ 2019 6/30/2022 3 Sacrament o $4,000,000 10/15/20'19 12/31/ 2022 3 Yuba County $300,000 6/9/2020 6/8/2022 4 City o f Vallej o $400,,000 10/9/2019 9/30/2022 4 Cont ra Cost a County Public Works Departm ent $250,000 9/10/2019 6/18/2022 4 Larkspur $300,,000 6/8/2020' 6/30/2022 5 County o f .Sant a Barbara Publ ic Works Transportat ion $4,000,000 7/1/2019 6/30/2022 6 City o f Bakersfi eld - Public Works Departm ent $786,225 1/1/ 2020 12/31/2021 6 City o f Orange Cove $330,,000 10/1/2019 9/30/2022 7 City o f Simi Valley $250,,000 10/14/20'19 6/30/2021 7 Los Angeles $10,000,000 11/20/20'19 11/19/2024 7 Los Angeles County $3,750,000 4/30/2020 4/29/2023 8 Riverside County Transportat ion Commiss ion $1,500,000 8/ 1/ 2019 7/ 31/2024 8 City o f Cat hedral City $600,,000 7/1/ 2019 6/30/2022 9 City o f Tehachapi $999,999 1/7/2020 1/7/ 2025 10 County o f M erced $2,500,000 1/ 1/ 2020' 12/31/2022 10 M erced County Public Works $500,,000 11/17/20'19 11/17/2022 10 City of Gust ine $999,999 10/1/2019 10/1/2022 11 County o f San Diego $5,000,000 7/1/2019 6/3/20'24


APPENDIX A

A&E Oversight Process Review Questionnaire Instructions: 1. Complete questionna ire by marlking your response to eac h question witn an X for t ne consultant contract procu rement specified in the attached Exh ibiit 10-C. 2. Provide the supporting documents to the file-sharing folder per the instructio ns in the e maill fo r ou r verificatio n. 3. Provide the fil.enames and page numbers to iincr,ease review efficie ncy and accuracy .. !

Filename & cite Acceptable No Description Yes No

page number Documentation Was there an indep endent cost estimate Independent Cost

1. prior to review of proposa ls - 23 CFR Estimat e {ICE)

172. 7(a)(1}(v)( B)? Did members invo lved in t he procureme nt Exhibit 10-1" Conflict of

process complete t he mandato ry Conflict Interest & 2. Co-nfi de nt ia I it y

of !Inte rest statements - 23 CFR St atement 172.7(6)(4), LAPM Exhiib it 10-T? Was pr ice/fee negotiations documented - Emai ls, letter showing:

3. 23 CFR 172.7(al(1l(v),(E)? t he negotiation process

Were score sheets signed and dated? Exhibit 10-B or 4. ,equivalent.

Secondary score If oral interviews were conducted, were sh eets. Docum entation

5. secondary sco re sheets retained - 23 CFR li sting eva luation criteria

172. 7(a)(1}(iv)( F)? for ora l interviews.

Was LAPM A&IE Consultant Cont ract No submittal 6. Reviewer Checklist (Exhibit 10-C) N/ A N/A requi red. Exhib it N/A

subm itted? 10-C received. Does contract cost proposal meet LAPM Cost propos.als e.g.

standard (Exhibit 10-H)? (e.g., labor costs Exhibit 10-H Cost

7. brolk:en down into direct, ind irect and Proposa l or equiva lent

profit components) - ILAPM Chapter 10 and 23 CF IR 172.11? Were sel'ection criteria and we ights Solicit at ion document,

properly defined in Rf P/RFQ - 23CFR e.g., RFP/RFQ cont aini rig 8. 172.7(a)(1}(ii),(C)? Used ILAPM Exhibit 10- t he se I ectio n/ evaluation

,cr iter ia w ith w eights. B?

Evidence of

Was RFP/RFQ so licitat ion by public announcement.. Posting

9. announcement/advertisement- 23 CFR ,documentation, e.g., newspa per, w ebsite

172. 7(a)(1}(i)? posting, et c.

·- --


APPENDIX B

Dl!saiption Vl!s No

D 5' u;i,::al ~ eney hav-e wriu en !Policy and . 0. Prf.l..:edyres; ·ft!r pTQ,;::urin;1; A _E iXJ:ntr a,:ts; =

:u rC F-R l'2,$ji;!1)( ll):1

Oi!!ll Local Age C!'/ dkm .1ment 150 ntract ll_ approva1l/ .i,ut h ori · on actio je .g. coun,:;i l

B .

re:50 ·on j '?

Wi:15' 5e, le,ttit.m m;;:1 de u~ing ..:riter•a stat,ed in RrP / RrO. = .a ffll '12. 'T{a)U] [iv (O)? Were ori · al score s eet s; retai ed - · 2:1 CFR 172:. 7{a) 1 [ivl(F]'?

liS im~ h..:id ..:if pi;!)' · ent i:learlv idenrt ffied in1 RF'P/RIFO, = 23 O'i:\R.172.7(a1l[l)liii1j(OJ=1 Ooes. contract term aYe• e-icpirat ion dlate = LAPM Cha teT 10 b h i b" UJ..iR?

t 7. l!)i!!JI Local Age c.v fol low Audit Proe:es:s?

Di!ll! Local .A~e ev r,e~a rg i;in:u::u rern ent 1-i. g;r~r!:e'S5 ·i:I d 5!.!!lirn i _ 1ls; 2! ~ CF-R ll 7:2.!ll and

2. CF'R 200-~ _$[i I°~

'19. 1.s t h e oo:ntractfor a C::onsul _am: in a Ma,n:a1gem ent Su[Pport Rale?

2,{).. Wi:15' a ,i:::ost anal~ i~ rJa ne ~ 2 CFR 200.32!fo)=1

F lename & cite Au:eptable page number Documentation

ti;.111, d;.iii,1~

'z:.itio (Ir n otice ee-d ram

ady to r;tr;ic , e.

UIKll

1 ~ il'l~' i,,~--

1~ r~~i'i.•~d gm~~~

I ~t~, O_E


APPENDIX B

,

2.1.

De,scription

Were Ta,sk Orders issued in ai=co d i:inoe wit 23 CFR Jl 72. 7, 172.9?

Were DBE reqw:irements. Fn~t ~ W M CtL 9 &10?

Rilil!iflame & cite pa.ge numbE!I'"

Aoceptable Documentation

ally, • dic.ite . •rl!;l:iD I


APPENDIX B

A&E Oversight A&EOversightprovidesguidanceonconsultant contract selection for local agencies.

dot.ca.gov/programs/local-assistance

Empowering Our Partners To Be Succesful dot.ca.gov/programs/local-assistance

https://dot.ca.gov/programs/local-assistance

https://dot.ca.gov/programs/local-assistance

2020 A&E Oversight PERFORMANCE MEASURES #20-01

Documents