MOBILITY MANGEMENT EVALUATION: The Long and Short of It

CIV ITAS TRAINING7 MAY 2014

FLORENCE, ITALY

MOBILITY MANGEMENT EVALUATION:

The Long and Short of It

Eric N. SchrefflerTransport ConsultancyESTC San Diego, CA USA

OUTLINE

1. My experience

2. Why evaluate? and for whom?

3. Forecasting vs. measurement

4. Principles of good evaluation

5. Overall framework for MM evaluation

6. U.S. examples – California and Washington D.C.

7. Transitioning from short-term to long-term evaluation

8. The politics of MM evaluation

9. Common mistakes

10. Top ten tips for effective evaluation

Quick Bio

MM/TDM evaluation for over 30 years Written/contributed to several seminar guidebooks Involved in several EC-funded projects

MOST – MOST MET MAX – MaxEVA CIVITAS GUARD CIVITAS II –final brochure CIVITAS PLUS training

Currently leading evaluation of TDM component of 6 U.S.

Urban Partnership Agreement projects

Why Evaluate? And for Whom?

Good management practice! Measure progress against objectives

Output or outcome objectives Program or public policy objectives Integral to performance-based planning

Why Evaluate? And for Whom?

Benchmark program against peers Compare cost effectiveness to other solutions

Marginal cost to accommodate a car vs. cost to reduce a car trip

e.g., LA Metro evaluation performance-based planning Satisfy funding entities Satisfy policy or oversight boards

Don’t be afraid of evaluation!!!

Know your evaluation,

Love your evaluation,

One day your evaluation may save your (life) program.

Forecasting vs. Results

Evaluation can be defined as: A priori estimation of expected impacts (forecasting) Ex post measurement of outcomes (results)

Forecasting Predicting what could/should occur Using comparative case studies, models,

sketch planning, SWAG Measurement

Quantifying what has actually occurred Using counts, surveys, etc.

Forecast

Actual

PROST! Principles for Sound Evaluation

Practical

Rigorous

Objective

Standardized

Timely

FRAMEWORK FOR EVALUATION

Important to plan for evaluation Have plan to frame approach and tasks Should include:

Purpose and objectives Data collection methods Analysis methods Budget Schedule Reporting

Use conceptual framework - MaxEva

MaxEva Assessment Levels

Assessment of Services Provided

A Project activities and outputsB Awareness of mobility services provided

C Usage of mobility services provided

D Satisfaction with mobility services provided

Assessment of Mobility Options Offered

E Acceptance of mobility option offered

F Take up of mobility option offered

G Satisfaction with the mobility option offered

Overall Effects H Long-term attitudes and behavior

I System impacts

MaxEva – The EPOMM Evaluation Tool – www.epomm.eu/maxeva

European Example – SwedenEmployee Public Transit “Test Rider” Pass Pilot

Assessment of Services Provided

A Outputs: posters, intranet info, meetings, 1,000 employees

B Awareness: 48% of public transit ticket offer (480 emp)

C Usage: 28% participated in meeting, sought info

D Satisfaction: 80% satisfied with info and pilot concept

Assessment of Mobility Options Offered

E Acceptance: 6% signed a contract to participate (54 emp)

F Take up: 85% of these used PT; 5% used before

G Satisfaction: 90% of participants satisfied with PT

Overall Effects

H Long-term: after one year, 40% still using PT

I System impacts: car use 110K km; CO2 20 tons/year

MaxEva – The EPOMM Evaluation Tool – www.epomm.eu/maxeva

TDM and Highway Reconstruction US101 Cuesta Grade CA

Reconstruction mitigation$730,000/year for TDMThree elements:

More commuter express buses Vanpool promotion/subsidies Carpool fuel incentive

Surveyed all participantsRemoved 310 cars dailyCarpool incentive most cost

effective ($3.36/VTR/day)

U.S. Examples

Regional TDM Program Commute ConnectionsWashington D.C.

Triennial evaluation since 1997 Consistent approach (MaxEva) Evaluate total impacts = travel,

emissions, energy (e.g., 118K fewer car trips per day)

Evaluate separate impacts Ridematching Employer outreach Mass marketing Guaranteed ride home Bike to work Carshare Telework Incentives

U.S. Examples

SHORT-TERM TO LONG-TERM

Short-run generally covers one year or year or duration of project funding duration.

Uses before and after data

Long-term can include projecting impacts into future (BCA, lifecycle)

Or can involve time series data for program over years

Key is planning and consistency

The Politics of Evaluation

There may be pressure to:

- subvert findings

- document desired outcomes

- spin the results

RESEARCH

Common Evaluation Mistakes

1. Putting off evaluation

2. Underfunding evaluation

3. Assuming “projected” = “actual” results

4. Incorrectly comparing projected to actual

5. Projecting target group findings to entire population

6. Ignoring causality and externalities

Common Evaluation Mistakes (con’t)

7. Confusing outputs with outcomes

8. Changing methods/assumption mid-evaluation

9. Assuming all mode shift from drive alone

10. Ignoring access mode to new travel options

11. Ignoring non-response in surveys

12. Giving into pressure to change findings

Top Ten Evaluation Tips

1. Get help, at least the first time

2. Plan, plan, plan Plan for evaluation Performance-based plan Stick to the plan

3. Budget for evaluation

Top Ten Evaluation Tips

4. Keep findings simple!

5. Don’t be afraid of what you may find!

6. Be confident! No whining!

Top Ten Evaluation Tips

7. Seek local default factors (e.g. trip length)

8. Use standardized methods/tools/guidance

9. Learn to talk like an engineer

Top Ten Evaluation Tips

10. JUST DO IT!

Key Resources

MaxEva – The EPOMM Evaluation Toolwww.epomm.eu/maxeva

Evaluation Matters: A Practitioners’ Guide to Sound Evaluation for Urban Mobility Measures

www.civitas.eu/content/evaluation-matters

Integrating Demand Management into the Transportation Planning Process: A Desk Reference (Chapter 9)

www.ops.fhwa.dot.gov/publications/fhwahop12035/index.htm

Canadian TDM Measurement Toolboxwww.tc.gc.ca/media/documents/programs/tdm-toolbox.pdf

Grazie per la vostra attenzione!

Eric N. Schreffler

Transportation Consultant

ESTC

San Diego, CA, USA

[email protected]