Smart Commute Evaluation Tools, Techniques and Lessons Learned in Monitoring and Evaluating Workplace-based TDM Programs
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 1/36
Smart Commute EvaluationTools, Techniques and Lessons Learned in Monitoring and
Evaluating Workplace-based TDM Programs
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 2/36
Outline
1. Background
2. Approach3. Monitoring Tools
4. Evaluation
5. Results
6. Challenges
7. Lessons
8. Next Steps
Ryan Lanyon
Team Lead, Smart Commute
Metrolinx (GTTA)
October 22, 2008
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 3/36
At-a-Glance
Build evaluation into your initial planning
Monitor activities, benefits and customer service Scrutinize and understand indicators
Learn from results and mistakes
Resources available at www.smartcommute.ca
Just do it!
Smart Commute works with various stakeholders to
relieve traffic congestion, improve air quality and health,
and reduce emissions that cause climate change.
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 4/36
Background
Smart Commute
Focus on commuters and workplaces
Partnership with municipalities
Staged development – 2001: Pilot TMA - BCRTMA
– 2004: Pilot regional implementation - Municipalities
– 2008: On-going implementation – Metrolinx + Municipalities
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 5/36
Background
Metrolinx (Greater Toronto Transportation Authority)
Formed in 2007
Coordinates transportation across GTHA
Regional Transportation Plan
6.13 million residents, 3 million commuters
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 6/36
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 7/36
Background 2001: BCRTMA / NTV
2004: 404-7
2005:
Mississauga
NE Toronto
2006:
Brampton-Caledon
Halton
2007: Central York
Durham
2008: Toronto-Central
2009: Airport?
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 8/36
Background
Employer program
Improve and encourage commuting options
Assessment of current conditions - Baseline
Development of a plan
Implementation – ‘Commuter benefits program’ – Carpool Zone and ERH
– Walking, cycling, transit, carpool, telework programs
– Campaigns, contests and special events
Follow-up assessment – Measured against baseline
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 9/36
Approach - Ideal
Goals
ObjectivesRegional Transportation Plan
Municipal Plans Implementation
Strategic Plan Monitoring
Evaluation
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 10/36
Approach - Actual
Goals Objectives
RTP
Municipal PlansImplementation
Strategic Plan
Monitoring
EvaluationFederal Funding Requirements
Implementation
Implementation
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 11/36
Approach
2004-2007 - Contribution Agreement
Negotiated between funder and municipalities 2008 - Monitoring Framework
Established by Smart Commute stakeholders – TMAs and Smart Commute Association (implementers)
– Municipalities and Metrolinx (funders)
Multipurpose – Measure activity levels for funders
– Track impacts to justify funding
– Collect statistics for marketing purposes
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 12/36
Approach
Inputs / Activities
Number of businesses – Active
– Engaged
– Supporting
Number of pamphlets distributed
Number of cycling programs Number of media releases
Number of commuters reached
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 13/36
Approach
Outputs / Benefits / Results
Brand awareness – % of employees aware; change from baseline
Commuter satisfaction – Ratings of Smart Commute program
Reductions
– GHG emissions
– CAC emissions
– Vkt and trips
Commuter cost savings
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 14/36
Monitoring
Levels of monitoring
Regional
TMA
Employer
Commuter / customer
Types of monitoring Behaviour change
Attitudinal change
Customer service
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 15/36
Monitoring
Regional
Existing sources – Statistics Canada
– Transportation Tomorrow Survey (TTS)
– Conducted by University of Toronto
– Detailed behaviour survey
– Conducted every five years
– Cordon Count Program
– Conducted by municipalities, compiled by University of Toronto
– Observed data
– Conducted every two to three years
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 16/36
Monitoring
Regional
New sources – Commuter Attitudes Survey
– Conducted by Metrolinx
– Detailed attitudinal questions
– Conducted every two years
– Carpool Zone
– Conducted by Metrolinx
– Monthly activity survey
– TMAs and Employers
– Compiled data from localized sources
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 17/36
Monitoring
TMA
Quarterly reports – tied to Metrolinx funding
Summary of activities per employers
Communications and outreach activity
Qualitative results
Comparison of activity levels between TMAs
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 18/36
Monitoring
Employer – Baseline Module
Baseline employee survey – Standardized across all TMAs and employers
– Census survey
Site assessment – Current conditions
– Inventory of advantages and deficiencies Vehicle and occupancy count
– Discreet sites
– 1,000+ employees
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 19/36
Monitoring
Customer – Commuter
Service-related – Monthly Carpool Zone microsurvey
– Emergency Ride Home post-ride survey
Employee surveys
Campaign-related data collection e.g. Car Free Day
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 20/36
Monitoring
ModeStatisticsCanada
(2006)
TTS
(2006)
CAS
(2008)
Surveys
(2001-08)
DriveAlone
65.90% 64.7% 48% 75.91%
Transit 19.93% 19.7% 28% 10%
Cycling 0.97% 0.7% 1% 0.89%
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 21/36
Evaluation
Benchmarks and Trends
75 employee baseline surveys
Statistics Canada
Transportation Tomorrow Survey
Cordon Counts
Commuter Attitudes Survey
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 22/36
Evaluation
External
Employer surveys – Biannual survey – program satisfaction rating
– Satisfaction with Smart Commute services
– Conducted by third party through Metrolinx
Commuter surveys
– Employee follow-up surveys – program satisfaction rating – Annual survey – Carpool Zone satisfaction
Internal
Smart Commute Technical Committee
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 23/36
Evaluation
External Review
Consultant evaluation of operational models
Research Partnerships
University of Toronto (Mississauga) – Factors for successful carpool formation
– Further research
Ryerson University – TMA Toolkit assessment
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 24/36
Results
Almost 100 members and partners
March 2007 – 50 employers
March 2008 – 75 employers
June 2008 – 89 employers
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 25/36
Results
Transportation indicators
76.4 million vkt
1.3 million trips
Equivalent of 10,000 cars
Increase in carpooling – 7% to 12% to 13%
Cars Off the Road
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 26/36
Results
Primary Mode Shift
2
5
7
34
50
0 20 40 60
%
1
6
13
28
48
0 20 40 60
%
1
4
12
31
48
0 20 40 60
%
Drive alone
Public Transit
Car/vanpool
Walk/Jog
Bike
2006 20052008
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 27/36
Results
Environmental indicators
17,500 tonnes of greenhouse gas emissions – Enough to fill Rogers Centre almost six times
100,000 kg of criteria air contaminants – Equivalent amount of NOx to smoking 3 billion cigarettes
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 28/36
Results
Communications and Outreach
400+ stories
75+ million media impressions
Brand recognition – Smart Commute ↑
– Carpool Zone ↓
5
9
14
27
17
0 10 20 30
%
Smart Commute
Carpool Week
Clean Air Commute
Carpool Zone Website
Commuter Challenge
Awareness
(2008)
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 29/36
Challenges
Pressures to implement right away
Show action
Pent-up demand
More interesting than planning or evaluating
All the pieces not always in place
Monitoring not appreciated until results urgentlyneeded
Objectivity
Of course my ‘baby’ was successful!
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 30/36
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 31/36
Challenges
Defining success
Setting targets without enough information – Are examples applicable?
Are targets achievable with the resources alloted?
Can these targets be effectively monitored?
What happens if we don’t reach these targets?
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 32/36
Lessons
Measurement and Evaluation
Some items simple, but time-consuming – Do you really need the data?
Project benefits less easy to isolate – Double-counting also possible through employee surveys
and Carpool Zone surveys
– External factors – environmental concerns, gas prices One clear conclusion: an impact was made.
Time and consistency open up opportunities – Benchmarking
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 33/36
Lessons Measure and Evaluation
Access resources at your disposal for help Universities
Municipalities
Provincial and Federal Governments
Evaluate what you do and how you do it
Give an adjustment period to monitoring Test! Test! Test!
Allow partners time to understand and see value
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 34/36
Lessons
Provide incentives for monitoring
Tie completion to access
Award and reward success
Measurement and Evaluation
Learn from mistakes – you will make them!
Learn from poor results – Was the program effective?
– Was the monitoring effective?
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 35/36
7/27/2019 Act Canada Tdm Evaluation 102208
http://slidepdf.com/reader/full/act-canada-tdm-evaluation-102208 36/36
Questions?Ryan Lanyon
Team Lead, Smart CommuteMetrolinx
416-874-5933