Lunch & Learns for Evaluators in Ottawa - … · Lunch & Learns for Evaluators in Ottawa: ... • Appreciative inquiry in ... groups of people who share a passion for

Lunch & Learns for Evaluators in Ottawa:

A promising model for professional development

and networking

CES Conference 2009 June 3

Anna EngmanNatalya KuziakKate Powadiuk

Ken StephensonLynda Weaver(Jane Whynot)

Discussant: Steve Montague/Charles Lufthaus1

Presentation Outline

• Ottawa L&L: History, Structure & Past Sessions

• Why Do the L&Ls Work: Theory of Learning

• Ottawa L&L: Survey Findings & Lessons Learned

• Toronto L&L: History, Structure, Past Sessions and Lessons Learned

• Considerations When Starting Your Own L&L

2

Ottawa Lunch and Learns

• Initiation: three evaluators met at 2007 CES conference

• Motivation: free, informal professional development and networking

• Resources: in-kind contributions (leading, hosting and participating)

• Growth: currently 100 participants, website

History and Structure

4

Past Sessions

• Reflective practice and evaluation

• Managing evaluation contracts

• Evaluability assessment• Vested interests in

evaluation• Attribution vs. contribution• Planning for gathering

performance results evidence for improved measurement, monitoring, evaluation and reporting of achievements

• Cultural competence in evaluation

• Feminist evaluation• The case study approach• Systematic lit reviews• Theory driven evaluation

and theories of change• Efficiency analysis• Appreciative inquiry in

evaluation I & II• Evaluation mistakes• Evaluation of networks and

partnerships

5

• Nutrition definitely improves learning!– Studies on children

• Eating with colleagues probably improves nutrition!– I’m not going to eat fries in front of you!

• We have different learning styles– E.g., visual, auditory, experiential, mixed

• We come with baggage!– So do the ‘teachers’

• Our attention span is about 11 minutes!

Theory for Continuing Education

6

• We decide what is important, what we will do with the info– Relevant, immediately applicable

• We learn best when info is for solving a problem

• We validate info against our beliefs, experiences– We may disagree

• We expect a collaborative learning environment

• We learn best when we participate actively

How Adults Learn

7Meloche, Hélène, www.hmeloche.com

Didactic lectures have little effect on changing behaviour

– Not to mention boring and not nutritious

Interactive, challenging, sequenced

– Enhance activity

– Opportunities to practice skills

– Learn…Work…Learn

Proof, Meta-Analysis

8

Davis, D., O’Brien, M. A. T., Freemantle, N., Wolf, F. M., Mazmarin, P. & Taylor-Vaisey, A. (1999). Impact of formal and continuing medical education. JAMA, (282(9): Internet document retrieved November 3, 1999 from http://jama.ama-assn.org/issues/v282n9/full/jrv90027.html

• Non-threatening, non-judgmental

• Community of Practice develops

…groups of people who share a passion for something that they know how to do and who interact regularly to learn how to do it better.

Peer Learning

9Wenger, 2002

1. Tell stories2. Play games3. Explore and experiment4. Use pictures5. Have a coach6. Learn with others7. Focus on what is important8. Take recess9. Have lunch10. Be passionate11. Keep learning

Eleven Common Sense Learning Principles

10Forman, D. C. (2003). Eleven common-sense learning principles. Training and Development , September, pp. 39-44

☻Lunch

☻Buddies

☻Active

Therefore..L&L best practices are:

11

• Online survey sent to 97 L&L participants in April 2009

• 54% response rate

• Of those who didn’t respond:– 42% had never attended a session

– 4% were too busy

– 53% unknown

Survey Methods

12

• Broad range of age and experience

• Occupation:

– 94% working; 24% studying; 20% doing both

• Majority of respondents are:

– Highly educated (70% Masters level; 14% PhD)

– Senior level at work (66%)

– In federal public sector (67%)

– CES members (85%)

– Female (76%)

Who are the L&L Participants?

13

Gender

* “2005 national survey” refers to: Borys, S., Gauthier, B., Kishchuk, N., Roy, S.N. (2005) Survey of evaluation practice and issues in Canada. Paper presented at the Joint CES/AEA Conference, Toronto, October 26, 2005. 14

0%

10%

20%

30%

40%

50%

60%

70%

80%

Male Female

Lunch & Learn survey (n=46) 2005 national survey (n=634)

Education

15

0%

10%

20%

30%

40%

50%

60%

70%

80%

Other Bachelors Masters or

post-grad

PhD

Highest Degree Completed

Lunch & Learn survey (n=49) 2005 national survey (n=638)

Employment Sector

16

0%

10%

20%

30%

40%

50%

60%

70%

Federa

l publ

ic

Not fo

r pro

fit

Private

Provinc

ial p

ublic

Mun

icipal p

ublic

Reg

iona

l pub

lic

College

/ uni

vers

ity

Lunch & Learn survey (n=48) 2005 national survey (n=647)

Experience (n=50)

17

Job Position (n=49)

“The ability to actually attend the sessions is the biggest obstacle for our

team at the moment... if messaging comes from the director level across...

that personnel should attend, and at least one staff member can be

released to attend on the team's behalf, we're all the better for it.”

* Officer / consultant

What is Appealing About the L&L?Interest in Topics

• 94% agreed that the topics are of interest to them

• 92% agreed that the topics are relevant to their work/studies

• Most well attended L&L sessions:– Evaluability Assessment

– Vested Interests in Evaluation

– Conducting Case Studies

– Cultural Competency in Conducting Evaluations

– Cost-efficiency Analysis

– Feminist Evaluation

– Appreciative Inquiry

19

What is Appealing about the L&L?Interest in Topics

“I always circulate the topics in advance to

members of my evaluation team, and encourage

representation if I think a topic is particularly

relevant for our work here. Anyone who attends

is then asked to present back to our whole

group, so we can all benefit from the learning.

We really appreciate this!”

20

What is Appealing about the L&L? Value for Participants

• 55% have benefited from the opportunity to network with other evaluators

– 22% were neutral, 10% disagreed

• 51% have applied the techniques or knowledge they gained from a L&L in their work or studies

– 18% were neutral, 16% disagreed

21

What is Appealing About the L&Ls?Applying What They Learned

22

L&L Topic How participants applied the knowledge or technique they learned

Evaluability Assessment, Case Studies, Cost-Effectiveness, Program Theory

Used to design or guide evaluation studies; used the methodology

Vested Interests in Evaluation

Dealing with requests to modify evaluation findings; addressing ethical issues; dealing with influence of difficult clients

Reference Materials Used the resources and reference materials; referred colleagues to them

Appreciative Inquiry Used in focus groups; incorporated into evaluation and planning practices

Systematic Lit Reviews Used the methods suggested, e.g., how to write literature search questions

What Other Value or Benefits Do You Get from Attending L&Ls? (n=20)

23

35% Networking professionally and socially; Staying connected; Being part of a community

35% The opportunity to share information with other evaluators, Ask them questions and learn from their experience

15% Learning how other evaluation firms or government departments conduct evaluations; Hearing about what is going on in evaluation in other shops

10% The opportunity to learn new strategies, tools, techniques and approaches in evaluation

10% It’s fun and motivational

10% Learning about good articles, publications and studies related to evaluations.

“A good way to hear what's going on in the field of evaluation. Good networking possibilities. Good to

feel part of a community.”

“By speaking with evaluators who completed similar projects, I avoided making mistakes and did not

have to reinvent the wheel.”

Other Value or Benefits of L&Ls

24

• The majority of participants (80-82%) are satisfied with the readings, the presentations and the format

• Slightly lower level of satisfaction (68%) with the locations of the L&Ls

• Factors that kept people from attending: too busy (86%), date not convenient (31%), locations not convenient (29%), time not convenient (20%)

• Only 31% of participants have led or co-led a L&L

What’s Working Well & What Could Be Better

25

• For the most part, continue!

• Challenge: Travel and time required

– Occasional evening sessions?

– Target evaluation managers and directors?

– CES accreditation?

• Challenge: Need more session leads

– Mentoring or teaming up to lead sessions?

– Session on facilitation?

– Promote more participatory formats?

– Promote the L&Ls as a spin-off opportunity?

Where do Ottawa L&Ls go from here?

26

• Challenge: Communications & awareness

– Made the Google group website public!

– Funding for a website?

– Collaborate with CES regarding promotions to NCC’s members?

• Challenge: Increasing interest in sessions

– Regular polls on what topics interest participants?

– Encourage participants to bring in experts occasionally?

• Challenge: Continuous improvement

– Evaluation forms after each session?

– Guest book to keep track of attendance?

Where do Ottawa L&Ls go from here?

27

The Toronto Lunch & Learns

28

• Started in 2006, ended in 2008

• Brought together a variety of individuals

• Originally held at Cathexis, then shift to different locations

• Variety of topics including:

– Building excitement about using data

– Capacity building

– Using existing data sets

– Innovative data collection methods

History of the Toronto L&Ls

29

What happened…

30

What worked well:

• Excitement about learning

• Diversity of experience and opinion

• Variety of presentation methods

• Collaborative planning

Challenges:

• Travel time

• Lack of varied attendees

• Loss of interest over time

• Competing deadlines

• CES Ontario conference – discussion session planned– Different format?

– CES involvement in organizing?

• Continuing to find new ways to promote professional development at Cathexis internally

What’s next for Toronto L&L??

31

• Where to find participants?

– Personal network, PD events, CES local chapter, university programs, managers, chain e-mails

• Where to find a facility?

– Multiple facilities (participants’ workplaces, community centers, libraries)

• How to increase likelihood of sustainability?

– A coordinator, a website, foster participant ownership, recognize contributions, participatory format, encourage spin-offs, make it a priority

Considerations when Starting Your Own L&L

32

• Where do you think the L&Ls in Ottawa & Toronto should go from here? Which suggestions do you agree with?

• How does the L&L as a learning model fit into your professional development?

Questions for Discussion

33

• Ottawa L&L:

– [email protected] http://groups.google.com/group/lunch-and-learn-for-evaluators-in-ottawa

• Toronto L&L:

– [email protected]

• Victoria L&L (starting up):

– [email protected]

For Additional Information

34