IFL Workshop-Personal Finance Outcome Assessments-04-15

Personal Finance Outcome Assessments: Methods and Metrics for Face-to-Face Classes and

Social Media Outreach Barbara O’Neill, Ph.D., CFP®, CRPC®

Distinguished Professor, Rutgers University [email protected]

Are You Evaluating Your Financial

Education Programs?

If so, how?

The Million Dollar “So What?” Question…

At the end of the day…did your financial education program make a difference?

How do you know?

The Current State of Financial Education Program Evaluation • Current evaluation efforts are still far from

satisfactory

• General lack of evaluation capacity

• Evaluation is often treated as an after-thought

• Outcomes (e.g., measures of changed behavior) are often confused with program outputs (e.g., number of participants)

• What gets measured gets funded

• With evaluation results, you can… – assess impact of programs on learners

– see if you accomplished what you planned

– know if a program was “worth it”

– celebrate success and learn from failure

– make informed decisions to improve, hold, or fold programs

– promote your program and win public support

We are in an “Accountability Era”

Consider a Third Party Program Evaluator

• Some funders require this (e.g., FINRA Investor Education Foundation and IMLS)

• No potential for conflict of interest by one entity performing both education and evaluation role

• Takes evaluation “monkey” off busy educators’ back

• Build into grant funding requests

• See http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html • Begins with the end in mind • Explains what a program is and what it will accomplish • Shows relationships between inputs, outputs, and outcomes

Introducing the Logic Model

Building a Strong Financial Education Program

INPUTS OUTPUTS OUTCOMES

Program investments

Activities

Participation

Short

Medium

What We

Invest

What We Do

Who We Reach

What Results

SO WHAT??

What is the VALUE?

Long-term

Staff

Money

Time

Volunteers

Partners

Equipment/Technology

Policies

Research

INPUTS

OUTPUTS

What We Do Who We Reach

ACTIVITIES Assess needs and assets

Design curriculum

Educate students

Conduct workshops

Facilitate learning groups

Sponsor conferences

Work with the media

Partner – collaborate

PARTICIPATION Participants

Clients

Customers

Users

Groups

Reactions - Satisfaction

OUTCOMES What Results for Individuals, Organizations, Communities..…

SHORT Learning

Awareness

Knowledge

Attitudes

Skills

Opinion

Aspirations

Motivation

MEDIUM Action

Behavior

Practice

Decisions

Policies

Social action

LONG-TERM Conditions

Human

Economic

Civic

Environment

OUTCOMES

Commonly Measured Items That Are Not Outcomes

• Participant satisfaction

• Number of people taught

• Units of education completed

• Number of events held

• Time and money spent

• Level of effort

Activity #1 What are some outcomes for your financial education programs?

Impact Evaluation Data Collection Methods

• Surveys (Paper or Online)

– Post-evaluation only (short programs)

– Pre- and post-evaluation

– Follow-up (e.g., 3 months later)

• Focus groups

• Interviews

• Observations

• Tests of knowledge/ability

• RARE: Control groups and longitudinal studies

Typical Survey Questions • General reactions to the program

• Changes in knowledge

• Changes in motivation, confidence, and abilities

• Intended changes in behavior

• Actual changes in behavior

• Future programming needs and preferences

• Demographics of participants

• Qualitative/open-ended responses

Post-Then-Pre (Retrospective) Evaluation

Method • Compares knowledge and attitudes before and

after a financial education intervention

• Administered once at the end of the intervention

• Helps avoid biases due to people thinking they know more than they know

Case Example: Humpty Dumpty

“Humpty Dumpty sat on a wall,

Humpty Dumpty had a great fall,

All the king’s horses and all the king’s men

Couldn’t put Humpty together again”

So Who/What Was Humpty Dumpty?

Probably NOT what you think!

Post-Then-Pre (Retrospective) Evaluation Method

• Why? Because people don’t know what they don’t know! • Who knew what Humpty Dumpty really was? • Five point rating scale

– 1 = Strongly Disagree – 5 = Strongly Agree

• Post: After listening to Barb, I know the history of the Humpty Dumpty nursery rhyme

• Pre: Before listening to Barb, I knew the history of the

Humpty Dumpty nursery rhyme

Post-Then-Pre Evaluation • Also known as a “Retrospective” evaluation • Helps identify changes in knowledge, attitudes, and

behavior

Another Post-Then-Pre Evaluation Example

Big Advantage: Avoids the problem of learners under-estimating what they don’t know before a program

Critical Incident Technique • Qualitative evaluation research method where

program participants tell personal stories

• Often used with “Train the Trainer” programs for professionals (e.g., teachers, librarians)

• Incidents are categorized and deemed successful (positive results) or unsuccessful (negative results)

http://www.joe.org/joe/2013june/tt2.php (Journal of Extension article)

More About the Critical Incident Technique

http://www.adb.org/documents/information/knowledge-solutions/the-critical-incident-technique.pdf

• Ask subjects to describe- through interviews-incidents that they handled well or poorly (need not be spectacular events)

– Example: How did library staff handle patrons’ personal finance questions?

• Provides rich personal perspectives

• Pre-intervention and Post-intervention comparison of number and type of incidents

Critical Incident Technique Example: NYPL Grant

Program Title: Money Matters

CIT Questions in NYPL Financial Education

Evaluation • Think about your experiences helping patrons with personal finance

questions. Remember a time when you had a successful experience helping someone with these types of questions. Please write down what happened.

• What made this a successful, positive experience?

• Think about your experiences helping patrons with personal finance questions. Remember a time when you had an unsuccessful experience helping someone with these types of questions. Please write down what happened.

• What made this an unsuccessful or challenging experience?

CIT Summary in Final NYPL Project Evaluation Report

“The change in the percentages of the types of incidents is indicative of change in attitudes and abilities. Also, a new category appeared in the successful incidents, Increased Knowledge/Confidence/Satisfaction, which was not present in the Pre-training Survey. Furthermore, for the unsuccessful critical incidents, the category Lack of Training/ Knowledge which was the most frequently seen theme for the Pre-training Survey was not found at all in the Post-training Survey. These findings resonate with quantitative findings and provide confirming evidence that the Money Matters training has been successful in improving participants’ knowledge of personal finance and their ability to be successful in handling patron inquiries. Additionally, staff members provided eloquent testament in their qualitative responses that their skills have been enhanced and their attitudes and behaviors became more positive for ably handling personal finance queries” (Radford, 2013).

Social Media Evaluation Methods

America Saves Week (ASW) Social Media Campaign

• Conducted by eXtension (online program of the Cooperative Extension System), 2011-2015

• Prepared “cut and paste” tweets and FB messages on saving, debt/expense reduction, America Saves, ASW

• Project cooperators from eXtension, America Saves campaigns, military, others

Triangulation (Multiple Methods) Evaluation

Approach • Unique Twitter hashtag: #eXasw

• Follow-up follower/friend survey

• Follow-up project participant survey

• bit.ly analytics to determine number of clicks on unique embedded links

• Pre- and post-ASW Twitter influence metrics (Klout score)

Quantitative Impact Data: 2011-2014

• #eXasw tweets recorded (1,091- 1,503)

• Facebook & Twitter message clicks (602- 2,814)

• Followers/friends reached (3,056- 8,163)

• Followers/friends reached (average): (192-254)

Follower/Friend Survey Results: 2012 Example

• Response: 50% “very helpful” and 50% “helpful”

• 54% visited ASW links; 38% planned to

• 8% joined AS; 38% planned to; 16% already joined AS; 38% had not joined AS

• Mostly positive comments about SM messages

• One negative comment about seeing repeated messages and the need for more action steps

Professional Collaborator Survey Results: 2012 Example • 22/22 Klout scores increased

– Average from 20.3 to 29.3 (vs. 11.22 to 19.68 in 2011)

• Positive comments (e.g., wanted more “cut and paste” messages, improved SM skills, ease of use, continue in 2013)

• State Governor and Senator started following one participant

• Reported likes, retweets, and direct messages

Klout Score Progression • Klout is a measure of a person’s “influence” on twitter

• Based on an algorithm with factors such as number of followers and number of retweets

• Go to www.klout.com and log in with Twitter or Facebook

– 2011: 11.22 to 19.68

– 2012: 20.3 to 29.3

– 2013: 32.6 to 39.6

– 2014: 29.76 to 38.84

Project Take-Aways • Social media messages are great for increasing

awareness but do not appear to prompt much action

• Project participants like having “cut and paste” social media messages

• Twitter influence (Klout score) of project participants became noticeably higher over time

• Social media is an effective outreach tool

Twitter Hashtags • A key component of Twitter use is the hashtag (#)

symbol followed by a series of unique characters

• The hashtag serves like a “magnet” to link together the tweets of Twitter users who are interested in the same topic (e.g., credit) or who are linked together in some way (e.g., professional conference attendees)

• By using a hashtag, users can effectively hold a “conversation” with one another about a topic of interest

Twitter Chats • Online users gather at a specified time (e.g., every

Friday at 3 pm ET) to “discuss” (read: tweet about) certain topics, questions, or issues.

• Synchronous learning environment where participants interact in real time using their Twitter user names (a.k.a.,”handles”).

• As with tweets at conferences, the “glue” that holds Twitter chats together is the designated hashtag

• Twitter chat hashtag creates a running stream of tweets so that everyone can follow the “conversation.”

More About Twitter Chats • Chat hashtags are often derived from a chat sponsor’s

name (example: #wbchat for WiseBread); industry (example: #creditchat); or program title (#SSHWchat for the Small Steps to Health and Wealth™ program)

• A Twitter chat hashtag creates a running stream of

tweets so that everyone can follow the “conversation” • The formatting convention used to follow Twitter chat

threads is Q1 for Question 1 and A1 for participant responses to that question

Twitter Chat Evaluation Methods

• Online survey (e.g., Qualtrics) link embedded into final tweets (with prizes as an incentive to complete)

Dr. Barbara O'Neill @moneytalk1Apr 29 Please let us know if U found #SSHWchat helpful & take this brief survey: https://rutgers.qualtrics.com/SE/?SID=SV_9X0V77kUKhOqlWB … Will pull winners at 1:30 pm #sshwchat

• TweetReach statistics: http://tweetreach.com/ (type in chat hashtag to pull a report)

• Follow-up contact from participants (Twitter direct messages, e-mail, etc.)

• Traffic to Web site during and after a chat

• Other?

Sample TweetReach Report

#eXASchat Evaluation Results

• Four consecutive Monday evenings in February 2014 to promote America saves and saving money

• Hashtag: #eXASchat

• 69 participants

• Estimated reach: 42,838 (estimated overall reach of tweets in report)

• Exposure impressions: 249,382 (based on participants’ followers; tells how many tweets were sent by users to their followers)

• Number of tweets sent by users with > 1,000 followers (96th percentile of all Twitter users): 38

#SSHWchat Evaluation Results

• Held at noon ET on 4/29/14 for Financial Literacy Month

• Topic: Strategies to improve health and finances

• 34 participants

• Estimated reach: 25,294 (estimated overall reach of tweets in report)

• Exposure impressions: 50,553

• Six-question Qualtrics survey: 29% response rate – 100% of respondents reported learning something new

– 100% of respondents reported planned behavior changes

• Comment: “Please have one monthly, if possible”

Archive a Twitter Chat With Storify and Track Use

https://storify.com/ Storify is an organized collection of tweets with links, photos, etc. to “tell a story” from Twitter content Video: https://www.youtube.com/watch?v=l9iHniFjiVc

Storify Samples: https://storify.com/RutgersNJAES/small-steps-to-health-and-wealth https://storify.com/RutgersSEBS/cook-douglass-community-day-2014-at-rutgers https://storify.com/wisebread/how-are-you-saving-for-retirement https://storify.com/JerryBuchko/mcpd2014-twitter-chat-archive-1

One Last Step: Share Your Program Evaluation Results

WHAT?

SO WHAT?

NOW WHAT?

Who Needs to Know About the Results of

Your Financial Education Programs? Why?

Tips for Telling Your Story • Use simple descriptive statistics (e.g., counts,

percentages, and averages)

• Don’t use jargon

• Don’t overstate your results (e.g., causality)

• Blend quantitative and qualitative data

• Clearly describe who the results represent (i.e., demographic characteristics of participants)

• Be honest about your program’s strengths and weaknesses, while highlighting the positive

Impact Statements: Intentions

As a result of participating in this financial education program, X% of participants reported that they…

• plan to do/use/adopt… • are more knowledgeable about… • are more confident in their ability to… • are more likely than before to do/use/adopt… • will do/use/adopt…

…a particular attitude, piece of information, or behavior.

Impact Statements: Actions

As a result of participating in this financial education program, X% of participants reported that they…

• are now doing… • did… • used… • increased their knowledge of… • adopted… • changed… … a particular attitude, piece of information, or behavior.

Social Media Impact Reports

• Number of participants • TweetReach report outreach numbers • Participant survey data • Data from other feedback methods

• Sample Evaluation Report:

http://www.slideshare.net/BarbaraONeill/sshw-twitter-chat-impact-statement-0414

Public Value Approach

Private benefits accrue to program participants

Public benefits are benefits to those who did not participate in a program

It is possible to exclude non-payers from participating in programs—and enjoying private benefits—but it is not possible to exclude non-payers from receiving public benefits of programs

How Do Financial Education Programs Create Public Value?

Does it narrow an information gap?

Does it address a crucial concern about fairness?

Does one person’s participation benefit people who do not participate in the program?

Does one person’s participation reduce costs for others?

Does the program improve upon market outcomes?

Program participants…

have been shown to change their behavior in specific ways…

that have been shown to lead to specific outcomes…

that directly benefit the participants.

that generate public value.

Participants in a nutrition program…

have been shown to increase their consumption of fruits and

vegetables…

which has been shown to reduce the incidence of diabetes…

which directly improves

quality of life for participants.

which reduces public health costs for all

citizens.

Participants

Changes

Outcome

Private benefits Public value

Public Value Statements • Focus on an outcome that matters to

stakeholder(s)

• Use stakeholder’s language

• Avoid jargon and empty words

• Should be short and believable

Public Value Statement Template

When you support _______________________program, participants will ______________________________, (changes) which leads to _______________________________, (outcomes) which will benefit other community members by _____________________________________________. (public value)

Activity #2 Write a public value statement for your financial education program?

Program Evaluation Resources

NEFE Evaluation Toolkit http://toolkit.nefe.org

Downloadable NEFE Financial Education Evaluation Manual

http://toolkit.nefe.org/Portals/0/Toolkit-Manual.pdf

Questions and Comments?

Barbara O'Neill, Ph.D., CFP®, CRPC

Extension Specialist in Financial Resource Management and Distinguished Professor, Rutgers University Phone: 848-932-9126

E-mail: [email protected]

Internet: http://njaes.rutgers.edu/money/

Twitter: http://twitter.com/moneytalk1