Managing Director Impact and Performance Assessment Department (IPAD) UJA-Federation New York October 28, 2015 Judith Samuels, PhD Measuring our Impact: State-of-the-Art Approaches to Evaluation Seminar at Meyers-JDC-Brookdale
Jan 21, 2018
Managing Director
Impact and Performance Assessment Department (IPAD)
UJA-Federation New York
October 28, 2015
Judith Samuels, PhD
Measuring our Impact:
State-of-the-Art Approaches to Evaluation
Seminar at
Meyers-JDC-Brookdale
HEADLINE HERE: PROSE CONTENTAgenda
• Our work
• Philosophy
• Methods
• Tools
• Models
IPAD Team: Jennifer Rosenberg, Jen Simbol, Alex Bebeshko
HEADLINE HERE: PROSE CONTENT
1. Develop and implement our impact measurement strategy
- Top-down approach starting with UJA-Federation NY mission
- Unified approach across all grants (over 1,000) to all grantees (over 400)
- Impact as related to MISSION
- Organization’s mission
- Commission/Department’s mission
- Initiative's mission
- Specific grant’s mission
IPAD works in 5 areas
HEADLINE HERE: PROSE CONTENT
1. Develop and implement our impact measurement strategy
2. Ensure every grant has proper, appropriate evaluation
- Level of sophistication of evaluation follows program/grant size
- Rethinking “chasing” the counterfactual
- Use of logic model for planning and to guide evaluations
IPAD works in 5 areas
HEADLINE HERE: PROSE CONTENT
Our Evaluation Framework:
Mission Impact
Goals Outcomes
Objectives Outputs
HEADLINE HERE: PROSE CONTENTLogic Model Template to Inform Planning:
Key (Color Code for Outcomes and Impact/Goals Boxes): Color – Impact on individual, Color – Impact on institution (agency, synagogue, etc.), Color – Impact on community
:
How to use this tool to inform your planning As you begin thinking about designing a new program/initiative, this logic model template will help you to frame your thinking on what you want to accomplish and what is the most effective and efficient strategy to do so. The logic model displays how investments link to results, and can inform planning, implementation, communication, and evaluation. When using this tool, start at the end and identify your goals and the impact you wish to have. To help you think this through, begin by asking yourself the following questions:
What is the impact we wish to have and on whom (i.e. individual, institution, community).
Do we want to see changes in an individual or a group/community’s behavior, action, attitude, perception or feeling?
Do we want to raise awareness on an important issue or for an individual or group/community to learn something new?
Do we want to see a change in an institution (i.e. the way it operates or makes decisions)? Once you have clearly defined goals/impact statements and outcomes, continue working backwards through the chart, asking yourself:
What is the best strategy or intervention (if you do not already have one) and how does it lead us to our desired goals?
What resources (i.e. financial, human, etc.) will we need to achieve it?
What are the outputs that will indicate whether or not we are on track to achieve our outcomes? Please note: when determining the appropriate output number, use your best judgement based on research from similar models in the field, or your previous experience funding similar types of projects/initiatives. Make sure that the output number is realistic in magnitude based on your goals (i.e. if you want to change the Jewish community of New York’s behavior, you likely need to reach far more than 100 people).
HEADLINE HERE: PROSE CONTENT
1. Develop and implement our impact measurement strategy
2. Ensure every grant has proper, appropriate evaluation
3. Increase the rigor & contribution of funded research studies
• ~ 40 Research/evaluation studies funded
• Expanding the pool of possible consultants, vetting along the way
• Explicit procedure to developing and reviewing RFPs
• Use of rubrics and explicit criterial to choose the best proposals
IPAD works in 5 areas
HEADLINE HERE: PROSE CONTENT
1. Develop and implement our impact measurement strategy
2. Ensure every grant has proper, appropriate evaluation
3. Increase the rigor & contribution of funded research studies
4. Move to state-of-the-art grants management & data usability
• Online grantee portal
• Grant reporting feeds directly into data warehouse
• Improved grantee relationships, dashboard capabilities
IPAD works in 5 areas
HEADLINE HERE: PROSE CONTENT
1. Develop and implement our impact measurement strategy
2. Ensure every grant has proper, appropriate evaluation
3. Increase the rigor & contribution of funded research studies
4. Move to state-of-the-art grants management & data usability
5. Provide analytic framework to assist in determining Core
(unrestricted) support to Network agencies
IPAD works in 5 areas
Well-managed,
excellent agenciesThat
Advance UJA-
Federation’s mission
Governance
Performance
Jewish Community
Building
Cooperation/Joint
Investment
Community Reach
History
JCC Pilot20% of the allocations to the JCCs
History80 % of allocations to the JCCs
Governance
25%
Performance
25%
Jewish Community Building
25%
Cooperation/Joint Investment
10%
Community Reach
15%
Weighting(Approved by the Impact
Committee)
Breakdown of Core
Allocations
Criteria Development
HEADLINE HERE: PROSE CONTENTExample of how we choose measures:Sources that Informed the Excellence Measures
Governance and Performance Measures
• Leadership Theory and Practice by Peter Northhouse
• Robinhood Leadership Assessment Tool
• Nonprofits Assistance Fund Financial Management Self-Assessment
• Assessment of Agency Executives from Nonprofit Leadership Services
• McKinsey & Co. Nonprofit Board Self Assessment Tool
• Hillel International’s Evaluation
• USAID: Leading and Managing Framework
• Community Resource Exchange Board Assessment Questionnaire
• Harvard Business Review
• NYC Mayor's Office of Contract Services Capacity Building and Oversight Unit
• Blackbaud Online Marketing Benchmarks for Nonprofits
• Independent Sector Principles for Good Governance and Ethical Practice
• Annual Hillel Assessment
HEADLINE HERE: PROSE CONTENTData Collection Methods:
• Data we have on hand
• Ie., Audited financial statements, targeted grant reports
• Surveys to Executive Directors and Board Presidents before
site visits (and additional informants as appropriate)
• Basic agency information
• Site visits – Trained Lay Leaders and Staff
• Interview guides with open-ended queries
• Interview with Executive Director, Board President, “consumers”
• Post site visit surveys
• Use of psychometrics to test reliability and validity
Governance Is the agency well managed and fiscally sound?
PerformanceDoes the agency track and evaluate program quality, outcomes, and impact? What is the agency’s
reputation for excellence overall and performance on targeted grants in particular?
Jewish Community BuildingDoes the agency inspire Jewish life and foster bonds and relationships across diverse groups of Jews
locally and globally?
Cooperation/Joint InvestmentDoes the agency meet the rules and responsibilities of being an affiliate of UJA-Federation and
contribute to the value of the network?
Community ReachHow large is the total community being served? Is the agency addressing needs in the broader
community?
History
Hillels Pilot Assessment Criteria
Criteria for Future Core Grant
Allocations
Governance Is the agency well managed and fiscally sound?
Performance
Jewish Community Building
Cooperation/Joint Investment
Community Reach
History
Governance
Leadership and Accountability
Management Practices
Financial Stability
Governance
Criteria for Future Core Grant
Allocations
Governance
PerformanceDoes the agency track and evaluate program quality, outcomes, and
impact? What is the agency’s reputation for excellence overall and
performance on targeted grants in particular?
Jewish Community Building
Cooperation/Joint Investment
Community Reach
History
Performance
User Satisfaction
& Enagement
Performance on UJA-Fed
Targeted Grants
Accreditations/
External Ratings
Program Outputs,
Outcomes and Impact(i.e. Technology
Infrastructure)
Performance
Criteria for Future Core Grant
Allocations
Governance
Performance
Jewish Community BuildingDoes the agency help Jews in need, inspire Jewish life, and foster bonds
and relationships across diverse groups of Jews locally and globally?
Cooperation/Joint Investment
Community Reach
History
Jewish Community Building
Jewish Impact
Quality/ Innovation of
Jewish Programming
Mission
Alignment(Jewish Focus)
Jewish Community Building
Criteria for Future Core Grant
Allocations
Governance
Performance
Jewish Community Building
Cooperation/Joint InvestmentDoes the agency meet the rules and responsibilities of being an affiliate of
UJA-Federation and contribute to the value of the network?
Community Reach
History
Cooperation/Joint Investment
Fundraising
Co-Branding
Exhibits Shared
Commitment to Communal Planning and
Action
Cooperation/Joint Investment
Criteria for Future Core Grant
Allocations
Governance
Performance
Jewish Community Building
Cooperation/Joint Investment
Community ReachHow large is the total community being served? Is the agency addressing
local community needs?
History
Community Reach
Use of Resources
Responsive to
Community Needs
Inclusive
Organizational
Practices(Community
Engagement)
Community Reach
HEADLINE HERE: PROSE CONTENT
Key: ▲▲ = Top Scores, ▲ = Positive/In top Quartile, ● = Neutral (Q1-Q3), ▼ = Negative/In Bottom Quartile (Below Q1) , ▼ ▼ = Below 70
Sample Individual Agency Score CardAgency: XXXX
Criteria Measure High Score
Low Score
Avg. Score
YourScore
Your Overall Criteria Score
E
X
C
E
L
L
E
N
C
E
Governance(25%)
Accountability Practices 93 61 80 62 ▼▼58Financial Stability 85 28 65 28
Leadership/Management 95 56 80 84
Performance(25%)
Accreditation/External Ratings 98 48 80 88
● 86User Satisfaction and Engagement 100 53 91 98
Program Outputs, Outcomes & Impact 90 47 72 71
Performance on UJA-Targeted Grants 92 56 83 88
M
I
S
S
I
O
N
A
L
I
G
N
M
E
N
T
Jewish Community Building
(25%)
Mission Alignment 92 46 78 77
●88Jewish Impact 98 54 83 90
Quality/Innovation of Jewish Programming 100 65 88 97
Cooperation/Joint Investment
(10%)
Exhibits Shared Commitment to Communal Planning and Action
97 55 83 96
▲ 95
Co-branding 98 45 82 97Fundraising 95 26 80 92
Community Reach(15%)
Size Within Sector 100 20 61 70
● 83Responsive to Community Needs 96 54 83 85
Inclusive Organizational Practices 100 77 85 93Your Overall Score: ● 80
FOR INTERNAL USE ONLY, NOT FOR CIRCULATION
HEADLINE HERE: PROSE CONTENTLocal Agency Assessments on a 2 year Cycle
• Hillel Pilot Assessment• In process, completion 3/2016
• To Core Allocation Committee 3/2016
• Impacting FY 2017 funding
• JCC Targeted Assessments• In process
• Overnight Camp Pilot Assessment• Model adaptations Spring 2016
• Assessment period summer 2016
• JCC Assessment (round 2)