West Coast Data Conference 2016: Las Vegas Toward a Measure of Mission Transparency Mark A. Hager Rodney Machokoto Chia-Ko Hung Terry Driscoll Arizona State University
West Coast Data Conference 2016: Las Vegas
Toward a Measure of Mission Transparency
Mark A. Hager Rodney Machokoto
Chia-Ko HungTerry Driscoll
Arizona State University
West Coast Data Conference 2016: Las Vegas
Gist of the Project Nonprofits vary on their level
of reporting “transparency” on Form 990.
So? Pressures Transparency Transparency Donor
decisions
West Coast Data Conference 2016: Las Vegas
Form 990
West Coast Data Conference 2016: Las Vegas
“Transparency” Quality of mission statement
and program descriptions Completeness of narrative
and financial fields Reporting on form rather
than attachments
West Coast Data Conference 2016: Las Vegas
Data 200 public charities drawn
from BMF with fiscal years ending in first half of 2011 and at least $200,000 in reported revenues.
Currently working on full coding of approximately 1200 cases.
West Coast Data Conference 2016: Las Vegas
Toward a Transparency Index Part I
Description
Max valu
e Quality of description of mission (Pt1) 5Reports number of board members (Pt1) 1Reports number of employees 1Reports number of volunteers 1Reports revenue streams 3Reports expense category totals 3Reports net assets or fund balance 3 17
West Coast Data Conference 2016: Las Vegas
Toward a Transparency Index Part III
Description
Max valu
e
Part 3, Line 1Quality of description of mission (Pt3) 5
Part 3, Line 4a
Quality of description of program service 5
Part 3, Line 4a Expenses
Reports primary program expenses 2
Part 3, Line 4a Revenues
Reports primary program revenues 2 14
West Coast Data Conference 2016: Las Vegas
Toward a Transparency Index Part VI
Description
Max valu
ePart 6, Line 1a Reports number of board members (pt6) 1
Part 6, Line 1bReports number of independent board members 1
Part 6, Line 8aReports whether board meetings are documented 1
Part 6, Line 12a Reports whether organization has conflict of interest policy 1
Part 6, Line 13Reports whether organization has a whistleblower policy 1
Part 6, Line 14 Reports whether organization has a document destruction policy 1
Part 6, Line 15aReports whether organization has independent review of compensation of CEO 1
Part 6, Line 15bReports whether organization has independent review of compensation of other key employees 1
Part 6, Line 18 Public availability of Form 990 3 11
West Coast Data Conference 2016: Las Vegas
Toward a Transparency Index Part VII
Description
Max valu
ePart 7, Line 1 et al
Lists officers, directors, trustees, and key employees 5
Part 7, Line 1 et al, Col. D Reports their compensation 5
Part 7, Line 1 et al
Lists board members 3
Part 7, Line 2
Reports number of individuals compensated more than $100,000 1
14
West Coast Data Conference 2016: Las Vegas
Description
Max valu
ePart 8, Line 1a Reports revenues from
federated campaigns 1Part 8, Line 1b
Reports revenues from membership dues 1
Part 8, Line 1c
Reports revenues from fundraising events 1
Part 8, Line 1d Reports revenues from related
organizations 1Part 8, Line 1e
Reports revenues from government grants 1
Part 8, Line 1f Reports other contributions 1
Part 8Revenues reported in relevant sections and generally add to total 2 8
Toward a Transparency Index Part VIII
West Coast Data Conference 2016: Las Vegas
Description
Max valu
e
Part 9Expenses reported in relevant sections and generally add to total 10
Part 9, Col. D
Penalty (-8) for organizations reporting substantial contributions and no fundraising expenses
010
Toward a Transparency Index Part IX
West Coast Data Conference 2016: Las Vegas
Description
Max valu
ePart 10, Lines 1-16
Assets reported on relevant lines and generally add to total 2
Part 10, Lines 17-26
Liabilities reported on relevant lines and generally add to total 2 4
Part 12, Line 2aReports whether financial statements had review by independent accountant 1
Part 12, Line 2bReports whether financial statements had audit by independent accountant 1 2
Toward a Transparency Index Parts X and XII
West Coast Data Conference 2016: Las Vegas
Distribution: 75 test cases
Range: 39 to 70
West Coast Data Conference 2016: Las Vegas
Coding Mission Statements
The Transparency Index asks:“How helpful is this mission statement?”
West Coast Data Conference 2016: Las Vegas
Coding Mission Statements Skipping the question is not
transparent If present, coders rate from 1
to 5:1: Very general; unhelpful2:3:4:5: Detailed; thorough
West Coast Data Conference 2016: Las Vegas
JK rates this a 1 Terry rates this a 4
West Coast Data Conference 2016: Las Vegas
Terry rates this a 1 Rodney rates this a 5
West Coast Data Conference 2016: Las Vegas
Inter-rater Reliability Degree of agreement among
raters Different approaches to
measurement:- Joint probability of
agreement- Cohen’s kappa, Fleiss’
kappa- Correlation - Limits of agreement- Krippendorf’s alpha
West Coast Data Conference 2016: Las Vegas
Krippendorf’s alpha
Kα = 1 – Not an option in standard
statistical packages Andrew Hayes has
publically available macros for SAS, SPSS, & Mplus
observed disagreement
chance disagreement
West Coast Data Conference 2016: Las Vegas
Krippendorf’s alpha DeSwert 2012: “Kα =.80 is
often brought forward as the norm for a good reliability test, with a minimum of .67 or even .60.”
Kα for P1 mission statement, 200 test cases: 0.592
West Coast Data Conference 2016: Las Vegas
DV: Variance in raters; n=75
M1 M2Expenditures (ln) * *% Revenues from Contributions nsTransparency Index nsTI: Part 1 nsTI: Part 3 nsTI: Part 6 nsTI: Part 7 - **TI: Part 8 nsTI: Part 9 nsConstant ns ns
West Coast Data Conference 2016: Las Vegas
Complexity = Variance The organization size
increases, variance in mission transparency assessment increases.
West Coast Data Conference 2016: Las Vegas
Opacity = Variance
As transparency of officer participation and compensation decreases, variance in mission transparency assessment increases.
West Coast Data Conference 2016: Las Vegas
Next Step 1 Finish all the data entry and
coding.
West Coast Data Conference 2016: Las Vegas
Next Step 2Come to grips with inter-rater variation Old school: Train coders
better, discuss and “fix” variant cases.
Alternative: Accept rater variance as part of the reality of user assessments. Ie, donors are going to differ in their assessments, too.
West Coast Data Conference 2016: Las Vegas
Next Step 3 Bring ratings for missions
(part 1) and programs (part 3) into Transparency Index.
Press forward on larger research agenda with the Index.
West Coast Data Conference 2016: Las Vegas
Thanks!Mark A. Hager
Rodney, JK, and TerrySchool of Community Resources & DevelopmentArizona State University