A web-based tool for assessing and improving the usefulness of community health assessments

Web-based Tool for Assessing and Improving the Usefulness of

Community Health AssessmentsMichael A. Stoto, Georgetown University

With Susan Straus, RAND; andCate Bohn and Priti Irani, NYSDOH

Assessment Initiative Conference Atlanta, GA, August 22, 2007

Stoto 2007 AI Conference 2

Background

• Project goal: Develop a web-based tool to assess and improve the usefulness of Community Health Assessments (CHA) in New York State and partners

• Partners– New York State Department of Health

Public Health Information Group– RAND Health

• Supported by CDC Assessment Initiative

Stoto 2007 AI Conference 3

Project Tasks

• Literature review of studies assessing the CHA process to identify factors/criteria for a useful CHA (www.rand.org/publications/TR/TR314/)

• Build consensus among stakeholders on the definition of “usefulness” of CHAs

• Develop and pilot test a web-based tool to– Assess the usefulness of CHAs to users– Feed the results back to CHA developers

• Refine, roll out, and evaluate the web-based tool to assess the usefulness of CHAs

Stoto 2007 AI Conference 4

Development of Criteria

• From literature review, created list of 85 criteria and 2 open-ended questions

• Obtained feedback from NY and national experts and practitioners

• Reduced list to 32 close-ended criteria in 3 categories plus 3 open-ended questions– CHA content, format, and impact

• Conducted e-mail survey of 5 NY counties• Reduced to 21 close-ended criteria in same

3 categories, plus 3 open-ended questions

Stoto 2007 AI Conference 5

CHA Document Content1. Clearly states the goals and purpose of the CHA2. Includes the most important aspects of the

community’s health3. Allows comparisons with data from other

communities or other appropriate benchmarks4. Allows comparisons over time 5. Presents data in meaningful subgroups of

population (e.g. to assess health disparities)6. Provides sufficient focus on positive

characteristics, e.g., community assets, as well as negative characteristics, e.g., death rates

7. Sufficiently documents the process and methods used to create the CHA

Stoto 2007 AI Conference 6

CHA Document Format8. Uses a consistent format to present information on

different topics9. Includes both summary and detailed versions to be

useful for a variety of audience10. Is well organized; it is easy to find content 11. Is easy to understand 12. Clearly indicates the relationships among related

indicators 13. Includes narrative and graphic representation of key

findings to meet the needs of varying audiences 14. Uses a similar structure or data elements as other

community planning tools in use 15. a) Is available online; b) includes appropriate links 16. Can be reproduced easily by photocopy 17. Clearly identifies data sources

Stoto 2007 AI Conference 7

CHA Document Impact18. Serves as a resource to prioritize and plan

services19. Serves as a resource for writing grant

applications20. Serves as a resource to guide a comprehensive

health promotion strategy

Open Ended Questions21. Are there any indicators that are not useful?

Why? How could they be made useful?22. Are there any useful indicators that are not

included in the CHA? What are they? Why?23. How will you use information from the CHA?24. Comments about this web-based survey

Stoto 2007 AI Conference 8

Pilot Test Procedures

• Five NY “counties”– Cortland, Dutchess, Monroe, Schenectady – Tri-County area: Clinton, Essex, and Franklin

• Process– Counties identified actual and potential users– County representatives sent e-mail invitation– RAND sent follow up e-mail with password– County representatives sent follow-up e-mail

• Response rate: 39% (70/180)

Stoto 2007 AI Conference 9

CHA Document Content

Stoto 2007 AI Conference 10

Statistical properties• Scales are Reliable

– Coefficient α

indicates whether responses to items on a scale are conceptually related

• Users Find Survey Easy to Use– “This survey has been easy to complete. It is

well thought out and presented. I am looking forward to receiving feedback on our Community Health Assessment.”

– Very easy and user friendly. Easy to navigate and respond to the questions.”

– “One of the better ones I have done--ease to use and follow. Thanks.”

Stoto 2007 AI Conference 11

Average Score by Question (1/3)

CHA Document Content 3.551. Clearly states goals and purpose 3.552. Includes important aspects of health 3.723. Allows comparisons with appropriate 3.524. Allows comparisons over time. 3.645. Presents data in meaningful subgroups 3.556. Focus on positive characteristics 3.437. Documents process and methods 3.43

Stoto 2007 AI Conference 12

Average Score by Question (2/3)

CHA Document Format 3.298. Uses a consistent format 3.759. Includes summary and detailed versions 3.4310. Well organized. 3.6911. Easy to understand. 3.6612. Indicates relationships among indicators 3.2113. Includes narrative and graphics 3.4614. Similar to other community planning tools 3.3816. Can be reproduced by photocopy 3.6517. Clearly identifies data sources 3.75

Stoto 2007 AI Conference 13

Average Score by Question (3/3)

Impact of the CHA Document 3.7018. Resource to prioritize and plan services 3.7319. Resource for writing grant applications 3.7520. Resource to guide health promotion 3.61

CHA Document Content 3.55CHA Document Format 3.29Impact of the CHA document 3.70

Average Score by Category

Stoto 2007 AI Conference 14

Average Score by County (NY Pilot)

2.75

3.00

3.25

3.50

3.75

4.00Q

1

Q2

Q3

Q4

Q5

Q6

Q7

Con

tent Q8

Q9

Q10

Q11

Q12

Q13

Q14

Q16

Q17

Form

at

Q18

Q19

Q20

Impa

ct

CHA Content CHA Format CHA Impact

Ave

rage

sco

re

Overall mean Cortland Dutchess Monroe Schenectady Tri county

Stoto 2007 AI Conference 15

Overall Results• Most useful aspects of CHA’s reviewed

Content2. Includes important aspects of health (3.72)

Format8. Uses a consistent format (3.75)

Impact18. Resource to prioritize and plan services (3.73) 19. Resource for writing grant applications (3.75)

• Higher ratings by those involved in preparing the CHA

• No or small differences by affiliation

Stoto 2007 AI Conference 16

Sample Open Ended ResponsesAre there any indicators in the CHA that are not useful? Which ones? Why? How could they be made useful? (21 responses)– “Everything in the CHA is quite useful”– DWI arrests and cirrhosis are poor measures of

substance abuse– Break down birth indicators by age of mother– Break out data for the City vs. rest of county– Include break down for all immigrant groups– Include community input data– Indicate whether change is due to change in health

status or better/worse reporting

Stoto 2007 AI Conference 17

Sample Open Ended ResponsesAre there any useful indicators that are not included in the CHA? What are they? Why? (24 responses)– “I found it to be entirely comprehensive”– Substance abuse and related behaviors– Communicable disease rates (HIV, STD, etc.) – Environmental health– Mental health, dementia– Domestic violence, sexual assault– Intendedness of pregnancy– Differentials in vulnerable populations– Lack of health insurance– Poverty, homelessness, crime

Stoto 2007 AI Conference 18

Sample Open Ended Responses

How will you use information from the CHA in developing, implementing, and evaluating your programs? (45 responses)– Planning (e.g. primary health care, services for

baby boomers, Consolidated Services Plan, Communities that Care, etc.)

– Prioritize goals and objectives– Advocacy (Coalition findings regarding youth)– Grant proposals (identify areas of unmet need,

state breast and cervical cancer grant)– Program management and improvement (MAPP)– Source of information for comparisons

Stoto 2007 AI Conference 19

Sample Feedback Report: High scoring CHA

2.75

3.00

3.25

3.50

3.75

4.00Q

1Q

2Q

3Q

4Q

5Q

6Q

7C

onte

nt Q8

Q9

Q10

Q11

Q12

Q13

Q14

Q16

Q17

Form

at

Q18

Q19

Q20

Impa

ct

CHA Content CHA Format CHA Impact

Ave

rage

sco

re

Overall mean Tri county

Stoto 2007 AI Conference 20

Sample Feedback Report : Low scoring CHA

2.75

3.00

3.25

3.50

3.75

4.00Q

1Q

2Q

3Q

4Q

5Q

6Q

7C

onte

nt Q8

Q9

Q10

Q11

Q12

Q13

Q14

Q16

Q17

Form

at

Q18

Q19

Q20

Impa

ct

CHA Content CHA Format CHA Impact

Ave

rage

sco

re

Overall mean Dutchess

Stoto 2007 AI Conference 21

2006: National roll out

• Locations: Nassau (NY), Dallas (TX), DeKalb (GA), Greenwood (SC)

• Limitations addressed: Ability to differentiate strengths and weaknesses of individual CHAs limited by sample size and ceiling effects

– Find a way to increase n for each county• Response rate: 32% (40/126)

– 39% (70/180) in pilot test– Revise question format to get more

precision and reduce ceiling effects• Increase number of response categories

from 4 to 6

Stoto 2007 AI Conference 22

Pilot and National Average Scores

4.6

4.8

5

5.2

5.4

5.6

Q1

Q2

Q3

Q4

Q5

Q6

Q7

Q8

Q9

Q10

Q11

Q12

Q13

Q14

Q16

Q17

Q18

Q19

Q20

CHA Content CHA Format CHA Impact

Ave

rage

sco

re

Adj. 2005 mean 2006 mean

Stoto 2007 AI Conference 23

Average Score by County (National)

3.5

4.0

4.5

5.0

5.5

6.0

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q16 Q17 Q18 Q19 Q20

CHA Content CHA Format CHA Impact

Ave

rage

sco

re

Dallas DeKalb Greenwood Nassau 2006 mean

Stoto 2007 AI Conference 24

National roll out results

• Increase in number of scale categories and loss in sample size operate in opposite directions

– Need larger samples of users!• “National” average similar to NY pilot after

adjusting for number of categories– Unexpected, given differences in state

CHA formats• No significant differences in usefulness

between counties in the national sample

Stoto 2007 AI Conference 25

Conclusions• Literature review and CHA user process developed

– Consensus criteria for CHA usefulness – Easy-to-use web-based tool to obtain feedback

• Users rate NY CHAs favorably– May have improved since 1998 critique

• More similarities across communities than differences– Lack of significant differences limits value of

feedback• Low participation rate

– Resulting small sample limits reliability– May reflect

• Logistical difficulties• Lack of interest in CHAs• Lack of a community involvement process