Participatory Evaluation Mary Phillips, BME Former Circles of Care Program Coordinator, Oakland and an Evaluator, Los Angeles, CA.

Participatory Evaluation

Mary Phillips, BMEFormer Circles of Care Program Coordinator, Oakland and

an Evaluator, Los Angeles, CA

It’s Not Just an Evaluation Methodology

• “We all have a lot in common. We all care very deeply about the communities that we come from and about American Indian and Alaska Native communities across the country. We all share deep concern about the well-being and future of Indian children and families. We are all very self-sacrificing. We are in this business because of our personal beliefs and our desire to make a difference. Our personal backgrounds, our training, our education, and our professional experience all contribute to our desire to make a difference.”

• Holly-Echo Hawk, M.S., Echo Hawk and Associates

It’s Not Just an Evaluation Methodology

• Participation Evaluation is a key component of the systems of care evaluation process that informs the mission, vision and authority of the Tribe/community throughout the Circles of Care program.

• Participation evaluation allows the community to voice its cultural values and promote cultural appropriateness in systems changes development at all levels.

•

It’s Not Just an Evaluation Methodology

• The Circles of Care initiative represented one of the first opportunities for American Indian/Alaska Native AI/AN families and their communities have substantial input into the design of services to address the needs of their children, from their own understanding of these needs.

• The Standard Infrastructure Program Announcement, INF 05 PA and the Circles of Care III NOFA have included community involvement and participation measures that you MUST address throughout the application and are considered in the review process.

How it Works

• blends evaluation process components to actively involve members of the community to collect qualitative data that better informs the system.

• Having members who are knowledgeable about cultural norms of the community bared an unobtrusive context to present systems change.

The Process of Participatory Evaluation

• Planning the assessment will require a working process between the project staff and evaluation staff and the key informants of each community. Taking into consideration the current status of delivery systems for Native youth and mental health, developing the assessment tool(s) was pivotal in getting necessary information on contextual issues of the system.

• Example: Community Readiness Model was developed at the Tri-Ethnic Center for Prevention Research at Colorado State University to measure systems change in Native American and other communities.

Community Readiness Model

• An innovative methodology to evaluate systems change and focus community efforts toward realizing its objectives. It identifies characteristics related to problem awareness and readiness for change within the community, fostering sound strategic planning, ongoing feedback, and realistic assessments of accomplishments.

Community Readiness Model

• The tool focused on collective information from these Native American resources: 3 community identified leaders, 3 agency directors, 2 cultural advisors, 3 community members and 2 youth group representatives.

• Input from key stakeholders and community members was collected through interview questions related to:– Community Efforts, Knowledge of Efforts, Leadership,

Community Climate, Knowledge About the Issues, Resources for Prevention Efforts.

Community Readiness Model

• The information collected allowed our program to measure where the urban Native American community stands with respect to its knowledge and readiness to implement a specific program (Strategic Plan for Urban Native Americans and their families Oakland, CA

Ongoing Particpation

• The evaluation participation process also incorporated resource development within the community. Linkages were made throughout the project as community coordinators and cultural leaders were provided information on the core elements of the system of care. This process required continuous interaction between agencies and cultural groups that concentrated the efforts to achieve child and adolescent behavior goals among agencies and family empowerment through focus groups.

SYSTEM OF CARE MODEL FOR AMERICAN INDIAN CHILDREN & FAMILIES

ADVISORY BOARD

RECOMMENDATIONS

VISION

COMMUNITY NEEDS ASSECOMMUNITY NEEDS ASSECOMMUNITY NEEDS ASSESSMENT SSMENT SSMENT Circle of Care Evaluation Process Flow

SER

VIC

E

PR

OVID

ER

S

CO

MM

UN

ITY

SURVEY

CIRCLE OF CARE EVALUATION

KEY INFORMANT INTERVIEWS

FOCUS GROUPS

AMERICAN INDIAN COMMUNITY

PRIORITY AREAS

Access

Outreach/Awareness

Family Support Services

Training Cultural Appropriate

Youth Services

Cultural Programs Community

Needs Assessment

Service System

Assessment

YOUTH, PARENTS, SERVICE PROVIDERS

Commitment Ownership Community

Items from Standard Infrastructure Program Announcement and Circles of

Care NOFA

• Respond to each of these required information items with statements to let the reviewer know what work has been done; the work that the program intends to do; and how you will address each issue.

• Though not all requirements listed here are given points, providing this information can help understand the ability of the program to connect and involve the community.

• These should be written in Sections B. Proposed Approach, C. Staff, Management and Relevant Experience, D. Evaluation and Data

Items from Standard Infrastructure Program Announcement and Circles of

Care NOFA

• Program Goals:– Engage their community members in assessing service

system needs, gaps, potential resources, and plan infrastructure development strategies that meet those needs.

– Increase the participation of families, tribal leaders, and spiritual advisors in planning and developing service systems and treatment options based on the values and principles of the AI/AN community served by the project.

– (Circles of Care III NOFA, p.2)

Items from Standard Infrastructure Program Announcement and Circles of

Care NOFA

• Grantees must use funds to carry out the following required activities:– Facilitate culturally respectful strategic planning

activities engaging community members, key stakeholders, youth, elders, spiritual advisors, and tribal leaders to identify outcome expectations and measures; (Circles of Care III NOFA, p.3)

• In “Section D: Evaluation and Data,” add the following bullet:– Describe how project staff will work with evaluation staff to

support the evaluation effort. The program planning activities and the evaluation activities will be expected to inform each other in a constant cycle, locally and collectively. (Circles of Care III NOFA, p.12)

• The Circles of Care process values participatory, community-based outcomes that are reflected heavily in the planning process.

Items from Standard Infrastructure Program Announcement and Circles of Care NOFA

Items from Standard Infrastructure Program Announcement and Circles of

Care NOFA

• Providing a venue for the Tribal/community to receive information and respond to with appropriate audience members can include meetings at all levels within the community.

• Example: Two community-visioning meetings were conducted at 2 separate locations within the community, and were documented based on invited quests, attendance, activities planned to collect feedback, and information passed on to community participants. This type of community visioning planning resulted in setting priority areas to build a Youth Development Facility.

CCoommmmuunniittyy VViissiioonn Native American Youth Development Facility

A Visioning Meeting On Community Priorities Held J uly 28th, 2001

The Community has spoken through valuable input on community needs at the first of

several community visioning meetings for the Youth Development Facility. Compounded interests created a face for delivery services to be later realized after the facility is established. Needs assessment of the community’s priorities have reached new perspectives as participants were given the opportunity to give a direct voice on what they consider to be a model vision for youth.

Priorities focused on monies to be spent on curriculum-based programs for youth

expanding the areas of academia, computer science, technology, fitness, sports, and a centralized community location for youth social and cultural events. Developing areas in innovative directions in response to needed youth services include fine arts, an American Indian public charter high school and affordable housing.

Goals in correspondence to the our Youth Services Department are to develop and

implement youth programs through assets within the community. Building consensus on every level of program structure and to maintain youth involvement throughout this process has created positive reinforcement in intervention and prevention, a basis for youth ownership.

Community Assessment Of Priorities | Youth Development Facility

Paper Money Amount Given to Community Total $13,733,000

$2,653,000

$2,150,000

$1,810,000$1,561,000$1,160,000

$1,129,000

$1,017,000

$908,000

$785,000$560,000

1) Housing 2) Computer Training Center 3) Gymnasium

4) AIPCS High School 5) Cultural Arts Center 6) Youth Business Center

7) Outdoor Space 8) Kitchen 9) Multipurpose Community Room

10) Fitness Center

Participatory Feasibility Assessment

• Complete the feasibility assessment and process evaluation, leading to adoption of the model. (Circles of Care III NOFA, p.4)– Strengthening Tribal capacity for evaluation of service systems

through ongoing involvement with staff and evaluation partners.

– Careful deliberation on setting up effective strategies with an “evaluation team”, to engage all participants that have a stake in the evolution of the infrastructure.

– This integration of tasks contributes to and OPEN style of communication that in turn enhanced the collaboration of evaluation and program staff.

Participatory Evaluation related to President’s New Freedom Commission

Goals

• The agenda of the President’s New Freedom Commission’s Report towards a transformation of mental health systems states:– 2. Mental Health Care is Consumer and Family Driven

• Involve consumers and families fully in orienting the mental health system toward recovery. (Circles of Care III NOFA, p.5)

• The consumer of the community has a voice in the process and is able to tell their story at the point in the process they choose.

• Parents are empowered through ongoing interaction in the process.

Participatory Evaluation related to President’s New Freedom Commission

Goals

• Circles of Care approach emphasizes flexibility as stakeholders shift positions and encourage converge of a the cultural knowledge base within children’s mental health.

• Example: A parent that attends parent focused support groups from the AI/AN clinic and has a child receiving services that is also attending the school in the same community will better inform on how services are impacting the service needs and accessibility.

Participatory Evaluation related to President’s New Freedom Commission

Goals

• The outcome of involving parents can result in a cultural definition of Severely Emotionally Disturbed:

– “Emotional disturbance is a temporary disharmony often involving the family, school, and community, which may affect the mental, physical, spiritual, and/or emotional well being of its members.” Choctaw Nation of Oklahoma, Circles of Care I grantee.