MASN, Evaluation, and PQAs • MASN Overview – Systems building • Quality Framework • Training and Technical Assistance – Site Visits – Quality Action Plans • Evaluation – Statewide Evaluation – Local/External Evaluation • Data Collection – Survey data collection – Kids Care Center data – PQA data collection
49
Embed
MASN, Evaluation, and PQAs MASN Overview – Systems building Quality Framework Training and Technical Assistance – Site Visits – Quality Action Plans Evaluation.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
MASN, Evaluation, and PQAs
• MASN Overview– Systems building
• Quality Framework• Training and Technical Assistance
Provides training and technical assistance to 21st CCLC and SAC grantees.
Supports successful implementation of grants and quality improvement process.
Technical Assistance
• 2 Program Visits per grantee– Program Visit Discussion Checklist– Program Level Action Plan
• 2 Site Visits per site (many grants have multiple sites)– Site Visit Discussion Checklist, Focused
Observation, Feedback– Site Level Action Plan
Training
• Advanced Planning with Data– Fall after PQA (Y2, Y4)
• Four clock-hour webinars this Spring– Parent and Family Involvement– Community Partnerships– Marketing Your Program– Afterschool Enrichment
• Methods Trainings– Based on action plans developed at the Advanced Planning
with Data session– This year, one training per grant (Cohort 6 and 7)
• Cohort 8 and SAC may attend if space available
Evaluation
Statewide and Local (Outside) Evaluation
Evaluation Work
• MASN is coordinating the new Statewide and Local Evaluation and processes– Sub contracting with Weikart Center and OSEDA
for survey development, data analysis, and report writing
• Quality Committee - looking at the resources being available for all programs
– Providing training/certification of external evaluators
Coordinated Evaluation
• Local evaluation parallels statewide evaluation– Same data and framework for all grantees
• Same data used for quality improvement efforts and evaluation efforts
PQA/Survey Report Fee
• $1500 per site to 4-H Center for Youth Development– Data:
• PQA(s)• Surveys
– Reports: • PQA(s)• Weikart Center Leading Indicators (Goal 2)• Missouri College & Career Readiness (Goals 1 & 3)
Logic Model
Afterschool Program Quality
Afterschool Program Quality
Youth Outcomes
Youth Outcomes
College and
Career Readiness/Success
College and
Career Readiness/Success
AS Staff Skills
AS Staff Skills
AS Program Structure
AS Program Structure
Training
Training
Coaching
Coaching
Self-assessment
Self-assessment
Goals
• Goal 1: Support or increase student achievement and sense of competence in the areas of reading/communication arts, mathematics, and science.
• Goal 2: Develop and maintain a quality program that includes a safe and supportive environment, positive interactions, and meaningful opportunities for engagement.
• Goal 3: Enhance youth’s college and career readiness skills and behaviors, including positive school behaviors, personal and social skills, and commitment to learning.
Data Sources used for the Statewide Evaluation
• Program Quality Assessments (PQAs)• Surveys • Kids Care Center (KCC) Data• DESE Core Data
Program Quality Assessments (PQA)
• Developed by the Weikart Center for Youth Program Quality– School Age Program Quality Assessment (SAPQA)
Walk-Through Method– Youth Program Quality Assessment (YPQA), with
STEM Add-on• On-site assessment by trained, reliable
external assessors
Surveys
• Survey links sent by e-mail March 2015 to grant contact– One per site – coded with Org ID so it is important to
make sure to use the correct links• Paper survey available for parents and Spanish
parent/youth version• Youth survey requires a student identifier and
“opt out” consent• Weekly e-mail with number of completed surveys
• Calculated multiple regressions predicting maintenance/increase of grades in three subject areas.
• All youth scales and demographic variables were entered.• Significant predictors shown below.
Reading (n = 3383) Math (n = 3351) Science (n = 2656)Time 1 reading grade (lower grade more likely)
Time 1 math grade (lower grade more likely)
Time 1 science grade (lower grade more likely)
Race (nonminority more likely) Grade level (younger children more likely) Gender (girls more likely)Grade level (younger children more likely)
Work Habits scale (higher scores more likely) Grade level (younger children more likely)
Work Habits scale (higher scores more likely)
Math Efficacy scale (higher scores more likely)
Work Habits scale (higher scores more likely)
Youth Engagement and Belonging scale (higher scores more likely)
Technology Efficacy scale (higher scores more likely)
Math Efficacy scale (higher scores more likely)
Statewide Evaluation
• Required by Federal Funding• Used to:
– Refine the grant requirements– Determine training and technical assistance needs– Plan for systems level changes or new development that
will support afterschool sites– Use aggregate data to communicate the importance and
impacts of afterschool
Tying it all Together
• Statewide and Local Evaluation
Afterschool Program Quality
Afterschool Program Quality
Youth Outcomes
Youth Outcomes
College and
Career Readiness/Success
College and
Career Readiness/Success
AS Staff Skills
AS Staff Skills
AS Program Structure
AS Program Structure
Training
Training
Coaching
Coaching
Self-assessment
Self-assessment
Goal 2Goal 3
Goal 1
Youth Program Quality Improvement (YPQI)
SurveysKCC
Core DataPQAs
Program Level Action PlanSite Level Action Plan
(Advanced Planning with Data)
Coaching VisitsMethods Trainings
SummitsConferences
Other Trainings
Instrument SelectionConsent
SchedulingReports
Coaching
PQA Specifics
Program Quality Assessments
• Developed by the Weikart Center for Youth Development
• Research based, reliable, and valid tool for self-assessment and external evaluation
• Used in many states and localities
l
Light Green- full-state implementation
Dark Green- place-based implementation
Gold- full-state + place-based
YPQI Quality Improvement Systems
Program Quality Assessments
• Four domains– Multiple
scales in a domain
• Multiple items in a scale
Ones, threes, fives, Oh My!!!
• 1 – Not observed on the day of the observation
• 3– Was observed at least once on the day of the
observation – Examples: once, not all youth, some youth
• 5– Was observed multiple times on the day of the
observation– Examples: all youth had opportunities
PQA Protocols
• School Age Walk Through (SAPQA)– K – 6th Grades– From start of programming, through departure– Limit of 3 hours – Welcome, multiple activities, educational offering,
snack, transitions, departure• STEM Program Quality Assessment (STEM PQA)
– 4th – 12th Grades– Two offerings – one STEM, other on same day
PQA Process
• Schedule the assessment• Confirm with program and assessor• Assessment takes place• Data Entry (2-4 weeks)• Quality Control• Report – available in Weikart Center Scores Reporter • Discuss PQA results with Coach at Program and Site Visits• Advanced Planning with Data – set site level action plans
(Sept/Oct)• Methods Trainings
During the Visit• Please tell your staff that someone is coming to observe• Assessors will dress casually
– 4-H Extension nametag/shirt• They will be a “fly on the wall”
– Once the program starts, they will be observing and taking notes so please do not try to hold a conversation with them. They will confirm with the site director and arrive early to hear about the program/schedule for the day.
• They will go from area to area – may not see everything, but that is ok… it is a “snapshot”– 2-3 hours is usually sufficient to have a good feel for the program even if they do not
observe every specific interaction• Assessors will say goodbye to a staff member, but it may not be the site director, if
not readily available• Assessor may ask on the confirmation call or before the assessment begins which
activities you would prefer to have observed if there are multiple offerings (except if older youth STEM)
Participation Agreement
• Key important policies• Sign and return to Anne Reeder