Education Policy Initiative at Carolina UNC Teacher Quality Research Initiative: Collaborating for Program Accountability and Improvement September 29, 2015 Kevin C. Bastian Director, Teacher Quality Research Initiative Education Policy Initiative at Carolina UNC Chapel Hill
33
Embed
Education Policy Initiative at Carolina UNC Teacher Quality Research Initiative: Collaborating for Program Accountability and Improvement September 29,
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Education Policy Initiative at Carolina
UNC Teacher Quality Research Initiative: Collaborating for Program
Accountability and Improvement
September 29, 2015
Kevin C. Bastian
Director, Teacher Quality Research Initiative
Education Policy Initiative at Carolina
UNC Chapel Hill
Education Policy Initiative at Carolina
The University of North Carolina at Chapel Hill
2
Outline• Background on the UNC Teacher Quality Research Initiative
• In-depth discussion on key research projects
• Individual-level data sharing and TPP improvement
Education Policy Initiative at Carolina
The University of North Carolina at Chapel Hill
3
UNC System Priorities
UNC System Overall Goal: • Preparing more, higher quality teachers and school leaders for North
Carolina Public Schools
Key Strategies to Address the Goal• Recruitment• Preparation• Beginning Teacher Support• Teacher Quality Research – Focused on outcomes/impacts
Education Policy Initiative at Carolina
The University of North Carolina at Chapel Hill
4
What is the Teacher Quality Research Initiative?
• Originated in 2008
• Partnership between the University of North Carolina General Administration, the Education Policy Initiative at Carolina (EPIC), and the 15 UNC System institutions that prepare teachers and school leaders
• Main goal: To conduct high-quality research and disseminate research evidence to improve the quality of teachers and school leaders prepared in North Carolina
• Integral component of the UNC system’s mission to prepare more and better teachers and school leaders for the public schools of North Carolina
Education Policy Initiative at Carolina
The University of North Carolina at Chapel Hill
5
Motivations for the Teacher Quality Research Initiative
• North Carolina as an accountability state
• Stalled progress in student performance and concerns about educational inequities
• Widespread attention on evidence-based reform
• Compelling evidence on the importance of teachers and teaching
Education Policy Initiative at Carolina
The University of North Carolina at Chapel Hill
6
Structure of the Teacher Quality Research Initiative
• Annual scope of research work between the UNC-GA and EPIC
• UNC Council of Education Deans Meetings—opportunities for dialogue, feedback and reflection
• Production of policy briefs, reports, and journal articles
• Benefits of working within a university system
Education Policy Initiative at Carolina
The University of North Carolina at Chapel Hill
7
Teacher Quality Research Initiative Studies
• Teacher portals analyses
• Program effectiveness analyses
• Recent Graduate Survey
• Job placement rates
• North Carolina Teaching Fellows evaluation
• Returns to teacher experience
• North Carolina New Teacher Support Program evaluation
• Graduate degrees in NC public schools
• edTPA validity and reliability studies
• NCTQ predictive validity study
• Personality traits and beginning teacher outcomes
• UNC MSA and Principal Fellows evaluations
Education Policy Initiative at Carolina
The University of North Carolina at Chapel Hill
8
Teacher Portals Analyses
• Static characteristic capturing teachers’ preparation prior to entering teaching
• Track demographic and compositional data on the NC teacher workforce
• Assess the performance of teachers entering the profession through different routes
• Value-added models: Statewide and within-school comparisons
• Teacher evaluation ratings
• Defined Portals• UNC undergraduate, graduate degree, and licensure only
• NC private undergraduate, graduate degree, and licensure only
• Out of state undergraduate, graduate degree, and licensure only
• Partnering with East Carolina University to conduct construct and predictive validity analyses on locally-evaluated edTPA data
• Locally-evaluated edTPA can provide faculty and staff:
• Common language and expectations for candidate performance
• Direct evidence about the extent to which candidates demonstrate specific knowledge and skills
• edTPA can leverage systems of continuous improvement
• Locally-evaluated edTPA may not guide TPPs to adapt/adopt more effective practices if the data are not valid and predictive of outcomes for program graduates
Education Policy Initiative at Carolina
The University of North Carolina at Chapel Hill
21
Construct Validity of Locally-Evaluated Portfolios
Planning for Content Understanding 0.73 0.22 -0.11
Planning to Support Varied Student Learning Needs 0.72 -0.04 0.18
Using Knowledge of Students to Inform Teaching 0.45 0.01 0.35
Identifying and Supporting Language Demands 0.77 -0.06 0.12
Planning Assessment to Monitor and Support Student Learning 0.69 0.16 -0.02
Instruction
Learning Environment 0.20 0.61 -0.03
Engaging Students in Learning -0.01 0.82 0.04
Deepening Student Learning 0.03 0.76 0.07
Subject-Specific Pedagogy 0.01 0.61 0.19
Analyzing Teacher Effectiveness 0.03 0.18 0.62
Assessment
Analysis of Student Learning 0.03 0.11 0.73
Providing Feedback to Guide Further Learning 0.04 0.03 0.71
Student Use of Feedback -0.09 0.02 0.89
Analyzing Students’ Language Use 0.14 -0.02 0.72
Using Assessment to Inform Instruction 0.13 0.03 0.73
Education Policy Initiative at Carolina
The University of North Carolina at Chapel Hill
22
Predictive Validity of Locally-Evaluated Portfolios
edTPA Measures Std. EVAAS Estimate
Planning factor 0.063
Instruction factor 0.213**
Assessment factor 0.178*
Planning for Content Understanding 0.121
Planning to Support Varied Student Learning Needs 0.050
Using Knowledge of Students to Inform Teaching 0.029
Identifying and Supporting Language Demands 0.092
Planning Assessment to Monitor and Support Student Learning 0.102
Learning Environment 0.156+
Engaging Students in Learning 0.271**
Deepening Student Learning 0.210**
Subject-Specific Pedagogy 0.229*
Analyzing Teacher Effectiveness 0.134
Analysis of Student Learning 0.124
Providing Feedback to Guide Further Learning 0.194**
Student Use of Feedback 0.195*
Analyzing Students’ Language Use 0.215*
Using Assessment to Inform Instruction 0.154+
Standardized Total Score 0.184**
Cases 209
Education Policy Initiative at Carolina
The University of North Carolina at Chapel Hill
23
Predictive Validity of NCTQ Ratings
• Established formal collaboration with NCTQ in spring 2014
• Assess the relationship between NCTQ’s TPP ratings and measures of teacher performance in North Carolina
• Study premise: An underlying justification for standards for teacher preparation programs is that meeting the standards should lead to higher quality preparation practices, better teacher performance, and better student outcomes
• If premise holds, TPP have an incentive to undertake reforms to increase their scores/ratings on standards
• If premise does NOT hold, TPP efforts could be better spent undertaking reforms that do lead to improve teacher and student outcomes
Education Policy Initiative at Carolina
The University of North Carolina at Chapel Hill
24
Research Background for NCTQ Analyses
• Full sample includes 4,513 first and second year teachers in 2011-12 and 2012-13 in NC public schools
• Ran overall program rating, standards, and indicators models
• Ran models for 1st year teachers only and for 1st and 2nd year teachers combined
• Took three approaches to address missing data—case-wise deletion, multiple imputation, and dummy variable replacement
Education Policy Initiative at Carolina
The University of North Carolina at Chapel Hill
25
Summary of NCTQ Results
NCTQ Ratings
Associations with Teacher Performance MeasuresTeacher Value-Added Teacher Evaluation Ratings
Total Tests Significant Positive Associations
Significant Negative
AssociationsTotal Tests Significant Positive
AssociationsSignificant Negative
Associations
NCTQ Overall Program Rating 42 1 0 30 8 0
Totals with NCTQ Standards 124 15 5 140 31 23
Selection Criteria 14 3 0 10 8 0
Early Reading 4 0 0 10 0 0
English Language Learners 4 0 0 10 8 0
Struggling Readers 4 0 0 10 0 1
Elementary Mathematics 4 0 0 10 0 0
Elementary Content 4 0 1 10 9 0
Middle School Content 4 0 1 10 0 8
High School Content 6 0 0 10 0 1
Classroom Management 14 0 0 10 0 10
Lesson Planning 14 0 0 10 0 3
Assessment and Data 14 4 2 10 0 0
Student Teaching 14 2 1 10 0 0
Secondary Methods 10 0 0 10 6 0
Outcomes 14 6 0 10 0 0
Education Policy Initiative at Carolina
The University of North Carolina at Chapel Hill
26
Limitations and Takeaways
• Limitations: Generalizability, missing data, and conceptual alignment between NCTQ standards and teacher outcome variables
• Takeaways
• In NC there is not a strong relationship between NCTQ’s ratings and meeting their standards and the performance of teacher preparation program graduates
• Results do not suggest that teacher preparation is unimportant or that the preparation components rated by NCTQ are unimportant
• Results suggest creating outcome-rich environments for preparation programs, setting higher standards for admission into programs, and continuing to identify ways to measure program quality
Education Policy Initiative at Carolina
The University of North Carolina at Chapel Hill
27
From Accountability to Program Improvement Research
• Current research may encourage programmatic reforms and help teacher preparation programs better consumers of evidence but it CANNOT formatively drive evidence-based improvements
• With what data can teacher preparation programs make evidence-based improvements?
• Teacher preparation programs need individual-level data on program graduates
• Helps teacher preparation programs determine whether preparation practices are aligned with the working environments of their graduates
• Helps teacher preparation programs assess how variation in graduates’ preparation experiences explain variation in their characteristics and performance
Education Policy Initiative at Carolina
The University of North Carolina at Chapel Hill
28
Data Sharing in the UNC System
• Initiative designed to stimulate a culture of evidence and program improvement by providing teacher preparation program with individual-level data on their program graduates
• In the UNC system we are providing teacher preparation programs with separate data files per school year, with each file containing data on all the individuals who were initially prepared to teach by a teacher preparation and employed as teachers in that school year
Education Policy Initiative at Carolina
The University of North Carolina at Chapel Hill
29
What Data are Being Shared?
Employment Status Teacher Characteristics Classroom Characteristics School Characteristics Teacher Outcomes
1. District and school2. Number of pay
periods 3. First pay period4. Last pay period5. Amount of time