Milwaukee Mathematics Partnership Program Evaluation MSP Regional Conference November 8 2006
Dec 25, 2015
Milwaukee Mathematics Partnership
Program EvaluationMSP Regional ConferenceNovember 8 2006
November 6-7, 2006 2
MMP Goals
Comprehensive Math Framework
Distributed Leadership
Teacher Learning Continuum
Student Learning Continuum
November 6-7, 2006 3
MMP Core Partners
University of Wisconsin—Milwaukee
Milwaukee Public Schools
Milwaukee Area Technical College
November 6-7, 2006 4
Evaluation Goals
Help the MMP better serve its constituents and improve its effectiveness
Serve the broader mathematics education community through documentation and dissemination of MMP activities
November 6-7, 2006 5
Evaluation Logic ModelStudent
Achievement
Teacher Content& Pedagogical
Knowledge
Math FacultyInvolvement
Learning TeamEffort
SchoolBuy-in
TeacherInvolvement
NewCourses
DistrictBuy-in
MPA Ownership
MATCBuy-In
UWMBuy-In
ClassroomPractice
MMPActivities
ProximalOutcomes
DistalOutcomes
November 6-7, 2006 6
2005-06 Evaluation Activities
MMP Online Survey MTS Survey Learning Team Observations Classroom Observations Assessment of Teacher MKT Social Network Analysis MPS Data Mining
November 6-7, 2006 7
Presentation Overview
Part I: District Wide Analysis
Part II: School Case Studies
November 6-7, 2006 8
Part I: District Wide Indicators
StudentAchievement
Learning TeamEffort
SchoolBuy-in
TeacherInvolvement
ClassroomPractice
November 6-7, 2006 9
Part I Activities
MMP Survey Designed to measure differences in the
quantity and quality of MMP related activities MTS Survey
Designed to measure how well MTS perceived school to be, in terms of meeting the goals of the MMP
MKT Assessment Designed to assess teachers’ mathematical
knowledge for teaching WKCE Tests
Designed to assess students’ mathematical content knowledge
November 6-7, 2006 10
Respondents
Number of MPS Respondents by Role in the MMP
Academic Year
2004-05 2005-06
Math Teacher Leader 124 140
Learning Team Member & Mathematics Teacher
167 284
LT Member (Administrative) 80 171
Math Teacher Only 676 1340
Literacy Coach 47 94
Total 1094 2029
November 6-7, 2006 11
Research Questions
1. Validity of MMP Survey2. Change in MMP Impact3. Characteristics of math-focused schools4. MMP Impact on student achievement5. Characteristics of quality learning teams6. Characteristics of quality MTLs
November 6-7, 2006 12
Research Question #2
Has the perceived impact or focus of the MMP changed since last year?
Analytical Approach Dependent t-tests conducted at the
school level for all school level variables obtained in both administrations of the MMP Survey
November 6-7, 2006 13
Results – Trends in Impact of MMP
This year statistically significant increases
MTLs reported discussing mathematics with others at their school (t(90) = 12.06, p < .001)
Teachers reported engaging in activities designed to align their curriculum to standards (t(111) = 8.53, p < .001)
Teachers reported engaging in activities designed around CABS or student work samples (t(106) = 7.04, p < .001)
November 6-7, 2006 14
Research Question #3
What variables characterize schools that are more focused on increasing student achievement in mathematics?
Analytical Approach Stepwise multiple regression
November 6-7, 2006 15
Results – Characteristics of Schools with Greater Math Focus
68% of variability in schools’ overall self-reported focus on mathematics could be explained by differences in: Teachers reported working together to improve
content and pedagogical knowledge (b = .46, t = 6.7, p < .001)
Teachers reported consistent instructional practices used at their school (b = .14, t = 2.4, p = .018)
Teachers perceived the Learning Team to be supportive of efforts to improve math teaching and learning (b = .38, t = 5.6, p <.001)
November 6-7, 2006 16
Research Question #4
What variables help to explain differences in the percentage of students classified as proficient in mathematics?
Analytical Approach Stepwise multiple regressions
controlling for previousachievement and SES
November 6-7, 2006 17
Results – Impact of MMP on Increasing Student Achievement
Schools with a stronger focus on increasing student achievement in mathematics are have a higher percentage of students proficient in mathematics, after controlling for SES and previous achievement (b = .26, t = 3.7, p =.001)
An additional 7% of variability in student proficiency rates explained by the addition of this predictor
November 6-7, 2006 18
Research Question #5
What variables characterize Learning Teams that are perceived to be more helpful in terms of increasing student achievement in mathematics?
Analytical Approach Stepwise multiple regression
November 6-7, 2006 19
Results – Characteristics of Supportive Learning Teams
64% of variability in schools’ overall perception of the level of support provided by the Learning Team could be explained by differences in: Teachers reported working together on
improvement activities designed around CABS or student work samples (b = .41, t = 5.5, p < .001)
Teachers reported a greater alignment between their school’s adopted curriculum and standards/learning targets (b = .18, t = 2.4, p = .021)
Teachers perceived the MTL to be supportive of efforts to improve mathematics teaching and learning (b = .46, t = 5.9, p <.001)
November 6-7, 2006 20
Research Question #6
What variables characterize Math Teacher Leaders that are perceived to be more helpful, in terms of increasing student achievement in mathematics?
Analytical Approach Stepwise multiple regression
November 6-7, 2006 21
Results – Characteristics of Supportive Math Teacher Leaders
42% of variability in schools’ overall perception of the level of support provided by the MTL be explained by differences in: Teachers reported working together on
improvement activities designed around CABS or student work samples (b = .38, t = 4.5, p < .001)
Teachers reported a greater alignment between their school’s adopted curriculum and standards/learning targets (b = .26, t = 3.0, p = .004)
MTLs perceived themselves as being supported by others at their school (b = .27, t = 3.2, p = .002)
November 6-7, 2006 22
Conclusions
MTSs in general have a strong sense of what is going on with school leadership, but less awareness about activity at the classroom level.
MMP efforts are being felt beyond the learning team and MTL by school staff
MMP activities are helping schools become more focused on increasing student achievement in mathematics
Schools that are more focused have increased the proportion of students proficient in mathematics
Adoption of MMP-related principles is reported to be related to supportive learning teams and to supportive Math Teacher Leaders
November 6-7, 2006 23
Part II: Case Study Schools
StudentAchievement
Teacher Content& Pedagogical
Knowledge
Learning TeamEffort
SchoolBuy-in
TeacherInvolvement
ClassroomPractice
Collaboration
November 6-7, 2006 24
Eleven Case Study Schools
Schools were diverse in terms of
Type Geography Student demographics
November 6-7, 2006 25
Case Study Data Collection
22 learning team observations—2 in each school
44 classroom observations—4 in each school; 2 teachers observed 2 times each
MKT Assessment for math teachers
SNA Survey for matheducation ‘stakeholders’
November 6-7, 2006 26
Results of Learning Team Observations
Team Functioning
LeadershipParticipation
Organization & StructureResults
Overall Functioning
MMP Issues
Math VisionIntegration
Math LeadershipMMP Work
Overall MMP
Overall, strongest areas were participation & mathematics leadership
Biggest areas for improvement were mathvision & results
November 6-7, 2006 27
Characteristics of Hi-Lo Scoring Learning Teams—Team Functioning
Distributed leadership Positional authority is less
important Multiple views are
represented and heard Multiple segments of the
school are represented Written agenda, note taker,
facilitator Explicit action items Participants have hi
knowledge and skill levels
Principal does all the talking A few individuals dominate
the discussion No agenda or team is easily
distracted from the agenda Little follow-through on
assignments No clear action items
Hi Lo
November 6-7, 2006 28
Characteristics of Hi-Lo Scoring Learning Teams—MMP Issues
Consistent curriculum Math is addressed
alongside and in combination with other subjects
Coherent within grades and (at times) across grades
MTL clearly in charge with respect to math
Attention to CABS; reference to MMP courses; reviewing student work
Variation in curriculum Math not addressed at the
meeting No clear math leader—i.e.,
hard to tell who the MTL is Confusion about the MMP
and CMF Too much non-academic
housekeeping School climate is
the priority
Hi Lo
November 6-7, 2006 29
Results of Classroom Observations
General Practice
Articulating Math TaskFormative Assessment
Overall
ComprehensiveMath Framework
UnderstandingComputingApplicationReasoning
Engagement
Overall, strongest areas were articulating the mathtask & understanding
Biggest areas for improvement were useof formative assessment & engagement
November 6-7, 2006 30
Characteristics of Hi-Lo Scoring Classroom Performance—General
Math task within the lesson was easy to identify
Math task was discrete and level-appropriate
Encouraging self-assessment and peer-assessment
Establish criteria for proficiency
Promoting problem solving and independent thinking
Math task was to complex or obscure
Only feedback provided was if answer was correct
Little teacher involvement in the lesson
Feedback focuses on student behavior
Hi Lo
November 6-7, 2006 31
Characteristics of Hi-Lo Scoring Classroom Performance—CMF
Student explanations sought
Computation is presented as a means to an end
Problem solving was emphasized
Students had to justify solutions
Lessons are made relevant by using everyday things like money or time and seeking examples from students’ lives
Close ended questions are emphasized
Only one way to solve problems presented
Minimal time allowed to share solutions
Students not accountable for responding to questions
Problems not presentedin context
Hi Lo
November 6-7, 2006 32
Results of MKT Assessment
Number &Operations
43 item assessment addressed 3 content areas:
Algebra Geometry
OverallScore
& &
November 6-7, 2006 33
Results of MKT Assessment
Average IRT Scores
nNumber & Operations Algebra Geometry Total
Lo 12 -0.78 -0.60 -0.65 -0.59
HI 32 0.12 0.31 0.20 0.22
Mean -0.36 -0.13 -0.32 -0.31
Median -0.38 -0.09 -0.39 -0.44
SD 0.70 0.77 0.79 0.75
November 6-7, 2006 34
Social Network Analysis
Math stakeholders in each school were asked to name individuals with whom the communicated about mathematics
Statistical analysis focused on1. Network and in-school density2. Importance of MTL and MTS
November 6-7, 2006 35
Metric n Total Named
Network density
Density in school
MTL Role--In Degree
MTS Role--In Degree
Lo 13 31 3.9% 7.6% 3.33 1.14
Hi 32 91 11.4% 31.1% 28.24 18.52
Mean 21.1 54.0 6.7% 17.6% 13.81 5.31
SD 6.8 17.6 2.6% 9.6% 7.2 4.9
Median 19 48 6.2% 15.4% 13.07 3.75
Overall SNA Results
Density—a perfect score is 100% where everyonenames everyone else
In-Degree scores are relative measures
November 6-7, 2006 36
Example Network
November 6-7, 2006 37
Example Network
November 6-7, 2006 38
Report Card Indicators
19 indicators in 7 domains based on in-school data collection, online surveys, and MPS data
1. MTS Assessment2. Collaboration3. Learning Teams4. Classroom Practice5. Professional Development6. Teacher MKT7. Student Achievement
November 6-7, 2006 39
Report Card Results
StudentAchievement
Teacher Content& Pedagogical
Knowledge
Learning TeamEffort
SchoolBuy-in
TeacherInvolvement
ClassroomPractice
WKCEMean % Proficient = 44%
Overall rating = 3.5Gap MTL v. other teacher = .2Teacher Engagement = 3.2
Overall IRT = -0.34Algebra IRT = -0.18
Team Functioning = 3.5MMP Principles = 3.6LT Quality = 3.1
PD Hrs. = 17.8Facilitation Hrs. = 1.0PD Quality = 3.1
Network density = 6.7% / School density = 17.6%MTL Role = 13.8 / MTS Role = 5.3
SR MTL Engagement = 4.4 / MTS Quality = 3.0
MTS Assessment = 38.3 of 55
November 6-7, 2006 40
Student Achievement & In-School Network Density
November 6-7, 2006 41
Student Achievement & Learning Team MMP
November 6-7, 2006 42
Student Achievement &Professional Development
November 6-7, 2006 43
Conclusions
MMP is advancing concepts, ideas, & principles that can help schools improve student achievement results in math.
Schools that score well with regards to MMP-related metrics have higher student achievement.
Learning team adoption of MMP ideas and dense in-school communication networks are predictors of high student achievement
November 6-7, 2006 44
Conclusions
At the same time… Some MPS schools are lagging behind
in terms of adopting MMP ideas. These schools perform do not score as
well on MMP metrics, which is consistent with student achievement results.
We know that other factors—prior year student achievement and SES—are stronger predictors
November 6-7, 2006 45
Evaluation Next Steps
District Wide Analysis Continue online survey & data mining Improve ability to link student and teacher data working with MPS
Case Study Schools1. Recruit case study schools Nov2. Plan observations Nov-Dec3. Observations Round 1 Jan-Feb4. Observations Round 2 March-April5. MKT Assessment May6. SNA Survey May