Top Banner
Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University
28

Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

Dec 26, 2015

Download

Documents

Beverly Powell
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

Evaluating Math Recovery: Investigating Tutor Learning

Sarah Elizabeth Green

Thomas Smith

Laura Neergaard

Vanderbilt University

Page 2: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

Rationale• As in many interventions, MR relies heavily on the

knowledge and practice of the tutors.• A substantial part of what “adopting” MR means for

districts is the training of tutors.– If tutoring is not effective for students we have to ask the

following:• Did tutors learn as intended from their training?• Did tutors conduct tutoring as intended with students?

• For the first question, this presentation specifically addresses three key issues:

• the effect of prior knowledge on tutor learning during training• the effect of tutor knowledge on the effectiveness of tutoring• change in tutor knowledge over the course of the evaluation study

Page 3: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

Math Recovery (MR) Tutoring

• MR is an unscripted intervention that is adapted to each individual student that is tutored.

• Tutors making ongoing diagnoses of students’ current mathematical thinking/strategies design instruction to fall within each student’s zone of proximal development.

• Central Tool: MR Learning and Instructional Frameworks– Based on 20+ years of research into the development of

children's thinking in the area of early number.– Learning Framework: Used for diagnosis and divided into stages

by content.– Instructional Framework: Used to guide tutoring instruction

based on a students current stage on the Learning Framework.

Page 4: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

Tutor Knowledge for MR• Math Content Knowledge

– This is a first grade intervention where the main content involves: number words, counting, numerals, structuring numbers, addition, and subtraction .

• Math Knowledge for Teaching– Math Knowledge for Teaching (MKT; Hill, Schilling & Ball, 2004)

is a specialized form of mathematical knowledge specific to how students are likely to solve math problems, which solution methods mathematically generalize, which will be most beneficial for their learning in the long run, etc.

• Knowledge of the MR Program– Specific knowledge the MR Frameworks and Assessments,

particularly how to use them to determine students’ current levels and plan tutoring instruction accordingly.

Page 5: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

Example: 12-9Student 1: Counts “11, 10, 9…”(looks at hands) and says, “3!”Student 2: Counts “11, 10, 9, 8, 7, 6 , 5, 4, 3…3!”Student 3: “It takes two to get to ten and one more to nine…3!”• Math Knowledge for Teaching:

– Understand which mathematical concepts are involved (backward number word sequence, one-to-one correspondence, relationships between quantities)

– Anticipate this range of solution strategies and more – Understand the relative sophistication of these strategies

• Knowledge of MR: – Naming these strategies, determining a student’s Stage of Early

Arithmetic Learning (SEAL) level on the Learning Framework and determining appropriate instruction using the Instructional Framework and other published resources (e.g. Wright, Martland, Stafford, 2006; Wright & Stafford, 2002)

Page 6: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

Example: 12-9Student 1: Counts “11, 10, 9…”(looks at hands) and says, “3!”Student 2: Counts “11, 10, 9, 8, 7, 6 , 5, 4, 3…3!”Student 3: “It takes two to get to ten and one more to nine…3!”

Taken from Math Recovery Teacher Handbook, 2007

Stage 3: Initial Number Sequence

Student uses counting-on rather than counting from “one” to solve addition or Missing Addend tasks(e.g. 6+x=9). The student may use a count-down-from strategy to solve Removed Item tasks (e.g. 17-3 as 16, 15 ,14- answer 14) but not count-down-to strategies to solve Missing Subtrahend tasks (e.g. 17-14 as 16, 15, 14 -- answer 3).

Stage 4: Intermediate Number Sequence

The student counts-down-to in order to solve Missing Subtrahend tasks (e.g. 17-14 as 16, 15, 14 -- answer 3). The student can choose the more efficient strategy of count-down-from or count-down-to strategy.

Stage 5: Facile Number Sequence

The student uses a range of what we refer to as non-count-by-one strategies. These strategies involve procedures other than counting by ones but may also involve some counting by ones. Thus in additive and subtractive situations, the student uses strategies such as compensation, using a known result, adding to ten, commutativity, subtraction as the inverse of addition, and awareness of the “ten” in a teen number.

Page 7: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

Example: 12-9Student 1: Counts “11, 10, 9…”(looks at hands) and says, “3!”Student 2: Counts “11, 10, 9, 8, 7, 6 , 5, 4, 3…3!”

Student 3: “It takes two to get to ten and one more to nine…3!”

Taken from Math Recovery Teacher Handbook, 2007

Phase 3:

Teaching the Figurative Child

Moving from Stage 2 to 3

FNWS 1 to 100

BNWS 1 to 100

Numerals 1 to 100

Counting-on and counting-back to solve additive and subtractive tasks

Combining and partitioning using 5 and 10 Combining and partitioning in the range 1-10

Early multiplication and division

Phase 4:

Teaching the Counting-on and Counting-back Child: Moving from Stage 3 to 5

NWSs by 2, 10, 5, 3, &4 in the range 1-100

Numerals 1 to 100

Incrementing by 10s and 1s

Adding & Subtracting to & from decade numbers

Adding & Subtracting to 20 using 5 and 10

Developing multiplication and division

Phase 5:

Teaching the Facile Child

NWSs by 10s on and off the decade

NWSs by 100s on and off the 100 and on and off the decade

2-digit addition & subtraction through counting

2-digit addition & subtraction involving collections

Non-canonical forms of 2- & 3-digit numbers

Higher decade addition and subtraction

Advanced multiplication and division

Page 8: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

Description of Tutors• All fully-certified, white teachers: 17 female, 1 male

– 17 certified for 1st grade (1 for 7th-12th math)• Specializations:

– Mathematics: 3 (two at upper grade levels)– Other Subjects: 2– Special Education: 2

• Before becoming MR tutors:– 10 classroom teachers, 5 Title I teachers– Other: 1 Special Education, 1 Instructional Coach, 1 Title I

Reading

• Teaching Experience:– General: Range = 3 to 30 years, Median = 11.5 years– Mathematics: Range = 2 to 30 years, Median = 7.5 years

Page 9: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

Tutor Training & Study Sites• 18 new tutors were trained at two different regional sites. • Training involved a five day summer institute and five additional pull out days in the

first months of school.• All “fresh” sites in terms of study schools.

Site A

( 5 Tutors, 5 Schools, 3 Districts)

Site B

(13 Tutors, 11 Schools, 2 Districts)• Midwestern: Rural/Suburban• No previous MR experience in Districts• Years Experience:

–General: (4,15) median: 9 yrs–Math: (4,15) median: 7 yrs

• Most (4) tutors were Title I Teachers•Regular Classroom Curricula: Envision by Scott-Foresman & Saxon Math

• Midwestern: Urban/Suburban• Schools in one district already using MR• Years Experience:

–General: (3,30) median: 15 yrs–Math: (2,30) median: 8 yrs

• Most (9) tutors were classroom teachers •Regular Classroom Curricula: Everyday Math & TERC Investigations

Page 10: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

Tutor AssessmentsIn order to measure the development of the new tutors the project used two

measures at the beginning of the study, at the end of year one and the end of year two:

1. Mathematics Knowledge for Teaching Assessment1 • The MKT was developed at University of Michigan in order to

measure Math Knowledge for Teaching.• We used a version of the test specific to elementary number and

operations.2. MR Tutor Knowledge Assessment (TKA)

• The TKA was developed in coordination with MR developers and experts specifically for this study.

• It is designed to assess a tutor’s understanding of the MR Frameworks and how to apply them in tutoring and assessment scenarios.

Both are multiple choice scenario based assessments intended to put the test-taker into the context of practice

1 (Hill, Schilling, and Ball, 2004)

Page 11: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

Descriptive Statistics: Pretest Scores

MEAN

(sd)

Site A (n=5) Site B (n=13) All Tutors (n=18)

MKT *

IRT score from the U of MI

-0.261 0.629 0.381

(0.803) (0.523) (0.716)

TKA *

Total Score out of 39 questions

25.4 30.9 29.4

(1.14) (3.04) (3.65)

*These initial differences between site are statistically significant.

Page 12: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

HypothesesH1: MR Tutors who begin with a higher level of Math Knowledge for

Teaching will more quickly understand how to use the MR frameworks and assessments. – Does MKT pretest score predict TKA pretest score?

H2: Tutors who have a higher level of Math Knowledge for Teaching/greater understanding of the MR frameworks will have a more positive effect on student achievement.– Does MKT pretest score or TKA pretest score predict variation in

treatment effect across tutors?

H3: As tutors gain experience tutoring and access to students’ mathematical thinking they will grow in both their Math Knowledge for Teaching and their understanding of the MR frameworks and assessments.– Do tutors’ scores on the MKT and TKA increase over time?

Page 13: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

H1 Theory: MKT and Knowledge of MR

Because students’ solution strategies are central to the MR Frameworks, we hypothesize that new tutors who know more about ways students typically solve problems in early number or understand the mathematics involved in solving early arithmetic problems (i.e. have more MKT) would be at an advantage in the initial training in terms of experiences to draw on in attempting to understand and use the MR frameworks and assessments.

Page 14: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

H1 Method & Results: MKT TKA• Linear Regression of MKT pretest scores on TKA scores

immediately following training.

Y = 0 + 1 (MKT) + e

MKT Pretest

1 2.56

SE 1.1

p 0.034

R2 = 0.25

Page 15: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

H1 : Controlling for Different Training

MKT Pretest Site B

0.78 4.83*

SE 1.13 1.76

p 0.69 0.015

• Site predicts TKA score following training.• The R2 for this model = 0.50•Site and MKT pretest are significantly correlated (0.57,p=0.01).• Trainers reported that the lower starting point of the teachers at site A led to the differences in training. • Therefore, some of the usefulness of site as a predictor is likely related to the lower MKT that may have led to the differences in training.

Page 16: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

H2 Theory: Knowledge and Student Outcomes

• If being an effective MR tutor requires the types of tutor knowledge that we measured then tutors with higher assessment scores should also be the most effective with students.– This relationship is mediated by what they do with students

during tutoring (fidelity).

Page 17: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

H2: Pretest Score Effects on Students• End of 1st grade and end of 2nd grade treatment-control

comparison using previously mentioned controls (e.g. pretest, race, gender, SES).

• Multi-level model nesting students within tutors and testing the significance of tutor pretest scores (on both MKT and TKA) on predicting the variation of the coefficient on treatment.

(Achievement) = 00 + 1(treatment) + 2-5 (controls) + 1 = 10 + 11(Tutor Assessment Score)

• Using all outcomes from the evaluation study: WJIII (Math Fluency, Applied Problems, Quantitative Concepts), MR Proximal, and the MR 1.1 Internal Assessment

• Given that tutors knowledge may be changing over time we tested this model separately for the end of first grade comparison by year.

Page 18: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

H2: MKT Pretest on Effect of Treatment• In year 2, at the end of first grade the MKT pretest scores predict the treatment effect as measured by the MR 1.1 Internal Assessment.

Year 1 Year 2

MKT on 0.40 1.24**

+ES = 0.31

Average Treatment Effect 3.37*** [0.71]

ES= 0.84

3.37*** [1.29]

ES = 0.85

% of Variation in Treatment Effect Explained

25 % 49 %

•There were no effects of MKT on treatment effect as measured by external student assessments.•There was no effect of MKT on treatment effect at the end of second grade.

*p< 0.1 , ** p <0.05, ***p < 0.01

Treatment Effect Mixed Model

Page 19: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

H2: TKA Pretest on Effect of Treatment

• TKA pretest score is not a significant predictor of the effect of the intervention on any of the student outcomes.

• We know that the relationship between knowledge and outcomes is likely mediated by practice

• This points to the importance of future analyses linking knowledge and fidelity for understanding where in the MR model the links are breaking down.

Page 20: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

H3: MR and Generative Tutor Learning

• It is a result of the theory and practice of MR tutoring that leads to the hypothesis of generative tutor learning.

• We would not expect other interventions, different in their core practices, to result in tutors learning in and from their tutoring practice in the same way as with MR.

• Elements of MR hypothesized to contribute to generative learning:– Focus on understanding students’ thinking (Franke, 1998)– Records and reflection about the link between student strategies

and problems posed by tutors– “Space” for tutors to be designers rather than strictly

“implementers” (Remillard, 2000)

Page 21: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

H3: Tutor Learning• Hierarchical Linear Growth Model with three repeated measures (on

both the MKT and TKA) nested within tutors.

• The predictor, time, is measured in years and used to predict MKT and TKA. If time is a significant predictor, then teachers improved their scores on these assessments over time, evidence of learning.– There is not formal training after the beginning of year one.

Ongoing support was limited in general, but this varied a bit based on site.

– Therefore, systematic growth by the tutors over the two years would indicate that they are learning in and from the practice of tutoring.

Page 22: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

H3: Tutor Learning

• On both measures there is evidence of tutor learning over time. • Given the prior group differences on these measures, what does

including the training site as a predictor on assessment score and as a predictor for rate of learning reveal about differences in learning by training site?

Coefficient on Time

SE p

MKT2 0.27*** 0.09 0.003

TKA 0.86** 0.39 0.03

*p< 0.1 , ** p <0.05, ***p < 0.01

2This model also controls for the form of the MKT that the teacher took by using a dichotomy indicating when they took form C (either time 2 or time 3, randomly assigned).

Page 23: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

H3: Tutor Learning on the MKT

• There is no difference in the rate of learning on the MKT by site. However, there is a significant difference in the average MKT score by site (as previously mentioned).

Beta SE p

Constant -0.05 0.23 0.83

Site B 0.67** 0.27 0.01

Time 0.29* 0.16 0.08

Time*Site B -0.04 0.19 0.85

*p< 0.1 , ** p <0.05, ***p < 0.01

Page 24: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

H3: Tutor Learning on the TKA

Possible explanations• Different levels of ongoing support led to different learning rates• Ceiling effect on the TKA assessment (39 points possible)• Teacher learning in practice is curvilinear, where once a certain level of

knowledge is reached, either there is little room left to grow OR direct intervention (e.g. more intensive PD) is required to continue to grow

Beta SE p

Constant 25*** 1.12 < 0.001

Site B 6.01*** 1.31 < 0.001

Time 2.2*** 0.65 0.001

Time*Site B -1.85** 0.76 0.015

*p< 0.1 , ** p <0.05, ***p < 0.01

Page 25: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

Summary of Findings• MKT pretest does predict TKA score after the initial training

– This suggests that Math Knowledge for Teaching might be one important consideration in choosing tutors who will learn MR quickly.

• MKT pretest also predicts the effect of treatment in year 2 as measured by the MR 1.1– Suggests the importance of Math Knowledge for Teaching for being an

effective MR tutor; the details of the conditions need further investigation.

• Tutors can continue learning (Math Knowledge for Teaching and knowledge of MR) through the practice of MR tutoring.– This suggests that even when highly skilled tutors may not be available

at the beginning of adoption, with experience the tutors can continue to improve in knowledge.

Page 26: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

Future Research & Broad Implications• Future analyses:

– Investigating MKT relationship to practice and student outcomes.– Investigating the relationship between knowledge of MR and

practice of MR using fidelity of implementation data.– Account for the differences in learning using additional data

sources: surveys, teacher lesson plans

• For future evaluators of teacher “delivered” interventions:– Those delivering an intervention can learn from more than just

their initial training. – This implies that thorough evaluations will have to question static

and simplistic views of teachers’ knowledge as something they learn in training and apply in practice.

Page 27: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

Contact Information:

Sarah Green

Graduate Student, Dept. of Teaching and Learning

Vanderbilt University

[email protected]

Page 28: Evaluating Math Recovery: Investigating Tutor Learning Sarah Elizabeth Green Thomas Smith Laura Neergaard Vanderbilt University.

References:Franke, M. L., Carpenter, T., Fennema, E., Ansell, E., & Behrend, J. (1998).

Understanding teachers' self-sustaining, generative change in the context of professional development. Teaching and Teacher Education, 14(1), 67-80.

Hill, H. C., Schilling, S. G., & Ball, D. L. (2004). Developing measures of teachers' mathematics knowledge for teaching. The Elementary School Journal, 105(1), 11-30.

Remillard, J. T. (2000). Can curriculum materials support teachers' learning? Two fourth-grade teachers' use of a new mathematics text. Elementary School Journal, 100(4), 331-350.

Wright, R. J., Martland, J., Stafford, A. K., & Stanger, G. (2006). Teaching number: Advancing children's skills and strategies: Paul Chapman Educational Publishing.

Wright, R. J., Martland, J., & Stafford, A. K. (2006). Early numeracy: Assessment for teaching and intervention: Paul Chapman Publishing.