PROJECT RENEW Evaluation Results Summary Sarah Hough.

PROJECT RENEWPROJECT RENEW

Evaluation Results SummaryEvaluation Results Summary

Sarah HoughSarah Hough

Purpose of this presentationPurpose of this presentation

Review the RENEW evaluation as Review the RENEW evaluation as whole.whole.

Focus on effects of project in Focus on effects of project in supporting beginning teachers as supporting beginning teachers as they better develop their ability to they better develop their ability to teach mathematics.teach mathematics.

Introduce the summative studies Introduce the summative studies underway.underway.

Talk about what data still needs to Talk about what data still needs to be collected in this last year of the be collected in this last year of the project.project.

Evaluation ResearchEvaluation Research

What is it?What is it? Basic Research=Basic Research=scientific investigation

to develop or enhance theory. Applied Research= Applied Research= testing theory to

assess its ”usefulness” in solving problems.

Evaluation Research=Evaluation Research=applied research plus a value criteria

Usually Evaluation Research Usually Evaluation Research is..is..

A study or series of studies A study or series of studies designed for the purpose of designed for the purpose of assisting project assisting project stakeholders in assessing stakeholders in assessing the projects worth.the projects worth.

Basics of RENEW EvaluationBasics of RENEW Evaluation

We are conducting a series of We are conducting a series of studies that (1) Describe the studies that (1) Describe the RENEW model and it’s impact RENEW model and it’s impact across district contexts; (2) test across district contexts; (2) test the effects of the PD activities on the effects of the PD activities on participants’ beliefs and participants’ beliefs and understandings of mathematics, understandings of mathematics, pedagogy and equity; (3) pedagogy and equity; (3) determine whether the project determine whether the project had an effect on retention rates had an effect on retention rates in the participating districts.in the participating districts.

Who gets to decide what is Who gets to decide what is valuable?valuable?

Evaluation of RENEW is “client-Evaluation of RENEW is “client-centered” in that we are interested in centered” in that we are interested in the describing the RENEW model the describing the RENEW model from all stakeholder perspectives from all stakeholder perspectives and examining effects of project and examining effects of project activities both seen and unforeseen activities both seen and unforeseen on all of its participants.on all of its participants.

Review of RENEW ModelReview of RENEW Model

Main Goal: Development of a Main Goal: Development of a professional development-mentoring professional development-mentoring model to keep effective teachers of model to keep effective teachers of

mathematics in the professionmathematics in the profession

Leadership teams offers intellectual Leadership teams offers intellectual and emotional support to Preceptors and emotional support to Preceptors who in term support the beginning who in term support the beginning teachers in their schools as they teachers in their schools as they implement National Standards based implement National Standards based reform in their classrooms.reform in their classrooms.

Objectives RENEW ModelObjectives RENEW Model

National standards-based reform is -- the type National standards-based reform is -- the type of mathematics classrooms envisioned by of mathematics classrooms envisioned by NCTM)NCTM)

All children have opportunities to work both All children have opportunities to work both alone and in collaboration with peers on alone and in collaboration with peers on important mathematics problems., important mathematics problems., developing both conceptual and procedural developing both conceptual and procedural knowledge that allows them to be knowledge that allows them to be mathematically powerfulmathematically powerful

Work of PreceptorsWork of Preceptors

MEPSMEPS

--to develop understandings of --to develop understandings of mathematics, pedagogy and mathematics, pedagogy and diverse student learners.diverse student learners.

Classroom VisitationsClassroom Visitations

--to support implementation in --to support implementation in classrooms.classrooms.

What Beginning Teachers Need in What Beginning Teachers Need in Order to Implement National Order to Implement National Standards-Based ReformStandards-Based Reform A relevant disposition towards A relevant disposition towards standardsstandards--

based reform that includes a teacher’s based reform that includes a teacher’s beliefs about teaching, beliefs about beliefs about teaching, beliefs about learning and beliefs about mathematical learning and beliefs about mathematical knowledge.knowledge.

The development of deep and flexible The development of deep and flexible conceptual understandings about subject conceptual understandings about subject matter.matter.

Pedagogical learner knowledge, relevant to Pedagogical learner knowledge, relevant to diverse learners, to teach in a way diverse learners, to teach in a way consistent with the standards.consistent with the standards.

Wang and Odell (2002)Wang and Odell (2002)

Does participation in RENEW Does participation in RENEW help preceptees acquire these help preceptees acquire these types of understandings?types of understandings?

Beginning TeachersBeginning Teachers

Particular questions asked and answered Particular questions asked and answered by evaluation studies.by evaluation studies.

Does RENEW have an effect on BT Does RENEW have an effect on BT understanding of mathematics concepts, if so, understanding of mathematics concepts, if so, how?how?

Are BTs beliefs about mathematics and its Are BTs beliefs about mathematics and its teaching in line with PSSM (NCTM, 2000)? teaching in line with PSSM (NCTM, 2000)? Does RENEW affect these beliefs over time?Does RENEW affect these beliefs over time?

How efficacious do BTs feel in implementing How efficacious do BTs feel in implementing PSSM in their own classrooms?PSSM in their own classrooms?

Using Concept Maps to Assess Using Concept Maps to Assess Content UnderstandingContent Understanding

Amount of knowledgeAmount of knowledge Way the knowledge is structuredWay the knowledge is structured

Triangulation using reflective Triangulation using reflective writingswritings

Every time I attend a math institute I Every time I attend a math institute I become confident and empowered in become confident and empowered in math and feel like a better teacher math and feel like a better teacher every second I am present in the every second I am present in the institute. I never knew statistics could institute. I never knew statistics could be this fun I’ve already planned most be this fun I’ve already planned most of my 1st month unit in statistics. of my 1st month unit in statistics.

(Preceptee, Summer Institute) (Preceptee, Summer Institute)

Before RENEW I never felt Before RENEW I never felt confident in solving math confident in solving math problems. I rarely understood problems. I rarely understood the WHY behind problems. The the WHY behind problems. The project offers a place that I can project offers a place that I can freely ask and feel comfortable freely ask and feel comfortable exploring exploring

(Preceptee, end of year reflection) (Preceptee, end of year reflection)

Beliefs ScalesBeliefs Scales

Beliefs About Teaching Mathematics Beliefs About Teaching Mathematics (cronbach’s alpha = .76) (cronbach’s alpha = .76) “In mathematics “In mathematics students cannot understand high level concepts until students cannot understand high level concepts until they have mastered the "basic" steps of a given they have mastered the "basic" steps of a given

procedure or algorithm”procedure or algorithm” Beliefs About Mathematics (cronbach’s Beliefs About Mathematics (cronbach’s

alpha = .61)alpha = .61) “Doing mathematics mostly involves “Doing mathematics mostly involves

memorizing facts and procedures”memorizing facts and procedures” Beliefs About National Standards Beliefs About National Standards

cronbach’s alpha = .83cronbach’s alpha = .83 “When doing “When doing mathematics you are discovering patterns mathematics you are discovering patterns

and making generalizations “and making generalizations “

Beliefs and UnderstandingsBeliefs and Understandings

0 years0 years One or One or more yearsmore years

Beliefs Beliefs TeachingTeaching

3.43.4 3.6**3.6**

Beliefs Beliefs MathMath

3.83.8 3.93.9

Beliefs Beliefs Standards-Standards-based based ReformReform

3.73.7 4.2**4.2**

BeliefsBeliefs

After one or more years in the After one or more years in the project Preceptee Beliefs About project Preceptee Beliefs About Teaching and Beliefs About Teaching and Beliefs About Standards-Based Mathematics Standards-Based Mathematics were not significantly different were not significantly different than that of their Preceptors.than that of their Preceptors.

BeliefsBeliefs Change in Preceptee Beliefs Change in Preceptee Beliefs

About Teaching and Beliefs About Teaching and Beliefs About Standards-Based About Standards-Based Mathematics were not mediated Mathematics were not mediated by number of years in the by number of years in the project. That is teachers in their project. That is teachers in their first 2 years of teachingfirst 2 years of teaching had had beliefs beliefs not significantly differentnot significantly different than those in their than those in their 33rdrd – 5 – 5thth years. years.

Triangulation with self-report data Triangulation with self-report data asking do you think your beliefs asking do you think your beliefs have changed?have changed?

62% stated positively yes, that their 62% stated positively yes, that their beliefs had changed; 16% stated that beliefs had changed; 16% stated that they had changed somewhat; 18% they had changed somewhat; 18% stated that their beliefs remained stated that their beliefs remained consistent but had been consistent but had been strengthened by the project or that strengthened by the project or that they were better able to act upon they were better able to act upon their beliefs as a result of their beliefs as a result of participation, and 4% stated that no participation, and 4% stated that no their beliefs had not changed.their beliefs had not changed.

Absolutely-I have applied many of Absolutely-I have applied many of the ideas and concepts presented in the ideas and concepts presented in this project. It has radically (I think) this project. It has radically (I think) changed my mind about the changed my mind about the approach to teaching math. approach to teaching math. (Preceptee, end of year reflection) (Preceptee, end of year reflection)

I think I always believed in these I think I always believed in these concepts but Project RENEW has concepts but Project RENEW has given me the tools to make given me the tools to make investigative learning possible in my investigative learning possible in my classroom.classroom. (Preceptee, end of year (Preceptee, end of year questionnaire) questionnaire)

Efficacy and Classroom Efficacy and Classroom PracticePractice

Link between efficacy and Link between efficacy and teaching practice.teaching practice.

Efficacy scale (Cronbach’s Efficacy scale (Cronbach’s alpha = .90) is administered alpha = .90) is administered every year.every year.

Not adequately prepared

Very well preparedSomewhat prepared

Fairly Well prepared

Manage a class of students Manage a class of students engaged in hands-on project engaged in hands-on project workwork

Not adequately prepared

Very well preparedSomewhat prepared

Fairly Well prepared

Help students take responsibility Help students take responsibility for their own learningfor their own learning

Not adequately prepared

Very well preparedSomewhat prepared

Fairly Well prepared

Lead a class of students Lead a class of students using investigative strategiesusing investigative strategies

Not adequately prepared

Very well preparedSomewhat prepared

Fairly Well prepared

Encourage student interest Encourage student interest in mathematicsin mathematics

Not adequately prepared

Very well preparedSomewhat prepared

Fairly Well prepared

Use strategies that specificallyUse strategies that specifically encourage minorities and femalesencourage minorities and females

Not adequately prepared

Very well preparedSomewhat prepared

Fairly Well prepared

Recognize and respond to Recognize and respond to student diversitystudent diversity

Changes in efficacy were Changes in efficacy were not not mediated by mediated by number of years number of years teaching.teaching.

Results triangulated with Results triangulated with reflective writingsreflective writings

I feel much more comfortable about I feel much more comfortable about discussing math with my students discussing math with my students and realize that there are many ways and realize that there are many ways for them to solve problems. for them to solve problems.

(Preceptee, reflection)(Preceptee, reflection) By collaborating with other math By collaborating with other math

teachers, I have learned better teachers, I have learned better teaching practices, as well as teaching practices, as well as strategies to help my students be strategies to help my students be more successful in math. more successful in math.

(Preceptee, reflection)(Preceptee, reflection)

What mediated the change?What mediated the change?

1. Effective Preceptors/those with 1. Effective Preceptors/those with beliefs, knowledge and practice beliefs, knowledge and practice in line with National standards. in line with National standards.

Relationships What mediated Relationships What mediated the change?the change?

2. Relationships between people 2. Relationships between people are key to the success of the are key to the success of the project.project.

Collaborative Relationships Collaborative Relationships Between Preceptors & PrecepteesBetween Preceptors & Preceptees

I have developed a good relationship with my preceptees built on trust and mutual respect. This is something I really didn't think of originally. I also found that our reflective discussions more often than not included issues they were concerned about that were not connected to math, which I felt was okay.

How Does Mentor Teachers How Does Mentor Teachers Leadership Develops Over Time? Leadership Develops Over Time?

3-stage cumulative developmental model

Stage I: Preceptor as Resource

Stage 2: Preceptor as Relationship Builder

Stage 3: Preceptor as Change Agent

Stage III: Teachers need to find ways to teach concepts that will help all students be successful. ..I am challenged personally about the equity of our school program. I consistently see the advanced math classes with fewer minority students than our population dictates. I see minority students scoring below proficient on the state standards. I must address these issues personally and share with our MEPS. I must continue to bring up all types of inequity, sometimes gently nudging others to examine their practice. Other times I must not be so gentle!

0%

20%

40%

60%

80%

100%

120%

Start of Year I End of Year I End of Year II End of Year III

Stage 3: Change Agent

Stage 2 with Beginnings of Stage 3

Stage 2: Relationship Builder

Stage 1 with Beginnings of Stage 2

Stage 1: Resource

Relationships Between Relationships Between Administrators and PreceptorsAdministrators and Preceptors

Ongoing, year-round Preceptor-Ongoing, year-round Preceptor-Administrator communication Administrator communication

Active Principal participation Active Principal participation during recruitment of during recruitment of PrecepteesPreceptees

Visibility of Principals at Visibility of Principals at mathematics professional mathematics professional development sessions.development sessions.

Summative Evaluation in Summative Evaluation in ProgressProgress

Tales of Three Districts: Case Tales of Three Districts: Case StudiesStudies

Retention ComparisonRetention Comparison What are Cohort I Preceptors What are Cohort I Preceptors

doing in their settingsdoing in their settings Matched Sample “Teacher Matched Sample “Teacher

Burn-Out” StudyBurn-Out” Study

Technical ReportsTechnical Reports

Hough, S., Sandoval, N. (2005). An Hough, S., Sandoval, N. (2005). An Evaluation of Project RENEW: Year Evaluation of Project RENEW: Year III Report Prepared for the National III Report Prepared for the National Science Foundation. February, 2005.Science Foundation. February, 2005.

Hough, S., O‘ Rode, N. (2004). An Hough, S., O‘ Rode, N. (2004). An Evaluation of Project RENEW: Year Evaluation of Project RENEW: Year III Report Prepared for the National III Report Prepared for the National Science Foundation. February, 2004.Science Foundation. February, 2004.

ArticlesArticles

Hough, S; N; Terman, N; Weissglass, J; Hough, S; N; Terman, N; Weissglass, J; (2005). Measuring Change in Beginning (2005). Measuring Change in Beginning Teachers’ Statistics Knowledge Networks. Teachers’ Statistics Knowledge Networks. To appear:To appear: International Journal of International Journal of Learning, Volume 12.Learning, Volume 12.

Hough, S; O’Rode, N; Terman, N; Hough, S; O’Rode, N; Terman, N; Weissglass, J; (2005). A Respectful Weissglass, J; (2005). A Respectful Approach to Assessing Change in Approach to Assessing Change in Teachers Understanding of Algebra: Teachers Understanding of Algebra: Concept maps. ManuscriptConcept maps. Manuscript in Revision:in Revision: Journal of Mathematics Teacher Journal of Mathematics Teacher Education.Education.

Conference PapersConference Papers

Hough, S; Sandoval, N; N; Terman, N; Weissglass, J; Hough, S; Sandoval, N; N; Terman, N; Weissglass, J; (2005). Measuring change: Experienced teachers’ (2005). Measuring change: Experienced teachers’ develop pedagogical content knowledge by linking develop pedagogical content knowledge by linking content and pedagogy. Paper to be presented at the content and pedagogy. Paper to be presented at the Annual Meeting of the American Educational Research Annual Meeting of the American Educational Research Association. San Diego, April 2006.Association. San Diego, April 2006.

O'Rode,N., Terman, N., Hough, S. (2005). Mathematics O'Rode,N., Terman, N., Hough, S. (2005). Mathematics and equity: developing the leadership of experienced and equity: developing the leadership of experienced teachersteachers. Paper presented at the Annual Meeting of the . Paper presented at the Annual Meeting of the American Educational Research, Montreal, 2005. American Educational Research, Montreal, 2005.

Hough, S; Erbes, S; O’Rode, N; Terman, N. (2004). Hough, S; Erbes, S; O’Rode, N; Terman, N. (2004). Understanding beginning teachers’ needs: The effects of Understanding beginning teachers’ needs: The effects of RENEW, a retention and renewal mathematics project. RENEW, a retention and renewal mathematics project. Paper presented at the Annual Meeting of the American Paper presented at the Annual Meeting of the American Educational Research, Montreal, 2005.Educational Research, Montreal, 2005.

Conference PapersConference Papers O'Rode,N., Terman, N., Hough, S. (2004). Mathematics O'Rode,N., Terman, N., Hough, S. (2004). Mathematics

and equity: developing the leadership of experienced and equity: developing the leadership of experienced teachers to work with beginning teachers. Paper teachers to work with beginning teachers. Paper presented at the New Teacher Center Conference, San presented at the New Teacher Center Conference, San Jose, CA.Jose, CA.

O'Rode, N., Hough, S. (2003). Revisiting the needs of O'Rode, N., Hough, S. (2003). Revisiting the needs of beginning teachers in the context of a mathematics beginning teachers in the context of a mathematics professional development project. Paper presented at The professional development project. Paper presented at The UCSB Education Conference.  Converging Realities: UCSB Education Conference.  Converging Realities: Embracing Diverse Disciplinary Perspectives . Santa Embracing Diverse Disciplinary Perspectives . Santa Barbara, California, February 2003.Barbara, California, February 2003.

Durias, R., Hough, S., Terman, N. (2002). Leadership for Durias, R., Hough, S., Terman, N. (2002). Leadership for excellence and renewal in mathematics education: An excellence and renewal in mathematics education: An emergent model for supporting beginning teachers in emergent model for supporting beginning teachers in mathematics. Paper presented at the California mathematics. Paper presented at the California Commission on Teacher Education.  San Jose, California, Commission on Teacher Education.  San Jose, California, March 2002.March 2002.

Next Steps in EvaluationNext Steps in Evaluation

2 minute each way dyad: What 2 minute each way dyad: What further information would you further information would you find valuable in assessing the find valuable in assessing the worth of the project?worth of the project?