Putting PVAAS Teacher Reporting Skepticisms to Rest Dr. John White, Manager, SAS EVAAS for K-12 PVAAS Statewide Team: Kristen Lewald,Joan Perekupka and.

Putting PVAAS Teacher Reporting Skepticisms to Rest

Dr. John White, Manager, SAS EVAAS for K-12PVAAS Statewide Team:

Kristen Lewald,Joan Perekupka and Jennifer Ross

Today’s Session

• Talk – Write – Share – Read - Ask Questions• Check for Understanding• Probe• Feedback and Clarification• What is the Real Issue of the

“Misunderstanding”?• Binder of Resources – 4 Sections• Lunch and Breaks

Key Areas

• Measuring Growth• Demographics• Transition in PA Assessment System• General: Teacher Value-Added• PA Specifics: teacher Value-Added• PA Teacher Evaluation System• Questions for PDE EE

When and How Did You LEARN PVAAS?When and How Do You USE PVAAS?

Intro by IU Team

Skepticisms -Misunderstandings

Need to Respond

Measuring Growth

How do you describe the concept of growth as measured by PVAAS?

How does PVAAS Measure Growth?

• Paper• Pencil• 3 minutes

How does PVAAS Measure Growth?

• Share with a Partner• 2 minutes EACH• Large Group Report Out

Learning about Measuring Growth

Measuring Growth

• How is there enough “stretch” on the PSSA to measure the growth of both low- and high-achieving students when we know that the PSSA test questions at any given grade level are limited to the “eligible content” for that grade? In other words, how can growth beyond (or below) the current grade level be shown when the PSSA is limited to questions from the eligible content for that grade level only.

Measuring Growth

Measuring Growth

1. Many educators believe that high performing students have a harder time making growth, so is this model setting us up for failure? How is it possible to show growth with high-achieving students?

– ½ table question #1– Partner• Underlying misunderstanding?• How would you respond to this issue?

Measuring Growth

2. How do you explain what happens when one grade level shows no growth in a given year and then the next year shows tons of growth? There are educators who say it is easy to show growth in some years/grades because the year/grade before the students “lost ground.” Can this be addressed?

– ½ table question #2– Partner

• Underlying misunderstanding?• How would you respond to this issue?

Learning about High Achievement and Growth


• PVAAS Misconceptions Document• First developed in 2011 and revised in 2012• Addressed in misconceptions #8 and #9

• New Concept of Growth graphics/slides• New slides address concept plus application for both low-

achieving and high-achieving groups of students• Appear in F2F Overview materials, webinar materials, and fall

session materials• One-pager on High Achievement & High Growth• Revised January 2013

• Webinar – High Growth, High Achieving Schools: Is it Possible?• Live webinars held twice in December 2012• Archived versions still available via Elluminate


• Virtual Learning Module – High Achievement & Growth• Released October 2012• Available on public and restricted sites• A shorter, more user-friendly version will be released by end of

June• Statewide Scatterplots and District/School Search Feature• Available on public site• Shows evidence of schools with many high-achieving students

making high growth• The Building Capacity for IUs webinar in March (#4 in the

series) specifically addressed how to use these features to answer the question, “Can high-achieving schools, or groups of students, show growth?”

Measuring Growth

• It is said that value-added modeling is inaccurate; that no one can truly explain how growth is determined; and that psychometricians do not believe value-added measures are valid. Can this be addressed?– Dr. John White– Reference Section 3 in Binder• Literature Review for Teacher Value-Added

Measuring Growth• How can you calculate value-added

estimates for grade 4 science? If grade 3 is the first year we test reading and math and we don’t have value-added estimates in grade 3, then why can we get value-added estimates for grade 4 science when that is the first year that subject is assessed?– PSSA Science as a GRADE level test– Feedback from Dr. John White

Measuring Growth• How do we tease out the factors that contribute

to growth? When districts are analyzing their value-added reports what should they look at to determine what they did in order to meet or not meet the standard for PA Academic Growth? How do we help districts that do not have or use a written curriculum? How do we help them interpret or make informed decisions based on the value-added data?– 1 minute think– Examples from 3 of You

DEMOGRAPHICS

Article

• “Adjusting for Student Characteristics in Value-Added Models”

• Section 1 in Binder• Read• Write Down 3 Key Points

Demographics

•What evidence exists about growth and demographics at the teacher level?–Dr. John White–Evidence over time

Demographics

• If achievement is influenced by demographics, and growth is based on achievement scores, then why isn’t growth influenced by demographics?

• How is the demographic makeup of classes (groups of students) taken into account, or not?

Demographics

• How are groups of students who learn differently taken into account, especially if there are high numbers of economically disadvantaged students or students with IEPs in the group?– Dr. John White– Using All Prior Achievement Data

Demographics

• What research has been done on the growth of students with emotional disabilities? The research lumps all special education students together.– PA’s Plan

TRANSITIONS IN PA ASSESSMENT SYSTEM

Transitions in PA Assessments

• Section 4 of Binder: PVAAS and the Transition of PA’s Assessment System

• 3 Minute Skim• On PVAAS Login

page


• Will teacher-specific value-added estimates be reliable for Keystone exams being that we are just now administering these for the first time?


• How will teacher-specific value-added estimates be reliable if the base year is changing each year? Can you explain the implications of resetting the base year each year for the next few years? Will this lead to larger standard errors? Are there ramifications for this in the 3-year rolling average for teacher reporting?

• Will the baseline for Keystone exams be changing each year?


• The grade 7 and grade 8 Math PSSAs and the Keystone Algebra I exam require two very different types of thinking (i.e., PSSAs seem to target more facts and processes, while the Keystone exams target higher-order thinking). Will the grade 7 and grade 8 PSSAs accurately predict Keystone exam performance?


• What will be the overall impact to PVAAS reporting now that the PSSA-M is not administered?


• If the projections to Keystone exams are for when students take the exam, how can we use the data to determine early readiness for the course (i.e., a high projection does not necessarily equate to early readiness)?

GENERAL: TEACHER VALUE-ADDED

General: Teacher Value-Added

• How do we reflect the fact that time after time we were told PVAAS would not drill down to the teacher and student level, and now it is going to that level?

• Act 82• PVAAS Prior to Act 82– No student-teacher linkages– Could NOT get teacher specific reporting


• What happens when a teacher has a year of “that class”? Teachers always have a year when that happens, and they don’t want to be judged by a single year.– PVAAS 3 Year Rolling Average in EE System– Is there a hidden need for professional

development?– Dr. John White• Findings from research on teacher effectiveness

measures? Consistency of measures? Effect as teachers change types of class/school/district?


• There are so many people involved in an individual student’s education. How is the teacher value-added measure for a single teacher credible for evaluation purposes?


• There are concerns that administrators could unfairly influence teacher-specific reporting by assignment of students to specific classes. Can this be addressed?– Turn to a partner– What is the underlying misunderstanding?


• How do administrators and districts combat the issue of teachers resisting teaching assignments for groups of students that are low performing as they won’t want to claim those students when it comes to value-added reporting?• What is the underlying misunderstanding?


• Teachers are now skeptical about taking on student teachers. We recognize there is research indicating there is no impact of that on value-added measures. Can this be addressed?

• Feedback from Dr. White


• Our district and schools have not met the growth standard for over 3 years in grade 5. Does this mean that our grade 5 teachers will not meet the growth standard either?

• District-School-Teacher Measures• Feedback from Dr. White


• How is class size taken into account with value-added measures? How are teachers who have small numbers of students affected?–Feedback from Dr. John White


• In a district where students stay and then go to the DR and come back a month later (and they have attendance issues), then they do poorly on the test, how is it fair to solely contribute that to the teacher?


• How fair is it to measure growth with three different groups of students when using it for teacher-specific reporting?– Stability of EVAAS Measures– Dr. John White


• How do the teacher value-added measures take into account a teacher who is co-teaching, and the co-teacher is harmful to the academic achievement of the students? The teacher may only have 50% instructional responsibility for those students, but the teacher’s value-added measure is still being affected by the co-teacher really “pulling these students down.”


• How do we tease out the teacher factors that contribute to growth? For example, in a grade 8 Algebra I course, it may be more rigorous and have higher achieving students than the grade 9 Algebra I course in the same district. However, if the same teacher is teaching both courses with two different sets of curriculum and the value-added results are all lumped together, how do we help teachers sort that information out?– PVAAS Custom Diagnostic Report

PA SPECIFICS: TEACHER VALUE-ADDED

PA Specifics: Teacher Value-Added

• Data Quality Concerns– From Collecting to Using– Addressing Data Quality• LEA SIS• PIMS• PVAAS Roster Verification

– PAIMS Tech Readiness– PAIU Data Quality Event Last Week• Planning any follow-up meetings with LEAs?• Involve Program and Data Staff

Impact in PIMS and Local SIS• Staff Data such as:

– NEW: Staff Work Emails– NEW: PVAAS Role for PVAAS Account Creation

• PIMS Course/HQT (Student-Staff-Courses)– NEW: All teachers linked to a student for a

grade/subject/course• Teachers beyond/different than HQT (HQT is a different PIMS need)• WHICH teachers REALLY had responsibility for planning the

instruction, delivering the instruction, and assessing the effectiveness of the instruction (Domains 1 AND 3)

• Could include LTS

– NEW: Keystone Related Course Codes/Course(s) that “trigger” a Keystone exam

– NEW: % Instructional Responsibility48

PA Specifics: Teacher Value-AddedData Quality

• Districts are concerned about the integrity of the data being used in calculation of teacher value-added estimates. Can this be addressed?

• There are concerns about the accuracy of student roster assignments to teachers. Can this be addressed?

• How is the attribution for the percentage of time that a student is enrolled in a class accurately reported and reflected?


• There are concerns with how to implement the roster verification system at a large scale once this becomes operational next year (i.e., the amount of paperwork and cross-checking that is required to attribute all students to a class is a lot). Can you address how districts are to accomplish this?

• There are concerns that enrollment data will not be accurately reflected for teachers. Can this be addressed?


• What data are we getting back and does that have to sync with PIMS after the teachers and administrators change the data?– Under discussion by PDE PIMS team


• There are concerns of the logistics of enrolling students and having teachers initially determine their percent of instructional responsibility. Can this be addressed?– LEA SIS– PIMS– Pre-population of PVAAS


• There are concerns regarding the how the information can be systemized so that the teachers and principals have as much data as possible pre-loaded into the roster verification system instead of having all rosters show 100%. Can the shared responsibilities somehow be represented in the pre-load of the verification tool? – LEA SIS– PIMS– Pre-population of PVAAS


• Given the numerous time, behavioral, management, and instructional demands placed on principals, teachers, and central office administrators, what is the anticipated time demand needed to affirm or correct the data input into the system?

PILOT SURVEY

PA Specifics: Teacher Value-AddedRole/Responsibility of Admin

• What happens to teacher value-added scores if teachers are reassigned after two years? Will this cause pressure to administrators to reassign teachers to a different assignment after two years so that teachers will never get the 15% for use in the evaluation system?


• There are concerns about potential union issues regarding teachers who have improvement plans. Could there be potential union concerns regarding how attribution of instructional responsibility of teachers is decided?

• There are concerns that principals might not have the time or know the teachers and their relationships to the students well enough to provide accurate validation of the data. Can this be addressed?


• There are concerns that teachers might skew the percentages they enter for instructional responsibility in order to show the best results. Can this be addressed?– What would you think this might look like??– Skewed how?– Underlying misunderstanding?


• There are concerns that administrators do not have a clear understanding of how to interpret PVAAS data, and therefore will not be able to correctly explain the methodology to teachers. Can this be addressed?

PA Specifics: Teacher Value-Added% Instructional Responsibility

• How do you attribute instructional responsibility for teachers involved in flexible grouping of RtII students?

• There are concerns about teacher calculation and attribution for long-term substitutes. Can this be addressed?

PA Specifics: Teacher Value-Added% Instructional Responsibility

• How do districts handle special education teachers if the students are itinerant?

• There are concerns about the percentage of instruction for students with IEPs who are on a teacher’s roster and are to receive support services from another teacher, but yet they do not. Can this be addressed?

PA Specifics: Teacher Value-AddedLEA Implementation

• There is concern that the PVAAS roster verification process will drive service delivery models rather than having the PVAAS roster verification process reflect service delivery models. Can this be addressed?

PA Specifics: Teacher Value-AddedLEA Implementation

• There are concerns about attendance issues, and districts not having tight enough attendance and enrollment policies and procedures in place. Can this be addressed?

PA Specifics: Teacher Value-Added

• Are there policies or guidelines regarding the breach of confidentiality and sharing of data for both student and teacher?– FERPA– Professional Conduct

PA SPECIFICS: TEACHER VALUE-ADDEDPVAAS PILOT FAQS

Section 4 in Binder

PILOT in Process Now: 273 LEAsAll PILOT materials are posted on PVAAS home page • https://pvaas.sas.com• IU PVAAS Contact for List of Pilot LEAs in Your region

65

https://pvaas.sas.com/

PA Specifics: Teacher Value-AddedPVAAS Pilot FAQs

• It is stated in the PVAAS Roster Verification web-based system that EVAAS will make adjustments if a student is over attributed in the final submission. This is a high priority concern. Can this be addressed?

• How will time, and therefore the percentage of student growth, be attributed to teachers who take a leave of absence during the school year?

PA Specifics: Teacher Value-AddedPVAAS Pilot FAQs

• How do you attribute instructional responsibility to homebound students?

• Why is enrollment data being used versus attendance data?

• Do teacher value-added scores follow teachers if they leave the district? For example, if a teacher was with a district for two years, do those PVAAS growth scores follow them if they enter another district?

PA SPECIFICS: TEACHER EVALUATION SYSTEM

PA Teacher Evaluation System

1. There are so many teachers who are not eligible for teacher value-added measures (i.e., they are in non-tested subjects and grades), so how is the evaluation system fair? Not all teachers will be evaluated in the same way.2. These systems discriminate against teachers who teach tested subjects and grades where value-added measures can be calculated. Other teachers are then “off the hook.” Can this be addressed?


3. There are concerns about inequity or challenges between teachers. Can this be addressed?4. There is concern that the assessments don’t measure total student learning (i.e., transfer, generalization, creativity, emotional intelligence), but yet teachers will be evaluated on these tests. Can this be addressed?


5. How can we articulate the weighting and importance of the formula used and its impact on student achievement?6. How will the teacher value-added reporting affect teachers as part of the teacher effectiveness rating system?


7. There are concerns that the public will be able to identify specific teachers who are effective and ineffective. Can this be addressed?8. How will the teacher-specific value-added data component be implemented into the entire teacher effectiveness model for PA? In other words, it is 15% of the model, but 15% of what?


9. Can you provide an explanation of teacher-specific reporting for the School Performance Profile (3-year average versus year-to-year reporting of the AGI)?10. Can you explain the School Performance Profile and its use, especially given the weight of PVAAS and its use in building-level data used in the teacher evaluation system?

QUESTIONS FOR PDE EE

Questions for PDE EE

• Will there be videos, webinars, and FAQs available for teachers specifically on the new teacher evaluation system?

• What will be the turn-around time for training on the reporting tool?

• Ask your regional IU EE Contact

When and How Will You Continue to LEARN PVAAS?

When and How Will You Continue to USE PVAAS?

Other Questions?

• DRAFT PVAAS Team Plans– PVAAS Pilot Training and Support• PVAAS Teacher Specific Reporting (Late Fall/Winter)

– SY13-14 Implementation• Preparing for Roster Verification (Early Fall)• PVAAS Teacher Specific Reporting- What to Expect• Roster Verification Training (Winter/Spring)

Other Questions?

• Next Steps?• Needs? • Ideas for Support/Follow-up with Your Group?• Building Capacity Sessions

THANK YOU!!!!

Putting PVAAS Teacher Reporting Skepticisms to Rest Dr. John White, Manager, SAS EVAAS for K-12 PVAAS Statewide Team: Kristen Lewald,Joan Perekupka and.

Documents

measuring growth slide

isnt growth

tons of growth

webinar high growth

pvaas measure growth

growth demographics

time slide

minutes slide