Discussion of Identifying Effective Teachers Using Performance on the Job (Gordon, Kane & Steiger) Henry Braun Boston College Teacher Quality Conference.

Discussion ofIdentifying Effective Teachers Using

Performance on the Job(Gordon, Kane & Steiger)

Henry Braun

Boston College

Teacher Quality Conference

Northwestern University

May 1 2008

GKS: A Modest Proposal

A plan to restructure the labor market and working conditions for teachers: Making “performance” a more significant factor in teacher advancement and compensation – with value-added measures given special prominence

• Comprehensive• Timely• Scalable• Costed • Defended

What really matters?

… the success of U.S. public education depends upon the skills of the 3.1 million teachers managing classrooms in the elementary and secondary schools around the country. Everything else – educational standards, testing, class size, greater accountability – is background … Without the right people standing in front of the classroom, school reform is a futile exercise. “ (p.5)

An Ecological Perspective on Reform

• What are the highest priority education goals?– e.g. Greater high school completion rates with students well-prepared

for life beyond high school• What are the most critical requirements?

– e.g. A teaching force that is : (i) demonstrably effective, (ii) adequately large, (iii) appropriately distributed

– e.g. Schools that are: (i) well led and collaborative, (ii) properly resourced, (iii) learning organizations

• What is the context in which the proposed reforms will be implemented?– e.g. (i) a technically flawed accountability system that generates

perverse incentives, (ii) problematic union-management relationships, (iii) concerns about over-reliance on test scores

• What are the warrants for “systems validity”?– e.g. (i) alignment of ends and means, (ii) coherence of system

components, (iii) feasibility, (iv) designs to mitigate negative consequences

Teaching: The Current Scene

• Initial entry based largely on credentials• Tenure an early and semi-automatic decision• Uniform salary schedules based on credentials and

years of experience• Seniority rights• High mobility rates (localized)• High levels of out-of-field teaching (localized)

The Argument

• Significant variation among teachers in effectiveness (as measured through value-added analysis)

• Existing barriers to entry are dysfunctional• Tenure decisions are now generally information-poor• Value-added measures of effectiveness are a credible

basis for decision-making• Current distribution of teacher quality is sub-optimal

Recommendation 1: Reduce barriers to entry

• A well-documented need to increase access to the teaching profession

• Current licensing standards are often “artificially” lowered

• How should we assess (pedagogical) content knowledge? What standards?

• How would this work in the elementary grades? • What about special education?• What would be the function of the M. Ed.?

Reflections on Teaching

“ I came to understand that teaching is hard and that being smart and well-educated doesn’t necessarily mean one will be good at it.”

(Checker Finn, on his first year of teaching

social studies in Newton HS, Newton, MA)

“The evidence indicates that neither an extreme central-ized bureaucratization nor a complete deregulation of teacher requirements is a wise approach for improving teacher quality.”

(Jennifer King Rice, 2008)

Recommendation 2: Modify tenure process

• A more rigorous evaluation for tenure is needed• Outcome-based indicators should be considered

• Better training for evaluators (e.g. Principals)• More equitable provision of support and mentoring• Lengthier probationary period• Salary consequences?

[The question we dare not ask: Why tenure?]

Recommendation 3: Introduce differential pay

• Inequitable distribution of experienced and more effective teachers

• Use substantial bonuses to shift incentives

• Need a stable index for “ranking” hard-to-staff schools that incorporates more than student disadvantage

• Avoid hard cut-offs• “Bonus” teachers should bring school resource dollars to

jumpstart improved working conditions• Must retain bonus-eligible teachers already in such

schools• Countervailing pressure of NCLB sanctions

Recommendation 4: Change teacher compensation

• Seven challenges are well worth considering!

• Improve credibility of all forms of evaluation• Pay attention to a number of desired outcomes (i.e., aim

for balanced accountability)• Don’t place excessive burden on test scores• Idiosyncratic school-level conditions are problematic

– School leadership– Out-of-field teaching– Grade-shifting

Recommendation 5: Develop data systems

• Yes, yes, yes [Molly Bloom (1922), Hillary Clinton (2008)]

• Federal government stepping up to the plate here!• Need substantial investment (and commitment) to build

(and to maintain) data warehouses• Many state education departments are under-staffed in

technology and data management

Using (Test-based) Performance Indicators (I)

• Intuitively attractive and commonsensical• Certain measures of performance contain useful

information • If indicators are seen as (relatively) fair, then attaching

stakes to performance focuses attention and can be constructive

• Provides a quantitative and comparable basis for decision-making

Using (Test-based) Performance Indicators (II)

• Cognitive outcomes are a subset of education goals• Current (state) tests are proxies of variable quality for the

target cognitive outcomes• Mis-alignment between proxies and targets can lead to

sub-optimal resource allocation (Murnane & Cohen, George Baker, and others)

• Disagreement on effectiveness of extrinsic motivation (Ed Deci, and others)

• Mixed findings on success of pay-for-performance schemes (Podgursky, Harris, and others)

• Implications of “Campbell’s Law”

Value-added Modeling (I)

• VAMs attempt to isolate the contributions of teachers to student learning as measured by test scores

• VAMs adjust current student performance for differences in prior performance and certain background characteristics (creating a “level playing field”)

• Superior to status models and growth models as a basis for performance evaluation

• Many models are in use and have been studied extensively with regard to their operating characteristics and sensitivity to departures from assumptions

• Policy use of VAM results implicitly relies on a causal inference

Value-added Models (II)

Fundamental technical problem in using VAMs

“In making causal attributions from observational data, no statistical model, however complex, and no method of analysis, however sophisticated, can fully compensate for the lack of randomization.”

(Braun, 2005)Practical Questions:(i) How close can we come?(ii) How do we know when we are close enough?(iii) What are the limitations on use?

Value-added Models (III)

Other technical considerations in the use of VAMs

• Test quality (substantive)• Test characteristics (psychometric)• Balancing model complexity against test information

density• Volatility• Bias

How do we go beyond math and language arts?

Value-added Models (IV)

Operational and political considerations

• Administrative complexity– VAM infrastructure– Multiple review systems– Training and auditing

• Fairness– Transparency– Level playing field– NCLB/AYP

• Core opposition to use of student results– Unions (e.g. New York state)– Concern with over-reliance on tests

T’was Ever Thus

For more than a hundred years much complaint has been made of the unmethodical way in which schools are conducted, but it is only within the last thirty that any serious attempt has been made to find a remedy for this state of things. And with what result? Schools remain exactly as they were!

John Amos Comenius, 1632(quoted by C. E. Silberman in“Crisis in the Classroom”, 1970)

The Status Quo Is Not Viable

• Demographic trends and static literacy distributions portend a downward shift in per capita human capital

(America’s Perfect Storm)

• Staggering opportunity costs in not increasing high school graduation rates

(The Price We Pay)

• More than 50% of non-capital school spending goes to teachers

(Understanding the Effectiveness of Teacher Attributes)

• Too many ships in the American convoy are taking on water

(Any newspaper)

CONCLUSIONS

• This is a thoughtful proposal to address a serious national need, grounded in the belief that data-based decision-making should play a greater role in shaping the teaching profession.

• I agree -- in principle!• As always -- the devil is in the details!• More attention is needed to:

– Full design of the new system– Infrastructure– School realities– Consequential validity

Parting Thoughts

Teacher compensation is the elephant in the room.

We need a constructive debate around various proposals to assist us in moving forward –

and rigorous evaluation to tell us what works and why!

Question yet to be answered:

Is the uniform salary schedule to teacher compensation as democracy is to forms of government --- the worst possible system, except for all the others?