The Role of Information in Systems · Professional development ... and Professional development) for the trees (of individual ... proficient over six years of implementing learning

Problem States, districts and schools are being asked to

conceptualize and implement broad, coherent systems Assessment Instruction Curricula Professional development

Goal of these systems is to increase student achievement

But the intricacies of these multi-component systems threaten to overwhelm the ability to understand and successfully plan these systems

Problem How are states and districts responding to the demand to conceptualize, award, develop and implement these coordinated and

complex systems? This session is intended to--

Help conceptualize the problem, Identify important aspects of assessment, curriculum,

instruction and professional development, Describe Systems Thinking as a conceptual tool, and Describe simulation as a statistical tool

Systems Thinking What is Systems Thinking? Systems Thinking means being able to see the underlying

web of ongoing, reciprocal relationships which are cycling to produce the patterns of behavior that a system is exhibiting

Systems Thinking means being able to see the forest (of relationships amongst Assessment, Instruction, Curricula and Professional development) for the trees (of individual tests)

Hard to do because we are crossing boundaries of expertise and authority

Session Jeff Barker, Executive Director, Student

Accountability and Assessment, Gwinnett County Public Schools

Jeff will discuss concerns in creating a coherent system of assessment, curriculum, instruction and professional development from the district perspective

Session Melissa Fincher, Associate Superintendent,

Assessment and Accountability, Georgia Department of Education

Melissa was going to discuss concerns in creating a coherent system of assessment, curriculum, instruction and professional development from the state perspective

Melissa was unable to attend

Session Paul Nichols, Principal Research Scientist, Pearson Paul will discuss using simulations to help design

coherent systems of assessment, curriculum, instruction and professional development

Session Kristen Huff, Senior Fellow, Regents Research Fund,

New York, NY Kristen will pull the two parts of this session together--

the state and district perspectives and a formal simulations-- and move toward incorporating systems thinking into assessment design and development.

Please Go in Order! Please rotate through the presentations in

order! (1) Jeff (2) Paul (3) Kristen

They will make more sense that way!

Paul Nichols

Center for Next Generation Learning & Assessment Pearson

Problem Recognition that improved student learning requires

coordinated assessment, instruction, curricula and professional development

States and districts are being asked to conceptualize and implement assessments as a component of broad and coherent systems

For example, comprehensive assessment systems or systems for learning.

New burdens on district and state staff

Problem Think beyond a single test or set of tests to

coordinated system of assessment, curriculum, pedagogy and professional development

Manage and held accountable for the award, development and implementation of multi-component systems

Cross department boundaries, engage multiple-vendors and involve many times the cost of a testing program

Status Quo State and district staff already have or can find the

training and tools to help with test development and implementation for individual tests

Currently no theoretical framework and no indices to help state, district and other educational leaders in planning, developing and evaluating systems for learning

Systems Approach Move beyond hand waving to offer a conceptual

framework and quantitative tools used by district and state staff to conceptualize and implement systems for learning (SFL)

Note the components that probably should be included in an SFL

Present simulation allowing district and state staff to predict SFL coherence across different configurations

Defining Systems What is a system? From the field of systems thinking:

A system is “a collection of parts which interact with each other to function as a whole”

An SFL is a set of components which interact with each other to function as a whole with the intention of improving student learning

Conceptual Framework The Learning Framework offers a conceptual

framework

Conceptual Framework Curriculum consists of the knowledge and skills in subject

areas that teachers teach and students learn. Instruction refers to methods of teaching and the learning

activities used to help students master the content and objectives specified by a curriculum.

Assessment is the means used to measure the outcomes of education and the achievement of students with regard to important competencies.

Professional development may be in schools of education or provided by schools and districts

Theory of learning is the model of student cognition and learning in the domain

Conceptual Framework Coherence is provided by the theory of learning The components must “talk to each other” in a

common language The language of communication is the learning theory

Quantitative Tool Simulation of the conceptual framework built with the

visually oriented graphical programming language STELLA

STELLA is a commercially available graphical simulation program

STELLA was chosen for this project because it is a powerful yet easy-to-use dynamic system modeling computer language

Quantitative Tool High level view of the simulation of the Learning

Framework under default assumptions

Leave Teaching Force Grd3

Year Begin InstrucPD

Leave Teaching Force Grd3

Receiving Instruc Innovation

Receiving Curr Innovation

Receiving Assess Innovation

Year Begin CurrPD

Year Begin AssessPD

InGrd3 ChCA

Grd3 ConCh Instruct PD

Grd3 ConCh Curr PD

Grd3 ConCh Assess PD

In Grd3 ChIPD ?

In Grd3 ChCPD ?

In Grd3 ChAPD ?

Grd3 ConCh Intruc Attmpt

Out Grd3 ChIPD

Out Grd3 ChCPD

Out Grd3 ChAPD

Grd3 ConCh Curr Attmpt

Grd3 ConCh Assess Attmpt

Grd3 ConCh Intruc NO Attmpt

Grd3 ConCh Curr NO Attmpt

Grd3 ConCh Assess NO Attmpt

InGrd3 ChIA

InGrd3 NoChIA

InGrd3 NoChCA

InGrd3 ChAA

InGrd3 NoChAA

OutGrd3 ChIA

OutGrd3 NoChIA

OutGrd3 ChCA

OutGrd3 NoChCA

OutGrd3 ChAA

OutGrd3 NoChAA

Grd3 ConCh Intruc Implmnt

InGrd3 ChII OutGrd3 ChIIGrd3 Assoc Intruc NO Implmnt 2

InGrd3 NoChIIOutGrd3 NoChII

Grd3 ConCh Curr Implmnt

InGrd3 ChCI OutGrd3 ChCIGrd3 ConCh Curr NO Implmnt

InGrd3 NoChCI OutGrd3 NoChCI

Grd3 ConCh Assess Implmnt

InGrd3 ChAI OutGrd3 ChAI

Grd3 ConCh Assess NO Implmnt

InGrd3 NoChAI OutGrd3 NoChAI

Leave Teaching Force Grd3Leave Teaching Force Grd3

Leave Teaching Force Grd3

Quantitative Tool View of the simulation of the Student Achievement

Subsystem

Grd3@A Grd4@A Grd5@A Grd6@A Grd7@A

InGrd3@A InGrd4@A InGrd5@A InGrd6@A InGrd7@A

OutGrd3@A OutGrd4@A OutGrd5@A OutGrd6@A OutGrd7@A

Cohort Size A

SuccessRate@A

SuccessRate@A

SuccessRate@A

CtchUp@A

SuccessRate@A

SuccessRate@A

SuccessRate@A

Grd8@ABlowGrd3@A

InBlowGrd3@A InGrd8@A

OutBlowGrd3@A

SuccessRate@A

OutGrd8@A

Remdl@ARemdl@A

Abv eGrd8@A

InAbv eGrd8@A

OutAbv eGrd8@A

Remdl@A

Remdl@ARemdl@ARemdl@A

InitializeTime A

InitializeTime A InitializeTime A InitializeTime A InitializeTime A InitializeTime A InitializeTime A InitializeTime A

CtchUp@A

CtchUp@A

CtchUp@A

CtchUp@A

Quantitative Tool A default static simulation Assumptions: Instruction, curriculum and assessment

implicitly or explicitly reflected an associationist learning approach

No professional development was taking place No changes in classroom practice across years

Quantitative Tool Distribution of student achievement across grades 3 to

8 under a default simulation

Grade

Proficient Below Grade

3 Proficient at

Grade 3 Proficient at

Grade 4 Proficient at

Grade 5 Proficient at

Grade 6 Proficient at

Grade 7 Proficient at

Grade 8

Proficient Above

Grade 8

3 100 1800 100 0 0 0 0 0

4 10 317 1587 86 0 0 0 0

5 1 38 415 1471 75 0 0 0

6 0 4 67 456 1407 66 0 0

7 0 1 10 90 468 1374 59 0

8 0 0 1 15 108 464 1359 53

Quantitative Tool Distribution of student achievement across grades 3 to

8 under a default simulation

0

200

400

600

800

1000

1200

1400

1600

1800

2000

1 2 3 4 5 6 7 8

Grade 3

Grade 8

Grade 5

Below 3 3 4 5 6 7 8 Above 8 Proficient at Grade

Quantitative Tool An evolving simulation in which teachers integrate an

innovation into their classrooms Innovative program where curriculum, instruction and

assessment based on a conceptual change theory of learning, i.e., a learning progression

Students are evaluated using a conventional standardized test, based on an associationist model, at the end of the year

Teacher retention 5% of grade 5 teachers leaves the district each year 5% are hired as replacements each year

Quantitative Tool Groups of 20 grade 5 teachers complete professional

development every year All teachers who attend professional development do

experiment with the innovations in their classrooms All teachers who experiment with the innovations find

that the innovation fit their classroom conditions Grade 5 students are slightly more likely to be increase

an achievement level on the standardized test by the end of the year

None of the newly hired teachers have received training in learning progressions

Quantitative Tool The distribution of grade 5 teachers across the

categories of implementing learning progressions

Year Receiving

Professional Development

Exploring Innovations

Implementing Innovations

Implementing Associationist

Approach

Ratio Implementing

Innovation/ Total

1 20 0 0 80 0.00

2 20 19 0 61 0.00

3 20 19 18 43 0.18

4 20 19 35 26 0.35

5 20 19 51 10 0.51

6 10 19 67 4 0.67

Quantitative Tool Distribution of Grade 5 students across levels of

proficient over six years of implementing learning progressions

Year

Proficient Below

Grade 3 Proficient at Grade 3

Proficient at Grade 4

Proficient at Grade 5

Proficient at Grade 6

Proficient at Grade 7

Proficient at Grade 8

Proficient Above

Grade 8

1 4 92 515 1323 66 0 0 0

2 4 92 515 1323 66 0 0 0

3 4 89 501 1339 67 0 0 0

4 2 87 488 1355 68 0 0 0

5 1 79 476 1373 71 0 0 0

6 0 69 462 1394 75 0 0 0

Quantitative Tool Distribution of Grade 5 students across levels of

proficient over six years of implementing learning progressions

0

200

400

600

800

1000

1200

1400

1600

1 2 3 4 5 6 7 8

Year 1 Year 6

Below 3 3 4 5 6 7 8 Above 8 Proficient at Grade

Conclusion Simulations can provide unexpected results Under the approach to professional development in

the learning progressions simulation, Entire grade 5 teaching force is not trained after 6 years Impact on EOY test performance is minimal

Suggestions for improving outcomes-- Hire teachers already trained in learning progressions Construct an EOY test that is more coherent with the

learning progression innovation

Summary State and district staff are asked to conceptualize and

implement SFLs State and district staff need a conceptual framework

and quantitative tools to help plan and evaluate SFLs Measurement theory focuses on individual tests or

groups of tests Instructional theory focuses on instructional

techniques and activities Simulations offer some help…