Assessing Change in Changing Times

Did we change?

Justin Reich

Doctoral Student, Harvard Graduate School of Education

Co-Director, EdTechTeacher.org

@bjfr

Blog: edtechresearcher.org

Why Change?

How do we change?

Did we change?

Using Data to Address Instruction

Identify a Problem of

Practice

Gather data about the problem

Examine Instruction

Develop an Action Plan

Act and Assess

Further Reading: Data Wise and Data Wise in Action

The lower school at the American School of Brussels is working on increasing connections between home and school and having parents more aware of and involved in children’s classroom experiences.

They are requiring all elementary teachers to create blogs to help build these connections. How could you assess, short term and long term, whether this initiative is working?

Scenario 1

InterviewsFocus Group

SurveysContent Analysis

Test ScoreAnalysis

How would you test French Fry Quality at

McDonalds?

The Fakename School District has a Wordpress install for classroom blogging.

Teachers have been asked to consider Will Richardson’s definition of blogging, emphasizing student autonomy, creativity, and attentiveness to a real audience

Bud Hunt wanted to write his MA thesis about blogging in the district. What should he do?

Scenario 2

What did Bud do?• He took a sample of all of his school’s blog

posts• He read them• He categorized them• He drew some conclusions about the

kinds of learning taking place based on that analysis

American School of Doha has a 6-12 one-to-one program. They are implementing a 1-1 program in the fifth grade.

The tech directors want to assess the efficacy of their intervention.

What questions do you have? What could we do?

Scenario 3

What does awesome look

like?

The Newmarket High School has just purchased 300 iPads to allow them to pilot a 1 to 1 program in the ninth grade.

Teachers have been asked to create “21st century learning environments,” and they can use the iPads however they wish.

How could you assess whether this pilot should expand to the entire high school?

Scenario 4

Using Data to Address Instruction

Identify a Problem of

Practice

Gather data about the problem

Examine Instruction

Develop an Action Plan

Act and Assess

Further Reading: Data Wise and Data Wise in Action

Six Steps for Research1. Ask a Question

2. Define Indicators

3. Identify a Sample

4. Identify a Method

5. Gather Data

6. Analyze the Results

Ask a QuestionGood questions ask about how technology changes people’s experience of learning.

Two helpful prompts:

What is the learning goal before technology?

Is technology supposed to change the learning goal?

Define Indicators• Patrick Larkin said he wants students to

be more engaged. What are indicators of engagement?

• Chris Lehmann said he wants students to be able to ask good questions. What are indicators that students ask good questions?

Identify a Sample• Whole Population• Convenience Sample- whoever you can

get• Purposive Sample- a few, well-chosen

people• Random sample- often the best method

Identify a Method• Online Survey• Interview (phone or in person)• Classroom observation (online or in person)• Teaching Materials Review (online or in

print)• Looking at Student Work (online or in print) • Web Analytics• Test Score Analysis

Gather Data• Structured Data Collection Allows for

Structured Data Analysis– Ask all of your random sample of parents the

same four questions, and you can compare their answers

• Unstructured Data Collection Allows for Surprising Insight

Analyze the Results• Carefully crunch the numbers OR• Just look at stuff and see what you think

• How will the data make your school better?

Rules• Write your name on the back of your cards• Gain points by presenting research plans• Start Research with a Question Card• All cards only go face up, cards only connect on lines• Extend Research with Sample, sampling, method, and

other cards• All current cardholders must agree to add a new card• Cards only add constraints, anything else is allowed• Call Justin when you present, total points times

multiplier is score

Presentations• What is the question?• What are the indicators?• What methods will allow you to assess

those indicators?• How will the data gathered help you

answer to the question?• What could you do with your results to

improve the school?

Final Thoughts• Identify learning goals connected to

technology investments and interventions• Measure progress towards those goals• Draw random samples rather than

assessing everyone– (and actually make them random)

• Leverage partnerships with universities and other organizations

• Use assessment data to reflect upon and improve instruction

Visit us at [email protected]@edtechteacher.org