Data use for improving teaching and learning in schools ... · • Quantitative data and qualitative data • Examples of data: demographic data, classroom observations, student surveys,

Data use for improving teaching and learning in schools: From challenges to opportunities

Zurich, 22-06-2017Kim Schildkamp, University of Twente, [email protected]

Content of this presentation

• Data-based decision making• Definition• Importance• Use of data

• Challenges in the use of data at policy level, school level and teacher level*

• An intervention to support data use: The datateam intervention

*Based on: Schildkamp, Karbautzki & Vanhoof, 2014; Vanhoof & Schildkamp 2014; Schildkamp, Ehren, & Lai, 2012;

Schildkamp, Heitink, van der Kleij, Hoogland, Dijkstra, Kippers, & Veldkamp, 2014; Schildkamp, Lai, & Earl, 2013.

Data-based decision making (DBDM)

• The use of data, such as assessment results, to improve education (Schildkamp & Kuiper, 2010)• Systematically collect• Analyze and interpret data• Use this information to improve education

• Quantitative data and qualitative data• Examples of data: demographic data, classroom

observations, student surveys, parent interviews, assessment results

Data in education

• Technological innovations in todays society• More and more data are available: e.g., Switzerland

after the introduction of a new competence-based curriculum (LP21) the basic competences will be measured all three years.

• Pitfalls: Data generated faster than school staff can use, data use as a goal instead of tool, lack of knowledge and skills, possibilities of misuse

• Promises of data: timely updates on the quality of education for different stakeholders, information on achievement, information on learning needs, tool in the school improvement process

Intuition or data-based decision making?

Important questions to ask:o Are decisions in education based on data, or on

intuition and assumptions?o If decisions are based on intuition, how accurate is

this intuition?

How accurate is your intuition? Let’s check: true or false

True or false?

Boys are better in mathematics than girls

A. TrueB. False

True or false?

Students have different learning styles to which you need to adapt your instruction in order to enhance achievement

A. TrueB. False

True or false?

Data-based decision making can lead to increased student achievement

A. TrueB. False

Challenges at policy level

• Access to relevant and high quality data (systems)• Data use as a balancing act:

• Amount of pressure (e.g., high stakes testing, sanctions)

• Amount of support (e.g., data systems, training)

• Amount of autonomy (e.g., centralized or decentralized)

• Accountability – school improvement (e.g., tension can lead to strategic use, misuse, and abuse)

• Data use as a goal instead of a tool• Important discussion: Who is accountable? To whom?

For what? In what manner? Under what circumstances? Different in different countries

Challenges at the level of the school

• Lack of collaboration around the use of data • Between school leaders and teachers• Between teachers

• Lack of expertise, for example, a data expert • Lack of a data use culture (e.g., vision, norms, goals)• Lack of school leader support in the use of data (e.g.,

facilitation, role model, distributed leadership)• Lack of training and professional development in the

use of data systems and in the use of data• Lack of time (or lack of priority?)

Challenges at the level of the teacher• Negative attitude: “I don’t belief in the use of data”• Social pressure: Data use done to the school, data use as

a goal (e.g., 80% of the school uses data) instead of tool • Lack of ownership over data and student learning• Lack of perceived behavioral control: lack of autonomy,

and/or “my measures will not influence student learning”• Lack of collaboration in data use• Difficulties in goal setting• Lack of data literacy: Knowledge and skills how to use

data to improve education (PD needed needed)

Too many data: where to start?

How problems often are solved

Problem Measure

The datateam® procedure• Teams 6-8 teachers and

school leaders

• Educational problem: low student achievement, safety

• Goals: professional development and school improvement

• Coach guides them through the eight steps (1-2 years)

• Data analysis courses

A little bit of historyyear Development

2009 Small scale pilot with 5 schools (secondary education)

2011 From regional to national 24 schools for secondary education 1 school for higher education (teacher training college)

2013 From national to international 10 schools for secondary education 4 schools in Sweden (secondary education)

2014/2015 Upscaling 15 schools in the Netherlands (primary and secondary education) 13 schools in Sweden (primary and secondary education) 1 school in England (secondary education)

2015/2016 Vocational education Belgium and the USA

Step 1: Problem definition

• Identify a current problem in the school• School-wide or subject-specific

• Proof that you have a problem• Collect data on current situation and desired situation • Three cohorts/years

• Example:• Current situation: ‘45% of our students is failing mathematics’• Desired situation: ‘Next year no more than 30% of our students is

failing, the year after that no more than 15%.’

Step 1 Problem definition examples

• Topics in the Netherlands, all in the cognitive domain:• Student achievement in a specific subject• Final examination results• Grade repetition

• Topics in Sweden, in the cognitive and social domain:• Student achievement in a specific subject• Stress• Safety• Classroom climate

Step 1 our problem definition

‘We are not satisfied with the number of students repeating the fourth grade of secondary education. Over the last three years, on average 20% of our students had to repeat the fourth grade (N=135)

Next year, we want to achieve that no more than 15% of our students have to repeat the fourth grade, and the year after that this should be no more than 10%.’

Step 2: Formulating hypothesis• Brainstorm possible causes

Ask colleagues for input Make a list

• Choose a hypothesis Based on criteria, such as: what can we influence as a school? Which

hypothesis do a lot of colleagues believe to be true? What is according to the literature a possible cause?

• Formulate a hypothesis Concrete Measurable

Assignment step 2

• In groups of three• You are working in a data team on the following problem:

‘We are not satisfied with the number of students repeating the fourth grade of secondary education. Over the last three years, on average 20% of our students had to repeat the fourth grade (N=135). Next year, we want to achieve that no more than 15% of our students have to repeat the fourth grade, and the year after that this should be no more than 10%.’

• Discuss possible causes of this problem, and make a list of possible causes• Choose one possible cause, and try to make it measurable

Step 3: Data collection

• Available data• Existing instruments• Quantitative and qualitative

• Examples:• Student achievement data• Surveys: motivation, feedback, curriculum coherence• Classroom observations• Student interviews, teacher interviews

Step 4: Data quality check

• Reliability and validity of the data

• Crucial step: not all available data are reliable and/or valid!

• Examples:

• Validity problems with survey

• Missing data

• Data of one year only

Step 5: Data analysis

• Qualitative and quantitative

• From simple to complex

• Extra support needed: course data analysis

• Examples:

• Average, standard deviation

• Percentages

• Comparing two groups: t-test

• Qualitative analyses of interviews and observations

Step 6: Interpretation and conclusions

• Is our hypothesis rejected or confirmed?• Rejected: go back/ further to step 2 • Accepted: continue with step 7

• 32 data teams (2012-2014):• 33 hypotheses: accepted• 45 hypotheses: rejected• 13 (qualitative) research questions• 13 hypotheses: no conclusion

due to limitations of the dataset

Step 7: Implementing measures

• Develop an action plan:• Smart goals

• Task division and deadlines

• Means

• Monitoring progress: how, who, which data?

Step 7 Improvement measures examples

• Netherlands• More intensive mentoring• Implementation of formative assessment• Instructional changes, such as improvement of feedback

• Sweden• Improvement of data collection and data sharing • Increased monitoring and follow-up of student absence• Improve the safety in places where students reported

feeling unsafe

Step 8: Evaluation (process)

• Process evaluation• Are the measures implemented the way we want?• Are the measures implemented by everyone?

• Example process evaluation:• Measure: start every lesson with a short repetition of

percentages in the form of a quiz to increase mathematic achievement

• Interview students: this is boring, start to detest percentages!• Adjust measures: repeat percentages only once a week

Step 8: Evaluation (effect)

• Effect evaluation:• Is the problem solved?• Did we reach our goal as stated in step 1?

• Example effect evaluation:• Did our measure(s) results in increased mathematics

achievement?

Assignment step 4-8

• In groups of three work on the assignment: Read the assignment Judge the quality of the data (step 4): If the data are of sufficient quality:

Look at the data analysis (step 5) Draw a conclusion (step 6) Develop an action plan (step 7) Think about the evaluation (step 8)

If the data are of insufficient quality: Determine how to collect new data (back to step 3) and/or investigate a new hypothesis (back to step 2)

Research results

• How do data teams function?

• What are the influencing factors?

• What are the effects of data teams?

• Results are based on three studies conducted in the Netherlands (Schildkamp, Handelzalts, & Poortman, 2015; Schildkamp & Poortman, 2015; Hubers, Schildkamp, Poortman, & Pieters, 2016) and one study in Sweden (Schildkamp, Smit, & Blossing, 2016)

Data team functioning• Difficult to formulate a measurable hypothesis

• Several rounds of hypotheses: first hypotheses often wrong

• Often external attribution: problem is caused by primary schools, by policy etc.

• However, this is necessary: need to create trust; practice with the eight step procedure; learning starts when you make mistakes; shows the importance of data

• From external to internal attribution

• Knowledge dissemination needs more attention

Functioning: depth of inquiry

• From intuition to data• From knowledge to

school improvement

Conditions and effects

Level 1: Teacher satisfactionregarding datateam procedure

Level 2: Teacher learning results from datateam procedure

Level 3: Teacher use of knowledge and skills from datateam procedure

Level 4: Student achievement

Conditions for effective

professional development (specifically in

data use)

Framework of effects from teacher satisfaction to increased student achievement (based on: Kirkpatrick, 1996; Guskey, 1988; Desimone, 2009; Desimone, Smith & Phillips, 2013)

Data team

Team:Attitudeknowledge and skillsShared problem goalsComposition of teamParticipation andcollaboration

Data:Access to dataHigh quality relevant data

School organization:FacilitationLeadershipVision, norms, goals

Policy: Municipality, inspectorate, coach

Effects (NL)Effects level Instrument(s)Level 1: satisfaction

• Satisfied about support, process and progress• ‘good’; ‘fun’

Level 2:knowledge, skills, attitudes

• Knowledge and skills increased significantly• ‘learnt how to use calculations in Excel’; what + how of

qualitative analysis; ‘you really need evidence’

Level 3: use of learning

• Data use for instruction: e.g., prepare students betterfor exam (explanation and practice)

Level 4: student achievement

• Five out of nine schools solved problem: Significantincrease in student achievement

Conclusion and discussion

• Data teams: From ‘intuition-based decision making’ to ‘data-based decision making’

• Change in school culture: “You want to take decisions based on assumptions, that is not the way we work here anymore”

• Support schools in solving problems and achieving goals• Importance of knowledge sharing within and outside the team• Need to invest in sustainability from the start: Data use as an

organizational routine • Increased student learning

ANY QUESTIONS?THANK YOU FOR YOUR ATTENTION!

Kim Schildkamp: [email protected]

Primary references• Campbell, C., & Levin, B. (2009). Using data to support educational improvement. Educational Assessment, Evaluation and Accountability, 21(1), 47–65. • Carlson, D., Borman, G., & Robinson, M. (2011). A multistate district-level cluster randomized trial of the impact of data-driven reform on reading and mathematics

achievement. Educational Evaluation and Policy Analysis, 33(3), 378–398. • Desimone, L. M. (2009). Improving impact studies of teacher’s professional development: Toward better conceptualizations and measures. Educational Researcher,

38(3), 181–199• Ebbeler, J., Poortman, C. L., Schildkamp, K., & Pieters, J. M. (2016). Effects of a data use intervention on educators’ use of knowledge and skills. Studies in Educational

Evaluation, 48, 19-31. • Ebbeler, J., Poortman, C. L., Schildkamp, K., & Pieters, J. M. (2016). The effects of a data use intervention on educators' satisfaction and data literacy. Educational

Assessment, Evaluation and Accountability.• Guskey, T. R. (1998). The age of our accountability. Journal of Staff Development, 19(4), 36–44.• Kirkpatrick, D. (1996). Great ideas revisited, Techniques for evaluating training pro-grams. Revisiting Kirkpatrick’s four-level model. Training & Development, 50(1), 54–

59• Lai, M. K., & Schildkamp, K. (2016). In-service Teacher Professional Learning: Use of assessment in data-based decision-making. In G. T. L. Brown & L. R. Harris (Eds.).

Handbook of Human and Social Conditions in Assessment (pp. 77-94). New York: Routledge.• McNaughton, S., Lai, M., & Hsaio, S. (2012). Testing the effectiveness of an intervention model based on data use: A replication series across clusters of schools. School

Effectiveness and School Improvement, 23(2), 203–228. • Poortman, C.L., & Schildkamp, K. (2016). Solving student achievement focused problems with a data use intervention for teachers. Teaching and Teacher Education, 60,

425-433. • Schildkamp, K., & Kuiper, W (2010). Data-informed curriculum reform: Which data, what purposes, and promoting and hindering factors. Teaching and Teacher

Education, 26, 482-496.• Schildkamp, K., & Poortman, C.L. (2015). Factors influencing the functioning of data teams. Teachers College Record. • Schildkamp, K., Poortman, C. L., & Handelzalts, A. (2016). Data teams for school improvement. School effectiveness and School Improvement, 27(2), 228-254• Schildkamp, K. Karbautzki, L., & Vanhoof, J. (2014). Exploring data use practices around Europe: Identifying enablers and barriers. Studies in Educational Evaluation, 42,

15-24. • Schildkamp, K., & Ehren, M., & Lai, M.K. (2012). Editorial paper for the special issue on data-based decision making around the world: From policy to practice to results.

School Effectiveness and School Improvement, 23(2), 123-132. • Schildkamp, K., Heitink, M., van der Kleij, F., Hoogland, I., Dijkstra, A., Kippers, W. & Veldkamp, B. (2014). Voorwaarden voor effectieve formatieve toetsing. Een

praktische review. Enschede: Universiteit Twente.• Schildkamp, K., Lai, M.K., & Earl (Eds.) (2013). Data-based decision making in education: challenges and opportunities. Dordrecht: Springer.• van Geel, M., Keuning, T., Visscher, A. J., & Fox, J. P. (2016). Assessing the Effects of a School-Wide Data-Based Decision-Making Intervention on Student Achievement

Growth in Primary Schools. American educational research journal.• Vanhoof, J., & Schildkamp, K. (2014). From professional development for data use to ‘data use for professional development. Studies in Educational Evaluation, 42, 1-4.