micro testing teaching learning analytics

Why micro testing is a necessity for teaching and learning. Learning analytics as teacher’s assistant.Microlearning 6.0 6th International Conference 2012, July 9th & 10th

Martin Schön Life Long Learning Graz University of Technology Mandellstraße 13, A-8010 Graz +433168734931 [email protected]

Martin Ebner Social Learning Graz University of Technology Münzgrabenstraße 35A/I, A-8010 Graz +433168738540 [email protected]

ContentOr: this is a story,

b) how we began to think about an educational problem (multiplication tables)

c) how we tried to solve it with a small application

d) and how this leaded us to Learning Analytics & Educational Data Mining

Idea

At the Graz University of Technology some students of Martin Ebner had to develop applications during their studies of information technology (Master of Science).

Task: Simulate a diagnostic situation: A student is introduced to a teacher. He/she tries to gain an overview of the tasks which the students can do reliable well. This should be done in an optimal way.

Task

• The application should test the competence in practicing the multiplication tables

• The system should provide appropriate tasks according to the competence grade of the learner.

• The system should ensure that already well-done exercises are repeated and practiced. After succeeding a problem the probability for a repeated display should decrease (similar

to a Leitner System )

Checklist cont. 2

• Nevertheless the tasks should tend to be challenging.

• In general the system should be motivating and show that learning can be fun.

• The system should record and safe fine-grained data of all done exercises, test results and the current competence grade of the learner in order to prepare the next sessions in an adequate way.

• It should relieve the teachers from the unsolvable task of remembering all data from every child – it should be user as an information processing tool

• That means the teachers should get help and no additional time consuming tasks

Checklist – cont. 3

• during this spring we defined another goal: We wanted to give teachers the opportunity to set up and watch a whole class - Today we are able to present a database for organizing schools, classes and individuals – with an interface for the conventional desktop browsers and also for Android or iOS operating mobile devices.

• the web-application is written in php/mysql –

• the apps in objectiveC (iPhone/iPad) & JAVA (Android).

Measurement

• the goal is to generate a complete table to inform learners as well as teachers in a deterministic way about the competences in every single task, in every single multiplication fact

This is opposite to our experience in interviews:• Teachers talk about quantity:

“I have 4 pupils who …. And 6 who … ….

Algorithm

The presentation of the tasks is not only randomly generated.. At the very first contact a moderate problem is presented. According to the results more or less difficult tasks are following.

The competence level of the test subject is afterwards estimated and after every task recalculated. This determines the difficulty of the next tasks.

Competence level

1. estimate the competence level:

2. After every new solved or unsolved problem we choose a smooth way for adopting this first estimation of competence to the experience during the sessions

Difficulty

• Without any empirical data we decided to use the ranking for the tables as follows (You´ll find this hierarchy in most didactical concepts):

easy …. 1, 2, 4, 3, 5, 8, 6, 7, 9 … difficult

• we transformed these ranking in difficulty levels between 1 and 0.

• We discussed to integrate a statistical founded ranking, but at

this point we fear, the teachers would be somewhere confused about the results. I’ll speak about this problem later.

Stored Data

• The answers of the learners are marked with 0,1 or 2:

• 1 shows that the user knew the correct answer once

• 2 indicates that the student had two consecutive correct answers (we say a question is “well known”)

• 0 indicates, the last answer was incorrect (or this item was presented never before)

Additional stored: a time line with needed time, results, task no., ever presented,

Selecting the next item

We use a random number between 0 and 1 to decide, which category is activated to generate the next multiplication problem: Therefore three cases are defined:

• Case 1: If the random number is 0<x<0.05 than a well-known question (marked with 2) is chosen.

• Case 2: The random number is 0.05 > x >= 0.15 than a known question marked with 1 is chosen.

• Case 3: The random number is x > 0.15 than an unknown question out of the extended and actual learning area is chosen.

These parameters can be changed for the total system, not for Indivuduals

Extended Area

• Extended area means the idea, that we choose items not only in the learning area, that means under the level of competence.

• To produce some dynamic and the chance to get a higher level, we add now 0.15 to the actual estimation of the degree of competence (0 …1) of the student

Prototype

Study

• first research study was carried out at a primary school in Austria.• Begin: summer semester 2011 • 43 pupils of the primary school Laubegg (age: 9-10). • at least 4 weeks. Some of the learners ignored this time restriction and

played the game again and again over months. • Learners learned on computers at the school as well as on their

personal computers at home. • 12.926 answers where given which means that on average each learner

answered 308 questions- they did 3.4 times the whole multiplication table.

• Bearing in mind that there was no real pressure from teacher’s side using the program it is a considerable pleasant high number.

• Furthermore it can be stated that pupils seemed to enjoy using the application or at least get not bored.

Highscores

Demotivated learner

One learner with a weak performance attracted our attention because of a very high number of trials (513). A detailed inspection showed that he did not work very intensively. In the first two tasks he/she failed, then eleven tasks were ok, his performance rose abruptly. However, afterwards, he continued approximately 400 times to wait the whole answering time without doing anything but asking for a new assignment.

Motivated captain

The most diligent learner. In the beginning, the assignments were solved correctly, then some mistakes occurred, afterwards a learning process can be recognized and finally with some occasional mistakes the learner works on a high performance level. Obviously, the learner was highly motivated to deal with the assignments given by the program.

Medium learnerIn the beginning, the learner made mistakes in every second assignment (0.5), followed by 7 mistakes consecutively. This is the reason for the big decrease (0.15). Afterwards, the learner gave a number of right answers and the rate of correct answers increased back to 0.5. In the following phase an up and down can be seen till a number of right consecutive answers helps to reach a level of 0.7. But then the number of mistakes rose again and the rate went down to about 0.5.

Medium learner 2

Weak learner

Classified as 2 id156

0

2

4

6

8

10

1 11 21 31 41 51 61 71 81 91 101

111

121

131

141

151

161

171

181

191

201

211

We don‘t know (now):More than half of the learners did not reach the 100 percent level (= 90 items are well-known). Therefore we have to think about this group of learners: Perhaps they

• didn´t get used to / have problems with the interface

• do not know the necessary operations; are not able to solve the learning problem correctly

• misinterpret an assignment

•are distracted by the environment, are badly concentrated for several reasons.

-> Could we gather more information with additional devices to know more?

Educational Data Mining

• At this point, after all the preparations and programming, with the first incoming data we began to realize that we had taken a step into a whole new age.

• We had begun to program and improve a simple cybernetic loop. At the beginning we saw just the necessity to store the performance data. Now we find us in the situation, that we want to collect “everything” – We want to store all tracks left by a learner. And we want even to produce more data, even in that we use additional instruments and sensors.

• We realized that this idea of collecting more and more data implies completely new perspectives to perceive and to reconstruct some learning processes (not any!)

Learning Analytics

Learning analytics is an educational approach aimed to improve education and facilitate learning by the systematic analysis and interpretation of digital tracks.

Through the use of more and more intelligent ubiquitous information technology for communication and organization, more and more events are created for leaving digital traces.

With the increasing data floods also the importance of this paradigm increases.

Interim conclusions

• It is as easy as never before in history to collect data

• We perceive much more details about the learners – the learning process – as never before!!

• Teachers get precise information

• This application was designed for diagnostic purposes, to get an overview on problems with multiplication facts. Now we see, it can be effectively used for training and facilitate learning

mathe.tugraz.at

Our latest design:

Informationprocessing

• The matrix at the left shows the history: The beige colour indicates “well known” results, the brown is “one right answer”. The rabbit comes from left to right to the carrot and catches one - if the pupil produces consecutive right answers.

DetailsIf you point over with the mouse on

the matrix, you see the multiplication fact behind the symbol.

Register (it`s simple!)

If you want to test this application you can use “no school” for an individual registration. If you want to administer a whole school and a couple of class lists … contact [email protected] to send you an administration-account.

Analysis

I´ll show here some graphs of the collected statstical data.

We made the experience, that the teachers in our interviewsare interested in very short compact information, overviews. Therefore the following graphs are only for scientifc purposes

We know, that if we would produce too much information many teachers will ignore the entire system ….

New findings This new applications are public available since March 2012.

Session time (s)

Well - known Perhaps someone is interested to get more information on the difference of signing the tasks with „known“ and well known:

Results for teachers

If the analysis shows more than 12 correct answers during the last 20 tasks the traffic light will show green, with less than 5 correct answers it is indicated red, otherwise the student and of course also teachers will see yellow. In retrospection to the current situation this would lead only to 5% red and yellow alerts.5% red and yellow alerts.

Future Work • closer look at the learners: perhaps more

intelligent analyses of the data? – more data – more concepts? Hints for Learners?

• Mathe-Multi-Trainer (multi digit multiplication)

• Cooperative Learning• Test and Training in Reading

Acknowledgements

We express our gratitude to the teachers of the primary school in Laubegg (Styria, Austria) as well as all participating school children. We are equally indebted to our funding agency “Internet Foundation Austria (IPA)” for supporting our ideas and helping us with netidee to work on the future of education.

Thank you!

Martin SchönTU Graz – [email protected]