LNCS 8658 - Monitoring Student Activities with a Querying ...nbaloian/UCAMI2014/... · Keywords: Monitoring students’ work, automatic assessment, improving classroom teaching, architectures

N. Baloian et al. (Eds.): CRIWG 2014, LNCS 8658, pp. 38–52, 2014. © Springer International Publishing Switzerland 2014

Monitoring Student Activities with a Querying System over Electronic Worksheets

Nelson Baloian1, Jose A. Pino1, Jens Hardings1, and Heinz Ulrich Hoppe2

1 Universidad de Chile, Department of Computer Science, Beauchef 851, Santiago, Chile 2 University of Duisburg-Essen, Lotharstr. 63/65, 47048 Duisburg, Germany

Abstract. Monitoring students’ work in the classroom has been recognized as one of the key factors for successful teaching since only a good real-time assessment enables the teacher to give proper and timely feedback. However, it is not an easy task to systematically supervise what students do in the classroom. It also might consume a considerable amount of teachers’ resources. This paper presents a work in which computer technology is used in classrooms by students working on electronic worksheets on their. We explore the possibilities of assessing students’ work during classroom by automatically analyzing the structure of the documents and the changes along time while students work on them. An experiment is described, showing the system is able to give the teacher valuable information. This information is intended to assess the students’ performance and provide them with proper feedback.

Keywords: Monitoring students’ work, automatic assessment, improving classroom teaching, architectures for educational technology systems, group workspace awareness.

1 Introduction

Despite progress in distance learning, classroom settings continue being massively used in education throughout the world. Of course, the context is not the same than even a few years ago. Students at all levels are aware of at least some Information Technology (IT) tools and thus, computer pervasiveness has made its definitive entrance in schools. Our vision for the future is The Collaborative Classroom (TCC), an evolution of our previous proposal, the Computer-integrated-Classroom (CiC) [2]. TCC includes IT hardware for all educational roles as they may be suitable [7] and recent educational software technology, implementing mechanisms supporting teachers and students’ work inside the classroom. One of them is Learning Analytics (LA) [5], of which we have our own version. As it is known, most LA efforts are not for real-time decisions; the work presented in this paper concerns LA for teacher’s use during classroom activity.

In particular, the focus of this article is on automatic monitoring of students’ progress for quick adjustment of the teacher’s work. It may also be considered as provision of teacher’s awareness on the students’ workspace activities. An example may clarify these statements. Suppose a high school mathematics class in which the

Monitoring Student Activities with a Querying System over Electronic Worksheets 39

students are supposed to individually solve sets of two linear equations; the teacher has previously explained how to do it and now the students are asked to solve their own equations. The problem we are dealing with is how to provide a suitable tool for the teacher to quickly grasp the solution progress of all students so that she can decide on whether to move ahead to more challenging equations or go back to reinforce the previous level of difficulty.

For many theories of learning and instruction, feedback is an essential part of the learning model, absolutely necessary to successfully achieve learning [3]. In [9] authors mention that feedback is one of the most powerful factors influencing learning and achievement, but this impact can vary in effectiveness depending on the type of feedback provided and the way it is given. Effective feedback requires teachers to make appropriate judgments about when, how, and at what level to provide it. Activities supplying teachers with relevant information to make these judgments are commonly known as assessment.

In general, the systematic monitoring of the students’ work should be a key success factor, since teachers will be better prepared to give meaningful and timely feedback when they are aware of the students’ current learning state [6]. According to [4], the existing educational research literature identifies the practice of monitoring student learning as an essential component of high-quality education. But monitoring students during in-classroom work may involve teachers moving around the classroom, being aware of how well (or poorly) students are progressing with their assignments, and working with students one-to-one as needed. These activities might be quite time consuming and sometimes difficult to perform even in classes with a reduced number of student. According to [8] due to the need to attend all students individually, teachers find it difficult to accomplish their role as facilitators in a classroom, and recommend the development of tools to support them in this task.

Some authors have developed systems intended to monitor students work mainly for the case of distance learning supported by a Learning Management System which are suitable for tracking the student’s activity since most of them provide at least a low level logging which registers all students’ actions. Log files can afterwards be automatically analyzed to extract high level information regarding students’ progress. However, there is little literature reporting the monitoring of in-classroom students work to support assessment, although the required technology is already available [11].

In the past, computer-based learning material has been developed in the form of “electronic worksheets” in order to implement in-class learning activities for the students [12]. These materials have been called “Active Documents” [14]. These active documents provide the students with a rich environment for interaction. Also, they allow collaborative work by making use of available networks. In most cases, an XML Document Object Model has been used as a way to manipulate these documents and to store them in permanent storage devices. If students work on these electronic documents by modifying their contents, then it is possible to do an automatic - and hence systematic - analysis of their work. For example, the analysis can be used to find out how the students are advancing in the completion of the tasks described in the active document, whether or not they are filling the document with the right answers, and so forth. This constitutes the basis of our proposal.

40 N. Baloian et al.

2 Query-Based Assessment for Monitoring Students’ Work

The specific goal of this work is to find out whether it is possible to develop a system in which a teacher can flexibly monitor the work of the students while working on electronic worksheets. The suitable worksheets are those which can be mapped to an XML Document Object Model (DOM) representation. Our proposal is a system that allows a teacher to “send” query agents through the network; these agents analyze the current state of students’ documents and deliver information back to the teacher; she can then use this information to assess how the students are performing. In order to illustrate this we will use a very basic example: let us consider again the scenario of a mathematics class where now K8 students have to individually solve a series of exercises related to the subject being taught, e.g., arithmetic multiplication. The exercises are distributed as an electronic worksheet consisting of 3 sections, with ascending degree of difficulty. For instance, the first section may contain multiplications of positive numbers of at most 2 digits each. The second section introduces exercises with multiplication of negative numbers and the third section introduces multiplication with many digits. The structure of these worksheets clearly defines each section containing several exercises each. Each drill exercise is structured as a question and answer, and the answering part is to be modified by the students during their work. In this case, the teacher could make use of the following information:

1. Students’ progress: the teacher wants to control how many exercises each

student has answered up to now, so she can query how many answer parts have been modified. This can be presented as a total number, as a percentage or as a table specifying which exercises have been modified.

2. Correct answers: in this case, we need to extract the contents of the answer for each modified exercise, and compare it to the corresponding entry in a table containing the right answers. The results of the comparisons can be presented as a total number, a percentage relative to total number of exercises or number of modified exercises.

3. Correct answers aggregated by section: the previous information can be presented divided by section.

4. Differences among students’ progress: we can apply query 2 for each student and present the numerical results in a table in descending order.

5. Solving order: using query 1 we can also determine whether the students are solving the exercises in the presented order or in another sequence.

6. Student’s work pace: we can apply the previous queries at various times and present the differences in the resulting information. This will show the progress during that interval.

This simple list of queries shows we may consider two types of them: those

gathering basic information (like the first two queries above), and those aggregating results from basic queries (like the rest of the queries above). The example describes a scenario for young children. The system we developed for this work was tested on a


scenario for high school or university students. This may hint the suitability of this methodology for a wide range of learning scenarios.

3 System Description and Architecture

In order to develop a system which allows a teacher to monitor the students work in the classroom we need in the first place a software and hardware architecture enabling the communication between students’ and teacher’s workplaces. This architecture should also allow sending agents from teacher to students and capturing their findings with the needed information. Instead of conceiving a new one, we base the present work on a previous architecture, namely, CiC version 2 (CiCv2) [2]. Both teacher and students use computers in face-to-face sessions in this framework, allowing them to interact at various levels. The teacher can present teaching material, typically using a large interactive display, distribute assignments, exchange individual or group messages and share documents created on-the-fly or taken from an archive. Assignments can include constructive or creative tasks on the part of the students. A central repository allows users to authenticate and to access files as well as to interact with other users’ applications. In the second place, we need software implementing electronic sheets allowing teachers to prepare the material students have to work on. We also opted for using an already existing tool called “FreeStyler” [10]. It implements a series of visual languages for modeling in a variety of subject domains such as Petri Nets and UML diagrams for computer science or system dynamics for physics, biology or economics (Fig. 1). On an abstract level, Freestyler can be seen as a graph editor, which allows the inclusion of various “palettes” defining a group of specific nodes and arcs with particular functionalities in order to model a certain system. Using FreeStyler and CiCv2 together the teacher can access documents from the repository, share them with the class, and send them to individual users or groups. By sharing a document we mean working simultaneously on the same document, propagating the changes to the participants as they occur.

The FreeStyler modeling tool can be used as a whiteboard application to present material and solutions to proposed problems, as well as an exercising environment for the students, who can work developing their own models according to instructions given by the teacher. FreeStyler manages content organized in pages and thus supports notebook-style usage pattern as well as page-based documents. These pages can be added, erased and copied. The Students’ FreeStyler (a FreeStyler version customized for students) allows them to interact with the repository by sending and receiving files. Within a session, the student’s application can also interact with the teacher and other students by exchanging messages in a chat, exchanging documents to be opened independently or sharing documents. In the latter case, they can modify a common document in real-time. The Teacher’s FreeStyler (a version customized for the teacher) has the same functionalities as the student’s application plus additional ones allowing monitoring the students’ performance while working on assignments proposed by the teacher using FreeStyler documents (Fig. 2).


Fig. 1. Screenshot of FreeStyler. At the right hand side we see the palette. In this example, it corresponds to the System Dynamics plug-in, which allows the modeling and simulation of dynamical systems. At the center we see an already constructed model (graph).

Fig. 2. CiCv2 Basic architecture

4 A Support System for Query-Based Assessment

As stated above, our interest in the plug-in inside the FreeStyler modeling tool is on the possibility for the teacher to monitor the classroom situation without cluttering the interface with external modules. This plug-in allows the teacher to access relevant information during the sessions, receiving this information from several sources. We call this plug-in the Querying System, since it is based on the visual composition of querying elements, as it will be shown below. Like other plug-ins in FreeStyler, a model consists of a graph. Nodes represent atomic queries which can be combined in a graph to form more elaborated queries. The inspiration for developing this graph


comes from the “pipe” metaphor first introduced by the Unix operating system: the results of applying a program to a data set is a new data set which is input to the next program specified by the “pipe”.

The teacher can access meaningful information sources available in the CiCv2 scenario by using the querying capability. A query is an object containing the specifications for gathering, composing and presenting certain information which is currently distributed in various files across the system. In order to process a query object and generate the answer, the system extracts information from files and log information from the central repository, the logs of the shared pages manager, the locally stored documents, and from the documents and logs of all student applications participating in the session.

The queries which will be used in a certain learning session are generally prepared in advance along with the designing of the learning activities and learning material that will be used. They can be specific to the activities in that particular session; they can also be general purpose queries which may be useful in any session. Either way, the query definitions are readily available as a way to minimize the teacher’s involvement in technical details during the classroom session. However, it is also possible for the teacher to adjust specific parameters on queries to achieve the desired results, as it will be presented below.

When the teacher has selected a query object, she asks for its execution by pressing a button, and the result will either appear beside the graphic representation of the query or generate some changes to the currently active document, such as adding new pages with results. It is also possible to program queries to be executed periodically or triggered at a specific time, having access to updated results without any further interaction.

5 Query Implementation

A core set of basic query objects, which we call Basic Queries, was developed during the implementation of our system. These contain the specifications for retrieving information which is recurrently needed during the monitoring of students’ work. These basic queries are the nodes contained in the palette implemented by the query plug-in. They can be directly used as they are by just dragging them from the palette to the working area or they can be combined to create new composed queries. The query composition is as follows: the output of one query object is connected as input to another query object by graphically drawing a directed arc between the nodes corresponding to those queries. The system checks the correctness of the composition by checking that the structure of the output data of the predecessor node matches the required input data structure of the subsequent node.

As an example, it is possible to obtain the difference between two documents by using the FileQuery twice for obtaining each file and connecting them to a DiffQuery. In order to hide complexity, these three queries can be encapsulated into a ComplexQuery, so the end user sees only one simple query which performs as expected. As a result, the teacher’s interface shows just one complete query hiding all complexity and delivering a result when needed.


A context is defined for each query. It describes and gives access to the relevant documents it needs to perform. When a query needs to process remotely located documents (e.g., documents that are located on a student’s computer) the querying engine sends the corresponding agent to the remote location. There, the context is set accordingly, so that the remotely executing agent has access to the local resources and it sends the results back to the original location.

6 Basic Queries

The basic query objects are the simplest building blocks that allow arbitrarily complex queries to be built and processed by combining them. Some queries do not have any input, and only generate output, such as a constant query always returning a fixed result and a “current document query” which always returns the current document as defined by the context where it is being executed. Other queries are terminal queries and do not provide any output, such as “Save Query”, which simply saves its input into a file whose name it also fetched from its input, or “Object Creation Query”, which creates a visual object being added as a new element in the modeling tool. The other queries have inputs that are processed to create a single output, such as XQuery, which executes a particular XQuery on its input, generating a single output.

Some basic queries are used to execute a particular query in a different context. As the teacher initiates the queries, the current context would always be the teacher application. In order to execute some query remotely on a student's application or at the repository, the basic queries RepositoryQuery and StudentQuery were defined. These are the objects implementing the query agents, since both take some query, send it to a remote location and trigger its execution in that remote context, receiving the result back at the original context. In the case of the StudentQuery, the remote location can be several students, either a list of predefined students, or all the students in the current session. This will trigger the execution of several agents. The results will be structured containing the result of the same query executed at the several locations.

7 Execution of Queries

Several queries can be combined by connecting the output of one to be the input of the next one, using the “pipe” metaphor mentioned above.

If we want to extract the information for the “differences among students’ progress” example, then it is necessary to use a Complex query containing a linked list of sub-queries. The first step inside this complex query is to execute the query corresponding to “correct answers” in the context of each student using a StudentQuery. The result is the information for each of the students within the session in a DOM. This information is afterwards transformed by the next sub-query into a table. A last query should take this table and show it in the application user interface.


Fig. 3. Control (from left to right) and data flow (from right to left)

The “last query” is automatically identified and triggered when the teacher executes the complex query. This last query in turn triggers the previous query or queries, until the query which requests the single data from each student (StudentQuery) is reached. At that point, a query in each student environment is triggered and the resulting information is returned to the StudentQuery. From then on, the information is processed pipeline-wise in the reverse order.

The execution for the general case can be seen in Figure 3, where the control flow advances from left to right. At any point, the control flow continues its execution remotely, as in the StudentQuery, sending the query and the remaining queries to a remote location. The query will be rebuilt at the remote location, it will have access to the local context and the query will continue its control flow. This process continues with the input queries, activating all of their respective input queries, until the leaf queries are reached, which do not have any external input, beginning the data flow of results in the opposite direction.

Then, the information flows in the opposite direction (right to left in Fig. 3), delivering the results being processed in every step, until reaching the “last query”. If the query was executed remotely, the result is sent back over the network, finishing the remote execution and continuing with the local processing. The “last query” receives the final result and it generally is one of the so called “terminal queries”, which present or save the result in a useful way for the teacher.

8 Trials and Observations

Our goal is to explore the utility of the proposed system in practice. For that purpose, we have developed a series of three sessions involving students working while the teacher uses the proposed system in order to perform the monitoring of the activities. We intend to show in a qualitative way that the system is effective in providing useful information to the teacher in a timely manner.

We developed three sessions lasting 90 minutes each, in order to test the monitoring activities stated above. Each experiment is situated in a specific context or scenario, where students are asked to perform some activities and the teacher uses the querying system to monitor the session. The group of students for all experiments is


the same, consisting of 16 university undergraduate and 2 graduate students, aged between 22 and 26 years old. All students were taking the course on distributed computing at the University of Chile, and none had previous knowledge of either the CiCv2 environment or the particular presented problems. As we intended to use the tool in as real an environment as possible, we used it in learning sessions with the same constraints a teacher would normally encounter. This included very short introductions of both the CiCv2 environment and each modeling plug-in, no more than 10 minutes in each experiment, which was enough to get the sessions started.

Session 1 In this session, the students were asked to use a collaboration framework for the Java programming language, called Matchmaker [13]. After an introduction, the students were handed out a document detailing the activities, one activity on each page, with some aspects of the activity being optional. The exercise on page 1 asked the student to create a collaborative session on a server by using some programming methods previously discussed in class. Optionally, the student can verify whether the session was successfully created, and the teacher can see which students have completed the optional part (“create + See” instead of “createOnly”). On page 2, the exercise requires the students to connect to the server and fetch all existing sessions, to print them out. On page 3, the students need to connect to a particular session, but many times the students forget to first check whether that session exists or not. On page 4, students are expected to create modifications in a session and on page 5, they are asked to fetch data from a session and print it out.

This session was set up to check the students’ state while they are working on their tasks. The state is available as a set of indicators by which the teacher may identify partial progress in specific sub-tasks as well as total progress, considering each student as well as the entire group. For this purpose, information needs to be gathered from diverse sources within the system, and then aggregated into a specific output. Figure 3 presents a query which is available to the teacher (a), and the output being generated by that query (b).

In this scenario, the teacher can perform a query searching for a particular solution in order to automatically identify students who have successfully solved a problem. This can also be applied for identifying students making common errors or omissions.

Results of Session 1 Figure 4 shows the result for each exercise and each student in a matrix depicting the level of progress. It starts with a “none” value, changing as the student completes subsequent programming steps. The teacher can see students “pedro” and “juan” only do the “join” part of the exercise, without getting the list first, which is an incomplete activity on page 3. Similar partial results exist for exercises on pages 4 and 5.

The teacher can identify which students are currently working on the system and which ones are likely to need assistance by using activity levels. She thus gets awareness on the pace of both individual students and the whole group. Students progressing either slower or faster than expected will be quickly noticed by the


Fig. 4. Determining students’ pace and specific solutions

teacher. She can take a closer look, either by directly approaching the student, or by using a new query to have a specific look at the student’s work. When a student is advancing faster than his peers, the teacher might want to provide additional challenges, ideas or assignments in order to keep stimulating him. She may also share the good work with other members of the class in order to discuss the solution and its alternatives. On the other hand, when a student is having problems to solve the proposed exercise, then the teacher can take adequate measures to overcome these specific problems.

In figure 4, part (b), the teacher can see that in the various stages, students have fulfilled none, a part, or the whole exercise. The columns show each student’s state on the exercise on a specific page, allowing the teacher easily compare each student’s activities. For example, the student with username “jorge” has not finished exercises on pages 3 to 5, but he was the only student to achieve more than the rest in exercise on page 1. In this case, jorge actually did more than was asked in the exercise, while pedro finished all exercises, but did not complete the exercise on page 3.

Figure 5 shows the evolution of a single student’s progress at different periods of time. We can clearly see how the results evolve according to the students’ work. In this case, student “pedro” first solved exercise on page 1, but only partially. Afterwards, the student worked on exercise on page 2, completing the first part and then the second one. The results are presented by the system automatically, re-calculating the query according to the period of time set by the teacher (2 minutes in this case). The hints given by the teacher to the students were relevant, and they helped to guide the class in a natural way. In particular, the teacher could do early


identification of at least five students who were not making any progress at all, despite the fact they seemed to be very busy working on the exercises. Only two of these students approached the teacher for help.

Fig. 5. Determining model complexity

Session 2 Let us consider a new scenario. We intend to show how the querying system is capable of identifying a student choosing one of several possible ways of solving a problem. University students were asked to model two stochastic processes to simulate lottery games, in order to determine which one had the highest probability of winning. The students would have to create a rather complex model by following the game descriptions literally. However, they could identify a simplification that radically reduced complexity of the model, without affecting the results. The teacher had access to a query that identified which of the paths each student seemed to have chosen. With this information, the teacher could start discussions in which students evaluate their peers’ solutions and learn several ways to solve the problem.

A second aspect considered in this scenario is to determine whether the student groups are doing real collaborative work or each student is advancing on his/her own. For this purpose, the teacher has access to data of each group of students, identifying active students and passive ones. We can see the query that provides the statistics (Figure 6a), and the results within one specific group (Figure 6b).

Results of Session 2 The gathered data in our experiment did not show major changes in the distribution of activities among the participants as the session progressed. Generally it was possible to identify one or two students in a group who had a participation that was slightly over the level of the other participants. As the teacher observed these figures, it was possible to take a closer look at the groups showing a large gap in the members’ participation. Perhaps the students were collaborating using direct face-to-face communication, or they were blocked because a difficult problem was encountered.

It was also possible to use queries to discover common students’ errors like using the wrong element for modeling a particular stochastic scenario. As expected, several users made a quite common mistake, resulting in wrong results. When the teacher detected this kind of errors, she determined the best way to handle the situation. Some of these ways were asking for differing results, starting a discussion and either letting students find out why the results are different and which result is correct, or taking a more direct approach by telling the correct solution.


Fig. 6. Student activity within a group

Session 3 The aim of session 3 was to find out if it is possible to find out how elaborated is the answer the students are developing. This might be an indicator about the correctness of the solution the students are working on, for example, when the model a certain student is developing is much more complex than the one they are supposed to construct as result of the task given by the teacher. This is with high probability a case when the student is working on a wrong answer. In the case of complex models, it is increasingly difficult to characterize a “right” or “wrong” solution, and the teacher has to use generic information to choose where to look for problems or right answers. The queries used in this scenario are generally applicable to any situation. Fig. 7 shows the teacher has sent a query in order to obtain the number of elements (nodes, arcs, strokes, etc.) contained in each student’s model. It is also possible to separate the number of nodes and arcs according to their type, which might provide even further information about the graph the student is developing. A documented example of this type of evaluation of model construction complexity can be found in [1].

Results of Session 3 During the session, it was possible to identify cases in which students were working on models that appeared extremely complex when compared to the model solution available to the teacher. In all cases the students used over twice as many nodes and edges than expected. A close look revealed they were creating a new model on the same page as the first one. Other students also decided to start a new model because the first one was unsatisfactory, creating the new model on another page, or by


Fig. 7. Determining model complexity

deleting the previous model before starting over. These three approaches provided different results. A teacher needs to be able to interpret these results, verify that the real cause of the results effectively matches the possible interpretations (e.g. by approaching a student or using a query to view the student’s model) and possibly take remedial actions.

9 Conclusions

The experiments described in the previous chapter show it is in fact possible to automatically extract valuable information from the worksheets the students are using. This information helps to assess their work and give meaningful feedback at the right moment at least in three different ways: • Tracking the students in order to monitor their advance on the work, as seen in

session 1. This gives the chance to assist students who might be working at a slower pace than the rest of the class or identify advanced students to give them positive feedback and/or provide them with additional problems to solve.

• Identify students that might have reached a correct solution in order to show them to the rest of the class in case their solution is a classic one or an unexpected one. This was the case of the session 2. This procedure can also be useful to identify cases where student are developing too complex solutions compared with the “classical” one the teacher might have in mind. In these cases, the teacher can assist the students guiding them in order to help them find simple solutions.


• Monitoring the students’ level of activity of the students either during individual or collaborative learning sessions. This was explained in session 3 (Fig. 6). A low level of activity can indicate the students are not well prepared to solve the proposed task or that the task itself could be ill-designed. It can also serve as an indicator that perhaps the switching between different learning activities (e.g. from individual solving problem to collaborative work) might be not well designed [2].

The last point of the previous list indicates this tool can also be used to validate or

discard a set of planned learning activities as “best practices”. In fact, a teacher can test whether a certain set of learning activities the students should follow in a predetermined order can be considered as a “best practice” candidate by monitoring the students’ work and checking if they advance as expected.

A very important feature of this approach is its generality. In fact, the developed query mechanism can be applied to various computer supported learning environments. The applicable environments must contain documents to be worked by students which are structured following the DOM standard and have their corresponding representation in XML. This requirement comes from the fact the query system only compares XML documents and it does not use their semantics. Therefore, this tool can be applied to monitor students’ work for any other discipline, not just the one mentioned here.

We used a modeling tool as an example to test our approach. It is shown that automatic assessment by querying the documents the students are working on is very powerful. This is indeed a complex scenario since highly structured tasks can be given to the students thus making the querying much easier in many cases. For example, the working sheet might be reading a text and answering questions about its content with multiple choices or a list of mathematical exercises with unique answers, like in the case of arithmetic basic operations.

The reported research is representative of the type of services we would like to develop for TCC. It provides services that could not be offered without standards such as DOM and XML and without computer devices for each student and teacher. Furthermore, devices such as tablets allow the teacher to monitor students’ activities while walking around the classroom and visiting students’ workplaces.

References

[1] Antunes, P., Pino, J.A., Xiao, L.: Assessing the impact of Educational Differences in HCI Desing Practice. International Journal of Technology and Design Education (in press, 2014)

[2] Baloian, N., Pino, J.A., Hoppe, H.U.: Dealing with the Students’ Attention Problem in Computer Supported Face-to-Face Lecturing. Educational Technology & Society 11(2), 192–205 (2008)

[3] Bangert-Drowns, R., Kulik, C., Kulik, J., Morgan, M.: The Instructional Effect of Feedback in Test-Like Events. Review of Educational Research 61(2), 213–238 (1991)


[4] Cotton, K.: Monitoring student learning in the classroom. School Improvement Research Series Close-Up #4. Northwest Regional Educational Laboratory, U.S. Department of Education (1988)

[5] Ferguson, R.: Learning analytics: drivers, developments and challenges. International Journal of Technology Enhanced Learning 4(5/6), 304–317 (2012)

[6] Grüntgens, W., Melzer, R.: Diagnostische kompetenz von lehrkräften im primar- und sekundarbereich (2004), http://pz.bildung-rp.de/pn/pb1_04/ diagnostischekompetenz.html (retrieved on September 2008)

[7] Guerrero, L., Ochoa, S., Pino, J.A., Collazos, C.: Selecting Computing Devices to support Mobile Collaboration. Group Decision and Negotiation 15(3), 243–271 (2006)

[8] Gutierrez-Santos, S., Mavrikis, M.: Intelligent Support for Exploratory Environments: Where are We and Where Do We Want to Go Now? In: Proceedings of the 1st Int. Workshop in Intelligent Support for Exploratory Environments on EC-TEL 2008, Maastricht, The Netherlands (2008), http://ftp.informatik.rwth-aachen.de/ Publications/CEUR-WS/Vol-381/ (retrieved on February 2011)

[9] Hattie, J., Timperley, H.: The power of feedback. Review of Educational Research 77, 81–112 (2007)

[10] Hoppe, H.U., Gassner, K.: Integrating Collaborative Concept Mapping Tools with Group Memory and Retrieval Functions. In: Stahl, G. (ed.) Computer Support for Collaborative Learning: Foundations for a CSCL Community. Proceedings of CSCL 2002, pp. 716–725. Lawrence Erlenbaum Associates, Inc., Hillsdale (2002)

[11] Pearce-Lazard, D., Poulovassilis, A., Geraniou, E.: The Design of Teacher Assistance Tools in an Exploratory Learning Environment for Mathematics Generalisation. In: Wolpers, M., Kirschner, P.A., Scheffel, M., Lindstaedt, S., Dimitrova, V. (eds.) EC-TEL 2010. LNCS, vol. 6383, pp. 260–275. Springer, Heidelberg (2010)

[12] Pinkwart, N.: Collaborative Modeling in Graph Based Environments. Ph.D. thesis, Universität Duisburg-Essen, Germany (2005)

[13] Tewissen, F., Baloian, N., Hoppe, H.U., Reimberg, E.: MatchMaker: Synchronizing Objects in Replicated Software-Architectures. In: Procs. of the 6th Collaboration Researchers’ International Workshop on Groupware (CRIWG), pp. 60–67. IEEE Computer Society Press, Madeira (2000)

[14] Verdejo, M.F., Barros, B., Read, T., Rodríguez-Artacho, M.: A system for the specification and development of an environment for distributed CSCL scenarios. In: Cerri, S.A., Gouardéres, G., Paraguaçu, F. (eds.) ITS 2002. LNCS, vol. 2363, pp. 139–148. Springer, Heidelberg (2002)