Syntactic REAP.PT Cristiano Jos´ e Lopes Marques Dissertation for obtaining the Master’s Degree in Information Systems and Computer Engineering Jury President: Professor Jo˜ ao Ant´ onio Madeiras Pereira Advisor: Professor Nuno Jo˜ ao Neves Mamede Professor Jorge Manuel Evangelista Baptista Evaluation Jury: Professor Bruno Emanuel da Gra¸ca Martins 2011
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Syntactic REAP.PT
Cristiano Jose Lopes Marques
Dissertation for obtaining the Master’s Degree inInformation Systems and Computer Engineering
Jury
President: Professor Joao Antonio Madeiras PereiraAdvisor: Professor Nuno Joao Neves Mamede
Professor Jorge Manuel Evangelista BaptistaEvaluation Jury: Professor Bruno Emanuel da Graca Martins
2011
Acknowledgements
First of all I would like to thank Prof. Nuno Mamede and Prof. Jorge Baptista for providing
me with this opportunity and for his excellent guidance, support, motivation, and knowledge
sharing. I would also like to thank Andre Silva and Joana Barracosa, for their optimistic support
and for the many conversations, work meetings and helpful insights along the course of this work.
Their help and collaboration strongly improved the journey. Furthermore, my thanks go to every
author for their work and contribution with inestimable background knowledge. To my friends,
a special thank you, for helping and providing their own knowledge and expertise, even when it
was not related with this work. To Fernando Pardelhas, Pedro Ruivo and Joao Dias, for their
continuous help, valuable feedback, support, motivation and and dedication firmly encouraged
me to move on. Lastly, but definitely not least, to my parents and family, a special thank you
very much for all these years of comfort and encouragement that made this work possible.
This work was supported by project CMU- PT/HuMach/0053/2008, sponsored by FCT.
Lisboa, 2011
Cristiano Jose Lopes Marques
To my parents, Jose and Cidalina.
To my brother, Nuno,
Resumo
Ensino da lıngua assistido por computador e uma area de pesquisa que se foca no desen-
volvimento de ferramentas que melhorem o processo de aprendizagem da lıngua. REAP.PT
(REAder-specific Practice Portuguese) e um exemplo destes sistemas, em que o objectivo e aju-
dar a ensinar a lıngua portuguesa de uma forma apelativa, abordando temas que sao do interesse
dos utilizadores.
Esta tese foca-se na geracao automatica de exercıcios sintacticos e de vocabulario. Para
tentar cumprir esses objectivo, um conjunto de tarefas foram desenvolvidas: um estudo de
formatos de questoes pertinentes e quais os sistemas que criavam estes exercıcios; um estudo
de sistemas CALL com exercıcios sintacticos e semanticos; e modelacao/desenvolvimento de
um solucao arquitectural para o problema, incluindo diferentes recursos: filtros, classificadores,
tabelas com informacao lexical adicional e geradores automaticos de distractores.
Outro objectivo deste trabalho e incluir este modulo no sistema CALL - REAP.PT.
Abstract
Computer-Aided Language Learning (CALL) is an area of research that focuses on devel-
oping tools to improve the process of learning a language. REAP.PT (REAder-specific Practice
Portuguese) is a system that aims to teach Portuguese in an appealing way, addressing issues
that the user is interested in.
This thesis focuses on the topic of automatic generation of syntactic and vocabulary ques-
tions. In trying to successfully accomplish this goal, there are several tasks that were developed:
a study on the format of pertinent questions and which existing systems create these exercises;
a study of some CALL systems with syntactic and semantic exercises; and the modeling of an
architectural solution of the problem, including other resources such as filters and automatic
distractors generation.
Another goal of the present work is to be able to include this feature in the tutoring system
1IntroductionThe field of Education is certainly one of the most important and discussed topics in the
world. During the past two decades, the exercise of spoken language skills has received increasing
attention among educators. According to the Lisbon Summit (Lisbon European Council 2010),
learning a foreign language is extremely important for the curriculum enrichment of any person
and is considered, by the same entity, as one of five key skills of a successful professional. With
the advancement of technologies for information systems, the use of CALL has emerged as a
tempting alternative to traditional modes of supplementing or replacing direct student-teacher
interaction, such as the language laboratory or audio-tape-based self-study. This context is the
starting point of the REAP1 project and also of the REAP.PT2 project.
1.1 Goals
The main goal of the REAP.PT system is to present to the students rich and authentic
study material (texts, exercises, etc) that is deemed interesting by the students (based on their
topics of interest), and adequate to their learning needs and current skills, thus being able to
advance their learning process.
The aim of this work is to develop a module of REAP.PT to help learning the syntactic-
semantic component of the language, through the development of automatic exercises drawn
from actual texts taken from the Internet, and according to the topic preferences of the student.
This module can use features already present in the REAP.PT system (see Section 2.1)
While the main focus of the original REAP project has been the acquisition of vocabulary,
the exercises to be present by this module of REAP.PT will focus too on aspects of syntax that
are especially problematic for students of Portuguese as a second language, for example, the
1http://reap.cs.cmu.edu (last visited in December 2010)2http://call.l2f.inesc-id.pt/ (last visited in December 2010)
2 CHAPTER 1. INTRODUCTION
placement of clitic pronouns, adjectives or adverbs, formation of interrogatives, the alternation
between direct speech and indirect speech, between active/passive sentence structure or nomi-
nalizations. Because the exercises are generated automatically, careful design of the generation
process is necessary in order to assure the linguistic adequacy and the relevance of the question.
Above all, the exercises automatically generated by the system should not present ambiguous
solutions. For example: in the case of gap-filling exercises, it is important that there is not more
than one possible solution so that the system does not invalidate the answer given by the user
just because this was simply not the expected input.
Additionally, it is necessary to create an interface for teachers and another for students.
The teacher will have permissions to see the answers given by students, save all the answers and
editing/viewing the exercises. The student interface will give him access to available exercises
and allow him to solve them. From a technical point of view, special care must be taken in
the development of both interfaces (teacher and student) so that it is not necessary to install
plugins on the client browser.
1.2 Requirements of Syntactic REAP.PT
This section defines the requirements of the problem, for adequate project planning and
management in the presence of changing requirements. Defining requirements precisely is im-
portant to identify what exactly should the system do. Requirements can be divided in two
groups (Rost 2005): functional requirements (see Table 1.1) and non-functional requirements
(see Table 1.2). The main requirements for this module of REAP are thus defined as:
1.3. COMPUTER-AIDED LANGUAGE LEARNING 3
Requirement Description
Automatic Exercise GenerationThe exercise generation must be done with-out human intervention
Different types of exercises
Clitic pronouns, adjectives or adverbs, for-mation of interrogatives, the alternationbetween direct speech and indirect speech,active/passive or nominalizations
Teacher permission Teacher can see students’ result
Students inputStudents can answer questions through abrowser
Table 1.1: Functional Requirements
Requirement Description
High AccuracyGenerated exercises don’t have ambiguoussolutions
SecurityDifferent permissions for teachers and stu-dents
Persistence Save students’ answers and results
UsabilityUsers do not need to have advanced knowl-edge in Technology
Table 1.2: Non-Functional Requirements
1.3 Computer-Aided Language Learning
Currently, CALL can be seen as an approach to language teaching and learning in which
computer technology is used as an aid to the presentation, reinforcement and assessment of
material to be learned, usually including a substantial interactive element. It also includes
the search and the investigation of applications in language teaching and learning (Hacken
2003). Gamper and Knapp (2002) define CALL as “a research field which explores the use of
computational methods and techniques as well as new media for language learning and teaching”
(Gamper & Knapp 2002).
However, various difficulties have to be overcome in order to developed CALL systems,
namely: the financial barrier resulting from the need of strong financial support required to
obtain human and technological resources; availability and access conditions to hardware and
software in the teaching institutions; and the reluctance – still quite strong – from many people,
both teachers and students, in using new technologies, specially in language learning.
4 CHAPTER 1. INTRODUCTION
One of the most important requirements for this CALL system is the ability to retrieve and
use texts with updated information, as well as to present correct and appropriate content.
This goal has already been achieved as a result form previous research (Marujo 2009) (Cor-
reia 2010).
The motivation for this requirements can be summarized as follows:
• Experiential learning: it is easier for the user to learn gradually from their mistakes;
• Motivation: generally speaking, the use of technology both inside and outside the class-
room, tends to make the learning process more interesting. However, certain design issues
may affect the way a particular tool creates influences motivation;
• Enhance student achievement: if texts shown to the student are to be selected according to
their topic preferences, it is then likely that their motivation be enhanced, thus improving
their learning performance;
• Greater interaction: computers can adapt to students; adapting to a student usually means
that the student controls the pace of the learning but it also means that the student can
make choices regarding the content and the manner of learning, skipping unnecessary
items or doing remedial work on difficult concepts; such control makes students feel more
competent in their learning (Okan );
• Completeness: a fully-fledged CALL systems provides the student the opportunity to
interact using one or more of the four basic languages skills (reading, writing, listening
and speaking).
1.4 Structure of this Document
The present thesis consists in 6 chapters, and it is structured as follows:
• Overview of the REAP.PT is described in Chapter 2
• Chapter 3 describes the work related to the theme of this thesis: Portuguese language
syntax exercises and automatic question generation systems;
1.4. STRUCTURE OF THIS DOCUMENT 5
• The solution proposed is presented in Chapter 4;
• Chapter 5 introduces how the system here developed will be evaluated;
• Chapter 6 presents the conclusion and future work.
6 CHAPTER 1. INTRODUCTION
2REAPREAding-Practice1 (REAP) project was initially developed by the Language Technologies
Institute (LTI) of Carnegie Mellon University2 (CMU). It is a system that implements the main
ideas of CALL and it is specifically focused on vocabulary learning.
The goal behind REAP, as described by Michael Heilman and Maxine Eskenazi “is to furnish
appropriate, authentic texts to students to help in reading and vocabulary learning. In REAP,
a student sees short reading passages that contain a number of words (usually ranging from two
to four) from his or her list of target words to be learned from context. The passages are Web
documents of about one to two pages in length, covering a wide variety of topics” (Heilman &
Eskenazi 2006). This reading is complemented by practice in series of automatically generated
exercises.
In April 2009, a Portuguese version of REAP - REAP.PT3 begun being implemented. The
target audience for this system is primarily students who want to learn Portuguese as a foreign
language.
2.1 Architecture of REAP.PT
In order to understand how the current work is integrated in the remaining modules already
developed within the REAP.PT project. This section describes the current system architecture
(see Figure 2.1) (Correia 2010)(Marujo 2009):
• Web Interface: Two interfaces, one for student and other for teacher have already been
implemented (Marujo 2009). These can be accessed through any web browser. The web
interface is also responsible for the communication between the database and the DIXI
1http://reap.cs.cmu.edu (last visited in June 2011)2http://www.cmu.edu/index.shtml (last visited in June 2011)3http://call.l2f.inesc-id.pt/ (last visited in June 2011)
lemmas (and over 33k inflected forms associated to them). This list defines the focus words that
students should learn.
To implement the automatic question generation system, the following resources have been
used: TemaNet9, PAPEL10 (Oliveira, Santos, Gomes, & Seco 2008) and MWN.PT11. Another
feature important is the dictionary of Porto Editora, the Infopedia12 that contains 920,000
entries.
Language Proficiency and Preferences
The level of language proficiency of the students is calculated through the resolution of a
set of exercises presented to students when they are first introduced to REAP.PT. Based on this
level, each student is assigned a set of P-AWL words that they must learn when interacting with
5http://www.infopedia.pt/ (last visited in December 2010)6http://www.linguateca.pt/PAPEL/ (last visited in December 2010)7http://www.instituto-camoes.pt/temanet/ (last visited in December 2010)8http://mwnpt.di.fc.ul.pt/index.html (last visited in December 2010)9http://www.instituto-camoes.pt/temanet/ (last visited in December 2010)
10http://www.linguateca.pt/PAPEL/ (last visited in December 2010)11http://mwnpt.di.fc.ul.pt/index.html (last visited in December 2010)12http://www.infopedia.pt/ (last visited in December 2010)
2.1. ARCHITECTURE OF REAP.PT 11
the system (Correia 2010). There is a direct correlation between the current level of student
and the keywords is supposed to learn.
When the students enter their personal area, they have the opportunity to choose which
topics they find more or less interesting, so that later the texts and exercises presented by the
system may be more suitable to their topic preferences.
Listening Module
Listening comprehension is a complex but critical process to the development of the skills
necessary to learn a foreign language. This module in particular can be viewed as the result
of integrating two components: Software Text-to-Speech and Multimedia Documents. The first
component is the software DIXI (see Section 2.1), through which students have the opportunity
to hear the words or word sequences (phrases) that they have previously selected. According
(O’Malley, Chamot, & Kupper 1989) listening helps to discriminate sounds, understand the
vocabulary, acquire grammatical structures, and to interpret intentions. All these factors are
responsible for the preeminence of listening comprehension, especially in the early stages of the
language acquisition (Stern ).
The second module, Multimedia Documents, complements the speech synthesis software. A
type of multimedia documents presented in REAP.PT and that has been already implemented
is the digital talking books. These are an excellent tool to improve language proficiency since
it allows students to follow the oral reading of a text by a nature speaker, thus being able to
associate the sounds to words, learn prosody and other speech features.
Another feature that is available in Listening Module is the broadcast news (BN) stories,
which consists of news subtitles from television through an automatic speech recognition system
(ASR). ASR, the AUDIMUS (Meinedo, Caseiro, Neto, & Trancoso 2003) “combines the temporal
modelling capabilities of Hidden Markov Models with the pattern discriminative classification
capabilities of multilayer perceptrons”. In short, AUDIMUS converts an acoustic signal to text,
allowing the processing of sound produced by one or more speakers (Meinedo & Neto 2003).
The news stories are presented in small sized videos (one of news piece) along with the subtitles
produced by AUDIMUS.
Although it is not the main objective of this module, the use of BN can also help students
12 CHAPTER 2. REAP
learn facial expressions, gestures and emotions, witch are essential elements of oral, face-to-face
communication. This is a side effect of the visual training implicit in the BN, which becomes
especially important for people with hearing difficulties, but who have the ability to interpret
facial expressions and lip movements.
Exercises
The development of (semi-)automatically built exercises has been the focus of much research
(L2F) in the improvement of the REAP.PT system. Initially a set of cloze questions have been
manually built by a team of teachers and linguists (Marujo 2009). Later on, automatic generation
of questions has been introduced (Correia 2010) like vocabulary exercises (see Figure 2.2) and
cloze question (see Figure 2.3) cases.
Qual das seguintes e uma definicao da palavra criar?; Which of the following is adefinition of the word create?
a) conceber algo original; design something originalb) coser a orla de um tecido para que nao se desfie; sew the hem ofa cloth so as not to shredc) temperar em vinha-d’alho com o intuito de aromatizar a comida,conserva-la e torna-la mais terra; had garlic wine in order to flavorthe food, keep it and make it more landd) causar a morte; cause death
Figure 2.2: Vocabulary Exercise
O livro e tao extenso e tao complexo que o editor decidiu que nao era para umtradutor e contratou dois profissionais para o fazer.; The book is so extensive andso complex that the editor decided it was not for a translator and hired twoprofessionals to do it.
a) palestra; lectureb) assunto; subjectc) receita; reciped) tarefa; task
Figure 2.3: Example of Cloze Question
The goal of this thesis is to continue the work already done, including more types of exercises
in REAP.PT.
3State of the art
3.1 Syntactic-Semantic Exercises
The literature review in this area is extremely important to ensure that the implemented
exercises are consistent with what is taught and practised by Portuguese students in their first
years of school. Thus, it will be ensured that this thesis focuses on the really important and
interesting exercises for learning the language.
In Table 3.1, a match is made between the skills explored and the exercises’ formats that
are presented in this thesis. These exercises are available in format in digital format (such as
software applications, online exercises) and on paper (such as grammars, books).
Skills were divided into four groups:
• Reading Comprehension - Ability to understand and interpret what you read;
• Syntactic knowledge of language - Ability to apply the language rules to construct sen-
tences;
• Semantic Knowledge of Language (Lexicon) - Knowledge of word meaning;
• Meta-Linguistic Knowledge - Knowledge about the language used to describe the natural
language.
It is important to mention that with a small amount of imagination, each exercise format
can explore other skills that are not represented in Table 3.1. Essentially this table reflects the
skills exercised by the systems presented in this work.
In the following sections, several syntactic exercises are presented.
14 CHAPTER 3. STATE OF THE ART
Reading Com-prehension
Syntacticknowledge oflanguage
SemanticKnowledgeof Language(Lexicon)
Meta-LinguisticKnowledge
Crosswords X X X
Fill-in-the-blanks
X X X
Correspon-dence Exer-cise
X X
”Right” or”Wrong”
X X
MultipleChoice
X X X
Short Answer X X
Word Sorting X X X
Identificationof Grammati-cal Category
X
Open Re-sponse
X X X
AlphabetSoup
X
Table 3.1: Exercises and language skills
3.1.1 Crosswords
Crosswords (see figure 3.1) are usually done on paper, but there are also computer versions.
Words must be discovered (remembered or guessed) following tips provided by the exercise.
The most common use of crosswords consists in identifying a word from its definition (e.g.
“profissional da area de ensino”/“professor” ; professional in the field of education/teacher).
Other tips may involve metalinguistic knowledge (e.g “verbo casar no gerundio”/“casando” ;
verb married in the gerund/married).
There is already a system (Aherne & Vogel 2006) developed by the Computational Lin-
guistics Group – University of Dublin1 that automatically generates crossword puzzles, using a
pre-installed database with the target clues/words pair needed to these exercises. This database
was obtained with the support of the information contained in WordNet (Stark & Riesenfeld
1http://www.scss.tcd.ie/disciplines/intelligent systems/clg/clg web/ (last visited in December 2010)
3.1. SYNTACTIC-SEMANTIC EXERCISES 15
Figure 3.1: Crossword (taken from http://www.prof2000.pt/users/amsniza/)
1998). Besides these systems, there are tools that do not allow the automatic generation, but
nevertheless help the user in the creation of the exercise in a digital format, such as the Jcross2,
Instant Online Crossword Puzzle Maker3 or the Crossword Generator for Teachers4, where the
clues and the words have to be inserted manually into the system by the user. There are other
systems that search the web for definitions of words and then through a program of constraint
satisfaction (CSP) constructs the puzzle (Anbulagan & Botea 2008).
Some characteristics of this exercise format:
• difficulties on producing automatically the word grid;
• the ludic aspect oh the exercise is a motivator;
• students may take advantage from the crossing of words to guess the missing words so
indirect help way be available and direct assessment of language skill being tested may be
hindered.
3.1.2 Correspondence Exercises
Correspondence exercises consist of two columns of items that are to be aligned by appealing
to a single linguistic relation (meaning, syntactic transformation, etc.). They aim at simulating
the student’s ability to differentiate between several concepts being presented, or to systematize
2http://hotpot.uvic.ca/ (last visited in December 2010)3http://www.varietygames.com/CW/ (last visited in December 2010)4http://www.theteacherscorner.net/printable-worksheets/make-your-own/crossword/crossword-puzzle-
maker.php (last visited in December 2010)
16 CHAPTER 3. STATE OF THE ART
the concepts involved in the linguistic relation under study. One of the most common uses of this
exercise format is to make the correspondence between the word and its definition. (example:
phone/mobile phone).
For example, in Figure 3.2, a correspondence exercise is shown, aimed at drilling the pro-
nounning of different noun phrases, in several syntactic functions (subject, object, beneficiary,
etc.).
Figure 3.2: Correspondence (taken from http://www.prof2000.pt/users/amsniza/)
To create exercises in this format there is JMatch, although it does not manage the exercise
automatically, it only allows teachers to build these exercises in digital format. Regarding the
(semi-)automatic generation:
• working the other way around (from pronouns to full-hedged Prepositional Phrases or
Nominal Phrases) raises the problem of anaphora resolution (Carbonell & Brown 1988), a
natural language processing (NLP) task that still has a limited success.
• the exercise may be automatically generated, depending on the more or less complexity
on the relation being tested. In this case, automatic identification of Nominal Phrase
(NP)/Prepositional Phrase(PP) and their function is likely to produce incorrect results
because, among other factors, the yet unsolved problem of PP attachment and the conse-
quent incomplete delimitation of the constituent being pronominalized.
3.1. SYNTACTIC-SEMANTIC EXERCISES 17
3.1.3 “True” or “False” exercises
True-or-false exercises are often used in text comprehension. They have the great advantage
of being aplicable in many other areas of language learning as well. For example, agreement
between nouns and adjectives can be tested, given the following instruction/example pair:
“Marca “correcto” or “incorrecto” as frases seguintes: Sempre julguei competente o gerente
e a directora” (Mark as “correct” or “incorrect” the following sentences: I have always considered
competent the manager and the director).
The main disadvantage of this type of exercises is the loss of specificity for each question,
since it may not reveal exactly the knowledge that has been activated to answer.
The most common systems for creating this type of exercise have a similar architecture to
(Bastos, Berardi, & Silveira 2004) where there is a database with a set of questions and answers
previously defined by a teacher.
3.1.4 Multiple choice exercises
Multiple-choice or cloze questions is probably the most widely used type of automatically
generated exercises. In their most basic form, they consist of a sentence (the stem), which is
provided with a blank space from where a word or string of words has been removed; and a set of
possible answers. Usually, the correct word is randomly placed among several incorrect choices
(distractors or foils) and while ideally only one possible solution exists, the set of distractors
are carefully designed to engage different linguistic strategies in order to pinpoint the language
knowledge that the student is supposed to have acquired and is being tested for. Much research
has already been devoted to the theoretical, psychological, cognitive and educational aspects
involved in the making of cloze question (Armand 2001), while several systems have been built to
• if the verb is in the infinitive — randomly pick one of the tenses of the indicative or the
subjunctive;
• in the selection of distractors the homographs of the target word should not be used.
4.2.1 Data
To generate the stems for this exercise, the corpus CETEMPublico (Santos & Rocha 2001)
was used, which contains over 190 million words and 7 million setences. In order to extract
the candidate sentences in a fast and expedient way1, the Hadoop2 plataform for distributed
computing was used. This platform is based on MapReduce and uses a cluster to process
large amounts of data. In this way, was possible to extract 720,784 sentences and generate the
corresponding exercises of this type.
1Initial experiments showed a 95,6% reduction of processing time using Hadoop against a sequential executionof the processes. Even so, it took about 1h30m to process the entire corpus.
2http://hadoop.apache.org/ (visited in Jul. 2011)
34 CHAPTER 4. OUR APPROACH
4.3 Collective Names and Nominal Determinants
The purpose of this exercise is to learn the subtle distributional constraints observed between
a determinative noun (Dnom) and the noun it determines (see Figure 4.9). This exercise also
serves to teach the classifying relationship between collective names and common names, since
collectives often function as Dnom on common nouns.
Figure 4.9: Example of Exercise of Nominal Determinant
These exercises are generated from real sentences, taken from the corpus. In these sentences,
a quantifying dependency (QUANTD) (see Figure 4.10) has been extracted by the syntactic
parser XIP (Hagege, Baptista, & Mamede 2010). This dependency holds between a Dnom
functioning as the head of a nominal or prepositional phrase (NP or PP) and the head oh the
immediately subsequent PP introduced by preposition “de” (of).
Again, the distributed computing platform Hadoop is used to retrieve from the large-sized
corpus the sentences that can be potential stems for this exercise.
Figure 4.10: QUANTD dependency
A new set of lexical information was added to the lexicon in order to generate adequate
distractors for the exercises. To do that, a list of determinative and collective names was added.
A set of semantic features was defined and accorded to these words. These semantic features
consist on the following categories: Human, Animal, Food, Organization/Institutions, Object,
Nature, Military and Local. By using these features, potential distractors that share the same
traits as the target word are never select, thus avoiding to generate unwanted correct solutions
4.3. COLLECTIVE NAMES AND NOMINAL DETERMINANTS 35
as foils. An additional set of constraints was also added to prevent the generation of distractors
that, in a figurative use, would be possible solutions. For example, Dnom associated to the
“Animal” feature (alcateia, “pack”, speaking of wolves) are not used for human nouns (e.g.
uma alcateia de polıticos “a pack of politicians”) since this combination might be used in a
ironic formulation.
To make the exercise more interesting, in the generation of distractors the same number
and gender of the target word is kept.
One of the important points in the CALL system is that the application is able to provide
feedback to the student. When the student incorrectly chooses a given word-definition pair, s/he
is given some feedback by the system. This feedback consists of three parts:
• the definition of the wrong answer;
• a real sentence as an example of the correct use of the selected word;
• pictures of the word selected.
This feedback system gives the indication that the student missed a question, and tries to
teach the meaning of the given nominal determinant and also its proper use (giving examples of
images, definitions, and uses the correct given determinative noun).
The illustrative images of the nominal determinant that appear on the feedback system
were selected in advance and stored in the REAP.PT database. To select these images was
necessary to retrieve the first ten images available in google images for each determinative noun.
Subsequently, through a voting system, the two most voted images (in 10 possible) are selected
and used by the REAP.PT system.
Following the same logic of the Mahjong Lexical exercise, it is important that there is some
repetition in the nominal determinants presented to the student. The Syntactic REAP.PT
presents sentences that are different but whose correct nominal determinant used to fill the
sentence has appeared in previous exercises. If the student hits the nominal determinant in
different contexts it means that s/he knows its meaning and knows how to make a proper use
of it.
36 CHAPTER 4. OUR APPROACH
4.3.1 Data
The syntactic REAP was able to extract from the corpus approximately 19,000 sentences,
of which 46,61% involve determinative nouns exercises and 53.39% correspond to the collective
names (see table 4.1). It should also be noted that determinatives names that appear more
frequently (see annex B) in the exercises are “percentagem”, “pacote” e “pedaco” and the
most common collective name (see annex B) in the generated exercises are “equipa”, “album”
e “comissao”. Although some nominal determinants /collective names do not appear in the
correct answers generated by this module, they are used as a distractors.
Exercise Number of exercises %
Collective names 9,035 46.61%
Nominal determinants 1,0345 53.39%
Table 4.1: Collective names and nominal determinants
The Hadoop3 plataform was used to process the corpus with the filters in order to generate
the exercise, in a similar way as with the previous game.
3http://hadoop.apache.org/ (visited in Jul. 2011)
4.4. TEACHER INTERFACE 37
4.4 Teacher Interface
This section presents the work developed in the teacher interface for the REAP.PT system.
At this stage, we felt the need to create an interface that would be simple and intuitive but
at the same time could give all the information needed by the teacher in order to assess the
learning process of each individual student and the class in general. This interface assumes
particular relevance because if teachers feel they can not control the progress of the students
(given answers, questions presented, results, etc.) they will not use the system. To avoid this
situation it is interesting that the system, in addition to providing a general feedback about the
students’ performance, may also give all relevant data to the teacher if s/he wishes to validate
any particular information or make an assessment using her/his own criteria.
The pages that allow the visualisation of the students’ results are composed of two parts:
the first entitled “General Information”, where the teacher a general overview of the current
status of the student. The second, called “Detailed Information”, which contains all data of
all exercises solved by student, separated by exercise. Thus, teachers can see the evolution of
students over the use of Syntactic REAP.
4.4.1 Mahjong Lexical
In the General Information screen, teachers can view the summary of a student’s perfor-
mance in percentage (0% to 100%). This performance is the weighted value between the number
of points earned versus the maximum number of points that the student could obtain.
In order to complement and help understanding this performance, the percentage is pre-
sented in different colors: red (poor performance), orange (can improve), blue (successful out-
come) and green (good result). On this page the teacher can also check:
• number of exercises solved;
• time spent solving exercises;
• number of errors;
• words whose meaning the student already knows;
• words whose meaning the student does not seem to know.
38 CHAPTER 4. OUR APPROACH
In the Detailed Information screen the teachers can view the student’s history. For each
exercise, the teacher can check the performance of the student, the mistakes, the points obtained,
and the retention rate associated with each exercise. In order to be simpler to understand, these
results are presented in a graphic and a table format. The graphics (see figure 4.11) show
the oscillations of student performance and the retention rate. As in the General Information
screen, graphics are colored with: red (bad performance), orange (can improve), blue (successful
outcome) and green (good result).
Figure 4.11: Teacher Interface
4.4.2 Choice of mood in subordinate clause
In the Detailed Information screen the teacher can analyze the results of the student for
each exercise, namely:
• the number of errors committed;
• the answers submitted by the student;
• check if the student failed to respond properly to the exercise.
4.4. TEACHER INTERFACE 39
4.4.3 Collective Names and Names determinative
On the page General Information screen the teacher sees an overview of the student
performance. Here it is possible to view:
• number of exercises presented;
• number of errors;
• number of correct answers;
• determinative names / collective names that the student already knows how to use cor-
rectly;
• determinative names / collective names that the student still does not know how to use
correctly;
On the Detailed Information page the teacher can monitor the answers given by the
students in each exercise (the system indicates the correct answer for each exercise). It is also
possible to view the question presented in each exercise, the number of wrong answers, the
retention rate and the associated information if the student successfully solved the exercise.
40 CHAPTER 4. OUR APPROACH
5EvaluationTo assess the performance of the Syntactic REAP.PT, two groups of students were given
the system to try it and comment on its use. This evaluation consisted in a Syntactic REAP.PT
Session, composed of three parts:
• initial form (see annex C): tracing the students’ profile (e.g. age, native language, etc.);
• exercises: three mahjong lexical exercises are presented (one for each difficulty level), three
mood selection for subclause verbs exercises, three nominal determinants exercises, and
three collective names exercises;
• final questionnaire (see annex D): aimed at qualitative assessment of the system (e.g.
system case of use, which exercises are most relevant or appealing, etc).
The exercises generated by the Syntactic REAP require some knowledge of Portuguese as
they call upon more advanced language contents. In this way, to evaluate the system, it is
required that the subjects of the evaluation comply with this condition: if they are non-native
Portuguese speakers, they already should have an elementary knowledge of the language. Due to
the impossibility of gathering a group with the characteristics described in time for this exercise,
the tests were conducted on a group with relatively similar characteristics - Portuguese native
speakers in the 3rd and 4th grade. In order be able to contrast this group performance, the same
test was also performed with another group of native speakers with at least a college degree.
Test Users
Therefore, 45 subjects performed this exercise, 18 in Group 1 and 31 in Group 2. A brief
overview of these two groups in presented in Table 5.1. Naturally, while de age of Group 1
subjects is quite homogeneous, Group 2 varied from 19 to 70, the average age being 24. Most
subjects from Group 2 know other languages (such as english, french, spanish, italian and
42 CHAPTER 5. EVALUATION
german). The choice of Group 1 can be justified by their knowledge of Portuguese as mother
language but their still limited vocabulary - this being one of REAP.PT main learning targets.
GroupNumber ofusers
% Average age Other languages
1 18 37% 8 –
2 31 63% 24 english, french, spanish, ital-ian, german
Table 5.1: Test Users
As we can see in the table, the nationality of the group differs between the portuguese and
brazilian, and most of the people that belong to the Group 2 (with college degree) speak other
languages such as english, french, spanish, italian and/or german.
Assessment – Test Environment
The testing environment was different for each group. Group 1 did the exercises one at time
in the room’s computer, and a team member of REAP.PT project helped them to access the
starting page and occasionally had to explain the exercises or some unknown word since some
children still hadn’t fully mastered their reading skills. Each child took about 26 minutes to
complete the exercises and the questionnaire.
Users from Group 2 did the test at their own homes, without the aid of any element of the
REAP.PT project.
43
Figure 5.1: The system was quick in giving an answer
Results and Discussion
As shown in Figure 5.2, move than 77% of the users found the system easy to use. In this
way we can say that one of the goals of this project was achieved, namely, to create a system
with a simple and easy to use interface, so that all users (even some inexperienced users) were
able to use the Syntactic REAP.PT. In Table 5.3 we can see that only 39% needed to use the
“Help” button.
Figure 5.2: The system was easy to use
The pre-processing of the exercises is made in advance and stored in the REAP.PT database.
Thus, the generation stage, which is the most time-consuming step of the processing, is already
44 CHAPTER 5. EVALUATION
Figure 5.3: The help menu contains useful information presented in a concise and clear manager
done when the user access the system. This is why the system is so quick when responding to
requests from users as is a reflect in their answer to the question shown in Figure 5.1.
Figure 5.4: Which exercise did you like best
Regarding the results obtained in the three Mahjong Lexical exercises that were presented,
Group 2 obtained better results with a performance of 84% (standard deviation = 6,6%), while
the users from Group 1 made more errors and obtained a performance of 54% (standard deviation
= 12,4%).
Users from both group reported that the exercise they liked most (44%) was the Mahjong
Lexical (see Figure 5.4), the next most enjoyed exercise was the “Collective Nouns” (27%) fol-
lowed by “Nominal Determinants” (17%) and the “mood of subordinate clause” (12%). However
45
it is remarkable that the 2 Groups had clearly different impressions about these three exercise.
While Group 1 liked “Collective Nouns” as 2nd best ( 40%), Group 2 shows similar preference
for each of these three (16%, 19% and 21%) this way have to do with the that collective names
are an explicit grammatical subject at “escola primaria” (elementary school) thus contributing
do a higher familiarity with the topic which entailed this preference trend.
Another interesting aspect to note is the error rate for each exercise. Starting at Mahjong
Lexical, in Table 5.2, we can see that in both groups the more the difficulty level of exercises
increases, more mistakes the students do, which seems to confirm the strategy followed to dis-
tinguish these levels from the sequence of definitions inside the dictionary entry of each target