SignMT: An alternative language learning tool

Page 1

Computers & Education 55 (2010) 118–130

Contents lists available at ScienceDirect

Computers & Education

journal homepage: www.elsevier .com/ locate/compedu

SignMT: An alternative language learning tool

Nadh Ditcharoen a, Kanlaya Naruedomkul b,*, Nick Cercone c

a Institute for Innovative Learning, Mahidol University, Bangkok, Thailandb Department of Mathematics, Mahidol University, Bangkok, Thailandc Department of Science and Engineering, York University, Toronto, Canada

a r t i c l e i n f o

Article history:Received 9 September 2008Received in revised form 22 December 2009Accepted 23 December 2009

Keywords:Computer-assisted language learningComputer-mediate communicationMedia in educationInteractive learning environmentsSign language machine translation

0360-1315/$ - see front matter � 2010 Elsevier Ltd. Adoi:10.1016/j.compedu.2009.12.009

* Corresponding author. Tel.: +66 22015444; fax: +E-mail addresses: [email protected]

a b s t r a c t

Learning a second language is very difficult, especially, for the disabled; the disability may be a barrier tolearn and to utilize information written in text form. We present the SignMT, Thai sign to Thai machinetranslation system, which is able to translate from Thai sign language into Thai text. In the translationprocess, SignMT takes into account the differences between Thai and Thai sign language in terms of bothsyntax and semantic to ensure the accuracy of translation. SignMT was designed to be not only an auto-matic interpreter but also a language learning tool. It provides meaning of each word in both text andimage forms which is easy to understand by the deaf. The grammar information and the order of the sen-tence are presented in order to help the deaf in learning Thai, their second language. With SignMT, deafstudents are less dependent on a teacher, have more freedom to experiment with their own language, andimprove their knowledge and learning skill.

In our experiment, SignMT was implemented to translate sentences/phrases which were collected fromdifferent sources including textbooks, cartoons, bedtime story, and newspapers. SignMT was tested andevaluated in terms of the translation accuracy and user satisfaction. The evaluation results show that thetranslation accuracy is acceptable, and it satisfies the users’ needs.

� 2010 Elsevier Ltd. All rights reserved.

1. Why’s TSL–Thai MT needed?

Deafness or hearing impairment affects not only a child who is deaf or has a hearing loss, but also the child’s family, friends, and teach-ers. Although deaf, hard of hearing and hearing signers can fully communicate among themselves by sign language, there is a big commu-nication barrier between signers and hearing people without signing skills. The biggest problem with sign language is that the vast majorityof non-deaf people do not understand it. Moreover, sign language does have its limitations, and these are, unfortunately, often in the areaswhere they are needed most – education, training and work. Therefore, deaf people in Thailand are encouraged to study Thai language, thewritten language along with sign language. Thai is an official language in Thailand. To Thai deaf, Thai is their second language.

Learning Thai is, however, quite difficult to deaf since Thai differs from Thai sign language (TSL) in both syntax and semantics. Althoughthe deaf and hearing-impaired are taught Thai language, most of them often faces with comprehension problems when they read and writeThai text. The primary causes of difficulty with Thai literacy are that Thai is a language that deaf people have not heard or have heard onlyin a limited way, the usage of the language and the lack of access to appropriate instructional methods.

It is widely acknowledged that computer technology when used to supplement traditional teaching methods has a positive impact onthe educational development of children (Passey, Rogers, Machell, & McHugh, 2004). Incorporating computer technology in learning lan-guages so-called computer-assisted language learning (CALL) can be found in five directions: (1) commercial software used for supple-menting to class instruction; (2) World Wide Web used as an incredible resources for language teachers; (3) other Internet applicationssuch as email; (4) presentation software used to make slides to accompany lectures and presentations, and to stimulate conversation inthe target language; and (5) authoring software which allow teachers to create exercises, language drills and activities for students. Thefunctionalities of CALL can be extended by integrating the machine translation (MT) technology into it (Dangsaart, Naruedomkul, Cercone,& Sirinaovakul, 2008).

MT is an automatic translation of natural languages by a computer. Two of the basic tasks of MT include understanding the source lan-guage (SL) and generating the target language (TL). A very simple example of these tasks in relation to the second language learning is the

ll rights reserved.

66 34243225.h (N. Ditcharoen), [email protected] (K. Naruedomkul), [email protected] (N. Cercone).

Page 2

N. Ditcharoen et al. / Computers & Education 55 (2010) 118–130 119

vocabulary/phrase drill exercises. Such exercises are that a computer prompts the student with a word/phrase on the SL then the studentresponds with the corresponding word/phrase in the TL, and vice versa. The computer then compares the student’s response with an an-swer generated by a computer and returns the feedback to the student.

Due to the difficulties in learning languages of deaf and the availability of computer technologies as mentioned above, we thereforedeveloped SignMT, the Thai sign to Thai machine translation system. SignMT was designed not only to facilitate deaf students in learningThai language and communicating among students and teachers in class but also support people who has no signing skills in learning Thaisign language.

SignMT is a student-centered learning based system which is aimed at increasing deaf students’ interest and motivation; improvingself-concept and mastery of basic skills; being more engagement in the learning process; and gaining confidence in directing their ownlearning and practicing. With SignMT, students are less dependent on their teachers and more comfortable to practice on their own pace.

The computer technologies in language learning, their advantages and disadvantages are presented in Section 2. The differences be-tween Thai and Thai sign language are described in Section 3. The architecture of SignMT is described in Section 4. The implementation,experiment, and translation samples are presented in Section 5. The evaluation and the conclusion are discussed in Sections 6 and 7respectively.

2. MT and CALL

Rost (2002) stated that the goals of language teaching profession are (1) giving students real opportunities to learn and helping themlearn more effectively; (2) increasing the enjoyment of language learning; (3) improving students’ ability to become better language learn-ers; and (4) making the teaching more enjoyable and rewarding. To achieve such goals, using computer technologies is one solution.

2.1. Computer in language teaching and learning

The use of computers in education has been emerged since late 1950s. With the advent of convenient microcomputers in 1970s, a com-puter used in schools has become widespread from primary education through the university level and even in some preschool programs.Computers have been used not only in educational administration but also in teaching process. The very first attempt to integrate the com-puter technologies into teaching process was found in PLATO. PLATO was developed at the University of Illinois, Urbana Champaign in 1959for pedagogy (Molnar, 1997). Afterwards, the projects of using computers in education for instruction and the research at Dartmouth andStandford Universities emerged in 1969.

During that time, Atkinson (1968) studied computerized instruction and learning process which pointed out how the computer couldsupplement the drill and practice of traditional instruction with relevant practice exercises while Suppes and Morningstar (1969) studiedcomputer-assisted instruction to improve student’s learning. Since then, computer technologies have significantly progressed towards edu-cation including languages learning.

Rost (2002) viewed teaching ideas as one of the three types: hard technologies, soft technologies, and instructional technologies. Hardtechnologies are the physical tools used in teaching e.g. chalk, magic markers, blackboards, whiteboards, notebooks, chair, audio players,video players, video cameras, and computers. In addition to these tangibles, hard technologies also include delivery systems, particularlyelectronic tools such as tapes, discs, CD-ROMs, email, websites, chat rooms, discussion boards. Soft technologies constitute the actual con-tent of communication, the form of the interaction used in teaching. Selection and use of soft technologies are considered the act and theart of teaching. Tools that teachers can use to enhance their teaching are actual course books, reference books, video, audio tapes, and thespecific websites. Instructional technologies are theories, models, techniques, and strategies that teachers develop and deliver through theirteaching. Among these three, instructional technology is the most interesting and most important one in language teaching.

In addition, teachers are currently excited with the developments that just 5 years ago seemed implausible: e-learning applications,smart libraries, asynchronous threaded discussion sites, synchronous multi-user virtual environments, interactive presentational media,video-conferencing, instructional media frameworks, interactive assessment, and online communities. However, before these develop-ments can be combined with teaching, they must be tested and evaluated to ensure that they serve the four main goals: (1) Do they givelearners more opportunities to learn? (2) Do they help learners learn more effectively? (3) Do they help learners more motivated, moresatisfied, more self-directed, and more intelligent? (4) Do they make teachers’ job easier, more enjoyable, and more rewarding?

These developments present the possibilities of enhancing both the quantity and the quality of self-access learning. Moreover, they canhelp teachers to focus on guiding and monitoring what learners do outside the classroom. CALL or computer-assisted language learning isone of instructional technologies used in learning languages. It is a student-centered learning material which promotes self-paced learning.

2.2. Computer-assisted language learning

CALL originated from computer-assisted instruction (CAI). It is a form of computer-based learning which carries two important features:bidirectional (interactive) learning and individualized learning. These features are meant to improve the learners’ knowledge. CALL isaimed at helping teachers to facilitate the language teaching–learning process and to reinforce what has been learned in the classrooms.It can be used as a remedial tool to help learners with limited language proficiency as well. CALL has contributed to second languageacquisition.

The technologies used in CALL instruction generally fall into two categories: software-based and Internet-based activities. In software-based activities, authoring programs allow an instructor to program parts or all of the content to be learned and how it can be learned. Someexamples of these programs include Hotpotatoes, WinCALIS, Clozemaster, Choicemaster and Multitester (Graham, 2002). Internet-basedactivities vary considerably, from online versions of software (where the learner interacts with a networked computer), to computer-med-iated communication (where the learner interacts with other people via the computer), to applications that combine these two elements.Some websites are based on the drill-exercise format, e.g., Dave’s ESL Café (http://www.eslcafe.com), some include games, e.g., Hangman(Graham, 2002), some derived from role playing games, e.g., Crimson Room’s (http://www.fasco-csc.com/index_e.php), Runescape (http://

Page 3

120 N. Ditcharoen et al. / Computers & Education 55 (2010) 118–130

www.runescape.com/), Quest (http://www.questrpg.org/), Fairly-land (http://www.1010game.com/asp/downloadpage.asp), and SecondLife (http://www.secondlife.com/).

The development of CALL was divided by Warschauer (1996) into three phases: Behavioristic CALL, Communicative CALL and Integra-tive CALL (Multimedia and the Internet). Behavioristic CALL, the first phase, was defined by Skinner’s dominant behavioristic learning the-ories. During this phase, CALL was confined to universities where programs were developed on big mainframe computers, like PLATOproject. Computers were considered ideal for this learning aspect since they are not tired or impatient with repeatedly presenting the samematerial to the students at their own pace or even adapting the drills to the level of the students. Communicative CALL, the second phase, isbased on the communicative approach which focuses on language usage rather than language analysis. The first CALL software in thisphase still provided skill practice but not in a drill format, for example, paced reading, text reconstruction and language games but com-puter remained the tutor. In this phase, however, computers provided context for students to use the language, such as asking for direc-tions to a place. Integrative/explorative CALL attempts to integrate the teaching of language skills into tasks or projects to provide directionand coherence. It also coincides with the development of multimedia technology which provides text, graphics, sound and animation, aswell as computer-mediated communication (CMC). CMC comes in two basic forms: asynchronous, e.g., email and forums, and synchronous,e.g., audio-and video-conferencing (using Skype). With such facilities, learners can communicate in the target language with other realspeakers cheaply, 24 h a day. Learners can communicate one-to-one or one-to-many as well as sharing audio and video files. Recent re-searches in CALL seem to favor a learner-centered integrative approach.

Besides the three approaches mentioned above, there are others including constructivism, whole language theory and sociocultural the-ory. With constructivism, students are active participants in a task in which they ‘‘construct” new knowledge based on experience in orderto incorporate new ideas into their already-established schema of knowledge. Whole language theory postulates that language learning,either native or second language, moves from the whole to the part; rather than building sub-skills like grammar to lead toward higherabilities like reading comprehension (Stepp-Greany, 2002). Sociocultural theory states that learning is a process of becoming part of a de-sired community and learning that communities rules of behavior (Mitchell & Myles, 1998).

All approaches, however, aim at using a computer to provide students opportunities to be less dependent on their teacher and havemore freedom to experiment on their own. Therefore, with CALL, the role of the teachers and the students changed (Thelmadatter,2007). Instead of handing down knowledge to students and being the center of students’ attention, teachers become guides; they constructthe activities for students and help them to complete the assigned tasks. Students must also learn to interpret new information and expe-riences on their own terms or through interaction and collaboration with someone other than the teacher. In such environment, any timidstudent is more comfortable to work on his/her own to improve his/her self-esteem and knowledge.

The successes in using CALL have shown in the development of learners’ four language skills including reading, listening, writing andspeaking. Most learners significantly gained reading and listening skills since reading and listening software are based on drills (Noemi,n.d.). However, gain in writing skill was not as impressive as it was expected since writing software was unable to assess the learner’s writ-ing abilities (Stepp-Greany, 2002). The speaking skills gained much attention since it was obtained via CMC in the forms of, e.g., chat, vid-eoconferencing. It helped the learner to develop the abilities to engage in meaningful conversation in the target language and to providecontrolled interactive speaking practice outside the classroom (Ehsani & Knodt, 1998). In addition, the speech processing technology wasaugmented into CALL to make an automatic pronunciation training which allows the learners to read the sentences on the screen and thecomputer returns the feedback as to the accuracy of the utterance.

2.3. The advantages and disadvantages of computer in language learning

A number of educators concluded that using computer technologies in second language acquisition has many advantages, e.g., the learn-ers are able to decide on what, when, where and how to learn; they have opportunities to study and practice at their own pace, as often asthey need; they have more independent from their teachers and classrooms. Currently, CALL can better serve the individual needs throughthe communicative and interactive activities. A lot of fun games were added to reduce the learning stresses and anxieties of the learner. Bycombining with the Internet, there is a channel for the learners to practice by having a real communication with people, especially the(second) language owners, in global community. Moreover, with the progress in computer technologies, CALL is able to capture and analyzethe learners’ performance during the learning process, and return feedback to the learner.

Although several research findings indicate that the use of computer in language learning has a positive effect on the achievement ofsecond language learners, it still has a few disadvantages in e.g., cost, computer related knowledge required and software limitations.

The first disadvantage of CALL is the ‘‘cost” of both computer and software. Once computer become part of education, it turns to be thefinancial burden not only at the low-budget school but also at home of the low-income family. The second disadvantage is the ‘‘knowledgerequired” in using CALL. Both the teachers and the learners are required to have computer related knowledge to fully utilize CALL. The thirddisadvantage is ‘‘software limitations”, e.g., none of computer technology is able to accurately understand natural language.

2.4. MT and CALL

Machine translation is one of the very earliest pursuits in computer science, it has proved to be an elusive goal, however today a numberof systems are available which produce output which, if not perfect, is of sufficient quality to be useful in a number of specific domainsincluding language learning. Besides, nowadays a number of computer-aided translation (CAT) tools exist, e.g., dictionary lookup programs,spell and grammar checker, translation memory and terminology management tools.

Either MT or CAT can be used to facilitate second language learning. It can be used to improve reading/writing skill in learning language.For example, in reading practice, MT can help translate word/phrase requested by the learners. In writing practice, MT can help check thespelling and grammar used by the learners. Translation output (in second language) can be used as an example of the correspondingphrase/sentence of the input (in the learner’s language).

Translation memory is a database that stores segments or sentences that have been previously translated for future re-use so that thesame sentence never needs to be translated twice. Since the source and the target languages are stored in pairs, the learner can study the

Page 4

N. Ditcharoen et al. / Computers & Education 55 (2010) 118–130 121

relationships between the languages from each pair in terms of both syntax and semantics. Translation memory is typically used in con-junction with terminology.

A terminology management tool is a searchable database that contains a list of approved terms and rules regarding their usage. It scanseach word of the source text in order to locate them in the particular dictionaries and, whenever possible, offers an equivalent to the trans-lator, automatically and in the target language. The efficiency of this function is therefore basically determined by the quality and the vol-ume of the particular dictionary. The second language learner can learn more words and their usages via this tool.

3. Some linguistic issues of TSL

Sign language is a three-dimensional visual language that uses combinations of signs (handshapes), gesture, facial expression, and inliterate communities, some fingerspelling to construct and convey meaning (William, 1960). Gestures can be characterized by both manualand non-manual parameters. Manual parameters include hand-shapes, hand-orientation, position, and motion while non-manual param-eters include posture of the upper torso, head-orientation, facial expressions, lip movement and position (Stokoe, 1978; Sutton-Spence &Woll, 1999; Wideman & Sims, 1998).

No one form of sign language is universal, e.g., Thai sign language (TSL) differs notably from ASL. Different sign languages are used indifferent countries or regions because of the differences in custom, tradition, and geography between countries. For example, Americansign language (ASL) is used by approximately one-half million deaf people in the United States and Canada (Lee & Kunii, 1992). British signlanguage (BSL) is used in England while Thai sign language (TSL) is used by approximately 3 millions deaf people in Thailand (Dailynews,2002).

TSL, like Thai, is considered to be a national language for Thai deaf people (Ministry of Education, 1999). Unlike Thai, however, TSL’ssyntax and semantics have not been standardized; they have been under studying by linguists. Thai is used as the standard spoken andliterary language in Thailand. It is basically monosyllabic in word form. In addition to consonants and vowels, each Thai syllable hasone of five phonemically differentiated tones (middle, low, falling, high and rising). Although TSL and Thai are originated from the sameculture and geography, their semantics and sentence structure are different in some espects.

3.1. Semantics issue

A sign represents an object, event, action, repeated process, an emotional situation, and so on. Some examples of signs are shown inFig. 1a and c–e represents objects ‘‘ –hat”, ‘‘ –elephant”, ‘‘ –boxing”, ‘‘ –nose” respectively. Fig. 1b represents an action ‘‘puton a hat”.

Fingerspelling is the representation of the letters of a writing system, and sometimes numeral systems, using only the hands. Thesemanual alphabets (also known as finger alphabets or hand alphabets), have subsequently been adopted as a distinct part of a numberof sign languages around the world. Fingerspelling may be used to represent words from a spoken language which have no sign equivalent,or for emphasis, clarification, or when teaching or learning a sign language. Fig. 1f shows fingerspelling as a part of the sign for ‘‘ –pumpkin”.

Every sign is comprised of basic components or parameters. There are five components to every TSL sign: hand-shapes, palm orienta-tion, location, body movement, and facial expression. In order to communicate in TSL most effectively, these parameters need to be incor-porated into the signs or sentences according to the structure of the sign language.

Fig. 1. Some examples of words/phrases in TSL.

Page 5

122 N. Ditcharoen et al. / Computers & Education 55 (2010) 118–130

Since a sign is a gesture or movement that conveys a concept, each sign is made with a specific hand configuration or handshape, placedat various locations on or near the signer’s body. If the handshape, movement, or location changes, the meaning of the sign also changes.Therefore, performing a sign correctly is as important as pronouncing a word correctly as illustrated in Fig. 2. In Fig. 2a, the three-parameter(hand-shape, movement, and orientation) signs presented in different locations refer to different meaning ‘‘ –hungry” and ‘‘ –like”.Fig. 2b shows two signs, for ‘‘ –uncle” and ‘‘ –aunt”, that share the same movement, orientation, and location but hand-shape. Fig. 2cpresents two signs which differ in palm orientation while Fig. 2d presents the other two which differ in movement.

TSL signs expressing emotions or opinions, e.g., joy, anger, tiredness, depression, doubt, questioning, need to be accompanied by theappropriate facial expression. Some of these signs are distinguished not by hand movements but by facial expression. For example, raisingeyebrows express ‘‘questioning”, frowning expresses ‘‘doubt”.

In TSL, a sign that comes after a noun is called a classifier (CLTSL). The appropriate CLTSL to be used is determined by the noun it follows.Some examples of CLTSL are shown in Fig. 3. CLTSL_1, CLTSL_H2, CLTSL_PL and CLTSL_PLANE are used with the nouns that refer to persons,standing persons, places and airplanes respectively. In Fig. 4, the classifier CLTSL_PL follows the noun ‘‘ –house” while the classifierCLTSL_A follows the noun ‘‘ –cat”.

One sign may not directly correspond to one word in Thai, e.g., the signs presented in Fig. 5a and b correspond to‘‘ –I disagree with your idea” and ‘‘ –Two people are engaged in discus-sion” respectively.

For some verbs, e.g., ‘‘ –walk”, ‘‘ –eat”, ‘‘ –drink”, how they are signed depend on their subjects. For example, the verb ‘‘ –walk”is signed differently for different subject ‘‘ –boy” and ‘‘ –horse”, even though it refers to the same activity which is ‘‘to move for-ward by putting one foot in front of the other”.

3.2. Syntactic issue

In TSL sentence structure, the temporal is simultaneous with spatial configuration while in Thai sentence structure, one word is fol-lowed by another. Table 1 shows sentence structures of TSL and Thai in pairs. In TSL, the negative ‘‘Neg” sign appears at the end of thesentence, an object ‘‘O” of the sentence comes before subject or verb except for the subject ‘‘ –I”.

Table 2 presents structures of noun phrases (NP), verb phrases (VP) prepositional phrases (PP) and some examples.

Fig. 2. One parameter difference signs.

Page 6

Fig. 3. Examples of CLTSL.

Fig. 4. Example of CLTSL usages.

Fig. 5. Examples of signs that does not directly correspond to one word in Thai.

Table 1Sentence structure of TSL and Thai.

TSL syntax Thai syntax

S + V S + VS + V + Neg S + Neg + VO (+CL) + S + VS (I) + O + V S + V + OO (+CL) + S + V + Neg S + Neg + V + OindO (+CL) + diO (+CL) + S + V S + V + diO + indOindO (+CL) + diO (+CL) + S + V + Neg S + Neg + V + diO + indOS + V + Q S + V + QO + V + Q V + O + Q

S = subject, V = verb, O = object, indO = indirect object, diO = direct object, Neg = negative sign, CL = TSL classifier, Q = question word.

N. Ditcharoen et al. / Computers & Education 55 (2010) 118–130 123

4. SignMT-system architecture

We designed the architecture of SignMT as illustrated in Fig. 6. SignMT performs the translation in four steps: word transformation(WT), word constraint (WC), word addition (WA), and word ordering (WO). WT was designed to allow the users to correctly input to Sign-MT. WC removes any word required no translation however the original meaning is preserved still. WA adds any necessary word requiredinto the translation process and WO arranges all the translated words into grammatical order. Each step leads to the generation of accuratetranslation output. Details of each step is discussed in the next sections.

4.1. Word transformation

The instrumented gloves, magnetic trackers and digital camera have been used as the input device in several sign machine translationsystems. Instrumented gloves are gloves containing sensors which measure the angle bend of the finger joints. With the gloves, approx-imate handshapes can be calculated. Magnetic trackers are sensors that calculate the position with respect to a fixed source. This sourcecan be worn on the body to give the position relative to the signer. These trackers are used to estimate hand position and motion. Theadvantage of using gloves and trackers is that it is easier and faster to calculate handshapes, position and motion from sensor data thanfrom video images. However, having the device attached to body is quite cumbersome. Moreover, instrumented gloves and magnetic track-

Page 7

Thai Sign Language

Word Transformation

Word Constraint

Word Addition

Word Ordering

Thai Language

Dictionaries Constraints Addition Rules

Ordering Rules

Fig. 6. SignMT architecture.

Table 2The phrase structure in Thai and TSL with examples.

TSL Thai

det = determiner, adj = adjective, prep = preposition.

124 N. Ditcharoen et al. / Computers & Education 55 (2010) 118–130

ers are rather expensive than a camera. Practically, a camera seems to be more available for deaf people than instrumented gloves andmagnetic trackers. If signs are captured on video, the hands and face must first be found and tracked in the image, and then handshapes,palm orientation, position, and motion must be determined from 2D image data, a recognition technique is required. Both tasks are difficultto model computationally and the error from recognition cannot be avoided.

In this research, we focused on the translation process which plays a crucial role in sign language translation. To avoid the commonerrors usually occurred in the recognition process, e.g., variation (different people can differently sign the same word), co-articulation(signs change depending on the preceding and following signs), word boundaries in continuous signing, background noise and sign inter-ference, we therefore designed the interactive WT module to facilitate the input process. The user can simply specify the sign parametersby selecting from the available pictures (Fig. 7). To reduce the variation error, the user can select the signs directly from the near-matchsign list generated from the specified parameters. To reduce the co-articulation error, users are allowed to edit the input sentence or changethe order of signs easily. With this input system; the word boundary, background noise and sign interference are no longer problems. Theexample input sentence is displayed in the last row of Fig. 7.

Each input sign is used as a key to search for its corresponding Thai word in the bilingual dictionary: TSL-Thai dictionary. In the dictio-nary, each sign entry is linked to its parameters, the corresponding Thai word, the related linguistic information including part of speechand WordAsso numbers. WordAsso number is a class number of the words which share the significant features (Naruedomkul & Cercone,1999). The entry ‘‘ –foot” sign and its information provided in the dictionary are shown in Fig. 8.

4.2. Word constraint

In word constraint step, the TSL constraints are applied to simplify the structure of TSL by removing some TSL signs which are not re-quired to be translated into Thai. TSL constraints are the linguistic characteristics of TSL which do not appear in Thai language. For example,classifier in TSL is not needed to be translated into Thai, therefore any CLTSL presents in the input sentence will be replaced with CLTSLfeature ‘‘<cltsl>” to retain its meaning (Table 3).

Page 8

Fig. 7. The user interface for user input.

Fig. 8. Example entry in the bilingual dictionary.

Table 3Some TSL constraints.

cheu

N. Ditcharoen et al. / Computers & Education 55 (2010) 118–130 125

If the sign ‘‘ –name” appears after the CLTSL, it indicates that the following noun is the proper noun. This sign is not required to betranslated into Thai either, it then replace with the feature <prop>.

4.3. Word addition

After applying all related constraints, some words that are needed to retain the meaning of original input and to make them grammat-ically correct in Thai, are added into the string. What and where to add depends on the additional rules. The rules were developed based on

Page 9

Table 4A small fraction of Thai unit-of-noun classes.

Class Sample word

2-4-2 Unit-of-noun2-4-2-1 Unit-of-noun of object2-4-2-1-1 Unit-of-noun of living thing2-4-2-1-1-1 Unit-of-noun of human2-4-2-1-1-2 Unit-of-noun of animal2-4-2-1-2 Unit-of-noun of non-living thing2-4-2-1-2-1 Unit-of-noun of tool

2-4-2-1-2-2 Unit-of-noun of room

24-2-1-2-3 Unit-of-noun of external body part

2-4-2-2 Unit-of-noun of collection2-4-2-2-1 Unit-of-noun of human

2-4-2-3 Unit-of-noun of time period

2-4-2-4 Unit-of-noun of frequency

126 N. Ditcharoen et al. / Computers & Education 55 (2010) 118–130

the semantic and syntactic differences between TSL and Thai. For example, the word ‘‘ –with” is added after the word ‘‘ –give” because‘‘ –give” is a di-transitive verb in Thai. The conjunction ‘‘ –and” is added between two phrases/clauses, unlike TSL in which the secondphrase/clause immediately follows the first phrase/clause without any ‘‘conjunction.” The unit-of-noun ‘‘ ” is added after the number‘‘ –two” in the string ‘‘ –fish –two <clth> –father –buy.” In Thai, a certain unit-of-noun is required for a noun which co-oc-curs with the determiner or plurality; the unit-of-noun ‘‘ ” is for a noun in the ‘‘animal” class while ‘‘ ” is for a noun in the ‘‘human”class (Table 4).

4.4. Word ordering

After all necessary words are added, word ordering rearranges the words in grammatical order according to the ordering rules. Theseordering rules are generated from the syntactic level differences between TSL and Thai (Section 3.2).

The structure of the TSL sentence which contains subject and verb in that order is similar to that of the Thai sentence. Therefore,the structure of the translated sentence remains the same as illustrated in Example 1. However, in the case that the TSL phrasecontains object, the words order must be rearranged as illustrated in Example 2 since the object locates in the different place inThai.

Example 1. [CL2] [ ] [ ] [ ] [ ] [ ] (Dang and Dum travel often.)(classifier) (cheu: NAME) (Dang: proper name) (Dum: proper name) (bpai-tieow: TRAVEL) (b�oi-b�oi: OFTEN).

TSL input

[CL2-classifier].

[ –name] [ –Dang] [ –Dum] [ –travel] [ –often] Word addition output [ –Dang] [ –and] [ –Dum] [ –travel] [ –often] Word ordering output [ –Dang] [ –and] [ –Dum] [ –travel] [ –often]

Example 2. [ ] [ ] [ ] [ ] [ ] (The father buys two fishes.)(bplaa: fish) (so�ng: two) (dtua: classifier) (por : father) (s�eu: bought).

TSL input

[ –fish] [ –two] [ –father] [ –bought] Word addition output [ –fish] [ –two] [ –unit-of-noun] [ –father] [ –bought] Word ordering output [ –father] [ –bought] [ –fish] [ –two] [ –unit-of-noun]

5. Translation samples

SignMT was designed to be able to translate any type of sentence, e.g., affirmative, negative, interrogative and imperative. In our pre-liminary experiment, the developed sign picture dictionary contains only 250 entries, and expanded to 500 and 1000 entries, respectively.There has been no significantly increase in processing time when running SignMT with the larger dictionary. The sample sentences werecollected from different sources including textbooks, cartoons, bedtime stories and newspapers. Some translation results are shown inTable 5.

In Table 5, rows 1–3, 4–5, 6–8 and 9–11 show the translation of affirmative sentences, negative sentences, interrogative sentences andimperative sentences, respectively. Rows 4 and 5 show the translation of negation. Rows 12 and 13 show the translation of sentences withThai and TSL classifiers. Rows 14 and 15 show the translation of sentences with preposition. Rows 16 and 17 show the translation of di-transitive verb. The translation of the compound sentences is shown in rows 18, 19 and 20.

Page 10

Table 5Examples of translation results.

N. Ditcharoen et al. / Computers & Education 55 (2010) 118–130 127

6. SignMT: an MT-CALL

Since SignMT was aimed at being a language learning tool not only for deaf people to learn Thai and TSL (in primary school), but also forhearing people who wish to learn TSL, the interface then must be designed for their convenience, and is able to display clear information.The interface of SignMT is comprised of two main windows: input and output windows.

Input window was designed to facilitate the TSL input process. For example, to input the TSL sentence ‘‘[ –chair] [ –aunt] [ –sit](An aunt sits on a chair)”, the user first selects the parameters (e.g., handshapes, location) provided in the combo boxes as shown in Fig. 7(Section 4.2) for the sign ‘‘[ –chair].” From the selected parameters, SignMT generates the suggested sign list which is the list of the nearmatched signs to the sign ‘‘[ –chair].” The user then selects the correct sign‘‘[ –chair]” from the list. The TSL input is shown in the lastrow at the end of the input process.

The history of the input sentence is kept for the user to conveniently select and edit in the next translation; the button ‘‘ (history)”is used to bring up the history form. Once the button ‘‘ (translate)” is clicked, the translation result in Thai is shown in the first row of theoutput window (Fig. 9) e.g., (An aunt sit on a chair). The grammar information of the output sentence is presented in the secondrow. It includes words with their corresponding POS and functions. The last row presents the meaning of each word in the output sentencein TSL.

7. Evaluation

The evaluation of SignMT was divided into two parts: translation accuracy and user satisfaction. The translation accuracy was examinedin terms of intelligibility and fidelity by linguistic teachers (both hearing and deaf), deaf students, and TSL interpreters. All testers are fromRatchasuda College, a college for persons with disabilities in Bangkok, Thailand. The goal of translation accuracy measurement is to deter-mine whether the system can generate a correct and reliable translation or not. The standard performance measures–accuracy, precision,recall and F-score, as shown below, were used to evaluate the intelligibility and fidelity.

Given Y is a set of candidate sentence and X is a set of reference.

Page 11

Table 6The intelligibility and fidelity evaluation.

Tester Accuracy (%) Precision (%) Recall (%) F-score (%)

1 94 100 96 97.962 93 100 95 97.443 95 100 94 96.914 95 100 96 97.965 94 100 94 96.91

Fig. 9. User interface of translation output.

128 N. Ditcharoen et al. / Computers & Education 55 (2010) 118–130

Accuracy ðY Xj Þ ¼ jYjjXj ; ðX; Y > 0Þ ð1Þ

Precision ðpÞ ¼ jX \ Y jjYj ; ðX; Y > 0Þ ð2Þ

Recall ðrÞ ¼ jX \ YjjXj ; ðX;Y > 0Þ ð3Þ

F1 ðp; rÞ ¼2pr

pþ rð4Þ

The 354 sample sentences and phrases which were collected from different sources including textbooks, cartoons, bedtime story, andnewspapers, were used in this evaluation. The intelligibility and fidelity were evaluated from six measures – (1) correct grammar, (2) cor-rect word usage, (3) inappropriate word usage, (4) incorrectly translated word, (5) convey the original meaning, and (6) convey the differ-ent meaning but can understand the original meaning. Table 6 shows the results of each measurement with the 93–95% accuracy, 100%precision, 94–96% recall and 96.91–97.96% F-score.

The goal of SignMT is to satisfy the user in both translation and language learning. We evaluate the user’s satisfaction in four aspects:preference, convenience, advantage, and needs. The questionnaires and interview were then designed to assess the student’s preference forSignMT (e.g., How do the students like using SignMT? How do they enjoy using SignMT?), the convenience of usage (e.g., how does the inter-face look? How clear does the communication?), the advantages of using SignMT (e.g., Do the students learn more vocabularies after usingSignMT? Can the students practice translation on their own?) and the need for SignMT (e.g., Can the automatic translation help the stu-dents in learning language? Which device/system do the students want in practicing language?). Some sample questions and users’ re-sponses are shown in Table 7.

Page 12

Table 7Sample questions and users’ responses.

Questions:With SignMT, students can concentrate on the studyWith SignMT, students can exchange their knowledge easily and naturalSignMT assists students to get enthusiastic to learn moreSignMT provides enough information to student in learning Thai languageWith SignMT, students are interested and enjoy being in the classroomSignMT is able to help students pay more attention to studySignMT allows students to be freedom to think out of boxSignMT is suitable to be used for self-learningWith SignMT, students can improve the literacy skillStudents want teachers to use SignMT to engage in learning languageResponses:This tool is friendly and easy to useI feel excited to using this toolIt provides the information about Thai grammar which I’m not good atThe tool helps me a lot in ordering Thai wordsThe tool helps me to practice writing Thai even at homeLearning Thai is boring, but with this tool, I feel enjoyable to play with Thai words. It helps me to create my own Thai sentencesI could communicate with my hearing teacher easily

Table 8The user satisfaction evaluation (5-excellent, 4-good, 3-fair, 2-poor, 1-very poor).

Mean Std. deviation Test value = 0

t df Sig. (2-tailed) 95% confidence interval of the difference

Lower Upper

Preference 4.15 .745 24.907 19 .000 3.80 4.50Convenience 4.35 .587 33.133 19 .000 4.08 4.62Advantage 4.25 .716 26.533 19 .000 3.91 4.59Need for user 4.15 .745 24.907 19 .000 3.80 4.50

N. Ditcharoen et al. / Computers & Education 55 (2010) 118–130 129

The questionnaires and interview were conducted at Ratchasuda College, with 98 deaf students. The evaluation results show that Sign-MT is beneficial to the users and satisfies their needs as shown in Table 8.

8. Conclusion

We present SignMT, an alternative language learning tool for deaf, which was designed to translate from Thai sign language into Thaitext. Its translation process is comprised of four steps: word transformation (WT), word constraint (WC), word addition (WA), and wordordering (WO). The distinction between Thai and Thai sign language in both syntax and semantic are concerned in each processing step.The translation begins with WT to transform the sign picture into matched text. WC removes untranslated words however, original mean-ing is still preserved. The semantic and syntax of target language are completed in WA, and WO make the structure of target language cor-rect. The knowledge-bases required in each processing step are developed separately for easy modification, expansion, and maintenance.The interface was designed to be used easily and conveniently for both deaf and hearing. The key features of SignMT are simplicity, mod-ularity, accurate, and user-friendly. SignMT is beneficial on increasing deaf students’ interest and motivation, improving self-concept andmastering of basic skills, and being more engagement in the learning process. The initial experiment and evaluation indicates that thetranslation accuracy is acceptable and the system performance satisfies the users’ needs.

References

Atkinson, R. (1968). Computerized instruction and the learning process. American Psychologist, 23, 225–239.Dailynews. (2002). Article from Dailynews on August 8, 2002. <http://www.dailynews.co.th> Retrieved on September 2008.Dangsaart, S., Naruedomkul, K., Cercone, N., & Sirinaovakul, B. (2008). Intelligent Thai text – Thai sign translation for language learning. Computers & Education, 51(2008),

1125–1141.Ehsani, F., & Knodt, E. (1998). Speech technology in computer-aided language learning – Strengths and limitations of a new CALL paradigm. Language Learning and Technology,

2(1), 45–60. July.Graham, D. (2002). CALL in the good practice guide at the website of the subject centre for languages, linguistics and area studies, University of Southampton. <http://

www.camsoftpartners.co.uk/docs/LLAS_Web_Guide.htm> Retrieved on May 2008.Lee, J & Kunii, T. L. (1992). Visual translation – From native language to sign language. In Proc. of 1992 IEEE workshop on visual languages, 15–18 September 1992, pp.103–109.Ministry of Education. (1999). Promulgation for acceptance of Thai sign language as national language for deaf, August 17, 1999.Mitchell, R., & Myles, F. (1998). Second language learning theories. London: Arnold.Molnar, A. (1997). Computers in education – A brief history. Technical Horizons in Education. <http://www.thejournal.com/articles/13739> May 2008.Naruedomkul, K., & Cercone, N. (1999). The role for word association numbers in machine translation. In Proc. of the conference pacific association for computational linguistics

(PACLING’99), Waterloo, Ontario, Canada, pp. 379–392.Noemi, D. (n.d.). Computer-assisted language learning – Increase of freedom of submission to machines? Retrieved on May, 2008.Passey, D., Rogers, C., Machell, J., & McHugh, G. (2004). The motivational effect of ICT on pupils. Report No. RR523.DfES. <http://www.dfes.gov.uk/research/data/uploadfiles/

RR523new.pdf> Retrieved on June 2005.Rost, M. (2002). New technologies in language education – Opportunities for professional growth.Stepp-Greany, J. (2002). Students perceptions on language learning in a technological environment – Implications for the new millennium. Language Learning and Technology

(January).Stokoe, W. C. (1978). Sign language structure (2nd ed.). Silver Spring, MD: Linstok Press.

Page 13

130 N. Ditcharoen et al. / Computers & Education 55 (2010) 118–130

Suppes, P., & Morningstar (1969). Computer-assisted instruction. Science (Reprinted in D. A. Erickson (Ed.), Educational organization and administration (pp. 236–254).Berkeley CA: Mccutchan)..

Sutton-Spence, R., & Woll, B. (1999). The linguistics of British sing language – An introduction. Cambridge: University Press.Thelmadatter, L. (2007). The computers are coming . . . are here! TESOL Greece Newsletter 1995 (July/September).Warschauer, M. (1996). Computer assisted language learning – An introduction. In S. Fotos (Ed.), Multimedia language teaching (pp. 3–20). Tokyo-Logos International.Wideman, C. J., & Sims, E. M. (1998). Signing avatars. In Technology and persons with disabilities conference. <http://www.csun.edu/cod/conf/1998/proceedings/

csun98_027.htm> Accessed January, 2009.William, C. Stokoe (1960). Sign language structure – An outline of the visual communication system of the American deaf. Burtonsville: Linstok Press.