MASARYK UNIVERSITY FACULTY OF EDUCATION DEPARTMENT OF ENGLISH LANGUAGE AND LITERATURE Some aspects of assessing pronunciation in EFL classes Diploma thesis Brno 2007
MASARYK UNIVERSITY
FACULTY OF EDUCATION
DEPARTMENT OF ENGLISH LANGUAGE AND
LITERATURE
Some aspects of assessing pronunciation in EFL classes
Diploma thesis
Brno 2007
Supervisor: Written by:
PhDr. Jaroslav Ondráček Simona Šebestová
Declaration
I hereby declare that this thesis is my own and that I used only the resources
stated.
Simona Šebestová
.......................................
2
Acknowledgements
I wish to thank to PhDr. Jaroslav Ondráček, the supervisor of my thesis, for his
valuable advice and guidance on the whole work. Thanks also belong to Petra
Hořejší and Štěpánka Zemanová for their help with the assessment and my
special thank goes to all my students who willingly cooperated.
3
Contents
Introduction...............................................................................................................6
1 Teaching and learning English pronunciation..........................................................7
1. 1 The role of the teacher......................................................................................8
1. 2 The role of the learner......................................................................................8
1. 3 Setting goals in English pronunciation.............................................................9
1. 4 What affects pronunciation learning..............................................................11
1. 4. 1 The native language.................................................................................11
1. 4. 2 The age....................................................................................................12
1. 4. 3 Phonetic ability........................................................................................13
1. 4. 4 Exposure to the language.........................................................................14
1. 4. 5 Attitude and sense of identity..................................................................15
1. 4. 6 Motivation...............................................................................................16
2 Giving feedback on pronunciation.........................................................................16
2. 1 Assessment methods.......................................................................................17
2. 2. 1 Impression-based pronunciation testing..................................................18
2. 2. 2 Atomistic testing......................................................................................19
2. 2. 3 Holistic testing.........................................................................................21
2. 2. 4 Atomistic versus holistic approach..........................................................25
3 Practical part...........................................................................................................26
3. 1 Introduction....................................................................................................26
3. 2 Evocation........................................................................................................27
3. 2. 1 Activity 1.................................................................................................29
3. 2. 2 Activity 2.................................................................................................31
3. 3 Realisation......................................................................................................34
3. 3. 1 Limited-response activities......................................................................35
3. 3. 2 Free-response activities...........................................................................38
3. 4 Reflexion........................................................................................................40
4 Research part.........................................................................................................46
4. 1 Plan and goals.................................................................................................47
4. 2 Methods..........................................................................................................47
4. 3 Evocation and pre-research............................................................................48
4. 4 Realisation......................................................................................................52
4
4. 4. 1 Atomistic testing results..........................................................................54
4. 4. 2 Holistic testing results.............................................................................57
4. 5 Reflexion........................................................................................................63
4. 5. 1 Assessors‘ agreement in atomistic testing...............................................64
4. 5. 2 Assessors‘ agreement in holistic testing..........................................67
Conclusion.......................................................................................................................70
Appendix 1......................................................................................................................73
Appendix 2......................................................................................................................75
Bibliography....................................................................................................................77
5
Introduction
English language plays a dominant role in modern
world. Each of us has ever been encountered with the
necessity to learn the language for many reasons. For many
people the most important function of any language is
communication with other people when travelling, studying
or working in a foreign country. Therefore, the trend of
learning English is increasing and the opportunities for
the actual use of the language are becoming more frequent.
The topic of this work is based largely on the
communicative function of language – speaking and
listening. We intend to deal with an aspect of English
which is, in our opinion, often neglected in English
lessons – pronunciation testing and its assessment.
The idea arose from our belief that teaching
pronunciation should be a part of a syllabus of any English
course, moreover, should be included from the very
beginning. One reason for this statement is that it is
generally believed that the age factor is influencial and
small children can learn pronunciation more easily than the
older or adults. The next benefit of pronunciation learning
is its possible effect on listening skills. In order that
our communication is comfortable for both sides, we need to
understand as well as our speaking should be
understandable. So the practice of our pronunciation helps
other people to understand us better.
Nowadays, almost every coursebook offers pronunciation
practice but any suggestions on giving such a structured
feedback like those on other aspects of language are rare.
We will try to present ways of testing pronounciation and
try to deal with the possible ways of the consequent
assessment.
6
Apart from the objective methods of testing
pronunciation which are not usually available in schools,
there are other – more often used – subjective methods.
These methods are based on the assessor’s impression, so
certain criteria should be followed to reach as reliable
results as possible.
The first part of this work will present the tasks
needed to be considered for efficient learning and teaching
– the role of the teacher and the learner and the factors
which have an effect on the whole pronunciation teaching
and learning process.
Next, we will give an overview of the testing methods
in connection with the two approaches to impression-based
assessment – atomistic and holistic.
The last research part will implement the theory into
practice and find out about what results can be actually
brought by the pronunciation testing and what problems in
assessment may be encountered.
1 Teaching and learning English pronunciationPronunciation seems to be sometimes a neglected part
in our English lessons. Many teachers are not used to
teaching it for many reasons. As most teachers in our
schools are not native speakers, there is no need to set
native-like pronunciation as one of our goals, moreover,
most teachers themselves do not feel perfect in this
language component and thus feel reluctant to show it. But
we do not need to be perfect to enable our pupils to
achieve their best. The next obstruction for teachers is
the lack of pronunciation tests and general unfamiliarity
with their assessment. Nevertheless, giving feedback on
correct speaking to our students should be involved.
Learning pronunciation will elevate their level of speaking
and undoubtedly will improve their listening skills.
7
Before teaching pronunciation, many aspects should be
taken into account. Among the most important ones are the
roles of the teacher and the learner. On the one hand, what
they aim to achieve and, on the other hand, what they are
willing to give to succeed in achieving it. Before setting
up goals and working out a plan, the teacher should know
about their learners’ skills and limitations. Needless to
say, the overview of the various aspects of English
pronunciation – sounds, stress, rhythm and intonation – is
essential.
1. 1 The role of the teacherAs pronunciation is a very complex language component,
there are many tasks for the teacher. Firstly, helping
learners hear and produce sounds from their native language
point of view. Secondly, it is establishing what to
concentrate on. It is advisable to set out priorities
according to the acceptability to the English speaker as
well as to what is beyond good intelligibility and is not
necessary to take one’s stand on. Thirdly, devising
activities, adjusting them for different learning styles,
and according to the aspects which influence pronunciation
learning as it is going to be described in the following
chapters. The last but not least important task is
providing feedback and assessing learners’ performance and
progress. Assessing one’s own production of speech is very
difficult as we tend to hear ourselves in a distorted way
and thus it is complicated to compare with the correct
production. Moreover, the feedback on how the learner is
doing and progressing is an essential motivation factor in
further pronunciation learning.
8
1. 2 The role of the learnerThe learner’s role is the same as in any other subject
which means taking one’s own responsibility and being
willing to learn. Here, the teacher’s possibilities are
limited, but still the choice of appropriate activities,
motivating learners and building the general awareness of
usefulness may be supportive. For example, one of the
methods for increasing motivation can be a class-discussion
based on one’s own experience with foreigners and their
pronunciation, what is acceptable and what is disruptive
while talking to a foreigner in any language. The issue of
motivation within all subjects of learning is definitely
very complex, depending on many inner and outer factors and
should not be neglected.
1. 3 Setting goals in English pronunciationPeople learn languages for many different purposes.
And therefore, the goals for individual learners may vary.
From the teachers’ point of view, the following aspects
should be taken into account: the age, natural ability and
motivation of the learners which is to be the base for
answering the questions about how much time we will devote
to teaching pronunciation and what level is needed for
obtaining efficient communication. This is difficult,
since, in contrast with e. g. grammar or vocabulary plans,
pronunciation does not enable this particular progressive
pace as all phonetic and phonological features occur from
the very beginning. Nevertheless, we can count on the
subconscious acquisition of the sound of English which will
be beneficial for both, teachers and learners, later on.
We can delimit two extreme targets in learning
pronunciation. On the one hand, some learners aim to obtain
native-like pronunciation, on the other hand, many
learners’ purposes are more practical in the way that as
9
long as their speaking is comprehensible, they do not have
the need to improve. Both these opinions have their
advantages and disadvantages. In practice, many learners do
not achieve native-like pronunciation and the question is,
if its obtaining is necessary. At the same time, learning
pronunciation does not only improve speaking, but has a
great influence on our listening skills, so its practice is
useful.
According to Gimson (1994, p. 273), the first extreme
target is achieving just such a level of pronunciation
which enables understanding. Gimson mentions so called
Minimal General Intelligibility as the lowest requirement.
This Minimal General Intelligibility:
possesses a set of distinctive elements which
correspond in some measure to the inventory of
the RP phonemic system and which is capable of
conveying a message efficiently from a native
English listener’s standpoint, given that the
context of the message is known and that the
listener has had time to “tune in” to the
speaker’s pronunciation. (p. 273)
In its opposition, Gimson describes High Acceptability as:
a form of speech which the native listener may
not identify as non-native, which conveys
information as readily as would a native’s and
which arrives at this result through precision in
the phonetic realization of phonemes and by
confident handling of accentual and intonational
patterns. (p. 273)
10
As the previously mentioned statements were the
extremes in attitude to pronunciation learning, our aim
should be somewhere in between, which means that we should
aim to reduce the time to “tune in” for the listener as
well as to put down the strain for the speaker in order to
make the conversation comfortable for both sides.
1. 4 What affects pronunciation learningLike in other subjects, there are many factors which
affect pronunciation learning, but some of them are
specific to this language section as well as to learning
foreign language as a whole. It involves both learning as
an organized and intentional process, and the unintentional
language acquisition, which proceeds subconsciously and
intuitively, however, is not less beneficial. Both,
learning and the language acquisition, are dependable on
the inner and outer factors, which the teachers should be
aware of when setting out goals and preparing methods and
materials.
The factors which cannot be influenced neither by the
learner, nor his or her surroundings are the native
language, the age and to a certain extent the phonetic
ability of the learner. In contrast, the amount of exposure
to English, the attitude and motivation may be supported
and changed a lot either by the teacher, or the learner,
but usually in cooperation and support from both sides.
1. 4. 1 The native languageThe influence of the native language is unquestioned.
It involves the individual sounds as well as combinations
of sounds and features such as rhythm and intonation. The
more differences there are between the native and the
target language, the more difficulties will be encountered
by the learner. The learner’s errors derive from various
11
sources. When a particular sound does not exist in the
mother tongue, the learners tend to substitute it by the
nearest equivalent from their native language. For example,
the sound [ð] will often be substituted by [d] or [z] as
these are the nearest. This first language application
affects the rhythm and the intonation too. Still, the
influence of the native language is only one of the factors
and does not need to be crucial.
1. 4. 2 The ageIt is commonly assumed that the age factor has a great
influence on pronunciation. If someone has a native-like
pronunciation in a second language, they are very likely to
have started learning it as children. And conversely,
people who started to learn a second language in adulthood,
will scarcely achieve a native-like accent, although their
grammar and vocabulary may be perfect. However, the results
of the studies on the age-relation issue which have been
carried out are conflicting. One should take into account
that it is very difficult to control other factors like
motivation, ability, opportunity to hear and use the
language and the learners‘ attitude, therefore the results
are bound to be distorted to a certain extent. Don Snow
(1992) comments on this issue in a contradictory way:
Research comparing children to adults has
consistently demonstrated that adolescents and
adults perform better than young children under
controlled conditions (e.g., Snow & Hoefnagel-
Hoehle, 1978). One exception is pronunciation,
although even here some studies show better
results for older learners. (p. 129)
12
The idea of pronunciation learning being somehow
dependent on the age has been supported by some researchers
who claim that language learning has a sensitive period and
that after a certain age people lose some abilities. This
certain age is said to be between ten and thirteen years.
(Kenworthy, 1987, p. 6) The age-relation is supported also
by others:
Pronunciation is one area where the younger-is-
better assumption may have validity. Research
(e.g., Oyama, 1976) has found that the earlier a
learner begins a second language, the more
native-like the accent he or she develops. (Snow,
1992, p. 129)
From the teacher‘s point of view, we assume, that for
learning pronunciation the age of primary and lower-
secondary level pupils is more convenient, as they are less
shy and often more willing to do activities which the adult
learners may find strange or silly. In children, we can
take the advantage of their spontaneity and willingness to
sing and chant, which is often not appropriate in older
learners. „Research suggests that older students will show
quicker gains, though younger children may have an
advantage in pronunciation.“ (Snow, 1992, p. 129) However,
when the adult learners are motivated and want to approach
native-like pronunciation, they are able to focus better on
their problematic areas and this fixed-target acquisition
may be also very efficient.
1. 4. 3 Phonetic ability
It is generally assumed that some people have a better
ability to hear foreign languages than others. This
“aptitude for oral mimicry”, “phonetic coding ability” or
13
“auditory discriminating ability” proved by researchers who
designed tests which measure this ability. Although the
results proved that some people can better discriminate
between two sounds, still it is not a crucial condition of
obtaining good pronunciation as training itself has a
significant effect. (Kenworthy, 1987, p. 6)
The point is, whether different types of learners
should be taught with different methods. Kenworthy (1987)
divides these learners into poor discriminators and good
discriminators and explains:
One study has indicated that those with good
phonetic abilities benefit from pronunciation
drills, tasks in which particular sounds are
heard and the learner has to imitate again and
again. Their innate abilities enable them to
exploit all the opportunities to compare what
they are doing with the model presented. “Poor
discriminators” do not seem to benefit from
drills very much. In fact, drills seem to cause
their attempts to stabilize before they reach an
accurate production of a sound. Because of the
complexities involved, this seems a factor which
is very much out of the control of the teacher.
We can only operate on the assumption that our
learners have the “basic equipment” and provide a
variety of tasks so that something will suit the
needs and abilities of each learner. (p. 7)
1. 4. 4 Exposure to the language
The exposure to the target language has a wide range
of meanings and intensity. The meanings may vary from
living in the country where the language is spoken to
talking to native-speakers or using authentic materials, e.
14
g. films, literature, music, television, so the term is
quite wide in its meaning. Living in the country of the
target language does not always mean the actual use of the
language. Many people may live in a non-English-speaking
environment, or use their mother tongue with their
families. Conversely, many learners who live in a non-
English speaking country may use English in many daily
situations like school or work, so the amount of exposure
basically depends on its quality and quantity.
Logically, being exposed to English in the English
environment should have a positive effect on the learner‘s
pronunciation and listening skills. “Research, however,
indicates that this increased exposure to English does not
necessarily speed the acquisition of English.“ (Snow, 1992,
p. 129)
It is considered to be a contributory factor, but is
not the most important.
1. 4. 5 Attitude and sense of identityIn the acquisition of pronunciation of a foreign
language the attitude and sense of identity of the learner
plays a major role. It is a personality-determined factor
and we can hardly predict whether people having come to the
environment with different accent will modify their own.
“Some seem to be “impervious” and even after a long time
will absorb only some turns of phrase and the pronunciation
of a few individual words; others seem very receptive and
begin to change their accent almost as soon as they step
off the plane.” (Kenworthy, 1987, p. 7)
Using or not using the accent may show different
attitudes to people. When imitating or adopting the accent
of the host country, we show our positive relation to their
language and culture as well as respect for them as people.
15
It is sign of how much we would like to integrate ourselves
in the new place.
As far as teaching pronunciation is concerned, the
attitude and sense of identity factor is closely connected
with the inner motivation factor which is going to be
explained next.
1. 4. 6 MotivationFor some learners pronunciation is more important than
for others. The learners who consider pronunciation to be
an important part of their English learning are usually
eager to be corrected and concerned about how good or bad
their pronunciation is. This can be also expressed by
unwillingness to talk when being uncertain about one’s
pronunciation: “If we cannot say it perfectly, we won’t say
it at all.” In these learners the teacher can perform
magic, because the learner wants to cooperate and improve.
To maintain this motivation from the learner’s perspective,
the teacher should set out goals and give the learner
feedback, so that both are aware of some kind of progress.
Conversely, the other extreme is, when the learner is not
aware that their way of speaking causes difficulty,
irritation or misunderstanding for the listener. The
teacher’s possibilities in motivating learners are limited,
nevertheless, the factor of motivation should not be
neglected as it is considered to be one of the most
important factors affecting pronunciation learning.
2 Giving feedback on pronunciationTeaching a language is inevitably tied with feedback
which involves either formal or informal testing, assessing
achievements, and offering suggestions for making
improvements.
16
As we have already mentioned, giving feedback on
pronunciation is essential for maintaining motivation and
providing students with information on how they are
progressing and what they should focus on. Compared to
learning grammar or vocabulary, where students are able to
assess themselves objectively having the correct answers at
their disposal, self-evaluation in pronunciation is
complicated since it is very often distorted by the
student’s own ear and phonetic ability. The importance of
teacher’s role in giving feedback on this language
component is thus unassailable and crucial. Yet, also from
the teacher’s point of view, assessing pronunciation, in
comparison with other language components seems to be a
difficult task as, at least, in the literature which we
have had at our disposal not much attention is paid to this
issue. The main reason for this negligence is the fact that
speaking, comprising pronunciation, is a skill which is too
complex to enable any realiable analysis which could be
considered to be objective.
17
2. 1 Assessment methods
Learning pronunciation is based on both production and
reception/ identification, therefore, it is closely
connected with oral exams and listening tests. Both
speaking and listening activities may be used for testing
and assessing our learners’ progress, although they do not
have to provide us with comparable and agreeable results.
Contrary to production, reception can be tested and
quantified by counting the correct answers in a written
test, whereas production is more difficult to assess as it
involves testing speaking or reading which, from the
listener’s point of view, include many other aspects of
communication, not pure sounds. It is always subjective to
a large extent and so the problematic matter of assessing
pronunciation production is its reliability. We are bound
to rely on the impression of the assessor. Therefore, both
the procedure and the assessment should be defined as
concretely as possible. Depending on the aim of our
assessment, two approaches of testing pronunciation are
going to be described – atomistic and holistic. Before
dealing with these, we look at the most common concept of
pronunciation testing.
2. 2. 1 Impression-based pronunciation testing
Taking into account our school environment, teaching
and technical conditions in our classes, testing based on
the teacher’s impression rather than using any objective
scientific methods is inevitable. Nevertheless, we should
be aware of the possible difficulties. According to Szpyra-
Kozlowska et al. (2005), the impression-based pronunciation
testing has many drawbacks. She mentions that different
assessors may have different criteria of evaluation and it
is also more general and therefore can be imprecise. In
opposition to such an opinion, Kenworthy (1987, p. 20)
18
believes that also assessments which are impressionistic
and therefore subjective can be both dependable and
accurate. She claims that the impression-based rankings
tend to agree with other objective ranking techniques.
What speaks in favour of the impression-based
pronunciation testing, is the fact that it is typical of
many international examinations in English, like Cambridge
English Examinations, where the assessors evaluate
"intelligibility and the amount of strain a candidate's
pronunciation puts on the listener.“ (Szpyra-Kozlowska et
al., 2005)
For example, the requirements on pronunciation in
First Certificate in English are defined: "although
pronunciation is easily understood, L1 features may be
intrusive"1, in Certificate in Advanced English, the same
is defined as follows: "L1 accent may be evident but does
not affect the clarity of the message"2 and in Certificate
of Proficiency in English: "pronunciation is easily
understood and prosodic features are used effectively; many
features, including pausing and hesitation, are "native-
like"."3 Obviously, the definitions of the requirements in
Cambridge English Examinations are very general and
impressionistic in nature. The examiners who are very often
non-native speakers of English are instructed as follows:
"when assessing pronunciation, examiners should try to put
themselves in the position of a non-EFL specialist, native
speaker of English and assess the amount of strain on the
listener and the degree of patience an effort required to
understand the candidate.“ (as cited in Szpyra-Kozlowska et
al., 2005)
On the basis of this instruction, the assessment may
have more interperetations depending on the assessor and is 1 Paper 3: Speaking. Cambridge Common Scale for Speaking [online; quoted 25. 1. 2007] http://www.cambridgeesol.org/support/dloads/ket/KET_HB_sampleS.pdf2 ibid3 ibid
19
bound to be always subjective to some extent, but we can
conclude that it is a method which can provide us with
information about the progress and achievement in quite a
plausible way and it enables us to compare the learners‘
performances within time or class.
2. 2. 2 Atomistic testingAtomistic approach requires a detailed marking scheme
in which specific aspects of pronunciation are evaluated
separately. It means reading aloud word lists based on
phonemic oppositions, short sentences containing minimal
pairs or it also enables testing appropriate sentence
accentuation or sentence stress and intonation. This
approach is claimed to be more objective than the holistic
one as it judges only segments of speech – particular
vowels, consonants, stress, rhythm, intonation, etc.
Nevertheless, the drawbacks of this approach are the
demands on the assessor – in our school environment – the
teacher. It requires recording the learners’ speech samples
and repeated listening to them, so it is extremely time-
consuming and thus unsuitable for large classes.
3. 2. 2. 1 Methods of testingAtomistic testing may apply to both production and
identification of sounds, stress patterns and intonation.
It is advisable for the teacher to choose a good material
and set out clear criteria beforehand. Atomistic testing
focuses always on a segment of speech, therefore we often
have to omit those wrongly pronounced segments of language,
which have not been aimed to assess. This is the only way
both to compare our learners’ performances, and to avoid
any confusion from the overload of information.
Repetition
20
On the beginner level, the easiest test to prepare is
repetition exercise. It is useful for learners who cannot
read or who are beginning with English. It is based on
hearing sounds, stress and intonation, and imitation which
gives the teacher the gist of learners’ potential and
phonetic ability. The test may consist of single words or
sentences checking particular items rather than all
pronunciation aspects at the same time in order to be as
much objective as possible.
Hearing identification
Another way of testing beginners as well as more
advanced learners is hearing identification (Madsen, 1983,
p. 61). Good pronunciation is dependent on our ability to
hear the language. It can consist of recognizing sounds in
minimal pairs, the fall or rise in intonation or
indentifying stress in words or sentences.
Reading aloud
Commonly used way of pronunciation assessment is
reading aloud. According to Madsen (1983) three points
should be kept in mind:
(1) When using lists of sentences, evaluate only
one or two points per sentence;
(2) use natural language;
(3) avoid signalling to the student which
pronunciation point you are testing. (p. 66)
Since reading tends to be longer and involve many
points to assess at the same time, it is advisable to
record the learners’ performances in order to listen to
them repeatedly and have the possibility to compare. The
material to read should enable natural sound, e. g. a
21
letter, instructions etc., and students should have time to
read the text silently before reading for assessment to get
the context. The reading aloud testing provides good
control and enables to test almost all aspects of
pronunciation including stress and intonation as well as
vowels and consonants. Nevertheless, we have to count on
the fact that reading and speaking skills are not the same
and, inevitably, the intonation and sentence rhythm in
reading is usually not as natural as in normal
conversation.
2. 2. 3 Holistic testingA higher level of achievement is testing
intelligibility and acceptability of the learner’s
performance. In this holistic approach to pronunciation
testing “examiners are asked not to pay too much attention
to any one aspect of a candidate’s performance, but rather
to judge its overall effectiveness.” (Alderson, 1996, p.
289)
The advantage of this procedure is that it can be
administered to large groups and is not as time-demanding
as the atomistic approach. As it has been already mentioned
this approach is used many international exams in English,
where the pronunciation is involved in so called
intelligibility and acceptability of the candidate’s
speaking performance. The definition of intelligibility is
very general: “Intelligibility is being understood by a
listener at a given time in a given situation. So it’s the
same as “understandability”.” (Kenworthy, 1987, p. 13) To
put intelligibility more concretely, we can say that: “The
more words a listener is able to identify accurately when
22
said by a particular speaker, the more intelligible that
speaker is.” (Kenworthy, 1987, p. 13)
In practice, words are made up of sounds and if the
speaker substitutes one sound for another and the result is
that the listener hears a different word, it means that the
speaker is unintelligible or not intelligible comfortably.
However, when the wrong substitution occurs in a speech,
but the word is anyhow understood, then we can say that the
speech is intelligible. Considering this should be the base
for setting out the aims of our pronunciation lessons,
which does not necessarily need to be native-like speaking
performance.
According to Kenworthy (1987; p. 14), as
intelligibility is very complex, there are many factors
influencing it. One of them is the counts of sameness. This
applies to the Czech practice for example in the sound [ð]
which is often substituted for [d] as it is the nearest
equivalent in the Czech language. But, when the listener is
aware of this and the incorrect substitution does not
produce another word, he or she will not have difficulties
understanding. The other factors affecting learner’s
intelligibility are, for example, the over-use of self-
corrections, hesitations or grammatical restructurings. All
previously mentioned factors may be annoying or disruptive
for understanding as well as for the whole communication.
Another speaker factor which can cause problems is that the
person speaks too fast. According to Kenworthy (1987, p.
14) it is usually not the speed itself which causes
difficulties, but incorrectly applied other features of
speaking, like sentence stress, rhythm, and intonation
which makes it difficult for the listener to pick out the
most important bits of the message.
So far, all the above mentioned factors were speaker-
focused. But we have to be aware of the fact that there are
23
always at least two participants in a communication.
Intelligibility also depends on the listener. Kenworthy
mentions two important listener factors: “first, the
listener’s familiarity with the foreign accent and, second,
the listener’s ability to use contextual clues when
listening.” (Kenworthy, 1987, p. 14)
To conclude, understanding is dependent on the amount
of exposure and being used to the foreign accent. A teacher
should be aware of this fact. Being exposed to the
learners’ pronunciation, the teacher’s assessment is very
likely to be distorted.
The issue of intelligibility is very complex and is a
major part of communication. Therefore, the goal is not
only the correct production of sounds, stress patterns and
intonation, but efficiency of communication without
irritation and difficulties understanding. So the goal of
pronunciation can be defined as comfortable
intelligibility.
2. 2. 3. 1 Methods of testingAs we have already mentioned, the main criterion for
holistic testing is the efficiency of communication between
two people. Therefore, the best method is interactive
testing including more than only one participant. All the
activities should be used in the interaction of the
assessor or another student to involve both sides of the
communication – the speaker and the listener – to function
as an oral interview including natural situations and
asking questions.
Re-telling stories
This kind of test involves first reading a story
silently and then telling the story using one’s own words
24
and sentence structures. The assessor may interefere giving
further questions.
Description of pictures
Pictures may be used for description of objects,
people or scenes, or for comparison of two similar
pictures, in which the learner looks for similarities and
differences.
Sequence of pictures
This test is based on telling a story involving
linking words expressing the cause and the result. It can
be applied to only one student or a pair where each of them
is given one half of the pictures and they should decide on
the correct sequence of the story.
Pictures with speech bubbles
In this test students are required to guess what the
people in the pictures are saying. It may be used
individually or in pairs.
Using maps
Many student’s books involve a unit dealing with
giving directions. This activity is to be done in pairs,
where one gives the directions and the other one follows
them.
2. 2. 4 Atomistic versus holistic approachWhen deciding between atomistic and holistic testing,
the purpose of testing should be considered. As far as
reliability is concerned, in Szpyra-Kozlowska (2005) Hughes
(1991) says that atomistic tests are more reliable for
diagnostic purposes in the language classroom and in cases
in which scoring is carried out by different assessors,
25
whereas holistic approach is faster and more appropriate
for experienced assessors.
26
3 Practical part
3. 1 Introduction
In this practical part we are going to deal with the
practical aspects of pronunciation testing and assessment
in class. On the basis of the theoretical part, a
qualitative research has been carried out to test students‘
pronunciation and their progress within a period of time.
The idea of assessing students‘ pronunciation arose
out of its teaching. In my English lessons we did
pronunciation exercises but the process was somehow not
complete. The feedback for the learners as well as for me
as a teacher was missing apart from me informally
responding to the individual performances. The conception
of pronunciation in our coursebooks did not enable me more
than assessing the learner‘s immediate performance. Before
starting our research, the students had been taught
pronunciation but had not been provided with any statement
on if they were progressing or not, or just a list of
points what they should focus on. So the need of a
structured feedback and the lack of instructions in our
coursebooks led us to try to design a complete
pronunciation course involving all the necessary steps.
Having studied literature dealing with the issue of
pronunciation tests, I found the instructions scattered in
many books, always being presented just in bits and pieces.
Therefore, I tried to put them together to be used in my
lessons in order to see to what extent they are beneficial
for the learners and achievable as far as their
reliability, time and demands on the assessors are
concerned.
The research itself and its results will be presented
as the last item of this part and will be preceded by three
27
parts generally relating to the three stages used in the
research.
The first part is the „evocation stage“. It deals with
the factors affecting pronunciation learning which have
already been described in the theoretical part, to be
precise, it deals only with such factors which the teacher
or the learner have some influence on. Further, we present
activities and recommendations which may be used in the
classroom.
The second part is called the „realisation stage“.
This stage involves the realisation of teaching
pronunciation and its testing. We omit the teaching
pronunciation phase and focus on the ways of testing
proceeding from the level of sounds production towards
testing intelligibility and considering their weaknesses
and suitability.
The third part is the „reflexion stage“. It presents
the description of criterial levels for atomistic and
holistic assessing and the requirements on the choice of
the assessors.
3. 2 Evocation
Setting out goals and criteria of assessment as well
as teaching pronunciation itself requires considering all
aspects affecting pronunciation and its learning. Having
been generally explained in the teoretical part, some of
these aspects are given and cannot be changed by any
participant of the learning process. Among these the
following are listed: the age, the native language and the
phonetic ability of the learner. Since we cannot influence
them, it is essential that we take them into account when
planning a course. The factors which the teachers or the
learners have some, however, limited control of, are the
exposure to English, the attitude and the motivation.
28
Nevertheless, the exposure to English during the course of
learning it, is somehow given depending on the English or
non-English environment, but we can still advise our
students on other possibilities which can improve their
English outside the class, e. g. help them use authentic
materials like film, newspaper or find a native speaker to
communicate with. The teacher’s control is quite limited,
because we cannot influence whether our learners really do
so. But we can contribute by using the right motivation
methods, either self-made, or inspired by those presented
in literature.
In the theoretical part an opinion was expressed that
motivation belongs to the crucial factors affecting
pronunciation. Kenworthy (1987, p. 57) claims that "a
higher level of awareness and concern usually has a
positive effect on motivation and therefore achievement."
For this reason the stage of „building awareness“ should
not be neglected.
What may be particularly helpful are the „General
awareness-building activities“ (Kenworthy, 1987, p. 54).
These should be used in such situations in which the
teacher encounters any negative attitute from the learners‘
side towards the pronunciation learning. Kenworthy mentions
two of such activities: „Questionnaire based discussion“
and „A tape-based activity“. „The first is based on a
questionnaire, the second centres on samples of the native
languages of the learners spoken with a heavy foreign
accent. Both aim to help learners develop a concern for the
pronunciation of English through their personal experiences
of language in use.“ (ibid)
The descriptions of the activities have been inspired
by various resources and the instructions are only
suggested and open to any adjustments.
29
3. 2. 1 Activity 1Aim: Increasing interest and motivation, discussion on
pronunciation
Resource: Questionnaire, questions and instructions in
Kenworthy’s Teaching English Pronunciation, Dalton, Ch.,
Seidlhofer, B. Pronunciation (1994, p. 155) and Ur, P. A
Course in Language Teaching (1997, p. 51)
Time: at least 30 minutes
Language used: mother tongue is acceptable for meaningful
discussion
Procedure:
1 Filling in the questionnaires individually;
2 Comparing the answers in pairs or groups;
3 Creating statements and questions based on the
pair/group discussion;
4 Whole-class discussion of the prepared statements and
questions.
Output: Teacher's evaluation:
- How involved were the students in the discussion?
- Did agreement or disagreement predominate, and what do
you think are the reasons?
- Were there questions which did not lead to an interesting
discussion? If so, why?
- Are there questions which you would like to add?
30
Activity 1 - Questionnaire4
1 Imagine you are talking in your own language with a foreigner. The person doesn’t speak
your language very well and is very difficult to understand. What do you do? Do you:
(a) pretend you understand even when you don’t?
(b) ask him or her to repeat everything slowly and carefully?
(c) try to get away?
2 What do you say when the foreign speaker apologizes for his poor accent? Do you:
(a) tell him his accent is very good even when it isn’t?
(b) tell him that his poor accent doesn’t matter?
(c) tell him that his accent is very bad and that he must work hard to improve it?
3 How do you feel when a foreigner pronounces your name wrong?
(a) very angry
(b) it bothers me a little
(c) it bothers me a lot
(d) it doesn’t bother me at all
4 How do you feel when you meet a foreigner who speaks your language with a very good
accent?
(a) surprised
(b) pleased
(c) not surprised
(d) full of admiration
(e) don’t care or think about it
5 In the future, who will you speak English to?
(a) mostly English5 people visiting my country who don’t know my language
(b) mostly English-speaking people in this country (Britain, USA, etc.)
(c) mostly non-English6 people who don’t know my language and whose language I don’t know,
so that we speak English together
(d) don’t know
6 Do you think it is more important to have good pronunciation when:
(a) you are speaking English to English people?
4 presented in Kenworthy (1987, p. 54)5 We would suggest using „English-speaking“ people for all English native-speakers rather than „English people“ which we consider to stand only for the people from England.6 Similarly, we would suggest using „non-English speaking“ people for the same reason as in the previous footnote..
31
(b) you are speaking English to non-English people?
7 Below are some situations. When is it most important to pronounce well? Put them in order of
importance with a number if you want.
(a) speaking on the telephone
(b) meeting someone for the first time
(c) talking to someone you know very well (a good friend) in an informal situation (e.g. at a
party)
(d) doing business in English (e.g. at the bank, post office, bus station, railway station, in
shops, etc.)
(e) talking to strangers (e.g. asking the way)
(f) chatting to a fellow student (e.g. during break time)
To sum up, on the basis of this activity, the teacher
should receive information on the students‘ attitude
towards learning pronunciation which is helpful to design
the course according their own needs and goals.
3. 2. 2 Activity 2Another awareness building activity is "a tape-based
activity". It consists of a recording of a strongly
foreign-accented speech and following activities based on
either discussion of given questions, or also a table to be
filled in preceding further discussion. As an example,
Penny Ur (1991, p. 51) presents a box for the students to
work with:
WORKSHEET: RECORDINGS OF FOREIGN PRONUNCIATION
32
Speaker’s mother tongue:
Words/phrases mispronounced Define or describe the mistake
Aim: Increasing interest and motivation, stimulate
discussion of the implications for communication and social
interaction of heavily foreign-accented speech.
Resource: Kenworthy (1987, p. 55), Ur, P. (1991, p. 51)
Time: at least 45 minutes
Language used: Mother tongue is acceptable for meaningful
discussion
Procedure:
1 Introduce the task asking: "I'd like to know what you
think about this speaker?"
2 Play the tape (2 - 3 minutes) of foreign accents (people
who are not very proficient in the target language)
and let students fill in the box or write down notes
concerning the aspects which make the people's
performances sound foreign;
3 Play the tape as many times as needed;
4 Let the students compare their notes in pairs or groups;
5 Let them discuss the answers to the following questions:
1. What seem to be the most common errors?
2. How would you describe this person's accent?
3. Would you like to have a conversation with
this speaker?
33
4. (If we have a tape of a foreigner speaking
Czech) Do you dislike hearing your language
spoken in this way?
5. Why do you think the person pronounces so
badly/in this way?
(a) The person doesn't really care about
pronunciation.
(b) The person hasn't been told how bad
their accent is.
(c) It's very difficult to pronounce this
language well.
(d) The person hasn't been told about
pronunciation.
6 Whole-class discussion
Output: Identifying words which are very important and need
to be pronounced well. The teacher gets a general idea of
how responsive the learners are likely to be to
pronunciation work, both as a group and as individuals.
An equally useful component of the "evocation stage"
is eliciting information from the learners about their
previous pronunciation and English learning experience.
Which questions we prepare depends on our needs, but we
should know if they want to improve their pronunciation,
what they think their weak and strong points are and how
much they want to improve.
Getting this kind of information may be either
included in one of the "building-awareness techniques"
mentioned above, or done separately in the form of a
questionnaire or an informal discussion. It is beneficial
for the learners themselves as well as for the teachers.
The task makes the learners consider and define their
individual goals which itself has a motivating effect.
34
The „evocation stage“ should provide us with the input
information for the realization stage including setting out
goals and criteria of further testing and assessing,
deciding on the amount of time to be devoted to
pronunciation and the form of a test.
3. 3 Realisation
The main part of the realization stage may involve
teaching pronunciation and carrying out a particular test.
As this work does not focus on teaching pronunciation
itself, we are going to present various techniques of
testing. All we can find in literature are just recommended
activities which may be used for testing rather than any
strict rules and exact instructions, so the teacher is free
in their choice and may adjust any material from literature
according to their own individual needs or create some
themselves.
Before designing a test it is important to consider
the aim. Like in any other language component, there are
two kinds of tests. The first one is a progress test, which
monitors the students‘ progress. The second one – a
proficiency test - deals with the learners‘ general level
of a language skill without reference to any course, which
is presented in the international exams.
In the practice of pronunciation testing this division
between the two kinds of tests infers the division between
atomistic and holistic testing although it does not have to
obtain in all cases. But the progress test is usually based
on what has been taught during the course and in most
coursebooks the pronunciation teaching course design has
some structure and deals with the aspects of pronunciation
in successive steps. So it is obvious that the test will
only include the points which have been covered. The
proficiency test does not take into account what has been
35
taught and is designed only according to the general level
of the learner’s English. It examines intelligibility
including all aspects of pronunciation where the main
criterion is comfortable communication between, the speaker
and the listener.
Another way of dividing tests is whether they test
reception or production. Reception deals with hearing -
identification of sounds, stress and intonation in the
atomistic testing and listening comprehension in the
holistic one. The advantage of the reception testing is
that the result can be quantified simply by counting the
correct answers. That is also the reason for their
sufficient amount in course books. More difficult is
examining production since it cannot be measured and
quantified but is entirely dependent on the assessor’s
impression. We will deal with the tests on production and
point out the rules which should be followed in order to
receive as reliable results as possible.
We will now look at the activities which may be used
for testing. They have already been briefly presented in
the theoretical part from the point of view of the holistic
and atomistic way of assessing. In this part we are going
to introduce them more thoroughly and focus more on their
advantages and disadvantages in connection with their
particular purposes.
3. 3. 1 Limited-response activities
In limited-response activities the teacher has total
control of the learner’s utterance. These activities are
usually easier to prepare as well as assess and their
results are comparable with other learners.
Test 1 - Words in isolation
This test is aimed at examine the learners’ ability to
distinguish between sounds. It is advisable to start
36
pronunciation learning based on minimal pairs and correct
production of sounds.
This is just an example which may focus on a limited
number of sounds. Students read a list of words containing
minimal pairs. The assessment form shows which sound is to
be assessed and the assessor puts a tick or a cross next to
each word. The test can be done in one lesson in class, or,
what is more reliable, we can record the learners‘
performances and let them be assessed by more people or by
someone else, not the teacher of the class who has become
familiar with the students‘ accents.
Assessment form7
think thingpen pan
sick thick
ran rang
vest west
Test 2 - Pictures
For pupils who cannot read, we can use pictures for
identifying objects, where there in each picture there is a
possible source of confusion. For example, a picture of a
pen can cause a confusion with the word pan. Which pictures
we choose is dependent on the words which our learners
know. Since reading is not involved, the advantageous
feature of the activity is avoiding any “spelling
pronunciations” and thus its suitability for also dyslexic
students.
Test 3 - Words in sentences 8
7 Inspired by UR’s publication A Language Teaching (1997). 8 Inspired by HEATON’s Classroom Testing (1990).
37
Reading aloud sentences containing the problematic
sounds is preferable because they provide context and
reading will be more natural.
There were several people standing in the
hole/hall.Are you going to sail/sell your boat today?Do you like this sport/spot?
Test 4 - Reading Aloud
Reading aloud is one of the most common ways of
examining pronunciation. On the one hand, it provides
control, on the other hand there is still a big difference
between reading and normal conversation, so we have to
count on the fact that it is not spontaneous. Learners are
often confused by the spelling and as they see pauses
between words they find it difficult to link, so we should
be aware that in real conversation the performance would be
better. Nevertheless this kind of test may give us an
approximate level of the learners‘ pronunciation. It is
important to choose the right material to be read. The best
is to prepare something that may actually be read aloud. It
can be a letter, instructions or a dialogue.
The following example is a short letter or a message.
The student will read only the left side, but the assessor
may get also the points which should be assessed. It is
better to divide the text into more lines and assess only
one or two points per line, because it is impossible to
notice everything and give such a thorough feedback when
there are many students in the class. The result of such a
kind of assessment would be too chaotic, because in case of
more assessors, each of them will point out and omit
something else and the results received by this kind of
38
assessment do not enable us to compare students‘
performances.
TextIt’s my girlfriend’s birthday
next week and I want to give her
a surprise with a special present
or a special evening somewhere.
What should I do?
Examiner’s notes´gз:l,frend; bз:θdeı
wi:k;
wið ə;
or ə; i:vnıŋ;
falling intonation
The above mentioned activities were based on a limited
response. Considering their advantages, they are easy to
prepare and their assessment enables comparison between
students, as the text is the same for everyone.
Further, we are going to present free-response
activities in which there is not one correct answer but the
student is free in the choice of his or her answer.
3. 3. 2 Free-response activitiesFree-response activities do not test pronunciation
during reading, but the student’s own utterance. Student is
free to choose a formulation for his or her answer. On the
one hand, the advantage of this method compared to limited
response is avoiding “spelling pronunciations” and the
student’s speech generally sounds more natural. On the
other hand, we should take into account the fact that
nervousness of a learner may affect his or her performance
and for this reason a lot of practice is required
beforehand.
Test 5 - Re-telling stories
As the title tells us, this test involves re-telling a
story or an incident. A student reads the text silently and
then is supposed to re-tell it in their own words. If the
text is chosen appropriately, the student should use such
39
elements of pronunciation which we want to examine. The
reading comprehension should not affect the result in case
of inaccuracy on the summary.
Test 6 - Using pictures
Pictures are useful for testing speaking skills
because their description does not involve any reading,
which could affect the performance. There are various ways
how we can use them.
For a description students should have time to see
them beforehand for a few minutes to prepare and order
their thoughts.
The international Cambridge examinations include
pictures for comparison in their speaking activity. The
purpose of this activity is to find out the differences or
similarities and their common topic. This activity can be
done comfortably as a part of a normal lesson better than a
part of a test and it also enables more than one
participants to work together.
Another variation of using pictures is the sequences
of pictures which is similar to telling a story. Students
are to look for the links between the pictures rather than
describe pictures in isolation.
For either individual or pair work there are pictures
with speech bubbles. In this activity students are supposed
to guess what the people are saying.
Almost every coursebook includes a „giving directions“
task and maps can be used for this purpose as well.
3. 3. 2. 3 Testing intelligibilitySo far, we have mentioned two levels of pronunciation
testing. The first dealt with limited-response activities
which are suitable for testing all aspects of
pronunciation, but the use of stress and intonation is not
40
as natural as in normal conversation. The second type
provided us with activities more suitable for testing
stress and intonation. The third type is testing
pronunciation in interaction. This involves production as
well as comprehension and the main criterion is the
intelligibility of the learner’s speech. It is not so
detail-focused as the previous two but involves all aspects
of pronunciation – segmental and suprasegmental.
The activities are mostly interview-like, either it is
the interview with the examiner, or with the peers. For
this technique, pictures, role-plays or discussion may be
used, but so that the speaker’s performance is as
spontaneous as possible, it is importnat to choose a topic
which the learner is familiar with.
In testing speaking, pronunciation is just one of
several aspects which are evaluated and which make the
speech comprehensible. So, apart from accent, there is
grammar, vocabulary, fluency and comprehension which make
the picture of the student‘s performance complete.
This kind of testing is demanding as far as the
assessment is concerned. As there are many aspects
involved, setting out clearly defined criteria is very
important, moreover, one assessor only is not enough for
achieving relatively reliable results. But this technique
enables the learner to „show their best“ in pronunciation
and so the results could be considered to be the most valid
and reliable.
3. 4 Reflexion
Having presented variations of pronunciation tests, we
are coming to their final part which is the feedback. This
part will deal with assessment and the task of the
assessor. As has already been mentioned, giving feedback on
pronunciation is a very difficult task because it is
41
impression-based. Another stumbling block is the complexity
of this language component, so what should be taken in mind
first is the aim of a particular test and on its basis
setting out concrete criteria, otherwise the assessor will
get overloaded with much information at the same time.
Assessing the limited-response tests is probably the
easiest. As it focuses on a limited number of items, we may
just tick or cross the correct or wrong production of
sounds.
Free-response tests are more difficult to assess and
so it is useful to record the learners’ performances and
listen to them more times. In the atomistic assessment
there will be only a few criteria, because it is very time-
consuming to cover everything. In the holistic assessment
the criteria are differently defined as they include both
speaking and listening.
Description of criterial levels
Setting out criteria of assessment should refer to the
design of the test. Learners are likely to make more
pronunciation errors when reading than when speaking
spontaneously, but there are many students who would become
self-conscious and shy to speak on a topic, so it is
recommended to use both methods.
Hughes (1989) describes the criterial levels of
obtaining valid and reliable scoring as follows:
Scoring will be valid and reliable only if:
Clearly recognisable and appropriate descriptions
of criterial levels are written and scorers are
trained to use them.
Irrelevant features of performance are ignored.
There is more than one scorer for each
42
performance. (p. 110)
The first point to consider before setting out other
criteria is the standard according to which we are going to
judge the learners’ performances. Is it the native-speaker
standard? It could be, but we consider it to be a
discouraging goal for our learners as this is almost
impossible for them to achieve. Moreover, what we intend to
suggest, are the possible ways of testing and assessing
pronunciation in our environment which entails the
prevalence of Czech teachers of English over native-
speakers, so it might seem quite daring of the Czech
teachers to give our learners feedback in the way of
comparison with native-speaker. This implies that the
criteria of marking should be set out so that it is
possible for the learners to obtain the highest score which
can correspond with comfortable intelligibility.
From the didactic point of view, the progress should
be appreciated more than the achievement itself, as hard
work and willingness to improve is more important in life
than having just a good starting level – i. e. good
phonetic ability.
As far as the approach is concerned, the descriptions
may be atomistic or holistic. In order to reach reliable
scoring we may use both methods as a check. The atomistic
approach usually checks only a limited number of items,
e.g. production of particular sounds, stress, linking and
the result is either „correct, or „wrong“, whereas the
holistic focuses on the quality rather than the total of
correct answers. In the theoretical part, we have already
mentioned the scale in Cambridge examinations and now we
will look at another example – the American Foreign Service
Institute interview procedure. Two testers are required to
assess the candidates holistically and rate them on a six-
43
point scale for each of the following: accent, grammar,
vocabulary, fluency and comprehension. Both results then
should agree and as Hughes (1989) mentions there is a high
level of agreement between holistic and atomistic scoring.
As an example, we present the six-point scale for accent:
1. Pronunciation frequently unintelligible.
2. Frequent gross errors and a very heavy accent
make understanding difficult, require frequent
repetition.
3. „Foreign accent“ requires concentrated listening,
and mispronunciations lead to occasional
misunderstanding and apparent errors in grammar
and vocabulary.
4. Marked „foreign accent“ and occasional
mispronunciations which do not interfere with
understanding.
5. No conspicuous mispronunciations, but would not be
taken for a native speaker.
6. Native pronunciation, with no trace of „foreign
accent“. (Hughes, 1989, p. 111)
We may notice that the criteria above refer mainly to
how difficult the speaker is not only to be understood, but
also to listen to. It is not an easy task to classify a
performance into one of the groups and what makes this kind
of assessment even more demanding is the number of
assessors needed. But the issue of the scorers is going to
be tackled later on.
Irrelevant features of performance
In assessing pronunciation it is important to focus
only on such aspects or items of pronunciation which have
been decided to be tested, otherwise the results will not
44
be reliable, but chaotic. This is important to be stressed
particularly in pronunciation testing, as it is difficult
to ignore other errors in a performance. An example has
already been mentioned in Test 5 – Re-telling stories where
the attention should be paid to pronunciation, not to wrong
comprehension. And the same applies to the test of
distinction between the sounds: if [θ] and [ð] is tested,
the result should not be affected by wrong stress, for
example.
The scorer
Unlike assessing other language components, assessing
pronunciation has its unique requirements and demands on
the scorers.
The first problem is, whether the class teacher is the
best assessor. No publication or article being referred to
in this work took the view that the class teacher is
suitable for assessing their learners. It is difficult for
the teacher to fulfil this requirement, but still, there
are some possibilities.
We believe that, if necessary, the class teacher is
able to assess according to the atomistic criteria, meaning
that they are able to distinguish betweent the correct and
wrong production of sounds, stress and intonation. For that
reason, if necessary, the class teacher may be helpful. In
order to get more reliable results it is better to ask a
colleague from the school to assess for checking.
The holistic assessment is more complicated. Here, one
teacher is never reliable, if two have different opinions,
the result is also pointless, because we can count only the
points of agreement between both assessors and the rest
does not give us any information which implies that three
assessors are needed to provide some level of reliability.
45
To make the method even more difficult, Kenworthy
(1987, p. 20) claims that all English teachers themselves
are not suitable judges, as they have a great exposure to
non-native accents and so they have developed special
skills as listeners which make them atypical listeners and
so unsuitable. „The ideal judges are listeners who have not
had an abnormal amount of exposure to non-native speech nor
any previous contact with the speakers being assessed. […]
Judgements by teachers of English are of limited value.“
(Kenworthy, 1987, p. 20). At least the good news is
according to Kenworthy that non-native listeners can be
used as judges, e. g. other learners of English in the
class. We think that this requirement is very difficult to
fulfil in our school environment, so we should at least
stick to the rule of three scorers.
46
4 Research part
On the basis of the theoretical part a qualitative
research has been carried out in order to consider the
importance of pronunciation testing in EFL classes.
Our research tried to implement the three previously
mentioned stages: evocation, realisation and reflexion into
practice.
As theory often differs from practice, we were not
able to fulfil all the requirements, rules and
recommendations, but followed the opinion expressed at the
beginning of the practical part that any methods and
techniques are just recommended and are open to any
necessary adjustments in order to be administered according
to our individual needs and conditions which were limited.
Our limitations lay in the time possibilities and the
number of students at our disposal. The whole procedure was
carried out over a span of four months including all the
mentioned stages.
The whole procedure consisted of several parts. The
first part dealt with receiving and analysing the input
information using the „building-awareness activity“,
questionnaires and pre-research. The second part involved
the research itself including the first and the second
recording, pronunciation practice, and subsequent analysis
of the results. The third part dealt with assessment and
analysing the results of the three assessors.
There were sixteen students involved in the research
so we are aware of the fact that such a limited number does
not allow us to fully categorize our findings as far as the
progress is concerned.
For the assessment there were three assessors – non-
native teachers of English. One can argue that the choice
of non-native English teachers, who may be familiar with
47
foreign accent as they have a large exposure to it in their
own classes, will lead to unreliable results. There is a
point in this argument, however, for assessing in our
English classes our conditions will hardly be much
different, so our way of testing pronunciation should
correspond to the possibilities in our classes.
Our research will analyse two aspects of our testing.
The first is the feedback on the two performances of each
student including the results of the atomistic and holistic
testing. The second aspect to be analysed are the results
of the three assessors who were involved. On the basis of
our results we will infer the rate of agreement separately
in atomistic and holistic part.
4. 1 Plan and goals
Firstly, we will deal with the methods and results of
our research. In my classes, having introduced the new item
in our lessons – focused pronunciation learning and
consequent recording - I found out that most students
welcomed it as they found it challenging and only few
remained indifferent. Consequently, their performances
obviously reflected their attitude and motivation.
The second point to be discussed in this work deals
with the issue of the assessor. We will try to point out
what the concrete problems in assessing are, based on our
experience, and how much our assessors‘ results differ from
one another.
4. 2 Methods
48
At the beginning, it is important to mention that we
have been restricted only to what we had at our disposal.
The first item to be mentioned is the time which limited us
as far as both teaching, and recording are concerned. The
next important item is the equipment and space. We did not
have any recording studio, so the sound was not always
perfect. Nevertheless, we dare say that we did not have any
better or worse conditions than in most schools, so in
practice we can hardly count on having much different
conditions.
4. 3 Evocation and pre-research
49
The evocation stage consisted of several steps. Most
of the students had not had any experience with being
taught pronunciation and they seemed quite unfamiliar with
this language component. So the first task was to make an
attempt to motivate them to learn.
In my classes we used the building awareness activity
– questionnaire - presented in one of the previous
chapters. It can be discussed in class informally, learners
can fill it in and compare in pairs or groups before the
whole-class discussion. It is not necessary to carry out
the discussion in English, since learners should not be
restricted only to their own level of English, but the aim
for the learners is to become aware of the meaningfulness
of pronunciation learning. The questions in the activity
fall into two groups: the first part is aimed to find out
about the learner‘s own experience and the following set of
questions is aimed to stress the importance or need of
pronunciation. It is difficult to make a concrete
conclusion out of an informal discussion but I found it
contributing to show the pronunciation issue also from the
point of view of the listener. When imagining or evoking a
situation with a foreigner speaking Czech with a very
strong foreign accent, students agreed on the fact that it
may make the conversation difficult and very demanding. To
conclude our discussion, we reached the view that learning
pronunciation is as important for speaking as for listening
and that at least, basic knowledge of the English
pronunciation principles may improve also our listening
skills.
Having introduced the learners to our general goals
and made them think of the possible benefits of
pronunciation learning for themselves, we moved on to
gather information about our students concerning the
factors affecting pronunciation. We have decided again on
50
the questionnaire form eliciting information about the age
and the length of learning English, the current or any
previous exposure to English, the actual experience with
learning pronunciation and their personal goals concerning
personal opinions of their strong points and weak points
which they would like to practise. We are going to describe
the input information which we consider to be relevant to
get the overview of the age and the possible influence of
other factors.
Level and age
The students differed in the level of English as well
as in the age at which they began to learn English.
As far as the level of English is concerned, students
belonged to three groups: A2, B1 and B2. The age at which
they began learning English ranged from 6 to 50, so the
whole group was very heterogeneous. Seven out of sixteen
students began learning English in adulthood and the rest –
nine students – before.
Exposure to English
None of the students reported using English out of the
classroom or having been to an English-speaking country for
more than a few weeks, so this factor can be considered as
insignificant.
Native language
The native-language factor will be mentioned later on
in connection with the problematic sounds. All students
were Czech and therefore in most cases the problems with
production of sounds, stress and intonation were similar
within the whole tested group.
51
Information on the learners‘ phonetic ability,
attitude and motivation is difficult to describe. But we
are aware of the fact that these factors are very
imprortant and that is also why we used the evocation phase
before the research itself.
Pre-research
Prior to starting the main research part, a pre-
research had been carried out in order to gather
information of each learners’ pronunciation and on the
basis of this a decision was made on setting out the goal
and the criteria for our further steps.
The pre-research part was a reading-aloud task. The
learners were given a text to read. This was a short
dialogue. They were given time to read the text silently
before the recording.
A Are you working at the moment?B Yes, I’ve got a job with a record company called the Mad Cat.A Really? Is that normal working hours?B Yeah, nine to five.A Do you share an office with other people?B Uhuh, there are three of us.A And do you get on well?B Yes, we have a good working relationship.(Natural English. Pre-intermediate. OUP9)
The recordings were given to three assessors, who were
asked to write down their first impression on the
performances in an unstructured way. In the pre-research no
strict criteria were given as their setting out was
supposed to be the result of the procedure.
Here is the example of one assessment given by one
assessor:
Student 14- links quite well (apart from ‘three of us’)- no weak forms- recognizable w, velar n; recognizable difference in there vs three
9 http://fds.oup.com/www.oup.com/word/elt/products/nepre_ts11.doc?cc=cz [online; quoted 16. 2. 2007]
52
- schwa – short, long – not clear- mad – æ is not open enough- no aspiration
To analyse the data received by the three assessors, I
used a table in which all the mentioned strong and weak
points were written down. The pluses stand for the aspects
which were considered to be pronounced correctly and the
minuses for those considered wrong. Further on, those
points on which at least two assessors agreed were counted
and the results were analysed.
Pre-research: results
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
S11
S12
S13
S14
S15
Linking + - -
- - -
+ + +
+ - +
+ + +
+ + +
- + -
+ - -
- - -
- + +
- - -
- - -
+ - -
- - -
- - -
[w], [v] + + - - + + + + + + - - + + + + + + - + + - - - -
- - - - - - -
[ł] - - + - - - - - + - - - + - -
[ŋ] + - - - + + - + + - + + + -
+ - - + - + + - - - - + -
++ - + - -
[θ] + - + - - + + + -
+ + - - + - - + - - - - - - - - -
[ð] + - + - - + + -
++ - + + - - - - + -
+ - - - - - - -
[ә] + - + + + - - - - - - - -
[З:] + - - + - + + + - + - - - - - -
[æ] - - - - - - - - - - - + - - - + - - - - - - - - -
Aspiration - - - - + - + - - - +
Aspect of pronunciation
Number of students who pronounced wrongly
Number of students who pronounced correctly
The order of the most problematic areas
Linking - 10 + 5 1[w], [v] - 7 + 8 3[ł] - 0 + 0 9[ŋ] - 5 + 6 5
53
[θ] - 6 + 2 4[ð] - 4 + 4 6[ә] - 1 + 0 8[З:] - 2 + 1 7[æ] - 9 + 0 2Aspiration - 0 + 0 9
According to the table the most problematic area for
the students was considered to be linking, production of
[æ] and [w], [v]. But the number of other wrongly
pronounced sounds was also not insignificant. Therefore,
the conclusion and the goal of our further steps was
decided to be linking on the first place and practising
some of the problematic sounds using exercises on minimal
pairs.
4. 4 RealisationAccording to the results of the pre-research, we have
designed a test consisting of several parts. The first two
parts applied the atomistic approach, the second three
parts holistic. The atomistic method of testing examined
the production of problematic sounds in minimal pairs and
linking in short phrases. All the students‘ performances
were recorded and given to three assessors to be evaluated.
In the holistic testing we used a reading-aloud
activity. Students were given time to read the test
silently before the recording.
Test on Pronunciation 1
sick thickfree threeden thenran rangsink singvest westvet wetpen panmen man
54
drink a cup of teaput it onAre you in the same place?Don’t crash into anything!read a booka big umbrellagoing outThere isn’t any doctor.I couldn’t do anything about that.The story is very interesting.
I am writing to ask for information about your language courses. I am especially interested in an intensive course of two or three weeks. I am thirty-one and I work in the library at Milan University. I can read English quite well but I need to improve my listening and speaking.
I have looked at your website, but there is no information about intensive courses next summer. Could you please send me information about dates and prices? I would also like some information about accommodation. If possible, I would like to stay with a family. My wife is going to visit me for a weekend when I am at the school. Could she stay with me in the same family?
Consequently, each of them was recorded reading aloud
all the three parts. The recordings were given to three
assessors to be evaluated and the results from the first
recording were given to students with comments on their
strong and weak points and suggestions for their further
pronunciation practice.
After three months of practice, students were
recorded again reading the same format of the test which
examined the same pronunciation points, just different
words for minimal pairs, phrases for linking and text were
chosen. The whole procedure was aimed to be the same and so
were the criteria of assessing. The test was supposed to
find out about the students‘ progress.
Test on pronunciation 2
veil whalevent wentthink sinkday theybeg bagsend sandsun sung
55
sin singclosing clothing
never againI saw it.Why am I leaving?may askHe’s quite old.An American car.Good evening!for everAre you enjoying it here?The shop isn’t open yet.too expensiveMind your own business.Thursday evening
For me the first good thing about the weekend is that I don’t have to go to work. I like my job, but I have to spend all day inside, in an office, and I’m a person who loves being outside. Another good thing is that I don’t have to get up early. During the week I have to get up at half past six every day. It’s not too bad in the summer but I hate it in the winter when it’s dark in the morning. But above all, I like the weekend because I have time to do all the things I really enjoy doing, like listening to music, reading, or going out with friends.
4. 4. 1 Atomistic testing resultsThe atomistic testing results consist of the
minimal-pair part and the part of linking. Each examined
point was evaluated by the three assessors who either put a
tick or a cross next to each point, according to whether
they considered the points to be rather correct or wrong.
We use the word „rather“ because in many cases it was
difficult to decide on the correctness as not all the
sounds were produced absolutely right or wrong. Therefore,
the assessors‘ evaluation reflected the quality of the
examined point which in many cases was not perfectly clear.
Minimal pairs
In testing minimal pairs, students were given
eighteen words – nine minimal pairs, examining the
students‘ problematic sounds. Such sounds were those, which
had been most often mispronounced in the pre-research: [v],
[w]; [s], [θ]; [d], [ð]; [e], [æ]; [n], [ŋ]. The table
below presents the results of the two recordings. In the
second and the fourth column, there is the total of correct
56
answers, i. e. answers which were ticked (considered
correct) by at least two assessors. The next columns show
each student’s percentage obtained in the test and the last
two columns refer to the differences between the first and
the second recording.
Student 1st recordingTotal: 18
% 2nd recordingTotal: 18
% Difference in correct answers
Difference (%)
A2/1 17 94.4 18 100 +1 +5,6A2/2 12 66.7 13 72,2 +1 +5,5A2/3 14 77.8 15 83,3 +1 +5,5
A2/16 11 61.1 15 83,3 +4 +22,2A2/9 17 94.4 15 83,3 -2 -11,1
B1/10 10 55.6 15 83,3 +5 +27,7B1/11 16 88.9 16 88,9 0 0B1/13 13 72.2 18 100 +5 +27,7B1/14 12 66.7 14 77,8 +2 +11,1B1/15 14 77.8 12 66,7 -2 -11,1B1/18 8 44.4 12 66,7 +4 +22,3B2/6 17 94.4 18 100 +1 +5,6B2/8 12 66.7 16 88,9 +4 +22,2
B2/17 13 72.2 17 94,4 +4 +22,2
Over the three months of practice, all students
apart from three improved by 16% on average. Nevertheless,
the assessors reported that in spite of such improvement in
this part, they had not noticed such significant progress
in the production of sounds in holistic part.
Linking
The test on linking was evaluated in the same way as
the test on minimal pairs. Students were given about
thirteen phrases in which fifteen points on linking were
tested. The assessors ticked or crossed all the points
according to their impression.
57
Student 1st recordingTotal: 15
% 2nd recordingTotal: 15
% Difference in correct answers
Difference (%)
A2/1 5 33.3 12 80 +7 +46.7A2/2 2 13.3 10 66.7 +8 +53.4A2/3 10 66.7 13 86.7 +3 +20
A2/16 4 26.7 6 40 +2 +13.3A2/9 0 0 6 40 +6 +40
B1/10 9 60 14 93.3 +5 +33.3B1/11 7 46.7 10 66.7 +3 +20B1/13 7 46.7 13 86.7 +6 +40B1/14 3 20 10 66.7 +7 +46.7B1/15 3 20 10 66.7 +7 +46.7B1/18 2 13.3 5 33.3 +3 +20B2/6 15 100 14 93.3 -1 -6.7B2/8 4 26.7 7 46.7 +3 +20
B2/17 4 26.7 12 80 +8 +53.3
As the result chart shows us, all students apart
from one improved significantly. The progress was by about
35% on average.
Conclusion on atomistic testing
The results obtained show improvement in both parts
of atomistic testing in most students. The improvement in
linking was more significant than in minimal pairs. This
may be caused by the fact that minimal pairs containing
problematic sounds are more difficult for students to learn
as they are often completely new to students, whereas in
linking students are supposed to use what they already can
just in a different way. The rule for linking is quite
simple for them and all they need is just concentrate on
its application.
4. 4. 2 Holistic testing resultsThe task for students in the holistic test was
reading aloud a short text.
58
The holistic assessment was divided into five parts:
overall impression, comprehensibility, production of
sounds, stress and intonation. All of these five parts were
evaluated according to a five-point scale in which each
point was defined in the assessor‘ sheet. The assessors
were supposed to use the five-point scale for each part
separately in accordance with their impression of the whole
performance.
Overall impression
The overall impression examined how natural or
difficult to understand the student’s performance was. The
scale for the assessors was the same as in the oral part10
of the Cambridge examination:
How does the performance sound to you?
1) pronunciation is heavily influenced by L1 features and
at times be difficult to understand
2) pronunciation is generally intelligible, but L1 features
may put a strain on the listener
3) although pronunciation is easily understood, L1 features
may be intrusive
4) L1 accent may be evident but does not affect the clarity
of the message
5) pronunciation is easily understood and prosodic features
are used effectively; many features, including pausing and
hesitation, are „native-like“
The assessors‘ task was to evaluate this part
without looking at the text which the students were
reading. In this part the marks of all assessors were
included in the results regardless of whether they differed
10 Paper 3: Speaking. Cambridge Common Scale for Speaking [online; quoted 25. 1. 2007] http://www.cambridgeesol.org/support/dloads/ket/KET_HB_sampleS.pdf
59
or not. The average mark was counted out of the three. The
table below shows the average mark of each performance from
the first and the second recording as well as the
difference between them. „Plus“ stands for improvement,
„minus“ for worse performance.
StudentOverall
impression 1
Overall impression
2Difference
A2/1 2.7 3.3 +0.6A2/2 2 3 +1A2/3 4 3.7 -0.3A2/4 3.3 - -
A2/16 2.3 2.7 +0.4A2/9 2.3 2 -0.3
B1/10 2.7 2.7 0B1/11 2.3 2.3 0B1/13 2 2.7 +0.7B1/14 2 2.7 +0.7B1/15 2.3 2.7 +0.4B1/18 1.7 2.3 +0.6B2/6 4.7 5 +0.3B2/7 3.3 - -B2/8 3.3 4 +0.7
B2/17 3.3 3.7 +0.4
The results showed slight improvement within one
level in most students and one student obtained the highest
mark – native-like performance.
Comprehensibility
The comprehensibility part tested how much of the
performance was understandable for the assessors. In this
part the assessment was based again only on hearing without
reading the text. The scale for assessment was given as
follows:
How much do you understand?
1) not most of the performance
2) not quite many words/phrases
3) not a few words/phrases
4) everything with difficulties
60
5) everything
Student Comprehensibility 1
Comprehensibility 2 Difference
A2/1 4.3 4.7 +0.4A2/2 3.7 4.3 +0.6A2/3 5 5 0A2/4 3.7 - -
A2/16 4.3 4.7 +0.4A2/9 3.3 4 +0.7
B1/10 3.7 4.3 +0.6B1/11 3.7 4 +0.3B1/13 4.3 4 -0.3B1/14 3.7 3.7 0B1/15 4 4.3 +0.3B1/18 2.7 4.3 +1.6B2/6 5 5 0B2/7 5 - -B2/8 4.7 5 +0.3
B2/17 5 5 0
Also in this part there was a slight improvement in
most students and the comprehensibility compared to the
overall impression has brought much higher scores for more
students which infers that despite a lower score in the
overall impression, the comprehensibility can still be
considered perfect.
Production of sounds, stress and intonation
The last three parts were assessed at the same time
and the assessors were asked to mark their overall
impression of the production of sounds, stress and
intonation according to the following scale:
1) Bad
2) Quite good
3) Good
4) Very good
5) Excellent
61
StudentProduction of sounds
1
Production of sounds
2Difference
A2/1 2.7 3.7 +1A2/2 2.7 3 +0.3A2/3 4.3 4 -0.3A2/4 3 - -
A2/16 2 3 +1A2/9 2.7 2.7 0
B1/10 2.7 3 +0.3B1/11 2.7 2.3 -0.4B1/13 3.3 3 -0.3B1/14 2 2.7 +0.7B1/15 2.3 3 +0.7B1/18 2 3 1B2/6 5 4.7 -0.3A2/7 2.3 - -B2/8 3.3 3.7 +0.4
B2/17 3.7 4.3 +0.6
As the production of sounds was more focused than
the overall impression and the comprehensibility, the marks
for this aspect were usually lower. But again, the results
showed slight improvement in most cases.
Student Stress 1 Stress 2 Difference
A2/1 2.7 3 +0.3A2/2 2.3 3 +0.7A2/3 3.3 3 -0.3A2/4 3 - -
A2/16 1.6 2.3 +0.7A2/9 2 1.7 -0.3
B1/10 2 2.3 +0.3B1/11 1.7 2.3 +0.6B1/13 2.3 2.7 +0.4B1/14 1.7 2.3 +0.5B1/15 1.7 2.7 +1B1/18 1.3 2.3 +1B2/6 4 4.7 +0.7A2/7 3 - -B2/8 2.7 3 +0.3
B2/17 2.3 3.7 +1.4
62
The assessment of stress showed improvement in all
students apart from one. Stress compare to the production
of sounds showed worse results in most cases than in the
test on production of sounds. We suppose that the reason
may be too much concentrating on fluent reading and correct
production of sounds and linking, as these aspects had been
involved predominantly in the previous practice.
Student Intonation 1
Intonation 2 Difference
A2/1 2 2.3 +0.3A2/2 2 2 0A2/3 2.7 3.3 +0.6A2/4 3.7 - -
A2/16 1.6 2 +0.4A2/9 2 1.7 +0.3
B1/10 2 2 0B1/11 1.7 2 +0.3B1/13 1.7 2 +0.3B1/14 1.7 2 +0.3B1/15 1.7 2.7 +1B1/18 1.3 1.7 +0.4B2/6 3.3 4 +0.7A2/7 3.3 - -B2/8 2.7 2.7 0
B2/17 3 4 +1
Out of all the previous parts, the test on
intonation proved the lowest scores. The reason may be the
same as in the test on stress. Moreover, the assessors
reported that they had found this part most difficult to
assess, as it is almost impossible to concentrate only on
intonation and they usually noted down their first
impression although there may have been other factors which
have distorted the evaluation.
Student Overall impression Comprehensibility Sounds Stress Intonation Total
A2/1 +0.6 +0.4 +1 +0.3 +0.3 +2.6A2/2 +1 +0.6 +0.3 +0.7 0 +2.6A2/3 -0.3 0 -0.3 -0.3 +0.6 -0.3
63
A2/16 +0.4 +0.4 +1 +0.7 +0.4 +2.9A2/9 -0.3 +0.7 0 -0.3 +0.3 +0.4
B1/10 0 +0.6 +0.3 +0.3 0 +1.2B1/11 0 +0.3 -0.4 +0.6 +0.3 +0.8B1/13 +0.7 -0.3 -0.3 +0.4 +0.3 +0.8B1/14 +0.7 0 +0.7 +0.5 +0.3 +2.2B1/15 +0.4 +0.3 +0.7 +1 +1 +3.4B1/18 +0.6 +1.6 +1 +1 +0.4 +4.6B2/6 +0.3 0 -0.3 +0.7 +0.7 +1.8B2/8 +0.7 +0.3 +0.4 +0.3 0 +1.7
B2/17 +0.4 0 +0.6 +1.4 +1 +3.4
Conclusion on holistic testing
According to the results in this part, most students
progressed in all five parts. The average highest score was
reached in the comprehensibility part which can be
considered good news as, for communication, it is more
important than the way someone’s speech sounds.
Conclusion on both – atomistic and holistic - parts
The results of both parts did not prove any
significant link between the atomistic and the holistic
part. This may be caused by a small number of students
tested or the fact that many students were able to learn to
produce the problematic sounds and linking in words or
short phrases. Accordingly, the progress in the atomistic
test was significant. But the holistic test revealed that
many students were not able to apply the items from the
atomistic test. Some may have achieved a very high score in
the minimal pairs and linking, but still the holistic test
showed wrong pronunciation of these items.
To sum up, according to our results, the
improvements in atomistic test may be a question of short-
term practice, whereas we consider any progress in holistic
testing a long-term matter as more factors are involved
like subconscious acquisition within some time and the
attitude and motivation of the learner.
64
4. 5 ReflexionThe last stage of our research dealt with the issue of
assessing. The feedback reflecting the students‘ results
has already been mentioned in the previous chapter.
In this chapter we will explain the reasons for our
choice of the assessors in reference to the criterial
suggestions presented formerly and analyse the results of
the assessors in the atomistic and holistic parts of the
test. The question that we would like to find the answer to
is whether according to our results our choice of assessing
methods can provide us with somehow reliable, nevertheless,
impression-based results which may help us in further
teaching English pronunciation.
Firstly, we will deal with the choice of assessors.
Having followed the rule of more than one assessor for each
performance we have opted for the number of three assessors
in case of different results of the two assessors. For
practical reasons three non-native teachers of English were
chosen. According to some authors this decision may not be
the ideal one, but not everybody opposes it. We think that
it was the best option which we had at our disposal and
also the easiest option when testing pronunciation in
school. Hardly ever can we count on a sufficient number of
native speakers in our schools, therefore, in our opinion,
we have administered an assessing method which we consider
to be the most likely applied in school practice in
testing pronunciation.
The most difficult task for us to tackle was setting
out concrete criteria for the assessment and following
them. We will see later how difficult it was for the
assessors to decide whether, e. g. the particular sound was
produced correctly or wrongly and decide on the level in
holistic assessment.
65
The last point supporting reliability is closely
connected to the previous one and means that any irrelevant
features of a learner’s performance which have not been set
as criteria should be ignored.
The assessors have been instructed according to the
suggestions mentioned above and in the next chapter we will
look at their agreeable or different results of assessment.
4. 5. 1 Assessors‘ agreement in atomistic testingThe test intended for the atomistic assessment
consisted of two parts. The first was aimed to examine the
students‘ recognition between problematic sounds and their
native-language nearest equivalents and the second part
focused on linking. The assessors were instructed to put a
tick or a cross next to each examined word or phrase and to
evaluate only the relevant features. This at first
obviously concrete criterion turned out to be difficult to
fulfil. The table below presents the results of one hundred
percent agreement of the three scorers.
Atomistic TestingPart 1 - Minimal pairs
Student
Assessors‘ 100% agreement
(number of the same answers out
of 18)
Percentage of assessors‘ 100%
agreement(%)
Assessors‘ 100% agreement
(number of the same answers out
of 18)
Percentage of assessors‘ 100%
agreement(%)
1st recording 2nd recording1 15 83,3 15 83.32 12 66,7 15 83.33 15 83,3 14 77.84 14 77,7 - -
16 16 88,9 13 72.29 13 72,2 16 88.96 18 100 18 1007 14 77,7 - -8 10 55,6 17 94.4
17 11 61,1 15 83.310 9 50 14 77.811 14 77,7 14 77.8
66
13 11 61,1 14 77.814 12 66,7 12 66.715 16 88,9 15 83.318 12 66,7 14 77.8
Assessors‘ agreement in total74%
Assessors‘ agreement in total82%
As we can see the total of assessors‘ 100% agreement
is 74 % in the first recording and 82% in the second one.
Their agreement varied on the scale between 50% and 100%
which is quite a significant difference. Even though the
criteria for assessment had been explained and all the
assessors had discussed them beforehand, they reported that
what they had found the most difficult was such a
production of a sound which they felt was not completely
correct but at the same time not completely wrong and
therefore some inclined to consider it correct and some
wrong. The next problem was the correct production of the
particular sound on the one hand, but mispronunciation of a
different part of the word or phrase which sometimes led to
the impression that the word had been pronounced wrongly.
The second recording compared to the first one showed
better results in the assessors‘ agreement. According to
the assessors, this is because as most students‘
pronunciation got better, more sounds were clearer and thus
easier to assess.
Atomistic TestingPart 2 - Phrases on Linking
Student
Assessors‘ 100% agreement
(number of answers out of
15 )
Percentage of assessors‘ 100%
agreement(%)
Assessors‘ 100% agreement
(number of answers out of
15)
Percentage of assessors‘ 100%
agreement(%)
1st recording 2nd recording1 8 53,3 12 802 13 86,7 10 66.7
67
3 7 46,7 9 604 8 53,3 - -16 14 93,3 5 33.39 15 100 10 66.76 11 73,3 14 93.37 10 66,7 - -8 9 60 7 46.717 7 46,7 12 8010 4 27 11 73.311 10 66,7 14 93.313 8 53,3 12 8014 13 86,7 10 66.715 10 66,7 10 66.718 15 100 9 60
Assessors‘ agreement in total67.5%
Assessors‘ agreement in total69%
The assessors reported the same reasons for different
evaluation in this test as in the first one. The results
differed mainly because of omitted and changed words in the
phrases so that the phrase lost its sense, however, the
student used linking correctly.
When comparing the agreement in assessment between the
first and the second recording, we come to very similar
numbers,
The impression-based assessment is highly subjective
and the „either-or“ evaluation can never be completely
reliable. The „tick“ or „cross“ assessment is too
restricted to such a subjective method of assessment. In
our opinion, the impression-based assessment should not be
restricted only to tick and cross answers as it has much
wider scale.
Nevertheless, according to the results in the table,
we are convinced that at least three assessors are
necessary to balance the impression of only one.
68
4. 5. 2 Assessors‘ agreement in holistic testing
In the holistic assessment the three scorers were
supposed to evaluate the reading-aloud task according to
the five-point scale in which the mark 5 represented the
highest score. As we have already stated that the standard
for the holistic assessment should not be native-like
performance, but comfortable intelligibility, the results
varied on the scale from the lowest to the highest scores.
The assessors‘ task was to choose such a definition of the
recorded performance which represented their impression
best. The table below shows the rate of agreement of the
three assessors in the five parts of holistic testing.
Although the categories Production of sounds, Stress and
Intonation tend to belong to the atomistic part, we decided
to put them at the end of the holistic test as the task for
the assessors was to give a mark for the overall impression
of these three categories.
Holistic Testing (the figures stand for the number of students out of 16)
Overallimpression Understandability Production
of soundsStress Intonation
100% agreement 4 4 4 1 0
2/3 agreement 12 10 11 10 10
Different results 0 2 1 5 6
Apart from the reasons for different assessments of
the scorers, we have come to another factor which
influenced the holistic assessment. The assessors reported
that in spite of good production of sounds, stress and
69
intonation it was often the fluency and the speed of one’s
reading which had a disruptive effect. A lot of pausing and
hesitating in the performance often made the impression and
thus the final mark worse.
Conclusion on the agreement in assessment
As we have seen, the agreement of the assessors was
not absolute in any of the parts. This comes out of the
impression-based assessment. Except for the criteria given
to the assessors, we have to count on each assessor having
his or her subjective criteria which are included in their
assessing. We have mentioned that in the atomistic test the
choice between „tick“ or „cross“ is too limited as we found
out that it was the quality of the tested item which was
considered rather correct or wrong. If the item was
produced rather well for at least two of the three
assessors it was considered correct. Due to the differences
in the three scorers‘ assessments, we assume that three
scorers are more reliable than only one as more scorers
offer more opinions and thus balance the subjective view of
only one impression. This conclusion has its relevance to
the holistic assessment as well.
70
Conclusion
The aim of this work was to present various testing
activities and ways of their assessing.
We believe that pronunciation as one of language
components should be included in any course design.
Moreover, structured pronunciation teaching should involve
giving structured feedback to learners as it simply makes
the whole process complete for both sides.
Pronunciation testing as one of the ways of feedback
presupposes setting out goals which help the learners be
aware of what is aimed to be achieved, therefore, the
practice itself becomes more meaningful for them. In
addition, structured pronunciation teaching is beneficial
in the way that it helps the learners build the general
awareness of the system of pronunciation as an important
language and communication component.
In our opinion, setting out goals and subsequent
working on their achieving, comparing results within time
and class is a significant motivation factor itself which
may enhance the whole process.
Arranging the right conditions will enable our
students to achieve their best. We admit that many of the
testing methods which have been presented are too time
demanding to be administered in large classes, however,
mostly not more than, for example, a correction of a
written assignment. Another problem could arise with the
number of assessors. On the one hand, it is assumed that
the number of three is supporting the reliability of the
assessment, on the other hand, this is just a recommended
procedure. The only recommendation is to try to follow the
advice given and adjust the activities to the possibilities
we have at our disposal. Any feedback which is aimed to
help our learners and which is accomplished thoughtfully
71
and responsibly is better than none. To compare, we often
assess spoken or written work of our learners, which is
also subjective and impression-based, therefore,
pronunciation is not as exceptional as it might seem.
In my own experience, I have found pronunciation
activities and the whole research procedure particularly
contributing as far as not only speaking and listening
skills of my students are concerned, but also their
attitude to many other activities involving reading,
speaking and listening. At the beginning of the testing
process, the students did not seem particularly
enthusiastic, but having received their first assessment
with their strong and weak points, information on the
overall impression and comprehensibility, their further
practice began to be more focused and motivated to achieve
higher scores in the following recording. This was a
turning point in our pronunciation practice, because it is
natural that people are eager to know about how they are
proceeding and consequently, willing to improve. I dare say
that they have become slightly less „scared“ of the
listening tasks as they are aware of the existence of weak
forms, linking and other aspects of pronunciation which
used to make them confused and discouraged from
concentrating before the actual listening.
In my opinion, the good point about pronunciation
teaching in general is that it may be a great time-filler
providing enjoyable activities, great grammar and
vocabulary drills and on the top of it, it can be used as a
meaningful leisure after any concentration-demanding
activity since it does not necessarily require too much
strain and attention.
The suggestion which should be followed in all
circumstances so as to motivate the learners and progress,
is maintaining nice atmosphere, encouragment and giving a
72
chance to everyone to experience their own success and
achieve their best.
73
Appendix 1
Assessor:
Instruction on assessment Student: ______________
Atomistic Testing
Part 1 – Minimal PairsPut a tick or a cross according to the correctness in pronunciation.
sick thickfree threeden thenran rangsink singvest westvet wetpen panmen man
Part 2 – Phrases on LinkingPut a tick or a cross according to good or bad linking.
drink a cup of tea k pput it on t tAre you in the same place? wDon’t crash into anything! ʃ wread a book da big umbrella ggoing out ŋThere isn’t any doctor. tI couldn’t do anything about that. w ŋThe story is very interesting. j j
Holistic Testing
1st listening: Do not read the text.How does the performance sound to you?Scale: natural -------------------------------------------------- very difficult to understand
5 4 3 2 11) pronunciation is heavily influenced by L1 features and at times be difficult to understand2) pronunciation is generally intelligible, but L1 features may put a strain on the listener3) although pronunciation is easily understood, L1 features may be intrusive4) L1 accent may be evident but does not affect the clarity of the message5) pronunciation is easily understood and prosodic features are used effectively; many features, including pausing and hesitation, are „native-like“
2nd listening: Do not read the text.How much do you understand?1) not most of the performance2) not quite many words/phrases3) not a few words/phrases4) everything with difficulties5) everything
74
Part 3 – Reading Aloud
I am writing to ask for information about your language courses. I am especially interested in an intensive course of two or three weeks. I am thirty-one and I work in the library at Milan University. I can read English quite well but I need to improve my listening and speaking.
I have looked at your website, but there is no information about intensive courses next summer. Could you please send me information about dates and prices? I would also like some information about accommodation. If possible, I would like to stay with a family. My wife is going to visit me for a weekend when I am at the school. Could she stay with me in the same family?
3 rd listening :Write down words/phrases you found difficult to understand.……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
4th listening:Scale: (1) Bad – (2) Quite Good – (3) Good – (4) Very Good – (5) ExcellentEvaluate the student’s:
production of sounds 1 2 3 4 5stress 1 2 3 4 5intonation 1 2 3 4 5
Further Notes:
75
Appendix 2
Assessor:
Instruction on assessment Student: __________
Atomistic Testing
Part 1 – Minimal PairsPut a tick or a cross according to the correctness in pronunciation.
veil whalevent wentthink sinkday theybeg bagsend sandsun sungsin singclosing clothing
Part 2 – Phrases on LinkingPut a tick or a cross according to good or bad linking.
never again rI saw it. wWhy am I leaving? j mmay ask jHe’s quite old. tAn American car. nGood evening! dfor ever rAre you enjoying it here? w ŋThe shop isn’t open yet. p ttoo expensive wMind your own business. rThursday evening j
Holistic Testing
1st listening: Do not read the text.How does the performance sound to you?Scale: natural -------------------------------------------------- very difficult to understand
5 4 3 2 11) pronunciation is heavily influenced by L1 features and at times be difficult to understand2) pronunciation is generally intelligible, but L1 features may put a strain on the listener3) although pronunciation is easily understood, L1 features may be intrusive4) L1 accent may be evident but does not affect the clarity of the message5) pronunciation is easily understood and prosodic features are used effectively; many features, including pausing and hesitation, are „native-like“
2nd listening: Do not read the text.How much do you understand?a) everythingb) everything with difficultiesc) not a few words/phrasesd) not quite many words/phrasese) not most of the performance
76
Part 3 – Reading Aloud
For me the first good thing about the weekend is that I don’t have to go to work. I like my job, but I have to spend all day inside, in an office, and I’m a person who loves being outside. Another good thing is that I don’t have to get up early. During the week I have to get up at half past six every day. It’s not too bad in the summer but I hate it in the winter when it’s dark in the morning. But above all, I like the weekend because I have time to do all the things I really enjoy doing, like listening to music, reading, or going out with friends.
3 rd listening :Write down words/phrases you found difficult to understand.……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
4th listening:Scale: (1) Excellent – (2) Very Good – (3) Good – (4) Quite Good – (5) BadEvaluate the student’s:
production of sounds 1 2 3 4 5stress 1 2 3 4 5intonation 1 2 3 4 5
Further Notes:
77
Bibliography
ALDERSON, C. J., WALL, D. & CLAPHAIM, C. Language Test Construction and Evaluation. Cambridge: Cabridge University Press, 1996. 310 p.
BACHMAN, L. F. Fundamental considerations in language testing. Oxford: Oxford University Press, 1990. 408 p.
BACHMAN, L. F., PALMER, A. S. Language testing in practice. Oxford: Oxford University Press, 1996. 377 p.
BAKER, A. Ship or sheep. 2nd edition. Cambridge: Cambridge University Press, 1981. 168 p.
BAKER, D. Language testing. London: Edward Arnold, 1989. 110 p.
DALTON, C., SEIDLHOFER, B. Pronunciation. Oxford: Oxford University Press, 1994. 191 p.
GAIRNS, R., REDMAN, S. Natural English. Pre-intermediate. Student’s book. 1st edition. Oxford: Oxford University Press, 2005. 176 p.
GILBERT, JUDY B. Speaking Clearly. Student’s Book. 1st edition. Cambridge: Cambridge University Press, 1990. 136 p.
GIMSON, A. C., CRUTTENDEN, A. Gimson’s Pronunciation of English. Sevenoaks: Arnold, 1996. 304 p.
HANCOCK, M. English Pronunciation in Use. 1st edition. Cambridge: Cambridge University Press, 2003. 200 p.
HEATON, J. B. Classroom Testing. London: Longman, 1990. 127 p.
HEATON, J. B. Writing English Language Tests. 1st edition. London: Longman, 1988. 160 p.
HUGHES, Arthur. Testing for Language Teachers. 1st edition. Cambridge: Cambridge University Press, 1989. 172 p.
JENKINS, J. The Phonology of English as an International Language: new models, new norms, new goals. 1st publ. Oxford: Oxford University Press, 2000. 258 p.
KELLY, G. How to teach pronunciation. 1st edition. Harlow: Longman, 2000. 154 p.
KENWORTHY, J. Teaching English Pronunciation. Harlow: Longman, 1987. 164 p.
McNAMARA, T. Language testing. 1st edition. Oxford: Oxford University Press, 2000. 140. p.
78
MADSEN, S. H. Techniques in Testing. New York: Oxford University Press, 1983. 212 p.
OXENDEN, C., LATHAM-KOENIG, C., SELIGSON, P. New English File. Pre-intermediate. Student’s book. 1st edition. Oxford: Oxford University Press, 2005. 159 p.
ROACH, P. English Phonetics and Phonology: a practical course. 3rd edition.
Cambridge: Cambridge University Press, 2000. 283 p.
SCRIVENER, J. Learning Teaching. 2nd edition. Oxford: Macmillan, 2005. 431 p.
SNOW, Don. Myths and Misconceptions About Second Language Learning [online]. Washington: Center for Applied Linguistics, 1992 [quoted 5 February 2007]. Available from: <http://www.cal.org/recources/digest/myths.html>.
SOARS, J. et al. Headway Intermediate Pronunciation. Oxford: Oxford University Press, 1990. 112 p.
SZPYRA-KOZLOVSKA, J., FRANKIEWICZ, J., NOWACKA, M., STADNICKA, L. Assessing assessment methods – on the reliability of pronunciation tests in EFL [online] . Lublin: Maria Curie-Sklodowska University, Lublin, Poland: 2005. [quoted 14th February 2007]. Available from: <http://www.phon.ucl.ac.uk/home/johnm/ptlc2005/pdf/ptlcp37.pdf>
UNDERHILL, A. Sound Foundations. Oxford: Macmillan, 2005. 210 p.
UNDERHILL, N. Testing spoken language. Cambridge: Cambridge University Press, 1987. 117 p.
University of Cambridge. ESOL Examinations. Paper 3: Speaking. Cambridge Common Scale for Speaking [online, quoted 14th February 2007] Available from: http://www.cambridgeesol.org/support/dloads/ket/KET_HB_sampleS.pdf
UR, Penny. A Course in Language Teaching. Cambridge: Cambridge University Press, 1997. 375 p.
79