CHAPTER 4 RESEARCH DESIGN In the previous chapters, ways of approaching how read defined from the perspective of test item specifications wer it has been examined and emphasized that, in investigating t with its relation to the latent structure of reading ability, th is on the“product”of FL reading as a result of FL reading“p Chapter 3 had described a way in which a constnlct of re defined by developing test items that elicit certain types o takers’reading comprehension. Reading“competence”was te constitutes a major part of reading“performance”, and i construct fbr the purpose of reading test item developmen although a test item is defined to be a tool which elicits a r performance should be accepted as something that all inferences and make generalizations about what sort of readin might be able to do. Furthermore, this should be conside interaction of his competence and the context rather than c holistic and content-representative. Tb continue along th the significance of specifying the components of a test i particular, in operationalizing the reading construct to be was f[耐her explored by reflecting on item diffriculty, or a quan item. The discussion had concluded in suggesting a possibili question type of a test item and its difficulty, which provid questions to the present research. 4.1Research questions Research Question 1: Is it valid to employ‘question types, as a prime compone items used in eliciting test takers, L2 reading performanc 46 東京外国語大学 博士学位論文 Doctoral thesis (Tokyo University of Foreign Studies)
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
CHAPTER 4 RESEARCH DESIGN
In the previous chapters, ways of approaching how reading ability could be
defined from the perspective of test item specifications were explored. In Chapter 2,
it has been examined and emphasized that, in investigating the nature of reading test
with its relation to the latent structure of reading ability, the scope ofthe present study
is on the“product”of FL reading as a result of FL reading“process”. Furthermore,
Chapter 3 had described a way in which a constnlct of reading ability could be
defined by developing test items that elicit certain types of reading product in test
takers’reading comprehension. Reading“competence”was termed to be a facet that
constitutes a major part of reading“performance”, and in defining the reading
construct fbr the purpose of reading test item development, it was proposed that,
although a test item is defined to be a tool which elicits a reading performance, that
performance should be accepted as something that allows the testers to draw
inferences and make generalizations about what sort of reading activities the test taker
might be able to do. Furthermore, this should be considered analytically as an
interaction of his competence and the context rather than considering it as something
holistic and content-representative. Tb continue along the same lines of apProach,
the significance of specifying the components of a test item,‘‘question types”in
particular, in operationalizing the reading construct to be tested was discussed. This
was f[耐her explored by reflecting on item diffriculty, or a quantitative aspect of a test
item. The discussion had concluded in suggesting a possibility of a link between the
question type of a test item and its difficulty, which provides the fbllowing research
questions to the present research.
4.1Research questions
Research Question 1:
Is it valid to employ‘question types, as a prime component that constructs test
items used in eliciting test takers, L2 reading performances?
46
東京外国語大学 博士学位論文 Doctoral thesis (Tokyo University of Foreign Studies)
What are the factors that constitute the L2 reading performances of leamers of
English at secondary education in Japan, when they are extracted from factor anal)rtic
studies of reading products elicited using reading test items? Would they differ
across learners with different reading abilities?
In an attempt to come up with a test item specification that effectively
operationalizes different reading performances to be tested, inspired by Negishi
(1996)and Wada(2003), the present study proposes the‘question typ♂of a test item
to be a prime component to constitute such a丘amework. At the same time, however,
because Negishi(1996)and Wada(2003)had not accommodated the interactions of
these constructing components with the latent reading structure of test takers, an
attention will be rendered to this aspect in much greater depth, as it is possible that
the prime factors could change in accordance with the test takers’reading abilities.
Research Question 2:
Is it valid to assume a certain relationship betWeen question types and item
difficulty in eliciting test takers, L2 reading performances?
Is the item difficulty of a test item, calibrated using Item Response Theory,
affected by its question type? If so, how? Wbuld this relationship differ across
learners with different reading abilities?
With an intere st in suggesting the facets of a reading te st item that would allow
the writers of test items to predeterrnine the difficulty of a test item, the present study
investigates the possibility of a link between the item diflriculty of a test item and its
question type. Attention will also be given to cases with different abilities of test
takers to see if the orders of perceived dif6culties across different question types
differ according to the different ability groups of test takers.
47
東京外国語大学 博士学位論文 Doctoral thesis (Tokyo University of Foreign Studies)
42Data Collection
4.2.1Subjects
Asample of 8301earners of English from senior high school and university in
Japan had participated in the main part of the present study. Of these,280 were
third-year high school students and 550 were first-year undergraduate students in
皿lverslty.
The maj ority of high school students had five years of English education under
the Course of Study provided by the Ministry of Education, Culture, Sports, Science
and Technology in English in a foreign language environment. They were told that
the test was administered to collect data on individual’s English proficiency. The
students had five English classes in a week;nothing was done in the classroom that
would help the students to prepare fbr the tests administered in this study.
For the university students, the circumstances were the same as high school
students except that the duration of time English was leamed was mostly six years.
All of the皿iversity students maj ored in one foreign language other than English and
were given the test early in April, immediately after they had entered university, as a
placement test fbr their English classes that were prerequisite in the university
curriculum. This was to ensure that the test takers did not have any special
knowledge of English or of any other academic field that would distort the outcome
of data collections.
There were some variations in both high school and university students’
background of how and how long English was learned(e.g. students who had
overseas experiences), however, the variation in the number of years they had spent
time abroad or the intensity of how much English they had leamed were so great that
it was not possible to come up with any generalizable criterion fbr omitting the scores.
Moreover, it could be assumed that tho se variations would be an inherent factor in
leamers’reading ability that enables them to score high on the test, so the present
author had decided to disregard such factors in the process of data collection as long
as it did not affect the distribution of scores too greatly.
48
東京外国語大学 博士学位論文 Doctoral thesis (Tokyo University of Foreign Studies)
4.2.2 Materials
The two sets of test instrument were employed in the main study.
4.2.2.1Test Set/l
Test Set A(presented in Appendix A)consists of nine passages, each passage
with three multiple-choice test items(one correct option and three distracters
provided)to be responded on the base of its comprehension. These nine passages
were selected after an item selection was done in the pilot study, providing 27 reading
test items. The features of these nine passages are as follows:
Table 4-1 The features of passages employed in Test Set A