1 Using a mixed methods research design to deconstruct the nature of low academic performance in primary schools facing exceptional challenge within the Republic of Trinidad and Tobago Jerome De Lisle School of Education, University of the West Indies, St. Augustine Abstract MacBeath et al. (2005) defined schools facing exceptionally challenging circumstances as institutions confronted by complexity resulting from higher levels of poverty, disadvantage, and turbulence among student, parent and staff, thereby threatening school performance. In the Trinidad and Tobago primary school system, this definition was operationalized by locating poorly performing schools located in disadvantaged areas that also experienced higher numbers of economically disadvantaged students. Overall, the study employed an explanatory sequential mixed methods research design, with a quantitative Phase I project and a mixed methods Phase II project. The Phase I project was designed to identify and characterize schools facing exceptional challenge and Phase II was a multi-site case study of three selected institutions, two single-sex and urban, one co-education and rural. The Phase II project used an embedded qualitative-dominant mixed methods design, with multiple- methods. However, qualitative themes were given priority. Metainferences were of three kinds: unique to one method, corroborative, or contradictory. It was not possible to resolve all contradictory findings, pointing to findings providing different lenses on the phenomenon. Overall, the evidence suggested that mechanisms and processes associated with “exceptional challenge” as operationalized in Trinidad and Tobago were complex and context dependent. Sustainable school improvement would require not just leadership training but also integrated services and interlaced interventions targeting the multiple levels and multiple deficits. Key words Mixed Methods Schools facing challenge Poverty concentration School effectiveness and improvement National Assessments of Educational Achievement
18
Embed
Using a mixed methods research design to deconstruct the - ICSEI
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Using a mixed methods research design to deconstruct the nature of low academic
performance in primary schools facing exceptional challenge within the Republic of
Trinidad and Tobago
Jerome De Lisle
School of Education, University of the West Indies, St. Augustine
Abstract
MacBeath et al. (2005) defined schools facing exceptionally challenging circumstances as
institutions confronted by complexity resulting from higher levels of poverty, disadvantage,
and turbulence among student, parent and staff, thereby threatening school performance. In
the Trinidad and Tobago primary school system, this definition was operationalized by
locating poorly performing schools located in disadvantaged areas that also experienced
higher numbers of economically disadvantaged students. Overall, the study employed an
explanatory sequential mixed methods research design, with a quantitative Phase I project
and a mixed methods Phase II project. The Phase I project was designed to identify and
characterize schools facing exceptional challenge and Phase II was a multi-site case study of
three selected institutions, two single-sex and urban, one co-education and rural. The Phase II
project used an embedded qualitative-dominant mixed methods design, with multiple-
methods. However, qualitative themes were given priority. Metainferences were of three
kinds: unique to one method, corroborative, or contradictory. It was not possible to resolve all
contradictory findings, pointing to findings providing different lenses on the phenomenon.
Overall, the evidence suggested that mechanisms and processes associated with “exceptional
challenge” as operationalized in Trinidad and Tobago were complex and context dependent.
Sustainable school improvement would require not just leadership training but also integrated
services and interlaced interventions targeting the multiple levels and multiple deficits.
Key words
Mixed Methods
Schools facing challenge
Poverty concentration
School effectiveness and improvement
National Assessments of Educational Achievement
2
Using a mixed methods research design to deconstruct the nature of low academic
performance in primary schools facing exceptional challenge within the Republic of
Trinidad and Tobago
Rationalizing the concept of “challenging context” in the South
Schools confront a variety of external contexts, some more challenging than others
(Harris et al., 2006; Levin, 2006; Michalak, 2009). These different contexts impinge on the
very nature and purpose of schooling, the pathway to successful school improvement,
essential processes such as leadership and teaching-learning, and the attainability of national
targets set for school improvement (MacBeath et al., 2005; Jacobson, 2008). For schools
within countries of the developing world, data from international assessments suggest that
variations in contexts are much higher (Mullis, Martin, Kennedy, & Foy, 2007). These
variations are attributable to uneven national development, the absence of policies fostering
equity, and the insufficiency of essential human and physical resources.
There is a large body of work on the relationship between contexts of poverty and the
underperformance of schools in the United States, Canada, and the UK (Ylimaki, Jacobson &
Drysdale, 2007). However, the issue has not been extensively studied within small
developing states. Such a study can address the transferability of key school effectiveness
constructs contexts (Harber & Muthukrishna, 2000). Trinidad and Tobago provides an
especially interesting case for investigating the nature of challenging contexts. Although now
classified as a high-income country, the legacy of unequal development has persisted. This
inequity has been identified as a notable problem in the English speaking Caribbean and in
Trinidad and Tobago (World Bank, 1993; World Bank, 1995).
Poverty concentration in Trinidad and Tobago schools arises from both geographic
location and the operation of the education market (De Lisle et al., 2009). At the primary
school level, there are various choice options in the education market, including private,
denominational or government institutions. However, private schools are not necessarily the
preferred choice among the socially advantaged because public schools can be equally high
performing. The great majority of public primary schools are, in fact, managed by the
different denominations (MacKenzie, 1991). The existence of this complex education market
has a downside with many low performing primary schools being vilified and rejected. The
Ministry of Education has recently labelled some schools as “underperforming” based on the
number of low performing students in the high stakes Secondary Entrance Examination.
Several labels have been used to describe schools performing below expectations
within accountability systems. However, not all descriptors give adequate attention to the
issue of context. One approach might be to label such schools as “ineffective” or
“underperforming” (Van de Grift and Houtveen, 2006; Houvteen et al., 2007). Muijs et al
(2004) focused more on context in the label “schools in socioeconomically disadvantaged
areas”. The label “high poverty-low performing” (HP-LP) is perhaps one of the more useful
because it captures both comparative underperformance and context. High poverty contexts
might refer to a variety of challenging contexts, with multilingual (Pretorius & Currin, 2010)
and urban dimensions (Jacobson, 2008). Ainscow, Muijis and West (2006) used the term
“schools facing challenge” to describe schools confronted by location, history, and pupil and
parental attitudes. MacBeath et al. (2005) added an important dimension of schools at the
extreme end of the continuum of contexts, with “exceptionally challenging” defined as
conditions of poverty, disadvantage, and turbulence among student, parent and staff that is
significantly higher than that of other schools.
3
Evidence and context within the Anglophone Caribbean
Policy formulation in Caribbean education systems has traditionally suffered from the
lack of high-quality, contextualized data. This lack of “indigenous knowledge” includes
evidence from both empirical data and qualitative case studies. Crossley (2008)
recommended the use of qualitative approaches to gather contextualized evidence within
small states. However, given the possibility of significant variations across geographic
location, case studies have limited utility. High quality evidence in the Caribbean requires
both generalizable data and information to elucidate the black box of operations. One
approach might be to use multi-site case studies and multistage purposeful random sampling
(Onwuegzubie & Collins, 2007). However, prior information on the characteristics of
institutions and communities must come from a large-scale quantitative study (Teddlie & Yu,
2007).
In the past decade, large-scale achievement data have only been available from public
examinations in the Anglophone Caribbean (World Bank, 1993). When used in evaluating
system performance, however, this type of data is limited because of the strong shadow
education market (Mizala, Pilar & Urquiola, 2007). Public examinations also do not provide
standards-referenced data to answer the question, “how good is good enough?” Only in the
last decade, have monitoring assessments been introduced into some Anglophone Caribbean
islands (UNESCO, 2008). Trinidad and Tobago installed a new national assessment system in
2004 designed to better to monitor achievement standards. From 2005, both norm-referenced
and standards-referenced data on education districts, schools and students were provided.
The context and design of the study
To operationalize “exceptional challenge” among primary schools in Trinidad and
Tobago, a measure of school performance was developed using the numbers of students at
different achievement levels in Language and Mathematics national tests administered at
Standards 1 and 3. Also obtained was a measure of community economic and social
disadvantage called the Basic Needs Index and the percentage of students experiencing
economic disadvantage in the school. The socioeconomic composition of the schools was
estimated from the percentage of the school population entitled to free school meals. Schools
in the category of exceptional challenge scored below 240 on the API, reported a free school
meals index of above 90, and are situated in communities with a BNI of below 50.
The overall purpose of the study was to identify schools facing exceptional challenge
and to deconstruct the nature of underperformance in these schools. Deconstruct means to
dismantle and analyze antecedents and consequences. Thus, the study was designed to gather
deep insight into the nature of low achievement within schools facing exceptional challenge
and to elucidate factors contributing to low school underperformance. The three research
questions guiding this study were:
(1) What are the locations and institutional characteristics of schools facing
exceptional challenge? (QUAN)
(2) What is the nature of key processes and systems, such as teaching-learning,
leadership, and the parent-school-family interface? (QUAL/MIXED METHODS)
(3) How do the contextual factors impinge on organizational efficiency and academic
achievement? (MIXED METHODS)
Design of the study
The overall mixed methods design was sequential explanatory, with quantitative and
qualitative data collected in two consecutive phases within the study (Ivankova, Creswell &
Stick, 2006). The design of the overall programme is illustrated in Figure 1. Using the
framework of Morse & Niehaus (2009), the Phase I and II components were separate
Phase II Project: Mixed Methods Qualitatively-Driven Embedded Study of three school sites designed to elucidate nature of underachievement in different challenging
contexts
Integrated analysis of nature and factors involved in
challenge
research projects within an overall mixed methods research programme. Although Phase I
was a mono-method quantitative study, Phase II was a qualitatively driven embedded mixed
methods study of underachievement at each school site. The theoretical thrust for the overall
programme was deductive. The theoretical drive of the Phase I project was deductive and the
Phase II mixed methods study inductive. The projects were sequential, with priority given to
the quantitative Phase I.
Figure 1: Overall Mixed Methods Programme Design
Rationale for the mixed methods research design
The rationale for the mixed methods design in the overall programme and in the Phase
II project was explored by considering the different purposes of mixing identified by Greene,
Caracelli, and Graham (1989). As shown in Table 1, the five purposes are triangulation,
complementarity, development, initiation and expansion. The complementary framework
proposed by Bryman (2006) extends and complements this framework. In the overall
programme, the main purposes were complementarity and development, with the qualitative
component in Phase II designed to enhance and illustrate the nature of processes within the
schools identified as exceptionally challenging. The qualitative component in the Phase II
project sought new frameworks while extending the breadth and range of inquiry.
Sampling strategy
The overall sampling strategy was sequential mixed method with multilevel sampling
used for Phase II (Teddlie & Yu, 2007). In Phase I, Census data 1 was obtained for all 477
primary schools in Trinidad and Tobago. The criteria used to select the schools facing
exceptional challenge were (1) 90% of the student population receiving free school meals and
(2) and API of under 150. A short list of six schools was developed and visits were made to
each school. The final list of three schools in Phase II was chosen based on criteria that
captured the variations for composition (coeducational/single-sex) and geographic location
(urban-rural). The schools came from two educational districts, which accounted for 74 of the
159 schools classified as “under academic watch” using the API rubric. All three schools
were enrolled in the Government’s Performance Enhancement Project for low performing
schools. The two urban schools (School A-Female and School B-Male) had populations just
over 100 (118 in both schools), but in the rural school, which catered for pupils from a small
very isolated village, the school population was just 41. In Phase II, probability samples were
obtained for some surveys and purposive samples used for interviews, observations and some
surveys.
5
Table 2: Rationale and purposes for mixed method design used to investigate challenging
schools in Trinidad and Tobago Primary Purpose of Mixing – Greene, Caracelli, & Graham (1989)
Programme/
Project
Additional Purposes – Bryman (2006)
1. Complementarity -sought
elaboration, enhancement,
illustration, and clarification of
the results from one method
with the results from the other
method.
Overall
Programme MM Completeness- A more comprehensive account of
the area of inquiry was provided.
Process- quan provided an account of structures but
qual provides sense of process
Different RQs- quan and qual each answered
different research questions
Explanation qual used to help explain findings
generated by the other
Sampling – quan used to facilitate the sampling of
respondents or cases.
Context - qual provided contextual understanding
coupled with generalizable, externally valid findings
in quan.
Diversity of views – quan uncovered relationships
between variables and qual revealed meanings
among research participants
Illustration - Used qual to illustrate quan findings.
2. Development –sought to use
the results from one method to
help develop or inform the
other method.
Overall
Programme MM
3. Initiation sought the discovery
of paradox and contradiction,
new perspectives of
frameworks, the recasting of
questions or results from one
method with questions or
results from the other method
Phase II MM Unexpected results - quan or qual generates
surprisingly results only explained by other.
Enhancement –Findings of quan or qual
augmented by gathering data using qual or qual.
Different RQs- quan and qual reach answered
different research questions
4. Expansion -sought to extend
the breadth and range of inquiry
by using different methods for
different inquiry components.
Phase II MM
5. Triangulation -sought
convergence, corroboration, and
correspondence of results from
the different methods.
Phase II MM
Instrumentation
In Phase II semi-structured interview schedules were prepared for individual and
focus group interviews with leaders, teachers and parents. Both structured and unstructured
observation methods were used. For teaching and learning, the Instructional Practices
Inventory was used to assess the nature of teaching and learning (Painter & Valentine, 1996).
Video and photographs from two of the three sites supplemented field notes, which contained
descriptions of teaching-learning events. Parents and teachers in the focus group were
required to complete survey questionnaires. Teachers completed questionnaires on collective
teacher efficacy and group organizational citizenship behaviour. Questionnaires were also
administered on student engagement to both students and teachers of all target classrooms.
The student engagement questionnaires included items from existing instruments on student
engagement (OECD, 2003).
6
Table 2: Distribution of schools facing exceptional challenge in the eight education districts of Trinidad and Tobago
Educational
District
Administrative Regions Urban-
rural
Exceptionally
Challenging
Context
Ownership Multigrade Status Classified by
Ministry as
Low
Performing No. % Gov’t Denom Full Partial Multi-
grade
POS & Environs City of Port of Spain Urban 5 13.9 1 4 4 1 0 5
Diego Martin Suburban 1 5.3 0 1 1 0 0 1
San Juan /Laventille Suburban 3 7.3 2 1 2 1 0 3
Victoria City San Fernando Urban 4 14.3 1 3 1 2 1 2
Princes Town Rural 4 3.0 0 4 1 2 1 3
Penal/Debe Rural 1 3.6 0 1 0 1 0 0
St. George East Arima Borough Urban 0 0.0 0 0 0 0 0 0
Tunapuna/Piarco Suburban 3 5.7 1 2 0 2 1 2
Caroni Chaguanas Borough Urban 1 4.0 0 1 1 0 0 1
Couva/Tabaquite/Talparo Rural 5 8.5 1 4 1 4 0 3
St. Patrick Point Fortin Borough Urban 2 10.0 2 0 2 0 0 1
Siparia Rural 2 6.1 0 2 2 0 0 1
North Eastern Sangre Grande Rural 7 17.1 2 5 0 3 4 5