Abstract · Web view< .001, respectively), thus providing validity of the two achievement tests in relation to school performance. The two Executive Function tests were the Tower

Evaluation of East Africa pre-primary schooling 2

Presented at Society for Research in Child Development, April 2017

An evaluation of the quality of pre-primary schooling in East Africa and its association with early primary outcomes

Frances E. Abouda*, Amina Abubakarb, Elias Kumbakumbac, Shehe Abdalla Moh'dd, Melanie Dirksa

aDepartment of Psychology, McGill University, Montreal, Canada

bPwani University and KEMRI, Kenya

cMbarara University of Science and Technology, Uganda

dState University of Zanzibar, Zanzibar

*Department of Psychology

2001 McGill College Ave., Montreal QC H3A 1G1

Telephone 514 398-6099; fax 514 398-4896

[email protected]

Acknowledgements

The research was funded by the SESEA project (Strengthening Education Systems in East Africa) with funds from Global Affairs Canada. The funders played no role in study design; in the collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the article for publication.

We would like to acknowledge the cooperation and support of personnel affiliated with the Education Ministries of Kenya, Uganda and Zanzibar, and the Madrasa Early Childhood Programme (MECP) of East Africa. Shekufeh Zonji, a program manager and researcher with wide experience, helped train assistants on the MELE measure of quality. Also we thank the staff and teachers of the pre-primary schools and primary schools, as well as the children and their parents, for providing important information.

Abstract

The purpose was to evaluate two common models of pre-primary schooling in East Africa, in terms of their quality and early primary academic benefits to children. Some 117 Madrasa pre-primaries were randomly selected from Kenya, Uganda, and Zanzibar, and 100 non-Madrasa pre-primaries feeding into the same government primary schools. Their quality was observed and rated using the 50-item Measure of Early Learning Environments (MELE), developed by an internationally sponsored team of experts and intended to be relevant for low- and middle-income countries. One year later, five randomly selected graduates from each observed pre-primary (N = 974, ages 70 to 117 months) were tested on the Early Grades Reading and Math tests, and on executive function and social problem-solving measures. The Madrasa pre-primaries were found to have higher overall quality, play, language, group activities, and program structure. There was no significant difference in first grade performance between graduates from the two programs. However, quality mattered. Based on a multilevel linear regression analysis, overall quality and some domain-specific qualities were associated with math and literacy performance. An increase in overall quality of one standard deviation led to an increased math score of 7.7% in Uganda and literacy score of 10.4% in Kenya.

Keywords: Measure of Early Learning Environments (MELE); early childhood development; Kenya; Uganda; Zanzibar

1. Introduction

Too many children around the world do not complete secondary or even primary schooling, and those who do often lack expected proficiencies in literacy and numeracy (UNESCO, 2017). Consequently, there have been concerted efforts by the international community to support improvements in education, especially access to pre-primary education and a smooth transition to primary school (e.g. Global Partnership for Education, 2016). It is expected that attending at least one year of pre-primary education will give children the kind of experience they need to succeed in primary school (Earle et al., 2018). For this reason, one of the Sustainable Development Goals (SDG) related to education is to have all children receive at least a year of free quality pre-primary education (UNESCO, 2015). Pre-primary education here refers to center-based organized instruction designed to introduce children to a school-type environment in the year ahead of primary schooling. In contrast, the commonly-used term preschool broadly refers to organized group care between the ages of 3 and 6 years. The current study compared two models of pre-primary education in East Africa by evaluating the quality of their teaching/learning program, using a quality measure newly developed for low- and middle-income countries (LMIC), and the academic, cognitive and social benefits to children one year later at the end of first grade.

Uganda, Zanzibar in Tanzania, and Kenya are low- and low-middle-income countries with Human Development Indices of 0.493, 0.531 and 0.555, respectively (United Nations Development Program, 2016). During the study, only Kenya had a widespread system of free pre-primary education where 77% of children 3-6 years participated; Zanzibar and Uganda had less than 20% of eligible children attending public pre-primaries with a similar number attending private faith-based programs (UNICEF, 2017). Yet, the three countries, along with others in East Africa, are working to align their education curricula at the pre-primary level (East African Community Secretariat 2014). The quality of these pre-primaries and the impact of quality on early grades literacy and math will help determine how close we are to meeting the SDG goals.

1.1 Background Literature

Until now, increasing access was the main goal of national and international organizations because of research showing that attending pre-primary led to cognitive gains with an effect size of 0.64 (Rao et al., 2017). Most of these studies assessed children at the end of a pre-primary program and found that they performed better on tests of school readiness than children who did not attend pre-primary (e.g., Aboud, 2006; Rao et al., 2012; Rao et al., 2012; see review by Nores & Barnett, 2010). There were also attempts to identify or directly implement improved pre-primary programs; as expected, improved or developmentally appropriate programs conferred greater benefits on children (Aboud et al., 2016; Moore et al., 2008; Mwaura et al., 2008; Opel et al., 2009; Opel et al., 2012). With attention now being paid to the concept of “quality pre-primary” we used Burchinal’s (2018) definition of early education quality, entailing sensitive and responsive interaction, setting reasonable limits to acceptable behavior, intentional teaching of age-appropriate skills and scaffolding of children’s learning, using a curriculum for instruction, and positive links with families.

1.2 Measuring Pre-primary Quality

In order to go beyond informed decisions about what makes a pre-primary have higher quality, a measure of pre-primary quality appropriate for low- and middle-income countries is needed. Our first question was: Would a newly developed measure of pre-primary quality be feasible in East Africa and reveal program differences? A metric of quality is important for a number of reasons. Although “developmentally appropriate” learning activities are generally known by educators and child developmentalists, a metric would permit a finer-grained analysis. A metric might also be used to describe features of a high quality program that in turn would be used to guide improvement. Finally, a measure of quality could be validated through associations with a direct assessment of children’s performance to confirm whether those qualities are indeed worth striving for.

Measures of quality developed for pre-primaries in LMIC have not been available until recently, so researchers have used ones developed in the United States and United Kingdom, sometimes with extensive modifications to accommodate a low-resource setting. When used in some Latin American countries such as Brazil, Chile, Colombia, Ecuador, and Peru, the Teacher Instructional Practices and Processes System (TIPPS; Seidman et al., 2014) and the Classroom Assessment Scoring System (CLASS; Pianta et al., 2008) provide information on the quality of teacher-child interaction. The TIPPS has also been used in Ghana (McCoy & Wolf, 2018; Wolfe et al., 2018) where, after being reduced to 14 items, it showed some significant but low correlations with preschool literacy and numeracy. However, the more general measure of quality, namely the seven-subscale Early Childhood Environment Rating Scale (ECERS-R; Harms et al., 1998) has been used in Latin America (Araujo and Schady, 2015) and extensively in Africa and Asia (Aboud, 2006; Brinkman et al., 2016) along with the ECERS extension to math, literacy, and science teaching (ECERS-E; Sylva et al, 2006). Yet researchers have acknowledged the unsatisfactory strategy of modifying measures in major and haphazard ways to suit low-resource contexts, or relying on the original quality measure without excluding context-unsuitable items. For example, Brinkman et al. (2016) found that 15 out of 43 items of the ECERS-R were not applicable to any of their rural Indonesian centers which lacked furnishings, computers, videos and televisions.

Consequently, along with four institutions supporting the MELQO (2017) initiative (Brookings Institution, UNESCO, UNICEF, and World Bank), we developed a measure of pre-primary process quality tailored for use in LMIC and based on the experience of international experts. This was the first effort to assess its use in East Africa and its value in predicting first grade literacy and math achievement.

1.3 Child Outcomes Associated with Quality

Our second question addressed the developmental benefits of attending high quality pre-primary education. Developmental benefits are usually measured in terms of academic achievement, or cognitive, language and social-emotional development. Using a school readiness test and other cognitive measures, several researchers have found a relation between ECERS-E and -R quality and pre-primary students’ performance in Bangladesh, Indonesia and East Africa (Aboud, 2006; Aboud et al., 2016; Brinkman et al., 2016; Malmberg et al., 2011). Studies in Latin America where an emergent approach to teaching/learning is more common found mixed results using the CLASS in Chile (Yoshikawa, et al., 2015). Similarly, in the United States, recent studies have found some evidence for preschool instructional quality predicting pre-primary (aka kindergarten) language and reading but not math achievement (Burchinal et al., 2008), though in most studies the associations were modest at best with effect sizes less than 0.20 (Burchinal, 2018). However, British children’s pre-academic and cognitive outcomes were more strongly associated with the ECERS-E literacy and math quality scores than with the ECERS-R total or subscale quality scores (Sylva et al., 2006). In sum, there is mixed evidence for the relation between measures of program quality and student learning in the pre-primary year. Less is known about how pre-primary quality relates to primary school learning, the focus of the current study.

The limitations of existing research on pre-primary quality and children’s learning concern first the measure of quality and second how it relates to school achievement beyond the pre-primary level. To address the first limitation, the Measuring Early Learning Quality and Outcomes (MELQO) Initiative (2016) held meetings with experts and set up a technical advisory group to develop a measure of quality appropriate for low- and middle-income countries. This measure, the MELE, was explored here for the first time to compare two well-known models of pre-primary education in East Africa. To address outcomes, which in past research were often not well-aligned with qualities or with future academic skills (Burchinal, 2018), we used a standard measure of reading and math for early primary grades, developed and used by governments in multiple countries with some cultural adaptation (e.g., Piper and Mugenda, 2012). Thus, previous limitations of quality and outcome measures were addressed in this study.

Another limitation of almost all the pre-primary research in LMIC, including this one, is the design, namely that children are not randomly assigned to pre-primaries of high or low quality, or to pre-primaries implementing different models. This happens because governments and organizations are already implementing pre-primaries; helping them to evaluate and improve existing programs takes precedence over researcher-controlled programs. Consequently, there may be selection bias if certain children attend certain pre-primaries. However, in LMIC children normally attend the pre-primary in their village with little choice. Only one published study included in a Cochrane Review (Brown et al., 2014) partly overcame this limitation by having pre-post assessments on the same children. This was the earlier study of our Madrasa and non-Madrasa preschools where children were pretested at entry when 3 or 4 years of age and retested 1 year and 2 years later (Mwaura et al., 2008). The study revealed no cognitive differences between the two groups of children at pretest, a significant difference one year later and a reduced difference two years later at the end of pre-primary. A problem associated with following up children even one year later in the same preschool is that of attrition which was 38% in the Mwaura study. This severely curtails statistical analyses. Considering this rate of attrition which one expects to be higher when following pre-primary children into primary, we used the post-only non-randomized design of other researchers in low-income countries, while minimizing and controlling for observed group differences, such as the mother’s education and family assets (e.g. Aboud & Hossain, 2011; Brinkman et al., 2016).

1.4 Aims of the Study

The present study evaluated the quality of two community-based models of pre-primary education in Kenya, Uganda and Zanzibar, namely the Madrasa program and other government or community programs. The term “Madrasa” in Arabic refers to school, though it has mistakenly come to be associated only with schools that teach Islam. In recent years, Madrasas in many countries have adopted the government curriculum or one that is even more progressive. The Madrasa preschool program in East Africa (also known as the Madrasa Early Childhood Program or MECP) integrated Islamic teaching into a secular curriculum. Though now managed and funded by the communities with parent contributions, the curriculum is developed by the MECP and teachers are trained and supervised by the MECP. They offer a 2-year certification course, sometimes open to government teachers as well, and a 6-month short course for those already certified. Their approach is based on a child-centered constructivist philosophy of active learning, where manipulation and exploration of materials and ideas are supported by high-quality teacher-child interaction (Mwaura & Marfo, 2011). These features of the program were to be observed as part of this research, and teachers’ experience and training were also assessed with an interview.

Graduating students from the Madrasa program feed into government primary schools, as do graduates from the other community and government pre-primaries. It was expected based on government documents and in-school observations (Plan International, 2014) that non-Madrasa pre-primaries would have a style of teaching-learning that would be teacher-led and where students would mostly respond to teacher requests. Tuition is free in these government schools though children are often expected to wear uniforms. Teachers were expected to have some secondary school education and at least one year in a certificate program, followed by regular supervision. At the time of the study, Kenya was working on a revised curriculum, Zanzibar was focused on expanding its offerings to rural regions and creating standards (World Bank, 2013), and Uganda relied mainly on local community groups (e.g. faith-based) to provide pre-primary education while the government sought to develop standards. Because practices often differ from policies, we aimed to evaluate what actually occurred in pre-primary settings and how the graduates of observed pre-primaries performed in first grade one year later.

Based on findings from past research, the hypotheses were:

Madrasa pre-primary programs would have a higher quality than non-Madrasa programs especially in the domains of free-choice play, pedagogical approach (e.g. program structure), and teacher-child interaction.

Graduates of the evaluated Madrasa pre-primaries, assessed at the end of primary 1, would have higher academic, cognitive and social outcomes than graduates of the evaluated non-Madrasa settings.

First grade literacy and math achievement would be associated with the quality of the pre-primaries attended by these students.

2. Method2.1 Setting and Participants

The three settings were Eastern Kenya (Mombasa and the coast), Central Uganda (north of Kampala), and Zanzibar, Tanzania, where the Madrasa Early Childhood Program supported Muslim communities from 1986, 1993, and 1990, respectively, In Uganda, non-Madrasas were non-profit community-managed pre-primaries. In Kenya and Zanzibar, non-Madrasas were government-managed, teachers were government-trained and supervised, and they used the government curriculum. The number of pre-primaries observed in October 2015 varied slightly by country from 78 in Kenya, 65 in Uganda, and 74 in Zanzibar. Approximately half were Madrasa pre-primaries, selected randomly as described below.

The Primary 1 graduates of the selected pre-primary schools came from a mix of disadvantaged urban and mainly rural communities. The education level of parents was generally low though most mothers had at least some primary school experience. Two-thirds of the families were Muslim and the rest were Christian. In October 2016 children were tested if they were between 6 and 9 years of age. Although this is a broad age range for the first grade of primary, it was representative of children who had graduated from the selected pre-primaries. Their caregivers (mothers) were interviewed after signing consent for their own and their child’s participation. Ethics approval was granted by McGill University, Canada, the Mbarara University of Science and Technology in Uganda, Pwani University in Kenya, and from the Ministry of Education in Zanzibar. The study was conducted independently of the funder and of the Aga Khan Foundation by university researchers with no connection to the education programs being evaluated.

2.2 Design

The design was a cluster nonrandomized two-group design (Madrasa; non-Madrasa) comparing the quality of Madrasa and non-Madrasa pre-primary settings (Phase 1) and a year later comparing randomly selected children, who had graduated from the observed pre-primaries, now at the end of their Primary 1 year (Phase 2). As mentioned previously, it was not feasible to get a baseline assessment of children, because they had started preschool at different ages, some at 3 years and some for pre-primary only, and because attrition was expected to be high.

Quality was assessed at the end of the school year, in the tenth month of 2015, and Primary 1 achievement in the tenth month of 2016 (see the flow graph in Figure 1). Pre-primary schools were selected as follows using a list of government primary schools with both Madrasa and non-Madrasa pre-primaries feeding into them. Of Madrasa pre-primaries feeding into government primary schools (80 in Kenya, 87 in Uganda, and 81 in Zanzibar), 36 were randomly selected from each of the three country sites; the non-Madrasa pre-primary that fed into the same government primary school used by these Madrasa graduates was then selected. So Madrasa pre-primaries were randomly selected and a yoked non-Madrasa was selected (in Uganda, comparison pre-primaries were community- or church-based because government-run ones were rare in this setting; in Kenya and Zanzibar comparison pre-primaries were government-run). One year later at the end of 2016, using the 2015 pre-primary lists of enrolled children, we randomly selected five primary school children who had graduated from the previously observed Madrasa and five from the yoked observed non-Madrasa. Thus, Madrasa pre-primaries were randomly selected, non-Madrasas were yoked, and their graduates were randomly selected. The cluster was the primary school. Reporting follows the TREND statement for Transparent Reporting of Evaluations with Nonrandomized Designs (Des Jarlais et al., 2004).

2.2 Randomization and Masking

Madrasa and non-Madrasa pre-primaries and students who graduated from these pre-primaries were not randomized; parents largely chose to which they sent their child. However, they attended the nearby government primary school for first grade. Selection bias due to non-randomization was minimized by including pre-primaries and children from similar and adjacent communities, feeding into the same primary school, and by statistically controlling for observed demographic differences, which had been few in the earlier studies (Malmberg et al., 2011).

It was not possible to mask data collectors to the pre-primary they were observing and rating for quality. Efforts were made to mask data collectors to the child’s pre-primary school when testing them in first grade. Local investigators gave data collectors lists of every fourth or fifth child plus replacements from the pre-primary rosters, without identifying whether it was Madrasa or non-Madrasa graduate. To minimize bias, we spoke to data collectors at length about the need to remain unbiased and shed any expectations about the pre-primary models. We avoided hiring data collectors who had prior experience and pre-conceptions about the education programs.

2.3 Phase 1. Pre-primary Quality Measure and Data Collection

A recently developed quality instrument, Measure of Early Learning Environments (MELE), was used to observe the pre-primary setting and to interview teachers (MELQO, 2016 www.ecdmeasure.org). To ensure content validity, conceptual domains were elicited from an international group of over 30 early education experts to cover constructs considered important in pre-primary settings. Domains often overlapped with those found in existing measures, such as Interactions and Language teaching/learning, so the following short headings were adopted: Physical environment, Interactions, Inclusiveness, Program structure, Language, Numeracy, Science, Games/art, and Free-choice Play activities. Fifty individual items were then formulated to fit these domains, selecting qualities from existing measures previously modified for use in LMIC, in particular, but not exclusively, play items from the ECERS-R (Harms et al., 1998), literacy, numeracy and science items from the ECERS-E (Sylva et al., 2003), teacher-child interaction items from the TIPPS (Seidman et al., 2014), and physical environment plus other items from the ECEQAS (Kaul et al., 2012) and TECERS (Isely, 2001). Program structure, similar to the ECERS-R with the same name, had three items but was kept as a separate domain; a group of items including songs, rhymes, art, and games when teacher-led and performed by children together was also kept as a domain separate from free-choice play activities. Items do not reflect any particular philosophy of early education but rather capture the definition of pre-primary education as the initial stage of organized instruction within a school-type environment, designed to meet the educational and developmental needs of young children (OECD, 2003; 2012). Likewise, domains were identified not as essential and distinct constructs, but rather to ensure inclusion of critical qualities.

Items were phrased in terms of observable and quantifiable events and resources that a pre-primary would be expected to provide for children of 4 to 6 years; they avoided phrases linked to child outcomes such as cognitive or social-emotional. Examples are: Scaffolding by teacher to help a child work through the steps to solve problems or errors (Interaction), Gender equality in class participation (Inclusiveness), Children are introduced to reading and/or writing letters (Language), Children’s new numeracy learning is connected to past learning and to its everyday application (Numeracy), and Children have access to different interest centers during indoor play, e.g., blocks, sand and water, books, art, games, dramatic, music (Free-Choice Play). Response options for each item are ordered from 1 to 4 to reflect low to high quality. Modifications of the measure underwent various iterations after sharing with stakeholders from around the world who evaluate or implement pre-primary programs, and again after trialing the measure in Tanzania where the initial 100 items were reduced to 50. Clarification of certain items again followed training before its use in this study. A manual is available with elaborations on the underlying features being measured and specific examples of each rating.

The full measure is freely accessible. A few examples of the 4-levels of quality indicate that quantity and frequency are included, as is the nature of the teaching method (e.g., group repetition vs individual work). Items from teacher-child interaction, program structure, language teaching, and numeracy teaching, respectively, are provided here.

Interaction. Behavioral indications of a positive environment between teacher and children.

1 Teacher rarely (<5 times) smiles, claps or verbally praises children’s efforts

2 Teacher smiles, claps or says “good” 5 or more times but it is automatic and routine, not heart-felt, and usually involves asking the whole class to clap

3 Teacher smiles, looks directly and has warm words of praise for 5 or more individual children’s efforts

4 Teacher’s smiles, looks directly and gives warm words of praise to 10 or more individual children

Program Structure. The daily routine, seen today, has a mix of activities including play (indoor, outdoor), arts & games (e.g. stories, songs, rhymes, art, games), and instructional (e.g. teacher-led child-directed language, numeracy).

1 Children receive teacher-led instruction most of the time (> 80%)

2 Teacher-led instruction occurs for approximately 2/3 of the time. The rest of the time is devoted to child-initiated play or arts & games but not both.

3 Teacher-led instruction occurs for approximately 2/3 of the time. The rest of the time is devoted to child-initiated play and arts & games activities (e.g. stories, songs, rhymes, games).

4 One-half or less of the time is spent in teacher-led instruction; the rest includes both child-initiated learning/play and arts& games activities

Language Teaching. Children are introduced to reading and/or writing letters.

1 Children's attention is not directed to written letters or written words in posters, books or blackboard

2 Children read and/or write letters or words by immediately repeating what the teacher says

3 Some children have the opportunity to read and/or write letters or words on their own (may copy from the board on to their paper), not simply repeating immediately what the teacher said

4 All children have the opportunity to read and/or write letters or words on their own, in at least one activity such as writing in an exercise book

Numeracy Teaching. Children use objects (e.g., blocks, sticks, bottle caps) for math concepts and patterns, and not simply for enumerating.

1 Objects are not used by individual children for this purpose

2 Objects used by less than half the children to reproduce what the teacher has shown

3 Objects used by more than half the children to reproduce what the teacher has shown

4 Objects used in an individual way by at least half the children to reflect math concepts without simply reproducing what the teacher has shown

On the MELE data form, there is space to enter information requested from the teacher and/or observed during the visit. This included: hours of operation and of observation; numbers of boys and girls enrolled and attending on the day of observation; number of teachers/aides; the weekly schedule of activities, and the duration of activities on the day of observation.

A short Teacher Interview using a structured questionnaire provided information on structural aspects of the pre-primary programs, specifically teachers’ education and training, mentoring and supervision, and parent engagement (MELQO, 2016). Again, most but not all responses were scored on a 1 to 4 scale while others required yes-no answers.

Training and data collection. Eight local, university graduates were recruited from each site based on their research experience, education, and communication skills. They were proficient in English and the local language.

Training on the MELE measure of quality was conducted by two researchers, one a developer of the measure and the second a local researcher who had obtained high reliability with the first. The items and the manual were in English given that all data collectors were proficient in this language. Over the course of a week, research assistants became familiar with the measure, with the manual explaining in detail what to observe for each item, and they practiced observing and rating pre-primary sessions using both videos and actual pre-primary settings. Inter-rater reliability with the measure developer was obtained at the end of each day and found to reach a minimum of 90% by the end of the third day. A fourth day was used to review additional videos and highlight some difficult items.

During data collection, local assistants visited one pre-primary a day, given that all operated in the mornings. They worked in pairs on the first day and individually thereafter. They were supervised by the local researcher, who was available particularly during the first week when logistic and other problems were solved. They were to observe a full morning session of activities, which normally lasted 5 hours. A subsample of 63 teachers (23 from Uganda, 16 from Kenya, 24 from Zanzibar), 35 from Madrasas and 28 from non-Madrasas, were interviewed after the session was finished for the day. The smaller sample was intended as the teacher measure had undergone less pretesting.

2.4 Phase 2. Primary School Measures and Data Collection

Caregivers (mothers) were interviewed to gather information on the child and the family, including the child’s birthdate, preventive health measures (e.g. immunizations, improved water, deworming), diet (recoded into seven food categories to calculate a dietary diversity score, Daelmans et al., 2009), past 2-week illnesses, along with family characteristics such as assets, parental education, religion, household size. Because the mother's education is often used as a covariate in analyses of children's achievement, we imputed missing scores for this variable based on the mean education level of other mothers in the cluster. Other covariates did not have missing data. Children’s height was taken twice and averaged to calculate a standardized height-for-age z score based on WHO growth references (World Health Organization, 2009).

Children were directly tested for math and literacy achievement, executive function, and social-emotional development. A fuller description now follows.

The Early Grades Math Assessment (EGMA) and Early Grades Reading Assessment (EGRA) have been translated into local languages and previously used with first, second, and third graders in East Africa (e.g., Piper, 2010; Piper and Mugenda, 2012). The tests were modified by our team of regional and international experts in early education and child development, to make them comparable across the three sites in East Africa, shorten them for first graders, and avoid floor effects with low variance. They were intended for research, not diagnosis. The Math test consists of 53 items in six subtests requiring children to read aloud numbers, identify the larger, complete a pattern, add, subtract, and solve word problems (alpha 0.70 Kenya, 0.76 Uganda, 0.78 Zanzibar). The Reading test, here called Literacy, consists of 155 items in eight subtests requiring children to read letters, syllables, familiar words, and novel words; read aloud a passage, comprehend words and a passage, and write words (alpha 0.84 Kenya, 0.81 Uganda, 0.89 Zanzibar). There were practice items and stopping rules but no time constraints within general limits. The Math score was the sum of correct answers across 53 items and the Literacy score the sum of correct answers across 155 items. Scores are presented out of 100 for comparability. High alpha coefficients and subtest-total correlations warranted combining subscores. Teachers' reports of the most recent Math and Language marks for a subsample of the students correlated significantly with their EGMA and EGRA scores ( r = 0.53 and 0.60 in Kenya; 0.37 and 0.40 in Uganda; 0.52 and 0.48 in Zanzibar, p < .001, respectively), thus providing validity of the two achievement tests in relation to school performance.

The two Executive Function tests were the Tower of London (Anderson, Anderson, & Lajoie, 1996) that requires planning and flexibility, and Digit Span that relies on working memory (Diamond, 2013). The Tower task requires children to move balls from one dowel to another to re-create the model in a minimum number of moves. There were 16 trials, each rated pass (1) or fail (0), summed for a total score. Digit span backwards (6 trials) and digit span sequences (6 trials) included listening to a sequence of 2, 3, and 4 numbers and repeating them backwards or re-arranging them in the proper forward sequence, respectively. Forward digit span sequences were given as practice before the test sequences. Scores were the number of trials performed correctly. The Digit Span and Social Problem Solving (to be described next) were the only measures newly used in East Africa and so translated into local languages before use. Research assistants during training discussed whether the instructions and scenarios were correctly communicated, and modifications were made when necessary.

Social problem solving consists of three vignettes describing a problem the child must solve (Shure & Spivack, 1982). They were modified to fit the context: how to reduce a mother’s anger after breaking her bowl, how to regain a teacher’s approval after performing poorly on a test, how to get access to a peer’s desirable toy. After each solution, the child was asked, “What else could you do?” Up to nine solutions, three for each problem, were recorded.

Training and data collection. Eight local university graduates were recruited for each site based on their research experience, education, and communication skills. They were proficient in English and the local language. Most had participated in the observation of pre-primaries.

Training on the Mother interview and direct individual child tests took one week of becoming familiar with the measures and practicing them on each other. Preliminary testing in Zanzibar, where we started, demonstrated high inter-rater reliability several days apart with high correlations and non-significant differences between means (e.g., Literacy r = .99, t = 0.95, p = 0.37; Math r = .90, t = 0.42, p = 0.69; Digit sequence r = .73, t = 0.60, p = 0.57; social problem solving r = .77, t = 0.36, p = 0.73). We used both correlation and difference tests to demonstrate the association between testers and the lack of a practice effect for children.

Data collectors were given names of primary schools and lists of children to test. During the previous year’s visit to pre-primary schools, names of all children in the pre-primary class were collected along with the corresponding catchment primaries they were likely to attend. From the 2015 list of pre-primary children, first grade students were systematically selected to be tested to make up the five needed for our sample (e.g., from a class of 40, every eighth child was listed for testing; from a class of 30, every sixth child was selected), along with replacements. Children were individually tested at school or at home after school. A quiet place was found for the testing; data collectors gave tests in the same order as described here to all children, and permitted rest periods if requested.

2.5 Sample Size Estimation

The sample size was based on child outcomes to be assessed when graduates of pre-primaries were in primary 1. A sample of 360 children per country was required, half from Madrasa pre-primaries and half from non-Madrasa pre-primaries. This was based on an expected difference of 0.40 standard deviations between group means, using a power of .80 and a significance level of .05 (based on previous research such as Aboud and Hossain, 2011; Brinkman et al., 2016; Malmberg et al., 2011; Nores and Barnett, 2010; Rao et al., 2017). The sample was then multiplied by 1.80 based on an intra-cluster correlation (ICC) of 0.20 because of clustering at the primary school level (based on previously obtained ICCs from Aboud et al., 2017). To obtain a sample of 180 children from each model of pre-primary in each country we selected 36 Madrasa and 36 non-Madrasa pre-primaries, taking 5 children from each a year later as they finished first grade. Because this was the first use of the full MELE in a LMIC, we did not intend to hypothesize or validate a factor structure, which would require a minimum of 250 pre-primaries (though available on request). Rather the analyses focused on a comparison of the two pre-primary models, and the construct and predictive validity of their setting quality.

3. Results3.1 Comparison of Madrasa and non-Madrasa Pre-Primary Quality

In order to compare the quality of the Madrasa and non-Madrasa pre-primary programs, analyses of covariance were conducted, using a 2 (Madrasa, non-Madrasa) X 3 (countries) design, covarying attendance numbers on the day of observation (which were significantly different by program and country). Table 1 presents means and standard deviations for attendance, overall quality and its domains for Madrasa and non-Madrasa pre-primaries. The overall quality of Madrasas was significantly higher than non-Madrasas (means were Madrasa M=2.66 and non-Madrasa M=2.46; effect size d=0.70). Despite this, ranges of quality ratings overlapped considerably especially at the low end, where Madrasa overall quality ranged from 1.88 to 3.40 and non-Madrasa quality from 1.92 to 3.00. As expected, Madrasas had significantly higher quality on items related to free-choice play activities (d=1.12) and language (d=0.60), and to a lesser extent on group activities (art, games) (d=0.40) and the program structure (d=0.25). There were no significant differences between Madrasas and non-Madrasas on the physical environment, teacher-child interaction, or math teaching.

There were significant country differences on the overall quality and on most domains, indicating that Kenyan pre-primaries scored significantly higher than Ugandan or Zanzibari ones except on the physical environment. Only Play showed a significant country x program interaction.

Regarding other data collected about the programs in the three sites, all had 5- or 6- hour daily morning sessions, with time reserved for language, math, and indoor or outdoor play generally 4 days a week. Nature/science teaching was typically given on only two days/week. Boys’ and girls’ attendance differed significantly but minimally across programs (Program x Gender, F(1,211)=4.39, p=.04), where girls slightly predominated in Madrasas and boys in non-Madrasas: Madrasa girls M=9.58, boys M=9.19; non-Madrasa girls M=12.28, boys M=13.51.

Insert Table 1 about here

Analyses of teachers’ responses to interview items revealed only a few differences: teachers in government pre-primaries were better educated (p = 0.02) whereas parents of Madrasas were more engaged in the program and its governance (p = 0.0002).

3.2 Comparison of Madrasa and non-Madrasa Graduates on Primary 1 Outcomes

Analyses of variance, using SAS 9.3 PROC MIXED to adjust for clusters, were first conducted on child and family variables derived from the mother’s interview to determine if Madrasa and non-Madrasa groups differed on child age or sex, and family variables. Group differences were then included as covariates in the analyses of child outcomes. Preliminary analyses of children showed large differences across the three countries, in literacy and math performance and other variables such as age (see Table 2). Concerning age, Kenyan children were 8.3 years of age on average with a range of 78 to 117 months, whereas Zanzibari children were 7.4 years on average (range 70 to 108). It was therefore decided to analyze children’s data separately by country.

Insert Table 2 about here

Table 3 provides information on children's first grade performance. Sex did not interact significantly with the type of pre-primary attended, so we combined boys and girls for these analyses (sex disaggregated data tables are available on request). Because of differences in children's ages and mother's education in some cases, these variables were covaried in analyses of achievement.

Child outcomes from primary 1 were analyzed to examine group and sex differences, using SAS 9.3 PROC MIXED analysis of covariance to adjust for covariates and clusters (ICCs are provided in the tables). Table 3 shows that on the Literacy and Math assessments, Madrasa and non-Madrasa graduates did not perform differently. Kenyan children attained higher levels of literacy and math than did Ugandan and Zanzibari children, though our intent was not statistically to test country differences. Although the age for entering primary school is similarly 6 years in each country, Kenyan children were on average older. However, their scores did not correlate with age; Kenyan children who were 8 or 9 years of age did not perform better than those who were 6 or 7 years. On the two tests of executive function, Tower Planning and Digit Span, again there were no group differences. There were also no group differences on the social problem test.

Insert Table 3 about here

3.3 Association of Primary Performance with Pre-primary Quality

Finally, children’s primary 1 literacy and math performance outcomes were correlated with ratings of overall quality and of four relevant quality ratings assigned to their former pre-primary while they were enrolled; namely interaction, language, numbers, and play quality. These domains are conceptually most meaningfully related to literacy and math performance, and have been empirically associated in past research (Aboud & Hossain, 2011; Aboud et al., 2016; Sylva et al., 2006). Again, because performance differed by county but not by Madrasa and non-Madrasa program, separate country analyses were conducted combining children who had graduated from both programs.

The multilevel linear regression program from MPlus 7 (Muthén and Muthén, 2015) was used. Adjustments were made for clusters, namely pre-primaries, and for covariates such as child’s age and mother’s education which were found to differ across pre-primaries. Adjusted standardized beta coefficient estimates, yielded by the analysis, served as effect sizes for the association between quality and child outcomes (Nieminen et al., 2013). Table 4 presents standardized coefficients and their p values by country for literacy and math. One measure of executive function, namely digit span, was strongly associated with overall quality, interaction and language (standardized coefficients ranged from 0.42 to 0.51, p < .01) in Uganda but not in Kenya or Zanzibar.

Insert Table 4 and 5 about here

The pattern for Kenyan data showed that students' Literacy scores were associated with the overall quality of their pre-primary, and Math scores were positively associated with the quality of math teaching. This would mean that if the overall quality rose one standard deviation from 2.68 to 2.98 on a 1 to 4 scale, students' literacy scores would be 10.4% higher; and if math quality rose from 2.43 to 2.98, students' math scores would be 6.44% higher. Ugandan students’ literacy scores were significantly related to Interaction, whereas math scores were associated with the overall quality. Consequently, literacy scores would rise by 9.36% and math by 7.7% if the corresponding quality was improved by one standard deviation. Zanzibari students’ scores were not associated with any quality ratings.

Two features of the Zanzibar data are worth noting: one is that their literacy and math scores were low, on average 50% or below for literacy and math. Another is that the internal consistency of their quality ratings was not as high as those for Uganda and Kenya. Across all countries, Madrasas showed more consistency than non-Madrasas (see Table 5), possibly reflecting a more cohesive program. Low consistency in a domain such as teacher-child interaction might be expected if teachers offered praise but no scaffolding. However, low consistency in literacy and math might reflect a fragmented program in these two areas of teaching/learning.

4. Discussion

The findings were that Madrasa pre-primary schools had higher quality than non-Madrasas on their overall rating and on the qualities of free-choice play activities, language teaching, games/art, and program structure. However, there were no differences between the two groups of graduates in their first grade literacy, math, executive function and social development. Finally, children’s first grade performance in literacy and math were associated with overall quality or with the quality of the corresponding domain in Kenya and Uganda, but not Zanzibar. We now interpret these findings in light of past research and implications for future improvements in pre-primary programs.

4.1 Quality of Pre-primary Schools

Overall, as expected, Madrasa pre-primary schools in the three countries had significantly higher quality scores than the non-Madrasa pre-primaries. Effect sizes were small to large, ranging from 0.25 in the domain of program structure to 1.12 in free-choice play. The program structure items included having a curriculum, using a mix of child- and teacher-led activities, and using a mix of small group and individual work in addition to whole group teaching. Madrasas were expected to have a better program structure and more free play compared to non-Madrasas, which were expected to have more whole-group teaching and less free play. Madrasas and non-Madrasas did not differ in adult-child interaction, inclusiveness, physical environment, numeracy or science teaching. Average ratings of Madrasas were just above the midpoint of the scale between 2.58 and 2.78, whereas non-Madrasa overall averages ranged from 2.35 to 2.55; ranges showed that quality between the two programs largely overlapped. In other respects, the programs were similar, especially regarding teacher experience and supervision, activities included in the daily routine, and numbers of children attending.

These results are comparable to others in the published literature in LMIC, which in most cases assessed quality with modifications of existing scales. For example, an earlier study on a smaller number of selected Madrasas and non-Madrasas in the same East African regions found that in the pre-primary year the former had higher quality with average scores for the 11 domains in the ECERS-R and –E of 4.11 and 2.93, respectively, on a 1 to 7 scale (Malmberg et al., 2011). At that time, the Madrasa pre-primaries averaged slightly above the midpoint as they did in our study; however, non-Madrasas performed better now compared to the earlier study. Non-Madrasas in Kenya and Zanzibar have benefited from collaboration with the MECP program in terms of teacher training specifically.

Despite its recent development, there are some unique strengths to the MELE in addition to its open access. Essentially the measure is a composite of items relevant to pre-primaries in LMIC, drawing from different measures with strengths in play, pedagogy, interaction, and physical environment. Each item describes four levels of quality with a short phrase of observable behaviors, thus providing clear guidance on how to improve. Items cover domains considered important, as judged by early education experts and as included in other measures, thus demonstrating content validity. Construct validity, in terms of known differences, was demonstrated here with significant differences between Madrasa and non-Madrasa pre-primaries on overall quality and several subscales. Certain clusters of items, such as those related to the physical environment, have been found to distinguish Indonesian preschools given assistance to strengthen this component (Proulx & Aboud, 2019). High correlations between the MELE and the ECERS-E in three China sites and also in Hong Kong support its convergent validity (Rao, personal communication). Other uses of the MELE, for example in Colombia, found associations with pre-primary math and literacy outcomes (Maldonado-Carreño et al., 2018). However, because many activities and materials were of low quality, they were rated simply as absent vs present. Such modifications might be necessary in order to conduct a statistical analysis, yet they do not provide guidance on improvement.

4.2 Early Primary Child Outcomes

Literacy and math skills at the end of first grade, along with executive function and social problem solving, showed no differences between graduates of Madrasa and non-Madrasa pre-primaries. The academic tests are ones commonly used by governments in the region to evaluate progress in early grades reading and math. With averages of 80%, Kenyan students achieved more literacy and math skills than those from Uganda and Zanzibar, whose scores averaged between 43% and 60%. However, in all three regions, subscale scores ranged from zero to 100%. Thus, teachers were trying to teach simultaneously children who do not know any letters or numbers and children who can read and comprehend passages. This is surprising in that all children in our samples had attended pre-primary programs where they received some language and numeracy instruction.

The lack of a significant difference between Madrasa and non-Madrasa students adds to a small but growing body of research examining whether pre-primary children maintain their advantage once they enter primary school. Most researchers find a difference at the end of pre-primary (e.g., Aboud, 2006; Brinkman et al., 2016; Martinez et al., 2012) but do not follow the children into primary school. For example, the earlier study comparing Madrasa and non-Madrasa children in the same East African sites found significant benefits for the former after the first two preschool years but less at the end of pre-primary (Mwaura et al., 2008; Malmberg et al., 2011). This suggests that the Madrasa curriculum may be more appropriate for 3- and 4-year-olds but lacks attention to early literacy and numeracy skills for 5- and 6-year-olds. Only a few LMIC studies have followed children into primary school. Aboud and Hossain (2011) found superior literacy and math performance in first and second graders who had participated in the pre-primary program only after the program raised its quality (see also Aboud et al., 2016; Montie, Xiang, & Schweinhart, 2006). Thus, for benefits to continue beyond pre-primary school, its quality must be more than adequate. This may explain the lack of significant differences in outcomes in first grade despite higher quality of Madrasa pre-primaries overall, and an association between quality and student outcome. Madrasa pre-primary quality was not sufficiently higher to impact first grade performance. This finding is important and draws attention to the need for researchers to follow children beyond pre-primary when examining relations between quality and academic performance.

4.3 Associations of Literacy and Math Performance with Pre-primary Quality

The importance of pre-primary quality in predicting early grades literacy and math received some support from the multilevel linear regression analyses. The overall quality regression coefficients for Uganda (0.35) and Kenya (0.50) were moderate, and comparable to Sylva et al.’s (2006) and Aboud et al.’s (2016) associations with the ECERS-E, and better than other findings using the ECERS-R (e.g. Abreu-Lima et al., 2013; Burchinal et al., 2008; Early et al., 2018; Gordon et al., 2013) or the TIPPS in Ghana (Wolf et al., 2018). This is the first such support for the MELE and promising evidence for an association between overall quality in LMIC and first grade achievement. The 50-item quality measure showed good internal consistency in all countries and so could be used as a measure of program evaluation. The associations found here and elsewhere emphasize the need to raise the quality of pre-primary education in order to have a more lasting effect on early grade literacy and math.

Beyond overall quality, associations with a few specific domains were interpretable. Kenyan math scores were predicted by math quality (std coeff 0.56), and Ugandan literacy and math were related to teacher-child interaction (std coeff 0.39 and 0.45). The former is to be expected; the latter might be explained in terms of the value of teacher-child open-ended and one-on-one conversations promoting language skills.

It is debatable whether one should expect discriminable and internally consistent subscales representing numeracy, literacy, interaction, and play quality. Regardless of the factor structure of the ECERS-R and -3, they continue to yield important information about preschool quality in high-income countries (e.g., Burchinal, 2018; Early, Sideris, Neitzel, LaForett, & Nehler, 2018). Analysis of the TIPPS in Accra, Ghana yielded a 3-factor structure of 14 items, but a set of 13 curricular content items showed stronger correlations with preschool literacy and numeracy outcomes than any factor (McCoy & Wolf, 2018). In these East African data, play and numeracy subscales showed good internal consistency; literacy and interaction items yielded only modest alphas, often because programs that included letter and word activities did not necessarily include story reading or discussion of new vocabulary. Likewise, teachers who praised good answers did not necessarily ask open-ended questions or scaffold new learning. Removing inconsistent items would raise the alpha level but leave an incomplete measure of literacy teaching, for example. Further use of the MELE in LMIC will help to clarify its value for program evaluation, as it did in Indonesia where attempts to improve physical infrastructure were reflected in higher physical environment scores but otherwise low scores pointed to missed opportunities to improve the pedagogical program (Proulx & Aboud, 2019).

It is not clear why significant associations were not found within the Zanzibar data. Their non-Madrasa program is somewhat new and there may have been less coherence in the program, as evidenced by lower alpha coefficients. Zanzibari children overall received lower math and literacy scores. Scores were similarly low among Madrasa and non-Madrasa graduates. Zanzibari families, however, had good diets, high maternal education and many family assets.

4.2 Strengths and Limitations

The research had a number of strengths, including the follow up of children into primary school to examine associations with pre-primary quality. Other design strengths were large samples of pre-primaries in each country, with Madrasa pre-primaries being randomly selected and matched with a non-Madrasa yoked to the same primary. The sample of children was randomly selected from within their respective pre-primary. The measures were appropriate, in particular the measures of literacy and math were previously used in the three countries and validated for our sample. The measure of quality was specifically developed for LMIC, yet tapped into domains that are considered important by experts in high-, middle-, and low-income countries. Built in is a strategy to raise the quality of one’s program based on the findings. Our analyses adjusted for covariates as well as clusters.

The design and procedures also had some noteworthy limitations, in particular selection bias due to non-random assignment, addressed here through matching pre-primaries and statistically controlling child differences. Pre-primaries were not randomly assigned to group. The children, likewise, were not randomly assigned to pre-primaries. The post-only design meant that children were not tested before their pre-primary program or at the end of pre-primary to see if they were comparable at baseline. Thus the lack of difference could be due to selection bias that put the Madrasa children behind non-Madrasa children at preschool or pre-primary entry, implying that they gained as a result of their pre-primary experience, though this is unlikely (Malmberg et al., 2011). We statistically controlled in analyses the few sociodemographic variables on which groups differed. However, groups may have differed on other unobserved variables that were not controlled Masking of data collectors was not possible when observing pre-primaries. Children covered a larger age range than expected, but we did not want to set an unrepresentative age limit.

Despite these limitations, the strengths give confidence to our conclusions that Madrasas were better in quality than non-Madrasas but that graduates from these two pre-primary programs performed similarly on measures of academic, cognitive and social development in primary school. There was some evidence that the quality of their pre-primary was significantly associated with children’s academic achievement at the end of first grade. This evidence supports the conclusion that program improvement and policy changes should be based on evidence of pre-primary quality.

References

Aboud, F. E. (2006). Evaluation of an early childhood preschool program in rural Bangladesh. Early Childhood Research Quarterly, 21, 46–60. doi:10.1016/j.ecresq.2006.01.008

Aboud, F. E., & Hossain, K. (2011). The impact of pre-primary school on primary school achievement in Bangladesh. Early Childhood Research Quarterly, 26, 237-246.

Aboud, F. E., Proulx, K., & Asrilla, Z. (2016). An impact evaluation of Plan Indonesia's early childhood program. Canadian Journal of Public Health, 107(4-5), e366–e372, doi: 10.17269/CJPH.107.5557

Abreu-Lima, I. M., Leal, T. B., Cadima, J., & Gamelas, A. M. (2013). Predicting child outcomes from preschool quality in Portugal. European Journal of Psychology of Education, 28, 399–420. doi.org/10.1007/s10212-012-0120-y

Anderson, P., Anderson, V., & Lajoie, G. (1996). The Tower of London test: Validation and standardization for pediatric populations. The Clinical Neuropsychologist, 10, 1, 54-65, doi: 10.1080/13854049608406663

Araujo, M.C., & Schady, N. (2015). Day care services: It’s all about quality. In S. Berlinksi & N. Schady (Eds.). The early years. Child well-being and the role of public policy. Washington, D.C.: Inter-American Development Bank.

Brinkman, S.A., Hasan A., Jung, H., Kinnell, A., Nakajima, N., & Pradhan, M. (2016). The role of preschool quality in promoting child development: Evidence from rural Indonesia. World Bank Group, Policy Research working Paper 7529, Washington DC.

Brown, T. W., van Urk, F. C., Waller , R., & Mayo-Wilson, E. (2014). Centre-based day care for children younger than five years of age in low- and middle-income countries. Cochrane Database of Systematic Reviews Issue 9. Art. No.: CD010543. doi: 10.1002/14651858.cd010543.pub2.

Burchinal, M. (2018). Measuring early care and education quality. Child Development Perspective, 12, 3-9. doi: 10.1111/cdep.12260

Burchinal, M., Howes, C., Pianta, R., Bryant, D., Early, D., Clifford, R., & Barbarin, O. (2008). Predicting child outcomes at the end of kindergarten from the quality of pre-kindergarten teacher–child interactions and instruction, Applied Development Science, 12, 140-153, doi: 10.1080/10888690802199418

Daelmans, B., Dewey, K., & Arimond, M. (2009). New and updated indicators for assessing infant and young child feeding. Food and Nutrition Bulletin, 30(2), S256-62.

Des Jarlais, D.C., Lyles, C., Crepaz, N. and the TREND Group (2004). Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND Statement. American Journal of Public Health, 94, 361-366.

Diamond, A. (2013). Executive functions. Annual Review of Psychology, 64, 135–168. doi:10.1146/annurev-psych-113011-143750.

Earle, A., Milovantseva, N., & Heymann, J. (2018). Is free pre-primary education associated with increased primary school completion? A global study. International Journal of Child Care & Education Policy, 12, 13-33.

Early, D. M., Sideris, J., Neitzel, J., LaForett, D. R., & Nehler, C. G. (2018). Factor structure and validity of the Early Childhood Environment Rating Scale – Third Edition (ECERS-3). Early Childhood Research Quarterly, 44, 242-256.

East African Community Secretariat (2014). Draft harmonised curriculum structures and framework for pre primary education. Arusha, Tanzania.

Global Partnership for Education (2016). GPE’s work for early childhood care and education. Policy Brief . Washington DC. https://www.globalpartnership.org/content/policy-brief-gpes-work-early-childhood-care-and-education

Gordon, R. A., Fujimoto, K., Kaestner, R., Korenman, S., & Abner, K. (2004). An assessment of the validity of the ECERS–R with implications for measures of child care quality and relations to child development. Developmental Psychology, 49, 146-160.

Harms, T., Clifford, R. M., & Cryer, D. (1998). Early childhood environment rating scale, revised edition. New York: Teachers’ College, Colombia University.

Isely, E. (2001). Tamil Nadu early childhood rating scale. Chennai, India: M.S. Swaminathan Research Foundation.

Kaul, V. (2012). Early Childhood Education Quality Assessment scale (ECEQAS). Centre for Early Childhood Education and Development (CECED), Ambedkar University, Delhi.

Maldonado-Carreño, C., Rey, C., Cuartas, J., Plata-Caviedes, T., Rodriguez, J. J., Escallon, E., . . . Dirección de Primera Infancia-MEN. (2018). Quality measurement of the institutional modality of early childhood education in Colombia. [Education Policy Notes, 1]. Retrieved from https://educacion.uniandes.edu.co/images/Notaspoliticaeducacion/Notes-In-Education.pdf

Malmberg, L.-E., Mwaura, P., & Sylva, K. (2011). Effects of a preschool intervention on cognitive development among East-African preschool children: A flexibly time-coded growth model. Early Child Research Quarterly, 26(1), 124-33.

Martinez, S., Naudeau, S., & Pereira, V. (2012). The Promise of Preschool in Africa: A Randomized Impact Evaluation of Early Childhood Development in Rural Mozambique. Available at: http://siteresources.worldbank.org/INTAFRICA/Resources/ The_Promise_of_Preschool_in_Africa_ECD_REPORT.pdf (Accessed February 6, 2012).

MELQO (2016). Overview of the Measuring Early Learning Quality and Outcomes (MELQO) Initiative. UNESCO: Washington DC.

Montie, J. E., Xiang, Z., & Schweinhart, L. J. (2006). Preschool experience in 10 countries: Cognitive and language performance at age 7. Early Childhood Research Quarterly, 21, 313–331.

Moore, A. C., Akhter, S., & Aboud, F. E. (2008). Evaluating an improved quality preschool program in rural Bangladesh. International Journal of Educational Development, 28, 118–131. doi:10.1016/j.ijedudev.2007.05.003..

Muthén, L.K., & Muthén, B.O. (2015). Mplus User’s Guide. Seventh Edition. Los Angeles, CA: Muthén & Muthén.

Mwaura, P. A. M., & Marfo, K. (2011). Bridging culture, research, and practice in early childhood development: The Madrasa Resource Centers in East Africa. Child Development Perspectives, 5, 134-139.

Mwaura, P., Sylva, K., & Malmberg, L.-E. (2008). Evaluating the Madrasa pre-school programme in East Africa: A quasi experimental study. International Journal of Early Years Education, 16, 237–255.

Nieminen, P., Lehtiniemi, H., Vähäkangas, K., Huusko, A., & Rautio, A. (2013). Standardised regression coefficient as an effect size index in summarising findings in epidemiological studies. Epidemiology Biostatistics and Public Health, 10, e8854-1.

Nores, M., & Barnett, W.S. (2010). Benefits of early childhood interventions across the world: (Under) investing in the very young. Economics of Education Review, 29, 271-282.

Opel, A., Ameer, S. S., & Aboud, F. E. (2009). The effect of preschool dialogic reading on vocabulary among rural Bangladeshi children. International Journal of Educational Research, 48, 12-20.

Opel A., Zaman, S. S, Khanom F, & Aboud F. E. (2012). Evaluation of a mathematics program for pre-primary children in rural Bangladesh. International Journal of Educational Development, 32, 104-110. doi:10.1016/j.ijedudev.2011.01.013.

Organization for Economic Co-operation and Development (2003). Education at a glance. Paris. Retrieved 11 September 2017 https://stats.oecd.org/glossary/detail.asp?ID=5409.

Organization for Economic Co-operation and Development (2012). Starting Strong III. A Quality Toolbox for Early Childhood Education and Care, Starting Strong Paris: OECD Publishing.

Pianta, R. C., LaParo, K. M., & Harms, B. (2009). Classroom Assessment Scoring System manual pre-K. Baltimore, MD: Brookes.

Piper, B. (2010). Uganda Early Grade Reading Assessment Findings Report: Literacy Acquisition and Mother Tongue. Research Triangle Park, North Carolina, USA: RTI International.

Piper, B., & Mugenda, A. (2012). The Primary Math and Reading Initiative: Baseline report. Prepared for USAID/Kenya under the Education Data for Decision Making (EdData II) project, Task Order No. AID-623-M-11-00001. Research Triangle Park, North Carolina, USA: RTI International. http://pdf.usaid.gov/pdf_docs/pa00hx75.pdf (Accessed April 1, 2016)

Plan International (2014). Report on visit to observe and discuss ECD Centers, Preschools, and Parenting Groups in Kenya. Nairobi, Kenya.

Proulx, K., & Aboud, F. E. (2019). Disaster risk reduction in early childhood education: Effects on preschool quality and child outcomes. International Journal of Educational Development, 66, 1-7.

Rao, N., Sun, J., Chen, E. E., & Ip, P. (2017). Effectiveness of early childhood interventions in promoting cognitive development in developing countries: A systematic review and meta-analysis. Hong Kong Journal of Paediatrics, 22, 14-25.

Rao, N., Sun, J. Pearson, E., Liu, H., Constas, M. A., & Engle, P. L. (2012). Is something better than nothing? An evaluation of early childhood programs in Cambodia. Child Development, 83, 864-876.

Rao, N., Sun, J., Zhou, J., & Zhang, L. (2012). Early achievement in rural China: The role of preschool experience. Early Childhood Research Quarterly, 27, 66-76.

Seidman, E., Raza, M., Kim, S., & McCoy, J. M. (2014). Teacher Instructional Practices and Processes System (ECD) – TIPPS: Manual and scoring system. New York University

Shure, M.B., & Spivack, G. (1982). Interpersonal problem-solving in young children: a cognitive approach to prevention. American Journal of Community Psychology, 10, 341-356.

Sylva, K., Siraj-Blatchford, I., & Taggart, B. (2003). Assessing quality in the early years: Early Childhood Environment Rating Scales – Extension (ECERS-E): Four curricular subscales. London: Trentham.

Sylva, K., Siraj-Blatchford, I., Taggart, B., Sammons, P., Melhuish, E., Elliot, K., & Totsika, V. (2006). Capturing quality in early childhood through environmental rating scales. Early Childhood Research Quarterly, 21, 76–92. doi:10.1016/j.ecresq.2006.01.003

United Nations Children’s Fund (2017). State of the World’s Children. New York: UNICEF.

United Nations Development Program (2016). Human Development Report 2016. New York: United Nations. http://hdr.undp.org/en/composite/HDI (Accessed April 1, 2017)

UNESCO (2015). Sustainable development goal 4 and its targets. https://en.unesco.org/education2030-sdg4/targets. (Accessed February 18, 2019).

UNESCO Institute for Statistics (2017). More than one-half of children are not learning worldwide. http://uis.unesco.org/sites/default/files/documents/fs46-more-than-half-children-not-learning-en-2017.pdf. (Accessed February 18, 2019).

Wolf, S., Raza, M., Kim, S., Aber, J. L., Behrman, J., & Seidman, E. (2018). Measuring and predicting process quality in Ghanaian pre-primary classrooms using the Teacher Instructional Practices and Processes System (TIPPS). Early Childhood Research Quarterly, 45, 1-13. doi.org/10.1016/j.ecresq.2018.05.003

World Bank (2013). Zanzibar SABER Country Report. World Bank Group. 2013. Zanzibar Early Childhood Development: SABER Country Report 2013. Systems Approach for Better Education Results. World Bank, Washington, DC. https://openknowledge.worldbank.org/handle/10986/24485

World Health Organization (2009). WHO AnthroPlus for personal computers Manual: Software for assessing growth of the world's children and adolescents. Geneva: WHO, 2009.

Yoshikawa, H., Leyva, D., Snow, C.E., Treviño, E., Barata, M.C., Weiland, C., … Arbour, M.C. (2015). Experimental impacts of a teacher professional development program in Chile on preschool classroom quality and child outcomes. Developmental Psychology 51, 309-322. doi.org/10.1037/a0038785

Figure 1. TREND Flow diagram for Zanzibar; the flow is similar in Kenya and Uganda

Eligible Gov’t Primary Schools (n=81)

with Madrasa & non-Madrasa Pre-primaries feeding in to it

Select yoked non-Madrasas feeding into same primary n=36

Recruitment of Pre-Primaries

Randomly select Madrasas n=36

Phase 2 Assessment October 2016

Phase 1 Assessment October 2015

Observation and rating of quality

using the 50-item MELE

Early Grades Reading Assessment

Early Grades Math Assessment

Executive Function & Social tests

Family demographic information

Analysis

Completed (n=712; 91.5%)

Lost to follow-up

District (n=0)

Children (n=66; 8.5%): Absent (37), Death (0), Migration (14), Refusal (4), Missed (9), Not found (2)

Follow-up of Pre-primary Graduates 1 year later at the end of Grade 1

Randomly select graduates of observed non-Madrasas (5 from each of 36 primaries)

Randomly select graduates of observed Madrasas (5 from each of 36 primaries)

Quality data on pre-primaries (n =38 )

Grade 1 achievement of Madrasa graduates whose pre-primaries had quality data (n=185)

Quality data on pre-primaries (n =36 )

Grade 1 achievement of non-Madrasa graduates whose pre-primaries had quality data (n=185)

Table 1. Comparison of MELE quality by country and program model in three East African countries

MELE conceptual domains (number of items; alpha)

Prog

Model

Means (SD)

Statistical significance F (p)

Kenya

Uganda

Zanzibar

Country

Model

Country x

Model

Sample size (n)

M

nonM

41

37

38

27

38

36

Overall Average (50; .80)

Physical environment (10; .51)

M

nonM

M

nonM

2.78 (0.32)

2.56 (0.28)

2.75 (0.34)

2.77 (0.52)

2.59 (0.31)

2.35 (0.26)

3.00 (0.33)

3.04 (0.34)

2.58 (0.26)

2.45 (0.19)

3.17 (0.32)

3.06 (0.31)

10.57

(<.0001)

13.80

(<.0001)

28.04

(<.0001)

1.17

(.28)

1.11

(.33)

0.69

(.50)

Interaction (8; .49)

M

nonM

3.07 (0.39)

2.99 (0.45)

2.83 (0.41)

2.62 (0.48)

2.87 (0.35)

2.82 (0.33)

10.51

(<.0001)

3.52

(.06)

0.67

(.51)

Inclusiveness (6; .42)

M

nonM

2.74 (0.41)

2.77 (0.38)

2.40 (0.42)

2.40 (0.35)

2.26 (0.22)

2.28 (0.29)

38.05

(<.0001)

0.03

(.97)

0.01

(.96)

Program structure

(3; .37)

M

nonM

2.78 (0.75)

2.36 (0.73)

1.96 (0.66)

1.69 (0.55)

2.17 (0.63)

2.19 (0.52)

22.96

(<.0001)

4.35

(.03)

1.68

(.19)

Language and Literacy (5; .53)

M

nonM

2.80 (0.57)

2.29 (0.46)

2.52 (0.52)

2.48 (0.55)

2.57 (0.65)

2.21 (0.50)

1.25

(.29)

14.78

(.0002)

3.05

(.05)

Numbers and Numeracy (6; .68)

M

nonM

2.43 (0.58)

2.43 (0.52)

2.37 (0.66)

2.21 (0.55)

2.23 (0.73)

2.02 (0.71)

3.33

(.04)

1.09

(.30)

1.07

(.34)

Nature and Science

(2; .63)

M

nonM

2.51 (0.90)

2.45 (0.81)

2.16 (0.65)

1.94 (0.58)

2.30 (0.65)

2.37 (0.67)

5.97

(.003)

0.38

(.54)

0.66

(.52)

Group Activities: games, songs, art (4; .37)

M

nonM

2.95 (0.78)

2.53 (0.65)

2.71 (0.69)

2.41 (0.67)

2.55 (0.61)

2.46 (0.63)

2.88

(.06)

9.08

(.002)

1.38

(.25)

Free-choice Indoor Play (6; .82)

M

nonM

2.82 (0.84)

1.91 (0.82)

2.47 (0.89)

1.25 (0.34)

2.16 (0.56)

1.89 (0.54)

9.62

(.0001)

69.08

(<.0001)

8.71

(.0002)

Attendance

M

nonM

12.66 (8.4)

27.38 (11.7)

17.58 (11.0)

19.55 (15.4)

26.84 (10.5)

28.52 (10.6)

13.75

(<.0001)

21.50

(<.0001)

9.10

(.0002)

Note. Ranges of overall quality showed large overlaps: Kenya Madrasas 2.20 to 3.40, non-Madrasas 2.10 to 3.00; Uganda Madrasas 1.88 to 3.02, non-Madrasas 1.92 to 2.96; Zanzibar Madrasas 2.14 to 3.26, non-Madrasas 2.12 to 2.84.

Table 2. Country means (SD) and significance levels of family variables and primary school performance

Variable

(theoretic max score)

Kenya

Uganda

Zanzibar

ANOVA effects

Country Prog.Model

Number of children

Madrasa; nonMadrasa

n = 108; 173

n = 173; 150

n = 185; 185

F ( p )

F ( p )

Child’s age (6 – 9 yrs)

99.35 (9.44)

88.94 (9.51)

92.34 (7.29)

100.75 (<.001)

0.39 (.53)

Diet diversity (7)

2.95 (0.78)

2.90 (0.97)

3.67 (0.80)

79.94 (<.001)

1.44 (.23)

Preventive health (14)

11.08 (1.39)

9.97 (2.19)

11.32 (1.44)

8.82 (<.001)

2.73 (.10)

Family assets (12)

5.58 (2.20)

7.12 (2.65)

9.01 (2.55)

123.91 (<.001)

0.41 (.71)

Mother’s education

6.22 (3.31)

8.18 (3.62)

8.64 (3.79)

24.24 (<.001)

0.01 ( .92)

Father’s education

8.21 (3.34)

9.64 (4.55)

9.09 (4.16)

4.68 (.01)

0.11 (.74)

Primary student performance

Literacy (100)

81.56 (20.73)

46.07 (24.60)

49.31 (30.70)

115.14 (<.001)

2.44 (.12)

Math (100)

79.27 (11.44)

60.00 (22.02)

43.53 (25.14)

161.80 (<.001)

0.53 (.47)

Notes. Prog.Model= Madrasa; non-Madrasa pre-primary schools

Primary school performance of children was analyzed with covariates (age, sex, assets, mother’s education) and clusters.

Table 3. Mean (SD) performance scores of Primary 1 students graduating from Madrasa and non-Madrasa pre-primaries, adjusting for clusters and covariates (child’s age, mother’s education)

Measure (max score)

Program Model

Effect

Kenya primary analyses

Madrasa

(n = 108)

Non-Madrasa

(n = 173)

Group

Group x Sex

ICC

M (SD)

M (SD)

F (p)

F (p)

Literacy score (100)

0.19

82.07 (20.32)

81.24 (21.03)

0.00 (.99)

0.15 (.70)

Math score (100)

0.12

78.42 (13.03)

79.80 (10.33)

1.28 (.26)

3.13 (.08)

Tower planning (16)

0.006

7.93 (3.97)

8.51 (3.83)

0.09 (.76)

0.27 (.60)

Digit Span memory (12)

0.04

7.72 (2.98)

7.66 (2.73)

0.53 (.47)

0.35 (.55)

Social Problem solutions(9)

0.09

4.93 (1.96)

4.92 (1.91)

0.45 (.50)

2.07 (.15)

Uganda primary analyses

Madrasa

(n = 173)

Non-Madrasa

(n = 150)

Group

Group x Sex

ICC

M (SD)

M (SD)

F (p)

F (p)

Literacy score (100)

0.30

43.60 (22.96)

48.92 (26.15)

0.96 (.33)

1.98 (.16)

Math score (100)

0.13

58.80 (22.19)

61.37 (21.81)

0.30 (.59)

0.57 (.45)

Tower planning (16)

0.09

8.90 (4.81)

8.77 (5.01)

0.33 (.57)

0.82 (.37)

Digit Span memory (12)

0.16

6.81 (3.28)

7.03 (3.33)

0.48 (.49)

0.43 (.51)

Social Problem solutions(9)

0.08

3.73 (1.71)

3.81 (1.70)

0.00 (.99)

0.24 (.63)

Zanzibar primary analyses

Madrasa

(n = 185)

Non-Madrasa

(n = 185)

Group

Group x Sex

ICC

M (SD)

M (SD)

F (p)

F (p)

Literacy score (100)

.11

48.37 (30.90)

50.25 (30.56)

0.77 (.38)

0.03 (.32)

Math score (100)

.15

43.58 (25.05)

43.48 (24.48)

0.00 (.99)

0.85 (.35)

Tower planning (16)

.05

8.54 (4.70)

8.20 (4.63)

0.65 (.42)

1.52 (.22)

Digit Span memory (12)

.18

5.45 (3.67)

5.36 (3.67)

0.04 (.85)

0.18 (.67)

Social Problem solutions(9)

.07

3.89 (1.83)

4.37 (1.72)

5.02 (.03)

2.70 (.10)

Table 4. Multilevel linear regression predicting children’s primary 1 literacy and math from their pre-primary classroom quality, covarying child’s age and mother’s education, adjusted standardized estimates (2-tailed p value)

MELE Quality (items)

Primary 1 Students’ Outcome Scores

Kenya

Uganda

Zanzibar

Literacy

Math

Literacy

Math

Literacy

Math

Overall quality (50)

.50 (.004)

.23 (.27)

.20 (.18)

.35 (.05)

.16 (.36)

.10 (.57)

Interaction (9 items)

.08 (.70)

-.21 (.32)

.39 (.008)

.45 (.06)

.07 (.28)

.08 (.64)

Language (5 items)

.27 (.14)

.16 (.50)

.23 (.07)

.28 (.10)

.02 (.88)

-.02 (.90)

Numbers (6 items)

.32 (.16)

.56 (.02)

-.28 (.01)

-.22 (.28)

-.02 (.88)

.04 (.82)

Free-play (6 items)

.34 (.08)

.06 (.79)

-.12 (.34)

.16 (.42)

.14 (.36)

.21 (.19)

Note. Significant coefficients are bolded.

Kenya analyses are based on n = 55 pre-primary clusters and n = 259 children.

Uganda analyses are based on 52 pre-primary clusters and n = 289 children.

Zanzibar analyses are based on 69 pre-primary clusters and n = 355 children.

Table 5. Alpha coefficients for total scores and four domains

MELE Domain

Item number

All data

N = 217

Kenya data

n = 78

Uganda data

n = 65

Zanzibar data

n = 74

All items

1 - 50

.80

.82

.82

.73

Madrasas All items

.81

.83

.82

.79

Non-Madrasas All items

.72

.76

.77

.59

Interaction

11 - 18

.49

.56

.55

.41

Language

28 - 32

.53

.57

.57

.57

Numeracy

33 - 38

.68

.54

.64

.80

Free play

45 - 50

.82

.86

.86

.66