Synthesis of Findings from Field Data Collection Observing ...

THIS POSTER IS MADE POSSIBLE BY THE GENEROUS SUPPORT OF THE AMERICAN PEOPLE THROUGH THE UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT (USAID). THE CONTENTS ARE THE RESPONSIBILITY OF AMERICAN INSTITUTES FOR RESEARCH AND DO NOT NECESSARILY REFLECT THE VIEWS OF USAID OR THE UNITED STATES GOVERNMENT.

SYNTHESIS OF FINDINGS FROM FIELD DATA COLLECTION OBSERVING MOTHER TONGUE LANGUAGE INSTRUCTION USE OF FORMATIVE CONTINUOUS ASSESSMENT IN GRADES 1-4

May 2016

(DELETE THIS BLANK PAGE AFTER CREATING PDF. IT’S HERE TO MAKE FACING PAGES AND LEFT/RIGHT PAGE NUMBERS SEQUENCE CORRECTLY IN WORD. BE CAREFUL TO NOT DELETE THIS SECTION BREAK EITHER, UNTIL AFTER YOU HAVE GENERATED A FINAL PDF. IT WILL THROW OFF THE LEFT/RIGHT PAGE LAYOUT.

TABLE OF CONTENTS

INTRODUCTION 3LEVEL OF COMFORT WITH THE NEW CURRICULUM 3COMFORT AND KNOWLEDGE OF FORMATIVE CONTINUOUS ASSESSMENT 5SUPPORT NEEDS BY REGION 6CLASSROOM OBSERVATION FINDINGS 7CONCLUSION AND RECOMMENDATIONS 10APPENDIX 1. TEACHER INTERVIEW PROTOCOL 12APPENDIX 2. CLASSROOM OBSERVATION PROTOCOL 14

3

INTRODUCTION After conducting a literature review on best practices in Formative Continuous Assessment (FCA) and reviewing examples of FCA manuals from Ethiopia, READ M&E in collaboration with RSEB representatives, conducted teacher interviews (see Appendix 1 for interview tool) and classroom observations (see Appendix 2 for observation tool) from the 21-25 of March 2016. During this period, the team visited nine schools: three schools in Debre Birhan of Amhara region, three schools in Modjo and Bishoftu of Oromia region, and three schools in Mekele of Tigray region1. From these three regions, READ M&E staff interviewed 30 teachers and observed 29 class sessions. All interviewed and observed teachers had previously attended training on the newly developed mother tongue instructional materials facilitated by READ II or READ TA. The following synthesis of findings examines teachers’ knowledge, instructional practices, and reported needs regarding FCA in mother tongue language instruction in grades 1-4.

LEVEL OF COMFORT WITH THE NEW CURRICULUM The teachers interviewed indicated that that the ten-day training by READ II or READ TA was essential in helping them properly utilize the teacher guide and student textbook for instruction. Prior to training, the connections between the teacher guide and student text was not evident. For example, when teachers were directed to read a passage to the students from the teacher guide and then refer to the student textbook for the related questions this was initially confusing. Teachers also had difficulty understanding; as well as making the connection between the illustrations found in the student textbook and the related narration and explanations in the teacher guide. Teachers indicated that the training they received helped to clear up these issues. Although most teachers described the newly developed mother tongue instructional materials as ideal compared to the previous versions, there were some discrepancies and fundamental issues identified within the newly developed MT materials. The primary complaint by the first grade teachers interviewed was that the incoming first grade students had either not attended kindergarten, and had no exposure to the alphabet, or had attended kindergarten but were still not sufficiently familiar with the alphabet to continue to the next phase in reading skill development. Teachers indicated that the first grade textbook was too advanced for the level at which students enter first grade. The level of emergent literacy behaviors and skills displayed by the incoming first grade students varied greatly; some students had yet to make the basic connection that books have stories or even how to handle books, nor had they learned how to hold pens/pencils to begin scribbling.

1In line with confidentiality agreement between respondent teachers and READ M&E, the names of the specific schools observed are omitted from this report to ensure the anonymity of respondents.

Students with varying learning disabilities, needing significantly more support, also present a challenge for teachers of all grade levels to manage while completing the prescriptive lesson plans. Additionally, large class sizes and an insufficient number of textbooks as well as poor quality black and white print of teacher guides and student textbooks were identified as significant obstacles to instruction.

The lack of mastery of grade level reading and writing skills is an issue throughout grades 1-4. Teachers in Mekele of Tigray region felt that the 3rd and 4th grade content covered in the materials was too broad and difficult for the pupils. According to the interviewed teachers, the content of the materials was beyond the ability and capacity of their students. In Amhara and Oromia regions, due to social promotion policies, students do not repeat primary grades and automatically pass onto the next grade even if they have not mastered the content. Teachers, in Debre Birhan of Amhara region, report having to rush through and even skip certain sections of the daily lesson plans because they have to reteach skills students should have mastered in earlier grades or that the students find too difficult. Teachers reported rushing thorough or skipping altogether: practice words for blending and segmenting, time spent reading passages to students and asking questions, as well as opportunities for students to give oral presentations. Teachers in Modjo and Bishoftu of Oromia region reported that although they feel rushed to get through the content, they try not to skip content but instead either merge similar or repetitive content or take time away from other subjects.

Despite the above mentioned issues, several strengths were also identified in the mother tongue curriculum. Blending and segmenting techniques, listening comprehension passages, and the “I do, we do, you do” methodology were all well received by teachers. Many teachers

Implication for FCA Teachers could benefit from:

Assessment tools which help them to identify reading disabilities and specific formative assessment strategies and techniques for large classes

Implication for FCA In order to make the best use of valuable time, FCA could be seen as a

way to help teachers determine the key reading skills they need to focus on in each lesson so that they can prioritize the content they are teaching.

Implication for FCA First grade teachers could benefit from a diagnostic reading skills assessment toolkit to help them identify the specific skills in need of remedial attention and which provides specific activities, tools or strategies for remediation.

5

praised the incorporation of questions that require critical thinking and implicit knowledge beyond what can be answered by simply reading the given passages. For example, the questions prompting students to imagine that they are characters in the stories and to predict what course of action they would take or asking students to predict what will come next in a

given story which can be found in the teacher guide and student textbook.

COMFORT AND KNOWLEDGE OF FORMATIVE CONTINUOUS ASSESSMENT The interviewed teachers provided similar answers in explaining what they believe Formative Continuous Assessment (FCA) is, its purpose, and its use in the classroom. FCA was generally described by the interviewed teachers as follows: monitoring of students’ comprehension of the lesson on a continuous basis to inform instruction as well as to inform students of their performance and what areas they need to improve. Teachers reported that they used exams, quizzes, classwork, words-per-minute and homework as formal means of FCA; and classroom participation and oral presentations as informal means of FCA. Teachers did not clearly describe how or if they scored informal FCA. Teachers also reported that they scored formal FCA approximately once a week across the schools. Teachers also reported having difficulty teaching mixed-ability groupings. Teachers in Debre Birhan of Amhara region report this heterogeneous composition of students makes the teacher’s job more difficult because the wide range of performance of the students warrants further adapting of the lesson plan to meet the needs. Teachers felt like they did not have the training or the time to adapt adequately meet the needs of all students. Teachers in Modjo and Bishoftu of Oromia region stated that the 1:5 grouping helped students to converse in small groups, support each other, and enhanced competition among group members; although they also acknowledged observing bullying of lower performing students by other students. In Mekele of Tigray region, teachers reported in some cases that they do not seat their students in mixed groupings but instead place their high achieving students in the same groups and lower achieving students in separate groups. One of the interviewed teachers stated that this will help the student to study hard to “become one of the good achieving student”.

Implication for FCA In order to further support the development of critical thinking and high

level reading comprehension skills, FCA tools can focus on before, during, and after reading comprehension techniques and questioning

strategies that teachers can use throughout all their subjects.

Teachers, across schools, report assessing their students as frequently as daily. Informal assessments during instruction, such as walking around and observing students and asking questions to gauge comprehension of the content of the lesson, was cited as a daily undertaking. When teachers perceive, through informal assessment, that a certain portion of their students have not mastered the lesson content, some teachers reportedly reteach the lesson content and/or encourage fellow students to help those students who do not grasp the lesson content. For this reason, many teachers report that mixed-ability group seating is favorable so that above average performing students can assist the below average performing students.

SUPPORT NEEDS BY REGION Although the teachers in Debre Birhan of Amhara region report having received support from principals and supervisors in implementing the mother tongue curriculum and FCA, it is in varying degrees. Some teachers stated that the cluster supervisor offered infrequent support by observing classroom sessions and giving feedback on scoring of FCA or managing of students, where as other teachers stated that no support on FCA was provided by the principals and/or cluster supervisors.

When teachers in Modjo and Bishoftu of Oromia region were asked whether they received support from principals and supervisors, they reported that they did not receive any technical support; they further explained that principals and supervisors had not received the 10 day MT language training. Similarly, teachers in Mekele of Tigray region reported that supervisors and school principals provided supervision on a regular basis in the form of classroom observation but did not provide expertise related to FCA.

The interviewed teachers in Debre Birhan of Amhara region had difficulty articulating what types of materials/tools they would find useful for helping their students’ progress along the continuum of literacy development. One of the interviewed teachers was observed using improvised flashcards with fidels for demonstrating blending and segmenting of words, but when asked during an interview what materials she uses beyond the textbook and teacher guide, she did not report the improvised flashcards as an additional material. When probed by the interviewer if she found the improvised flashcards as a useful tool, she stated that she found it very useful. Although additional supplementary reading materials were reported as being available, the grade-level appropriateness could not be confirmed. When asked whether they had supplementary reading materials other than the curricular materials, most of the interviewed teachers in Modjo and Bishoftu of Oromia region reported that they had no additional reference materials to strengthen their teaching. Most of these teachers highlighted the following as tools they would find useful to teach the MT curriculum;

Implication for FCA Teachers could benefit from peer to peer FCA tools and

techniques which can easily be used by students in the mixed ability groupings of five particularly in grades 3 and 4.

7

• Flash cards with letters for teaching segmenting and blending of words, • Small black boards for children in grades 1 and 2 to practice writing, • Charts that show frequently appearing words in the textbooks, • Colored pictures of animals with their names, • Reference materials such as an Afan-Oromo dictionary, and an • FCA instructions

Teachers in Mekele of Tigray region explained that their students need supplementary reading materials. Most teachers reported the mini chalkboards they utilize in classrooms as very useful materials but indicated that they are not available in adequate number. Altogether the observed schools in the three regions have similar needs and available resources. They all need supplementary materials to support their teaching, additional training on incorporating FCA in their daily instruction, and FCA tools to help them identify skills to focus on.

CLASSROOM OBSERVATION FINDINGS The following section presents the summary of the observational study undertaken in a total of 30 mother tongue classroom sessions in grades 1- 4 across the select schools in Amhara, Oromia and Tigray regions. The observational instrument used is a checklist composed of a series of items that are related to the availability of mother tongue instructional materials, the use and type of formative assessment practices, and the interaction between teachers and their students. The main findings are presented below and an item level analysis is available upon request.

The teacher observation form collected qualitative data on how often and in which ways teachers clarify topics for the students. In four out of five classrooms, teachers were observed giving clarifications ‘often’ or ‘always’ during the instructional time as can be seen in Figure 1. The clarifications were given either by re-teaching the topic (33%) or explaining the idea in a different way (50%) (see Figure 2).

Figure 1: Level of clarification of the lesson

Figure 2: Type of clarification

37.90% 34.50%

17.20%10.30%

0.00%10.00%20.00%30.00%40.00%

Always Often Sometimes Never

Does the teacher attempt to clarify when the student(s) appear not to understand what is being taught?

Over 40% of the time teachers were observed reteaching the lesson in the exact same way when they noticed through informal formative assessment that students were not mastering the content. This is something that could definitely be addressed through training because chances are, if the pupils did not understand it the first time, they still won’t understand it the second time unless they are helped to think about it in a different way.

During observations, the method of assessment most often observed was Stimulus, Response and Evaluation (SRE). SRE is defined as a stimulus from the teacher, followed by a response from an individual student, and then the evaluation of the response by the teacher. Individual assessment was observed on average about 6 times per observation followed by paired assessment 3 times and group 2 times. In almost all cases teachers were following the gradual release model commonly known as: “I do, we do and you do” and the ‘you do’ part was used as a means of assessment. From a list of different assessment techniques, the most popular one was asking questions followed by calling on students to present their learning orally. Teachers were also observed informally checking students’ activities and no formal quizzes or tests were observed.

The classroom observations revealed that teachers were giving an equal chance for both active and passive students as well as girls and boys and the textbook to student ratio in most cases was 1:1 or 1:2. About 90 percent of teachers had the Teacher Guide.

There was evidence that teachers were reviewing students’ exercise books as evidenced by checkmarks (100%), grades (55%) and comments (40%) written in the books. In almost all cases teachers were following the prescribed lesson plan but in about 40% of cases they were unable to cover the lesson within the allotted time.

About 55% of classrooms have visual aids but they are not frequently used by the teachers. Visual aids observed were sometime old, sometimes related to other topics such as biology, or inappropriate to the level of the students. For example, a fourth grade classroom had a poster defining nouns and verbs in English whereas English is not taught in that school until fifth grade. Despite the limited resources, most teachers made an effort to make the classroom colorful and enjoyable as can be seen in the Selam Choro School photos below.

40.70%55.60%

7.40%0.00%20.00%40.00%60.00%

Reteach that part of the lessonin the same way?

Explain in a different way NA

How does the teacher clarify when the student(s) appear not to understand what is being taught?

9

Selam Choro School, Debre Birhan Selam Choro School, Debre Birhan

CONCLUSION AND RECOMMENDATIONS Most interviewed teachers had a general knowledge of FCA and demonstrated their understanding by continuously assessing students in the classroom. Although teachers claimed that they use various assessment techniques to assess their students, the classroom observation revealed that the dominant method of assessment was oral questioning/presentation.

Regarding support needed to enhance the FCA for Mother Tongue instruction, the interviewed teachers suggested that the following may be useful; flash cards with letters for teaching segmenting and blending of words, small chalk boards for children to practice writing, charts that show frequent words, and grade-level appropriate supplementary reading materials.

Based on teachers’ interviews and classroom observations the summary of findings and actionable recommendations for future interventions are presented below. Many of the findings lead to recommendations which are beyond the current scope of work for READ M&E but which are never the less important. Perhaps the recommendation may be addressed by READ M&E or other programs in the future. Recommendations that READ M&E intends to address through development of an FCA toolkit and training are highlighted in bold font (see Table 1 below).

Table 1. Findings and Recommendations

Findings Recommendation

Interviewed teachers stated that they had difficulty understanding and making a connection between illustrations found in the student textbook to related narration/ explanations in the teacher guide.

Provide continuous professional development training and follow up mentoring on identified thematic areas in order to properly implement the lessons. This is not in our current FCA plan but may be addressed in the future.

Most new first grade students had either not attended kindergarten, and had no exposure to the alphabet, or had attended kindergarten but were still not sufficiently familiar with the alphabet to continue to the next phase in reading skill development.

Prepare a diagnostic reading skills assessment toolkit and build the capacity of first grade teachers to help them identify the specific skills in need of remedial attention and which provides specific activities, tools or strategies for remediation. This is not in our current FCA plan but may be addressed in the future.

Students with varying learning styles, large class sizes, and an insufficient number of textbooks as well as poor quality black and white print of teacher guides and student textbooks were identified as significant obstacles to high quality instruction.

Prepare assessment tools which could help teachers identify students with learning disabilities and specific formative assessment strategies and techniques for large classes. This is not in our current FCA plan but may be addressed in the future.

Due to the broad scope of the content to be covered, poor capacity of teachers, and ability of students, compounded by the policy of free promotion of grades 1-3 students, many teachers rush through and even skip certain sections of the

In order to make the best use of valuable time, FCA could be seen as a way to help teachers determine the key reading skills they need to focus on in each lesson so that they can prioritize the content they are teaching.

11

Findings Recommendation

prescriptive daily lesson plans. Teachers report they have to re-teach skills student should have mastered in earlier grades or that the students find too difficult.

Specifically the following FCA materials are recommended by interviewed teachers. ü Flash cards with letters for assessing

segmenting and blending of words, ü Small chalk boards for children to

practice writing and reading, ü Charts that show high frequency words

in the respective language ü Grade level specific FCA

instructions/instruments Teachers have difficulty teaching critical thinking skills. Different questions that prompt students to predict what course of action they would take had they been the character(s) in the stories or what the students predict will come next in a given story found in the teacher’s guide and student textbook are regarded as best practices in developing critical thinking.

In order to further support the development of critical thinking and high level reading comprehension skills, FCA tools can focus on before, during, and after reading comprehension techniques and questioning strategies that teachers can use in promoting critical thinking, inference making and high-level comprehension.

Many teachers reported that mixed-ability group seating (1:5) is favorable so that above average performing students can assist the below average performing students.

Mixed ability grouping should be strengthened with peer to peer FCA tools and techniques which can easily be used by students in the mixed ability groupings of five particularly in grades three and four.

APPENDIX 1. TEACHER INTERVIEW PROTOCOL

FCA Teacher Needs Assessment Grades 1-4

Sample: • Teachers from 3 clusters in each region (Amhara, Oromia & Tigray),

o We will pre-select teachers who have completed the 10-day READ TA MT training and who have received the new MT materials (student books and teacher guides)

• 1 school per cluster o Focus on semi-urban schools (not in the big cities but also not very rural. Accessible

from the cities. • 4 teachers per school (1 per each grade – assuming there are 4 different teachers for each grade

– sometimes a teacher may teach multiple grades)

3 regions x 3 schools per region x 4 teachers per school (grade 1-4) = 36 total teachers

FCA Teacher Interview Tool Demographic Information:

• Date:____________________ • Observer name: ___________________________________ • Name of school: ___________________________________ • Cluster: ___________________________________ • Unique teacher ID: ___________________________________ • Grade (circle): Grade 1 Grade 2 Grade 3 Grade 4 • In what region (circle)? Amhara Oromia Tigray

Teacher Background Information:

• How many years have you been teaching? ______________________________ • What is your highest level of education? _______________________________ • What year did you graduate? ______________ • Are you teaching in your mother tongue (Circle)? Yes / No • Have you received any formal education in the teaching in your MT (Circle)? ? Yes / No

Place a check in each box once you feel that the respondent has answered the question. You may skip around or allow the respondent to move through different answers without following the exact order. Comfort with the new curriculum

¨ How easy or difficult are the teachers’ guides for you to follow and use in implementing your lessons & why?

¨ What do you like about the new textbooks and teachers’ guides? ¨ What are the biggest challenges you face in teaching this curriculum? ¨ Are you able to get through the full MT lesson in the allotted time period or do you feel rushed

to get through the full content of the lesson?

13

a. If you get through it all, do you feel you have enough time to get through the lesson and ensure that all students have grasped the material – i.e. also providing remediation activities as needed?

b. If you are rushed, why do you feel that is? What parts of the lesson do you usually skip?

Comfort and knowledge of FCA ¨ What does formative continuous assessment (FCA) mean to you? ¨ What do you see as the purpose of continually assessing students? ¨ How do you think assessment helps students? ¨ How do you think assessment helps you, the teacher? ¨ How does grouping students in mixed-ability groupings (1-5) help or hinder students’ learning? ¨ How do these groups of 5 play a role in how you assess your students? ¨ How often would you say you assess your students learning?

c. Multiple times per lesson d. Once per lesson e. Several times per week f. Once per week g. At the end of each unit h. Rarely i. Never

¨ What kind of method, if any, do you use to keep track of students’ progress or monitor their learning throughout the school year?

¨ What strategies do you use to find out if your students are learning the content of your lesson? ¨ What are some ways that you informally assess your students’ learning? ¨ What do you do when you determine that a certain portion of your students have not mastered

the lesson content? ¨ How do you identify your highest and lowest performing students and how does that affect your

teaching? ¨ What do you do to close the gap between lower preforming students and higher performing

students? ¨ How do you provide remediation to students who do not grasp the material?

Support Needs

¨ What are the areas you feel your students’ struggle with most in learning to read and write? ¨ What kind of support do you receive from your principal, supervisor or others around

implementing the MT curriculum or formative continuous assessment? ¨ What kinds of tools do you have to help you assess your students’ learning in relation to the

curriculum on a continual basis? ¨ What kind of learning materials (beyond the curriculum and textbooks) do you have, and what

are they? ¨ What kinds of tools would be most useful for you to help support your teaching of the MT

curriculum?

APPENDIX 2. CLASSROOM OBSERVATION PROTOCOL

Teacher Classroom Observation

*Interviewer should have a copy of the MT teacher guide and textbook during observation for reference

Demographic Information: Date: ____________________

Observer name: ___________________________________

Name of school: ___________________________________

Cluster: ___________________________________

Unique teacher ID: ___________________________________

Grade (circle): Grade 1 Grade 2 Grade 3 Grade 4

In what region (circle)? Amhara Oromia Tigray

1. What is the textbook to student ratio?

2. Does the teacher have a teacher guide in the classroom? ¨ Yes

¨ No

3. Are the students seated in groupings of 5? ¨ Yes

¨ No

4. Does the instructor attempt to clarify when the student(s) appear not to understand what is being taught?

¨ Always

¨ Often

¨ Sometimes

¨ Never

5. How does the teacher clarify when the student(s) appear not to understand what is being taught?

¨ Reteach that part of the lesson in the same way

¨ Explain in a different way

¨ N/A

6. Put a tic (e.g. IIII) below (individual/pair/group) every time you see a teacher assess students in one of the following ways.

Individual Pair Group

15

7. Does the teacher use the “You do” portion of the “I do, we do, you do” as an opportunity to assess students’ learning as evidenced by their correction of the learners?

¨ Yes ¨ No

8. Put a check next to each of the types of assessment activities you see:

¨ Teacher informally assesses students by walking around to look at their work. ¨ Teacher gives a quiz/test ¨ Teacher assigns individual classwork ¨ Teacher assigns group work ¨ Teacher asks questions ¨ Teacher asks students to indicate kinesthetically their understanding of the teaching (i.e. thumbs

up thumbs down, raising hands or fingers, standing up or sitting down, etc.) ¨ Teacher calls on students to present their learning (oral presentations, etc.)

9. Which students does the teacher call on throughout the lesson? ¨ The same few students ¨ Different students each

time

10. Does the teacher call on both boys and girls? ¨ Yes ¨ No

11. Are there any visual aids on the wall? ¨ Yes ¨ No

12. If there are visual aids on the wall, does the teacher refer to them or incorporate them into the lesson?

¨ Yes ¨ No ¨ N/A

13. Does the teacher use any other visual aids? ¨ Yes ¨ No

14. What other visual aids does the teacher use? Please describe the format, content and how the teacher uses the visual aid in the lesson (i.e. flashcards with fidels which the teacher used to teach the fidels and their sounds, or a big book with a story in MT which the teacher used to teach listening comprehension). Write N/A if not applicable.

15. Review a student’s notebook. Is there evidence that the teacher has viewed it?

¨ Yes

¨ No

16. What is the format of that evidence? ¨ Check mark

¨ Grade

¨ Comment

17. Does the teacher follow the prescribed lesson plan outlined in the Mother Tongue teacher’s guide?

¨ Yes

¨ No

18. Did the teacher complete the entire lesson within the designated time?

¨ Yes

¨ No

Other Comments: