Page 1
DEVELOPING A WRITING OBSERVATION PROTOCOL 1
Developing a Writing Instruction Observation Protocol:
Implications for Practice, Research, and Policy
Andrea Winokur Kotula and Cynthia Mata Aguilar
Education Development Center, Inc.
Terrence Tivnan
Harvard University Graduate School of Education
Page 2
DEVELOPING A WRITING OBSERVATION PROTOCOL 2
Suggested citation:
Kotula, A. W., Aguilar, C. M., & Tivnan, T. (2014). Developing a writing instruction
observation protocol: Implications for practice, research, and policy. Waltham, MA:
Education Development Center, Inc.
Copyright © 2014 by Education Development Center, Inc.
Education Development Center, Inc. is a global nonprofit organization that creates learning
opportunities for people around the world, empowering them to pursue healthier, more
productive lives. For more information, visit edc.org.
Page 3
DEVELOPING A WRITING OBSERVATION PROTOCOL 3
Abstract
The focus of this article is on the design of a Writing Instruction Observation Protocol to capture
the details of writing lessons; the components of effective writing instruction, as indicated by
research; the practices that teachers are using when they teach writing; and the effectiveness of
those practices, as documented by classroom observers using our protocol. This singular focus
on writing was essential, not only because it could inform our research but also because of the
Common Core State Standards’ focus on writing, and the potential need in the field for such a
protocol. Over a two-year period, we observed fourth-and fifth-grade writing instruction in 259
classrooms in 56 randomly selected treatment and control schools in six districts. What we
discovered in the classroom can guide teacher preparation programs, shape professional in-
service training, direct curriculum, influence policy, and ultimately support the development of
improved writing skills among students.
Page 4
DEVELOPING A WRITING OBSERVATION PROTOCOL 4
Developing a Writing Instruction Observation Protocol:
Implications for Practice, Research, and Policy
If today’s students expect to compete in a global economy, they must possess the skills to think
critically and creatively, solve problems, collaborate, and especially communicate (Partnership
for 21st Century Skills, 2011). “Writing has never been more important than in this digital age. It
is almost inconceivable to achieve academic success without good writing skills” (College Board
Advocacy and Policy Center, 2010, p. 2). Writing is essential in the business sector as well (cf.
Levy & Murnane, 2004; Murnane & Levy, 1996; National Governors Association Center for
Best Practices, 2005), with many arguing that along with being extremely important for daily job
performance (National Commission on Writing, 2003), good writing affects hiring and
promotions (ACT, 2005).
However, “despite the existence of solid models of how to proceed . . . writing is often
poorly taught. The reality is that a lot of writing instruction is perfunctory” (College Board’s
National Commission on Writing, 2006, p. 12). Teachers in grades 4-6 report that they devote
only about 1.25 hours per week on writing instruction; their students spend about two hours each
week writing. Moreover, these teachers indicate that they felt unprepared to teach writing after
graduating from their teacher preparation programs (Gilbert & Graham, 2010). While much has
been written about what constitutes “good” reading instruction, writing instruction has taken a
back seat. The National Reading Panel (National Institute of Child Health and Human
Development, 2000) has found more than 100,000 published studies about reading. In contrast,
the landmark meta-analysis of writing research by Graham and Perin (2007a, 2007b) identifies
Page 5
DEVELOPING A WRITING OBSERVATION PROTOCOL 5
approximately 600 documents on writing instruction for students in grades 4–12. In fact, writing
has been called “the neglected element of American school reform” (National Commission on
Writing, 2003, p. 9).
The 10 College and Career Readiness Anchor Standards for Writing, part of the Common
Core State Standards for English Language Arts & Literacy in History/Social Studies, Science,
and Technical Subjects (CCSS) (National Governors Association Center for Best Practices,
Council of Chief State School Officers, 2010), become increasingly challenging as students
progress in school. In the early grades, students begin to learn how to write opinion pieces,
informational/explanatory texts, and narratives. However, in grade six, for example, they must
write not only opinion pieces, but arguments that link a concluding statement to the argument
and that incorporate claims, logic, evidence, and credible sources. By the time they reach grades
eleven to twelve, students need to “write arguments to support claims in an analysis of
substantive topics or texts, using valid reasoning and relevant and sufficient evidence” (p. 45).
They are expected to develop counterclaims, include words, phrases, clauses, and syntax that
“link the major sections of the text, create cohesion, and clarify the relationships between
claim(s) and reasons, between reasons and evidence, and between claim(s) and counterclaims”
(p. 45). Writing under pressure adds another dimension to the task, and students must “have the
flexibility, concentration, and fluency to produce high-quality first draft text under a tight
deadline as well as the capacity to revisit and make improvements to a piece of writing over
multiple drafts when circumstances encourage or require it” (p. 41).
Student Writing Outcomes
While calls to improve student writing are not new, the stakes are far higher today than ever
before. However, across all grades, only about one-quarter of the students in our nation’s schools
Page 6
DEVELOPING A WRITING OBSERVATION PROTOCOL 6
are proficient in writing. The most recent National Assessment of Educational Progress (NAEP)
Writing Assessment at grades 8 and 12 (National Center for Education Statistics [NCES], 2012)
found that only 27% of students at both grades performed at or above the proficient level (“solid
academic performance”) in writing; just 54% of eighth graders and 52% of twelfth graders
performed at the basic level, and 20% of eighth graders and 21% of twelfth graders obtained
writing scores below this level.
The NAEP most recently assessed fourth-grade student writing in 2002. Findings from
this assessment revealed that only 28% of fourth graders obtained scores at or above the
proficient level. Fully 58% of fourth graders scored at the basic level—revealing only a “general
grasp” of the writing task—with 14% demonstrating writing skills below this level (Persky,
Daane, & Jin, 2003).
The writing of students among certain racial and ethnic groups suffers even more. In
2011, 34% of white eighth-grade students obtained scores at or above the proficient level, while
only 14% of Hispanic and 11% of black students did so; similarly, 35% of white twelfth-grade
students obtained scores at or above the proficient level compared to only 11% of Hispanic and
9% of black twelfth graders (NCES, 2012). There was also a significant gap in writing
performance among different racial and ethnic groups for fourth graders in 2002. Whereas 34%
of white students obtained scores at or above the proficient level, only 17% of Hispanic and 14%
of black students scored at those levels (Persky et al., 2003).
While one would expect student writing proficiency to improve over time, findings from
the NAEP (Persky et al., 2003) suggest that it does not. By twelfth grade, one in five students
still cannot compose texts without a significant number of grammar, spelling, and punctuation
errors (National Commission on Writing, 2003). Even among college-bound students, just over
Page 7
DEVELOPING A WRITING OBSERVATION PROTOCOL 7
one in three high school graduates (36%) cannot meet the readiness benchmarks set for college-
level writing (ACT, 2013).
Overview of Research
In response to the national student writing proficiency crisis, we engaged in a four-year research
project funded by the US Department of Education, Institute of Education Sciences, to study a
writing curriculum for fourth- and fifth-grade classrooms in six school districts. As part of that
study, we developed a Writing Instruction Observation Protocol (see Appendix) for classroom
observations specific to writing instruction. In this article, we detail our design of the protocol—
among the first such instruments in the field and the first that is based on research, was shown to
have moderate to high inter-rater reliability, and was used in a study with large numbers of
teachers to gauge its feasibility. We also examine one set of findings from our research: the
strategies that our study’s teachers used when they taught writing—as documented by classroom
observers using our protocol—and the effectiveness of those strategies, as determined by
research on the components of effective writing instruction. We examined the observations
across treatment conditions to capture a “snapshot” of current writing instruction in our study
schools. What we have discovered in the classroom, if validated in future studies, can guide
teacher preparation programs, shape professional in-service training, influence policy, direct
curriculum, and ultimately support the development of improved writing skills among students.
Method
Participants
We observed writing instruction of 259 teachers (203 fourth-grade classes and 56 fifth-grade
classes) in 56 randomly selected treatment and control schools in six school districts in
Massachusetts (see Table 1). Districts ranged in size from 13,373 students to 4,496 students.
Page 8
DEVELOPING A WRITING OBSERVATION PROTOCOL 8
Districts are classified by NCES locale codes according to their distance to urbanized areas. Five
of our study districts are considered “suburbs, large,” or a “territory outside a principal city and
inside an urbanized area with population of 250,000 or more.” The sixth district is considered a
“city, small,” or a “territory inside an urbanized area and inside a principal city with population
less than 100,000” (NCES, n.d.) All are Title I districts.
There was a high percentage of low-income students in the study (those eligible for free
or reduced lunch), 76%. Thirty-two percent received Limited English Proficiency (LEP) services
during the study or up to two years prior to it, and 17% were enrolled in special education during
either or both years of the implementation. Twenty-five percent of the students were White, 52%
Hispanic, and 6% Black. Teacher experience varied from beginning to veteran teachers with over
20 years of experience within and across schools and districts, and participating teachers taught
between one and five writing classes in their schools.
Procedure
The data collection for the study took place over two years. In Year 1, we studied the
students in fourth-grade classrooms. In Year 2, we studied a second cohort of fourth-grade
classrooms and followed the first cohort of students into their fifth-grade classrooms. We
observed all of the fourth-grade teachers who taught writing in Year 1 (2010—2011) in five
districts in Massachusetts, or 121 lessons, with a few exceptions because of scheduling or
availability issues.
In Year 2 of the implementation (2011–2012), one district dropped out of the study due to
competing interventions; thus, we observed teachers in four of the Year 1 districts engage a new
cohort of fourth-grade students in writing lessons, and we also observed fourth-grade teachers’
writing lessons in a new district that joined the study. In addition, we observed fifth grade
Page 9
DEVELOPING A WRITING OBSERVATION PROTOCOL 9
teachers. We observed all of the new fourth- and fifth-grade teachers of writing in Year 2 and
about 40% of the returning fourth-grade teachers. Of the 178 lessons observed in Year 2, 54 were
new fourth grade teachers, 96 were new fifth grade teachers, and 28 were returning fourth grade
teachers. We determined the order in which we observed teachers by randomizing within each
district within time blocks. If a teacher taught more than one section, we observed one of the
classes, making adjustments to the random order of observations for geographical constraints.
Five districts participated each year—or a total of six districts across the two years.
Measures
Development of an observation protocol for writing lessons. To capture the details of
the writing lessons, we needed a teacher observation protocol that focused specifically on writing
instruction. When it became apparent that no such protocol existed for our purposes, we created
our own. In the process of developing it, we needed to (1) identify the elements of “good”
writing instruction; (2) examine national and state writing standards, frameworks, and
assessments; and (3) examine other observation protocols for formatting suggestions.
Elements of good writing instruction. Over the past two decades, a series of meta-
analyses and additional research studies have found that teachers need to learn how to teach
writing effectively (Coker & Lewis, 2008; Graham & Perin, 2007a). Unfortunately, teachers and
curriculum designers tend to believe that writing instruction is easier than it is (Hillocks, 2008).
The research identified the following effective teaching practices that support positive writing
outcomes:
Providing explicit instruction as part of a systematic writing curriculum that uses a
process writing approach (Graham, McKeown, Kiuhara, & Harris, 2012; Graham &
Page 10
DEVELOPING A WRITING OBSERVATION PROTOCOL 10
Perin, 2007a, 2007b), maintaining a balance among materials, activities, learning tasks,
and teacher instruction (Hillocks, 2008)
Scaffolding students’ writing practices to help them while they are learning to be better
writers (Graham & Perin, 2007a, 2007b)
Teaching writing strategies for the writing process, including such stages as planning,
drafting, revising, and editing (Bui, Schumaker, & Deshler, 2006; Graham et al., 2012;
Graham & Perin, 2007a, 2007b; Hillocks, 1986). Providing comments about student
writing paired with an opportunity for revision. (Hillocks 1986 found that students gain
almost twice as much on measures of their writing as those who just receive comments
for the teacher.)
Building students’ capacity to evaluate their own writing using guides or scales (Hillocks,
1986)
Providing instruction of prewriting, summarization, and sentence combining skills
(Graham et al., 2012; Graham & Perin, 2007b; Hillocks, 1986).
Ensuring that students have time to study and emulate good models of writing (Graham
& Perin, 2007b)
Engaging students in inquiry activities (Graham & Perin, 2007b; Hillocks, 1986)
Helping students to set product goals (Graham et al., 2012; Graham & Perin, 2007b)
Teaching text structure, transcription (spelling, handwriting, keyboarding), and self-
regulation (e.g., goal setting, self-assessment); using imagery and creativity; assessing
student writing and providing feedback; and increasing time spent writing (Graham et al.,
2012)
Page 11
DEVELOPING A WRITING OBSERVATION PROTOCOL 11
Providing opportunities for peers to assist each other in collaborative writing (Graham et
al., 2012; Graham & Perin, 2007b). Engaging students in student-led small-group
discussions (Hillocks, 1986, 2008).
Using word processing equipment (Graham et al., 2012; Graham & Perin, 2007a, 2007b)
Including multiple purposes for writing, using varied genres, and reflecting real-world
writing needs (Coker & Lewis, 2008; Graham & Perin, 2007a, 2007b)
Focusing on the quality of the writing, not just on the lower-level skills of writing
mechanics or word count (Deatline-Buchman & Jitendra, 2006)
All of the above findings on the elements of good writing instruction informed our design of
items for our Writing Observation Protocol.
National and state standards, frameworks, and assessments. To ensure that our Writing
Instruction Observation Protocol reflected expectations for “good writing” in the fourth and fifth
grades as set by national and state standards and assessments, we compared the fourth-grade
NAEP writing objectives and scoring rubrics to the writing standards and scoring rubrics from
our study state’s English language arts curriculum the Massachusetts English Language Arts
Curriculum Framework. The NAEP is generally considered the gold standard writing
assessment. We also examined the scoring rubrics from the five states with the highest fourth-
grade NAEP writing scores in 2002.
We found a very close alignment between the Massachusetts standards and the NAEP
objectives. In both, students need to write for a variety of purposes; use both narrative and
informational formats; write for different audiences; make use of various stimulus materials;
employ different stages of a writing process that includes revision; demonstrate organization in
Page 12
DEVELOPING A WRITING OBSERVATION PROTOCOL 12
their writing; and apply standard English conventions (Massachusetts Department of Education,
2001; Persky et al., 2003).
Our examination of the NAEP and MCAS scoring rubrics indicated that the rubrics
disagree only on the role of spelling in good writing and writing instruction. The NAEP employs
one six-point holistic rubric, while MCAS uses a six-point rubric for “topic/idea development”
and a four-point rubric for “standard English conventions.” The NAEP includes spelling only as
part of conventions—which is just one element of the holistic score. MCAS does not assess
spelling on its rubric for “standard English conventions” and only considers spelling if those
errors interfere with communication. We did not include a spelling item in our observation
protocol.
We examined the writing scoring rubrics of the five states with the highest 2002 NAEP
writing scores for three reasons: to determine how they define good writing; to see if they share
skill foci with NAEP or MCAS, or both; and to ascertain if there are additional essential areas
that they assess. We found close alignment between the NAEP, the MCAS, and the writing
scoring rubrics of the five states—Connecticut, New York, North Carolina, Delaware, and
Massachusetts. (Note that Delaware assesses writing at the fifth—not fourth—grade level. Also,
New York does not assess writing from a stand-alone prompt. Fourth graders in New York
answer two extended-response questions: one each in the reading and listening sections.) The
writing scoring rubrics for the NAEP and the top-scoring states were perfectly aligned in the skill
areas of Organization, Details, Sentence Structure, Word Choice, Revision, and Editing. We
have already discussed the alignment with Massachusetts. The differences among the other states
are as follows:
Mechanics: not assessed in Connecticut and New York.
Page 13
DEVELOPING A WRITING OBSERVATION PROTOCOL 13
Spelling: only assessed in North Carolina.
Grammar and Usage: not assessed in Connecticut or New York.
Audience and Purpose: assessed in New York and Delaware.
Because of the close alignment of the scoring rubrics, we did not add items for mechanics,
spelling, grammar and usage, or audience and purpose to our protocol.
Sample classroom observation protocols. To inform the format of our Writing
Instruction Observation Protocol, we reviewed numerous classroom observation protocols to
identify their structure, categories, and rating systems. While the list is not comprehensive, a few
examples follow:
The Oregon Teacher Observation Protocol (OTOP) (Wainwright, Flick, & Morrel,
2003), which consists of 10 items that are scored holistically along a 1–4 scale, or N/O
when not observed. We found the use of a holistic scale especially helpful.
PEP Observation Protocol (Howard, Shoemaker, Peach, & Tomlinson, 2007), which is
an adaptation of the OTOP and adds two capsule description items that summarize the
lesson. We incorporated the check-off format of the capsules in some of our items.
English Language Observation Development Tool (Sapien-Melchor, 2006), which targets
English language development classes but provides rich detail in a check-off format.
Four Dimensions of Classroom Practice (Disston, 1997), which begins each item with
“How does the teacher,” a question that guided our team’s focus toward what both the
teacher and the student are doing during the lesson.
In summary, developing our Writing Instruction Observation Protocol was a recursive
process; we continuously cross-checked the sources to determine the appropriate placement and
wording of items. In addition, we added items the research literature identifies as good
Page 14
DEVELOPING A WRITING OBSERVATION PROTOCOL 14
instruction, although neither NAEP nor our study state measures them (e.g., planning, revising,
and providing feedback to students).
The observation protocol. Our Writing Instruction Observation Protocol includes a
series of questions and checklist items that represent relatively objective considerations about the
organization of the class, the way teachers explain lessons and interact with students, and the
types of interactions and feedback included in lessons. The observer’s task is to notice and record
these activities during writing lessons. The protocol includes the following four sections
(Sections 2 and 3 constitute the two primary blocks of the instrument):
Section 1—Introduction: This brief section focuses on the teachers, how they introduce
the lessons, and how set the stage for the day’s work. The section addresses three
questions fundamental to any lesson: (1) “What instruction is the teacher providing?”;
(2) “What are students doing during skill/strategy instruction/practice?”; and (3) “What
is the teacher doing while the students are working?”
Section 2—Skills/Strategies Instruction/Practice: Section 2 provides 14 questions about
what the teachers are doing during this phase of the lesson; what the students are doing
and how they are organized; and whether activities involve the whole class, small
groups, or individual students. If the lesson focuses on composing, and not on
“instruction/practice,” the observer has a space for checking off the section as “not
applicable.” The observer also notes whether or not the teacher gives students feedback
on their writing as part of the skills/strategies instruction/practice and, if so, the kinds of
feedback.
Section 3—Composing: Section 3 consists of 10 questions on the portion of the lesson
that involves student writing and composing. In this section, the observer notes how the
Page 15
DEVELOPING A WRITING OBSERVATION PROTOCOL 15
students are working as they compose and in what part of the composing process they
are engaged. As in Section 2, the observer has a space to check off the section as “not
applicable”—if the lesson focuses on instruction/practice instead of composing—and
space to document if the teacher provides feedback and, if so, what kinds of feedback.
Section 4—Closure: Section 4 contains miscellaneous questions about whether the
teacher provides closure to the lesson and assigns homework that involves writing.
Section 4 also asks the observer to document what other adults in the room, aside from
the teacher, are doing during the lesson, and provide space for the titles of other adults.
The protocol provides space at the end of the page for the observer’s comments.
All sections have subcategories for the observer to check off. Some sections record the
number of minutes spent on the task and include a category for “other.” Throughout, space is
included for observer comments as needed.
We designed the protocol so that an observer could easily record whether the teacher
provides instruction in skills or strategies, practice, or both and whether the lesson includes
composing. The protocol also enables the observer to determine what the teacher and students
are doing during different parts of the lesson. We recognized that some features of the questions
in the protocol are not appropriate or necessary to all lessons. For example, a writing session that
focuses entirely on completing a draft might not include an instructional component and, in that
case, guided practice would not be relevant. Similarly, it might be unnecessary to show an
example or model a skill or strategy for a lesson that is a continuation from a previous day.
Protocol rating scale. After completing Sections 1 through 4 and documenting the
details of the writing lesson, observers use the following scale to rate the effectiveness of nine
Page 16
DEVELOPING A WRITING OBSERVATION PROTOCOL 16
specific features of the lesson (e.g., “How effective was the lesson objective?” “How effective
was the feedback?”):
1 = not effective
2 = somewhat effective
3 = effective
4 = very effective
Observers indicate if they did not observe a category or if it did not apply to the lesson.
Then, to capture the effectiveness of the writing instruction, observers provide an overall rating,
which enables them to rank lessons along the same four-point Likert scale.
Observers were trained not to average the individual category ratings because some
categories are more essential to a specific lesson than others. For a number of reasons—different
lessons have different objectives, students vary in their knowledge and skills, some teachers use
a writing curriculum and others fashion their own, lessons may share characteristics and yet
differ notably in their overall effectiveness—the category ratings were to be used as guidelines
and the overall rating scored holistically. The observers provided the summary rating of the
overall effectiveness of the writing session using the same 4-point scale of effectiveness.
We relied on the observer’s overall rating rather than on an average of the nine individual
ratings for two reasons. First, an average of the nine ratings might not capture the variability
among lessons of each category’s importance. A second reason for relying on the observer’s
overall rating rather than on an average of the nine individual ratings emerged when we looked at
the results. As it turned out, the ratings of the nine categories of lesson effectiveness were highly
consistent with one another and also highly correlated with the overall rating. The internal
consistency reliability (Cronbach’s coefficient alpha) for a composite variable based on an
Page 17
DEVELOPING A WRITING OBSERVATION PROTOCOL 17
average of the nine ratings was .87, indicating that the lessons rated as more effective on one
dimension (e.g., effectiveness of the lesson objective) also tended to be rated as more effective
on other dimensions (e.g., feedback or classroom climate). In addition to this relatively high level
of internal consistency, the correlation between the average of these ratings and the observer’s
overall rating was .86 (p < .001); in other words, the observers’ ratings of overall effectiveness
were also consistent with their ratings based on the separate features of instruction. We found
both the overall summary rating and the ratings of separate features to be useful parts of the
observation protocol and believe they help the observers. Moreover, the separate ratings could be
valuable in enabling observers to examine specific features of the lessons. In principle, using an
average of several ratings provides an assessment of effectiveness that is potentially more fine-
grained, but the single overall rating offers a simple and easy-to-use assessment with virtually
identical results.
Observer training. During the initial implementation of the classroom observations,
observers received a full-day training on the use of the protocol conducted by the principal
investigator and site coordinator, both experienced literacy researchers. Because of the
importance of consensus on terminology, our research team created common definitions for
observers to apply to the following terms: “direct vs. indirect instruction,” “discussion,”
“modeling vs. examples (models),” “feedback,” “guided practice,” “lesson integrity,” and
“classroom climate.” For example, we defined “guided practice” as “a purposeful means by
which students can learn, practice, and/or apply new skills/strategies.” To be considered using a
guided practice, a teacher needs to be guiding the students—walking around and helping or
leading a whole-class or small-group practice. Reviewing a worksheet or homework was not
considered guided practice.
Page 18
DEVELOPING A WRITING OBSERVATION PROTOCOL 18
Inter-rater reliability. As part of the process of developing the observation protocol,
multiple observers viewed a sample of writing lessons at the beginning of each year so we could
assess the levels of rater agreement and reach an acceptable level of inter-rater reliability. A
member of our research team observed classrooms with the new observers, comparing and
discussing results after each observation. Two observers rated 34 lessons and three observers
rated 24 lessons for a total of 58 lessons. Each rated the lessons independently, with discussions
occurring only after the lessons and the ratings were completed. We randomly selected
classrooms in order to ensure a variety of settings in this phase of the study.
The observers tended to show high levels of agreement when rating the objective features
of the lessons—whether the teacher introduced the lesson (from 89 to 96% for the three
questions) or whether the instruction included feedback to students about their writing (91%), for
example. We examined the percentage of agreement, using Cohen’s kappa to adjust for
agreement “by chance” when analyzing the classroom activities variables. Sections 1-4 contained
30 questions that included some subquestions, resulting in a total of 39 items for analysis. For
these 39 items, the inter-rater agreement averaged more than 90% (mean = 91, median = 93, SD
= 5.3). We analyzed the effectiveness ratings by determining the percentage of agreement, the
correlations of the ratings, and the intraclass correlation. For the nine ratings that focused on the
effectiveness of the instruction, the inter-rater agreement was still adequate, with an average
percentage of exact agreement of just less than 80% (mean = 78, median = 78, SD = 5.9). For the
summary rating of the overall effectiveness of the lesson, the raters showed exact agreement of
82%, and the average correlation of the raters’ summary ratings was also adequate (r = .73).
Administering the protocol. Each year after we completed our inter-rater reliability
study, two observers conducted the remaining observations, one per classroom. Before we
Page 19
DEVELOPING A WRITING OBSERVATION PROTOCOL 19
conducted any visits, we sent a letter reviewing the procedures and assuring teachers and
principals of the confidentiality of the data we collected. To minimize scheduling conflicts, we
informed teachers, principals, and district liaisons one week prior to the proposed class visit. Our
observers did not share the protocol or provide feedback to the teachers.
Results and Discussion
Descriptive statistics and overall reliability information on the ratings of lesson
effectiveness are presented in Table 2. Observers rated most lessons as “effective” (about 72%
of all lessons), while they rated about 11% as “very effective” and 18% as “somewhat effective.”
Introducing the Lessons and “Skills/Strategies Instruction Practice”
Table 3 presents information on how frequently teachers—across treatment and control
schools—used a variety of instructional features while introducing the lessons and during the
“Skills/Strategies Instruction/Practice” portion of the lessons. This table indicates the overall
frequency of each feature (the “most popular” features are listed first) and also shows the relative
frequency of the features of the lessons rated as “very effective,” “effective,” and “somewhat
effective.” We created three sets of classrooms based on their effectiveness rating and compared
them with a series of cross tabulations. Chi-square tests of independence were used to identify
which differences were statistically significant.
Introducing the lessons. One of the significant findings was in the way that teachers set
the stage for the day’s work and introduced the writing lessons. In almost all cases, they provided
an explicit statement of the lesson objective(s) (in 91% of all of the lessons observed). However,
they did so in 100% of the lessons rated as “very effective,” but in only 82% of the lessons rated
as “somewhat effective.” Similarly, teachers linked a previous lesson to a current lesson in 96%
of the lessons rated as “very effective,” but in only 65% of the lessons rated as “somewhat
Page 20
DEVELOPING A WRITING OBSERVATION PROTOCOL 20
effective.” Activating prior knowledge was also a strategy found significantly more often in
lessons rated as “very effective.” We observed it in 74% of those rated “very effective,” but in
only 46% of the “somewhat effective” lessons. These three features of lessons—explicit
objectives, connections with earlier lessons, and activation of students’ prior knowledge—were
the clearest features that distinguished the lessons that were given the highest overall ratings
from lessons given ratings of only “somewhat effective.”
Skills and strategy instruction and/or practice. While significant variation appeared in
the portion of the lessons that focused on the instruction and practice of skills and strategies, the
most common approach emphasized direct skill or strategy instruction (96% of the lessons) and
practice (94%) during class. In many lessons, teachers also provided examples, asked open-
ended questions, explained how the skills would be helpful, and modeled the process of using the
skills. These approaches to explaining, questioning, providing models, and modeling, while
observed in 74%, 59%, 50%, and 47% of the lessons, respectively, were observed only slightly
more often in the highest-rated lessons; and the differences were statistically significant only in
the case of “providing examples of writing related to the lesson”; teachers provided examples in
89% of the lessons rated “very effective” compared to only 63% of the lessons rated as
“somewhat effective.” The use of discussion, small-group work, or computers (i.e.,
keyboarding), as identified by the literature as important elements of effective writing
instruction, was rare.
It was not common for teachers to engage in class discussions (14% in small groups; 8%
whole class).Teachers modeled the skill or strategy they were teaching in fewer than half of the
lessons (47%). Because this practice was observed only slightly more often in the highest-rated
lessons, our findings did not indicate a statistically significant difference on the effectiveness of
Page 21
DEVELOPING A WRITING OBSERVATION PROTOCOL 21
teacher modeling as an instructional practice. However, 89% of the lessons in which teachers
provided examples of the focal skill or strategy were rated “very effective,” strengthening the
evidence in favor of using that practice in writing instruction.
In summary, direct instruction was the predominate approach to teaching writing skills
or strategies, and teachers usually provided time to practice the relevant skills or strategies. They
often posed open-ended or probing questions and elicited student responses. In general, our
findings point to the effectiveness of three specific practices in teaching writing skills and
strategies: working directly with individual students, providing individualized feedback, and
providing examples of the skill or strategy that is the focus of the lesson.
Student activity. During the “Skills/Strategies Instruction/Practice” phase of the lesson,
students tended to work alone (72%) and write on paper (76%), often giving oral responses to
teacher questions (63%). They worked less frequently in small groups (13%) or with partners
(26%), and we rarely observed them using computers (1%).
Teacher activity. When students were engaged in seatwork, teachers generally circulated
around the classroom (76%), often stopping to engage with individual students (77%), and
addressed the whole class (70%). We found that when teachers worked with individual students,
the effectiveness rating of the instruction increased: 82% of the lessons rated “very effective”
included teachers working with individual students compared to 59% of the lessons rated
“somewhat effective.” It was much less likely for teachers to work with small groups of students
during the “Skills/Strategies Instruction/Practice” phase of the writing lessons (14%).
Feedback to students. During the “Skills/Strategies Instruction/Practice” phase of the
lessons, most teachers provided feedback to students about their writing (98%); this feedback
was mostly verbal (94%), sometimes given to individuals and sometimes to the whole class. We
Page 22
DEVELOPING A WRITING OBSERVATION PROTOCOL 22
saw the same pattern when teachers gave feedback to individual students as we did when they
worked with individuals. Teachers provided feedback to individual students in about 90% of the
lessons rated “effective” or “very effective” compared to 68% of the lessons rated “somewhat
effective.” Teachers very rarely gave written feedback or any formal grades to students during
this part of the lessons. In sum, our findings show that the effectiveness of lessons increased
when teachers both work individually with students and provide them with individualized
feedback.
Student composing and writing in class. In addition to lessons on writing skills and
strategies, most of the lessons that we observed also included time for students to write and
compose in class. Table 4 presents the results from this phase of the lessons in order of
frequency, along with the relative frequencies for the lessons rated as “very effective,”
“effective,” and “somewhat effective.” Students benefited from the same kinds of instruction in
this part of the lesson that effectively supported their efforts to learn skills and strategies. For
instance, 83% of the lessons rated “very effective” included teacher-provided examples of
writing that related to instruction compared to only 29% of the lessons rated as “somewhat
effective.” Interestingly, this was not a common strategy, occurring in less than one-third of the
lessons. Similarly, highly rated lessons were also more likely to have teachers actively model
writing as part of the lesson; 33% that were rated “very effective” compared to 7% that were
rated “somewhat effective,”
Student activity. Similar to what we saw when observing the skills and strategies portion
of the lessons, students almost always worked alone (98%) and wrote on paper (97%) during the
composing part of the lesson. We occasionally saw them working with partners (12%) or in
small groups (8%), but the predominant mode was for them to write on their own. The fact that
Page 23
DEVELOPING A WRITING OBSERVATION PROTOCOL 23
they almost always worked on their writing tasks alone during the composing part of the lessons
cannot be taken as indicative of any inadequacy in the instruction; writing is in and of itself a
solitary activity. However, given clear direction and rubrics, small-group work may well succeed
at any grade level, and discussion and collaborative writing have been noted in the literature as
important components of good writing instruction. Because of its limited use in our study (8%),
we can draw no conclusions about its effectiveness.
In the lessons that we observed, students generally wrote drafts (78%) about a specific
topic (76%) and occasionally engaged in the planning part of the writing process (23%) or in
revising earlier work (15%). We very rarely observed students using computers, tablets, and/or
laptops to compose (1%). Word processing is an essential writing practice, and it is a concern
that it was not observed in more lessons.
Teacher activity. Because the focus was on having students work on drafts, teachers tended to
circulate around the classroom (93%) and work with individual students (94%) during the
composing phase of the lessons. Teachers occasionally addressed the whole class (33%),
although this was less common while students were composing, and they sometimes worked
with small groups of students (13%).
Teacher feedback to students. Virtually all teachers provided some type of feedback on
writing (97%) to at least some of the students, almost always in the form of individual comments
given verbally (91%) as the teachers moved about the room. Much less frequently, teachers
provided whole-class feedback (34%). Written feedback was also rare (17%).
Sharing writing. Students frequently shared their writing (in 74% of the lessons). Sharing
usually involved individual students reading their compositions aloud to the whole class (68%);
only rarely did we see students sharing with partners (8%) or in small groups (3%).
Page 24
DEVELOPING A WRITING OBSERVATION PROTOCOL 24
Writing genres. In most cases (81%), students worked on imaginative or literary writing.
Writing in these genres was very common in the most effective lessons (95% of the “very
effective” lessons compared to 72% of the “somewhat effective” lessons). But many lessons also
focused on informational writing (43%), and over the course of the school year most classes
undertook a variety of writing tasks and topics, such as poetry, opinion pieces, or responding to
literature. For some lessons, teachers used a mixed approach in which some students wrote
informational reports while others composed fictional or imaginative pieces. The fact that
imaginative or literary writing was the most common mode of writing in the lessons we observed
suggests potential challenges for teachers and students alike as they come to terms with the
CCSS. While the CCSS were not fully implemented at the time of these observations, they had
been published, and our study state’s schools have agreed to adopt them. Although it should be
noted that the fourth-grade statewide composition test writing prompt was a narrative format, this
still does not prepare students for the more demanding genres that constitute the backbone of the
CCSS: the expectation that by sixth grade students will be able to write opinion and
informational pieces that establish a clear, supporting point of view. Students who are not
proficient in writing informational text will not be able to meet the new standards. Even more
importantly, they will not be prepared for higher education or the workplace, in which
informational text predominates.
Closure. About two-thirds (68%) of the lessons came to a formal end with the teachers
summarizing some of the key ideas that were covered. Teachers provided closure in 81% of the
lessons rated “very effective” compared to 56% of the lessons rated “somewhat effective.” The
assignment of writing instruction homework was very rare—in only 8% of the lessons we
observed. However, we are not aware of any research that demonstrates a relationship between
Page 25
DEVELOPING A WRITING OBSERVATION PROTOCOL 25
the assignment of homework and effective writing. Of that 8%, homework involving writing was
commonly used as a way to give more practice in specific skills (50%) and to provide practice
with other kinds of writing (31%).
Study Limitations
One limitation of this study is that we only observed each teacher once during the study, with the
exception of about 40% of the returning fourth grade teachers in Year 3. Consequently, we may
not have adequately captured typical teaching practices in all cases. Our hope is that because we
observed teachers in random order and because we observed so many lessons, we were able to
obtain a “snapshot” of all the instruction. However, we note that it is possible that we
overestimated or underestimated the prevalence of some of the teaching practices. A second
limitation is that the classroom observers were trained by the same research team that created the
observation protocol. Therefore, the observers rated the effectiveness of each lesson according to
the nine features that the team selected and deemed important and were influenced by the other
features on the protocol that they observed or did not observe. While we think that all the items
on the protocol are driven by the research literature on writing and effective instruction, we may
have inadvertently introduced error into the ratings by our own beliefs. Third, we are aware that
the relationships we found between the features of instruction and the effectiveness ratings were
determined through a series of cross tabulations and that these correlations do not necessarily
imply causality. Finally, although the items in our protocol are based on what the professional
literature identifies as good writing instruction, we suggest the need for an analysis of student
outcome data to explore the predictive validity of the protocol.
Implications for Practice, Policy, and Research
Page 26
DEVELOPING A WRITING OBSERVATION PROTOCOL 26
We created the Writing Instruction Observation Protocol to capture the details of lessons
that focus specifically on writing instruction in the intermediate grades. This singular focus on
writing was essential not only because of the needs of our research study but also because of the
importance of writing in general—both in school and in adult life—and because of the emphasis
on writing in the CCSS. Researchers, principals, writing coaches, and other teacher leaders need
such a protocol to evaluate and recommend practices, as do teachers themselves if they intend to
examine their own classroom effectiveness.
The Writing Observation Protocol makes it possible to accurately observe how teachers
engage in a very challenging task: helping students improve their writing. The NAEP results, the
multiple reports citing the urgency of improving writing instruction in our schools, and the new
rigor within the CCSS all compel us to examine our findings with an eye toward practice, future
research, and policy. The ability to write a personal narrative, once the centerpiece of many state
assessments at the fourth-grade level, is now not enough to demonstrate writing proficiency and
growth. Our project’s 259 classroom observations provide baseline data for the state of writing
instruction in today’s classrooms. Below are some implications and recommendations based on
our findings.
Practice. The following practices—which our observers rated as effective, based on the
findings from the literature that inform our protocol—should be included daily in writing
instruction:
Explicitly state the objective of the writing session, explain how the lesson connects to
previous instruction—even to instruction from prior years—and activate students’ prior
knowledge about the topic.
Page 27
DEVELOPING A WRITING OBSERVATION PROTOCOL 27
During the instruction or practice of skills or strategies, work with individual students and
provide feedback to them about their writing. Feedback needs to be focused, clear, and
timely. Written feedback can provide students with documentation to which they can
refer when reviewing their writing.
When students are composing, provide examples of the skill or strategy that is the focus
of the lesson or of the type of writing. Engage students in a discussion about the
challenges and successes they experience when using the examples to inform their own
writing.
Model how to compose the type of writing that is required of students. Help students
understand and talk about the different types of writing and the approaches to
implementing each of them.
We also recommend research-based practices that we did not observe or only observed
rarely:
Include more informational/expository writing instruction and practice.
Work in dyads or small groups, as collaboration has been found to be an effective
instructional practice.
Provide small-group or whole-class discussions in which students reflect on what their
peers and their teachers have said and build on the discussion with new comments.
Use computers and/or tablets in the composing process.
Research. Our findings indicate many implications for further research. Although there
was no significant difference among the various effectiveness ratings for many of the items on
our protocol, additional research with a larger and especially more varied sample is needed to
test this out. For example, how important is it to tell students how or why what they are learning
Page 28
DEVELOPING A WRITING OBSERVATION PROTOCOL 28
will help them as writers? As well, are lessons more effective when the teacher asks open-ended
and/or probing questions? How important is it for students to share their writing? Are lessons
more effective when students share their writing with the whole class than when they work with
partners or in small groups? Are discussions about writing in small groups more effective than
they are with the whole class? While the amount of instructional time for writing was not a focus
of our study, what exactly is the optimal amount of writing instruction and practice per day or
week?
Research has identified some effective practices. However, that research is scant. We
would like to see more research designed to discover additional effective practices, along with a
concerted effort by researchers to partner with school districts in randomized control trials to
determine which writing practices are truly effective. Moreover, some of the current decisions
about effectiveness have been based on a limited number of studies, several of which were of
poor quality. It is important to determine—in real classrooms—which combinations of practices
help students become better writers. Finally, while we identified several clusters of instructional
features, we do not know if any particular combination leads to more effective lessons. There
was no single pattern or approach that was consistently associated with higher observer ratings of
overall effectiveness. Are there particular clusters that help to improve student writing more than
other clusters? Additional research might tell us.
Policy. Several policy implications can be drawn from our findings. School
administrators need to review their current writing curricula to ensure that they reflect the CCSS
and the research on writing. By fourth grade, students are required to write opinion papers on
literature and expository texts; their curriculum needs to support the progressive development of
these skills. Thus may be important for administrators to examine the types of writing instruction
Page 29
DEVELOPING A WRITING OBSERVATION PROTOCOL 29
provided in the early grades to ensure it includes more than lessons on writing personal narrative.
It is also essential that districts and schools provide professional development that focuses on
building teachers’ knowledge and expertise in teaching writing, particularly in using the
practices that we (and others) have found to be effective. This training needs to be intensive and
ongoing, with mentoring by trained personnel.
To enable these changes, school leaders will need to create a culture of writing by
allowing staff to examine student writing, perhaps with a structured protocol. They might also
consider cultivating in-house teacher-leaders who can support all teachers in writing across the
curriculum. School leaders of all kinds could take advantage of the protocol used in this study to
examine and inform writing instruction in their classrooms.
Finally, to bring about significant change in teaching practices, colleges and universities
with teaching preparation programs will need to stress writing instruction, with an emphasis on
the CCSS and research on effective writing practices.
Page 30
DEVELOPING A WRITING OBSERVATION PROTOCOL 30
NOTE:
The research reported here was supported by the Institute of Education Sciences, US
Department of Education, through Grant Number R305A090479 to Education Development
Center, Inc. The opinions expressed are those of the authors and do not represent views of the
Institute or the US Department of Education.
Page 31
DEVELOPING A WRITING OBSERVATION PROTOCOL 31
References
ACT. (2005). Crisis at the core: Preparing all students for college and work. Washington, DC:
Author. Retrieved from http://www.act.org/research/policymakers/pdf/crisis_report.pdf
ACT. (2013). The condition of college and career readiness 2013. Washington, DC: Author.
Retrieved from http://www.act.org/research/policymakers/cccr13/index.html
Bui, Y. N., Schumaker, J. B., & Deshler, D. D. (2006). The effects of a strategic writing
program for students with and without learning disabilities in inclusive fifth grade classes.
Learning Disabilities Research & Practice, 21(4), 224–260.
Coker, D., & Lewis, W. (2008, spring). Beyond writing next: A discussion of writing research
and instructional uncertainty. Harvard Educational Review, 78(1), 231–251.
College Board Advocacy and Policy Center. (2010). Teachers are the center of education:
Writing, learning and leading in the digital age. New York: College Board.
College Board’s National Commission on the Writing. (2006). Writing and school reform.
New York: College Board.
Deatline-Buchman, A., & Jitendra, A. K. (2006, winter). Enhancing argumentative essay writing
of fourth-grade students with learning disabilities. Learning Disability Quarterly, 29, 39–54.
Disston, J. (1997). Seeing teaching in videos: Using an interpretive video framework to
broaden pre-service teacher development. (Master’s project) MACSME Program,
University of California, Berkeley.
Gilbert, G., & Graham, S. (2010). Teaching writing to elementary students in grades 4–6: A
national survey. The Elementary School Journal, 110(4), 494-518.
Page 32
DEVELOPING A WRITING OBSERVATION PROTOCOL 32
Graham, S., McKeown, D., Kiuhara, S., & Harris. K. R. (2012). A meta-analysis of writing
instruction for students in the elementary grades. Journal of Educational Psychology,
104(4), 879–896.
Graham, S., & Perin, D. (2007a). What we know, what we still need to know: Teaching
adolescents to write. Scientific Studies of Reading, 11(4), 313–335.
Graham, S., & Perin, D. (2007b). Writing next: Effective strategies to improve writing of
adolescents in middle and high schools – A report to Carnegie Corporation of New York.
Washington, DC: Alliance for Excellent Education.
Hillocks, G., Jr. (1986). Research on written composition: New directions for teaching.
Urbana, IL: ERIC Clearinghouse on Reading and Communication Skills and the National
Conference on Research in English.
Hillocks, G., Jr. (2008). Writing in secondary schools. In C. Bazerman (Ed.), Handbook of
research in writing: History, society, school, individual, text (pp. 311–329). New York:
Lawrence Erlbaum.
Howard, M., Shoemaker, B., Peach, H., & Tomlinson, D. (2007). PEP observation protocol.
Appalachian Mathematics and Science Partnership, Partnership Enhancement Program.
Retrieved from http://www2.research.uky.edu/amsp/pub/
Sharepoint%20Toolkit%20Documents/OTOP%20adapted%20for%20PEP%202.doc
Levy, F., & Murnane, R. (2004). The new division of labor: How computers are creating the
next job market. Princeton, NJ: Princeton University Press.
Massachusetts Department of Education. (2001). Massachusetts English language arts
curriculum framework. Malden, MA: Author.
Page 33
DEVELOPING A WRITING OBSERVATION PROTOCOL 33
Murnane, R., & Levy, F. (1996). Teaching the new basic skills: Principles for educating
children to thrive in a changing economy. New York: Free Press.
National Center for Education Statistics. (n.d.). Common core of data: Identification of rural
locales. Retrieved from http://nces.ed.gov/ccd/rural_locales.asp
National Center for Education Statistics. (2012). The Nation's Report Card: Writing 2011
(NCES 2012–470). Washington, DC: U.S. Department of Education, Institute of Education
Sciences. Retrieved from http://nces.ed.gov/nationsreportcard/pdf/main2011/2012470.pdf
National Commission on Writing. (2003). The Neglected R: The need for a writing revolution.
Retrieved from http://www.host-collegeboard.com/advocacy/writing/publications.html
National Governors Association Center for Best Practices. (2005). Reading to achieve: A
governor’s guide to adolescent literacy. Retrieved from http://www.nga.org/cms/home/nga-
center-for-best-practices/center-publications/page-edu-publications/col2-content/main-
content-list/reading-to-achieve-a-governors-g.html
National Governors Association Center for Best Practices, Council of Chief State School
Officers. (2010). Common core state standards for English language arts & literacy in
history/social studies, science, and technical Subjects. Washington DC: Author.
National Institute of Child Health and Human Development (NICHD). (2000). Teaching
children to read: An evidence-based assessment of the scientific research literature on
reading and its implications for reading instruction: Reports of the subgroups (NIH
Publication No. 00-4754). Washington, DC: U.S. Government Printing Office.
Partnership for 21st Century Skills. (2011). Framework for 21
st century learning. Washington,
DC: Author. Retrieved from http://www.p21.org/overview/skills-framework
Page 34
DEVELOPING A WRITING OBSERVATION PROTOCOL 34
Persky, H. R., Daane, M. C., & Jin, Y. (2003). The nation’s report card: Writing 2002 (NCES
2003-529). U.S. Department of Education. Institute of Education Sciences. National Center
for Education Statistics. Washington, DC: Government Printing Office.
Sapien-Melchor, R. (2006, January). English language development observation tool. San
Diego County Office of Education. Retrieved from http://www.sdcoe.net/lret2/eld/pdf/
ELDobservationtool.pdf
Wainwright, C. L., Flick, L., & Morrel, P. (2003). The development of instruments for
assessment of instructional practices in standards-based teaching. Journal of Mathematics
and Science: Collaborative Explorations, 6, 21–46.
Page 35
Table 1
Schools and Classroom Sample for the Observations of
Writing Instruction Lessons
Frequency Percent
Teachers observed
259
Grade level 4 203 78.4
5 56 21.6
School district A 5 schools
B 9 schools
C 15 schools
D 14 schools
E 7 schools
F 6 schools
Total 56 schools
School year 2010–2011 119 45.9
2011–2012 140 54.1
Page 36
DEVELOPING A WRITING OBSERVATION PROTOCOL 36
Table 2
Descriptive Statistics for the Summary Ratings of the
Effectiveness of the Lessons
Rated item* Observed
n of
Lessons**
Mean SD Min Max
1. Lesson objective 231 2.88 .54 1 4
2. Activation of prior
knowledge
219 2.81 .60 1 4
3. Skill/strategy
instruction
232 2.07 .60 1 4
4. Models or modeling 196 2.92 .69 1 4
5. Guided practice 224 2.94 .60 1 4
6. Feedback 250 2.93 .52 1 4
7. Academic challenge 255 2.98 .44 2 4
8. Lesson integrity 257 2.93 .68 1 4
9. Classroom climate 257 3.04 .74 1 4
Composite (mean of all
nine ratings) a
258 2.92 .42 1.4 4.0
Single item overall rating
of effectiveness b
257 2.93 .53 2 4
*Rating scale for each item: 1 = not effective; 2 = somewhat effective; 3 = effective; 4 =
very effective.
** Observed number of lessons varies slightly because certain ratings were not applicable
for some lessons. The composite rating was based on the mean of the relevant ratings.
a. Cronbach’s alpha (internal-consistency reliability) for composite scale = 0.87.
b. Correlation of composite rating and single-item rating = 0.86, p < .0001.
Page 37
DEVELOPING A WRITING OBSERVATION PROTOCOL 37
Table 3
Features of Writing Instruction during the Introduction and the
Skills and Strategies Phase of the Lessons
Percentage of Lessons in Which Feature Was Noted
Phase of
the
Writing
Lesson
Observed Variable Across All
Lessons
Lessons
Rated
Somewhat
Effective
Lessons
Rated
Effective
Lessons
Rated
Very
Effective
Introducing the Writing Lesson
Teacher explicitly stated
the writing session’s
objective
91 82 92 100 *
Teacher explained how
previous lessons relate
to today’s writing
session
80 65 80 96 *
Teacher activated prior
knowledge relevant to
today’s writing session
63 46 66 74 *
Skills and Strategies During
Instruction/Practice
Teacher provided direct
skill/strategy instruction
96 90 97 96
Teacher provided
students with in-class
time to practice the
focus skills/strategies.
94 95 93 100
Teacher provided
example of writing
related to the instruction
and/or practice.
74 63 75 89 *
Teacher asked open-
ended and/or probing
questions.
59 65 56 78
Page 38
DEVELOPING A WRITING OBSERVATION PROTOCOL 38
Table 3
Features of Writing Instruction during the Introduction and the
Skills and Strategies Phase of the Lessons
Teacher explained how
or why skills/ strategies/
processes will help
students as writers (the
“why” of instruction).
50 39 51 58
Teacher modeled the
process of using focus
skills/ strategies.
47 48 46 58
Teacher provided
indirect skill/strategy
instruction.
19 21 20 19
Teacher engaged
students in small group
discussion.
14 20 11 23
Teacher adjusted whole
class and/or small group
skill/strategy instruction
and/or practice based on
observations of
perceived student needs.
12 5 14 11
Teacher engaged
students in whole class
discussion.
8 15 5 19
What Were Students Doing?
Student Activities During
Instruction
Students worked on
paper.
76 64 79 70
Students worked alone. 72 56 76 67 ~
Students gave oral
responses.
63 73 60 70
Students worked with
class.
60 64 59 67
Students wrote single
words or phrases.
37 33 38 41
Students wrote single
sentences (not
connected).
27 20 31 19
Students marked text. 27 30 27 19
Page 39
DEVELOPING A WRITING OBSERVATION PROTOCOL 39
Table 3
Features of Writing Instruction during the Introduction and the
Skills and Strategies Phase of the Lessons
Students worked with a
partner.
26 18 26 33
Students wrote more
than one sentence
(connected text).
25 15 28 19
Students worked with
small group.
13 23 11 15
Students worked in
“other” way
12 10 12 26
Students worked on
computer.
1 0 1 4
What Was the Teacher Doing?
During student work
time, the teacher
worked with
individuals.
77 59 80 82 *
During student work
time, the teacher
circulated around the
room.
76 62 79 78 ~
During student work
time, the teacher
addressed whole class.
70 72 68 74
During student work
time, the teacher
worked with small
groups.
14 18 14 7
During student work
time, the teacher did
“other.”
4 5 4 4
Teacher Feedback to Students
During the Lessons
Teacher gave students
feedback on their
writing or oral
responses.
98 100 98 96
If teacher gave 96 97 95 96
Page 40
DEVELOPING A WRITING OBSERVATION PROTOCOL 40
Table 3
Features of Writing Instruction during the Introduction and the
Skills and Strategies Phase of the Lessons
feedback, was the type
verbal?
If teacher gave
feedback, was the type
in comments?
94 92 95 96
If teacher gave
feedback, was format
individual?
87 68 91 89 *
If teacher gave
feedback, was format
whole class?
72 76 69 82
If teacher gave
feedback, was format
small group?
14 21 15 4 ~
If teacher gave
feedback, was the type
in writing?
7 5 9 4
If teacher gave
feedback, was the type
in ‘other’ (explain)?
3 8 2 0
If teacher gave
feedback, was the type
in grades?
0 0 0 0
If teacher gave
feedback, was the type
in rubrics?
0 0 0 0
~ p < .10.
* p < .05, comparisons across the lessons rated somewhat effective, effective, and very
effective.
Page 41
DEVELOPING A WRITING OBSERVATION PROTOCOL 41
Table 4
Features of Writing Instruction during the Composing Phase of the Lessons
Percentage of Lessons in Which Feature Was Noted
Phase of
the
Writing
Lesson
Observed Variable Across
All
Lessons
Lessons
Rated
Somewhat
Effective
Lessons
Rated
Effective
Lessons
Rated
Very
Effective
Composing and writing
instruction: Student composing
and writing in class
Teacher provided
example of writing
related to the
composition lesson.
31 29 29 83 *
Teacher explained how
or why
skills/strategies/processes
will help students as
writers (the why of
instruction)?
28 38 22 40
Teacher modeled
relevant process of
writing.
18 7 22 33 ~
Teacher adjusted whole
class and/or small group
instruction based on
observations of perceived
student needs.
12 15 12 6
Students during the composing
time
Students worked alone. 98 95 98 100
Students worked on
paper.
97 95 97 100
Students worked with a
partner.
12 13 13 5
Students worked with
small group.
8 5 7 20
Students worked with 7 5 8 5
Page 42
DEVELOPING A WRITING OBSERVATION PROTOCOL 42
Table 4
Features of Writing Instruction during the Composing Phase of the Lessons
class.
Students worked on
computer.
1 0 2 0
Students worked in
‘other’ way (explain).
1 3 1 0
Student tasks during composing
time
Students wrote to a topic. 78 80 80 57 ~
Students drafted
composition.
76 80 77 62
Students
planned/organized
composition.
23 28 22 30
Students revised
composition.
15 20 12 24
Students edited
composition.
6 10 6 5
Students gave or received
feedback on composition.
3 3 3 5
Students completed final
copy.
3 3 4 5
Students reflected on the
what-why-how of the
lesson.
1 0 2 0
Students freewrote.
0 0 1 0
Teachers during the composing
time
During student
composition time, the
teacher worked with
individuals.
94 95 94 91
During student
composition time, the
teacher circulated around
the room.
93 93 94 91
During student
composition time, the
teacher addressed whole
33 43 32 24
Page 43
DEVELOPING A WRITING OBSERVATION PROTOCOL 43
Table 4
Features of Writing Instruction during the Composing Phase of the Lessons
class.
During student
composition time, the
teacher worked with
small groups.
13 13 12 19
During student
composition time, the
teacher did ‘other’
(explain).
2 0 4 0
Providing feedback on writing
Teacher gave students
feedback on their writing.
97 98 97 95
If teacher gave feedback,
was format individual?
94 93 95 91
If teacher gave feedback,
was the type verbal?
91 93 90 91
If teacher gave feedback,
was the type in
comments?
90 88 91 90
If teacher gave feedback,
was format whole class?
34 45 30 43
If teacher gave feedback,
was the type in writing?
17 18 17 19
If teacher gave feedback,
was format small group?
12 10 12 19
If teacher gave feedback,
was the type in ‘other’
(explain)?
1 2 0 5
If teacher gave feedback,
was the type in grades?
0 0 0 0
If teacher gave feedback,
was the type in rubrics?
0 2 0 0
Sharing of writing
Teacher had students
share their writing.
74 67 75 82
Teacher had students
share their writing with
the whole class.
68 62 68 77
Page 44
DEVELOPING A WRITING OBSERVATION PROTOCOL 44
Table 4
Features of Writing Instruction during the Composing Phase of the Lessons
Teacher had students
share their writing with a
partner.
8 8 9 5
Teacher had students
share their writing with a
small group.
3 0 4 4
Writing genres
Students are working in
imaginative/literary
genre.
81 72 81 95 ~
Students are working in
informational/expository
genre.
43 48 43 25
End of lesson steps
Teacher provided closure
to lesson.
68 56 69 81 ~
If teacher assigned
homework that involved
writing, the purpose was
to practice the focus of
targeted instruction.
50 60 50 0
If teacher assigned
homework that involved
writing, the purpose was
to engage in other kinds
of writing.
31 40 30 0
Teacher assigned
homework that involved
writing.
8 12 7 4
~ p < .10.
* p < .05, comparisons across the lessons rated somewhat effective, effective, and very
effective.
Page 45
Appendix
Writing Instruction Observation Protocol
All observation questions refer to the writing instruction
Introduction
Does the Teacher:
1. Explicitly state, verbally, the writing session’s objective? Yes No
2. Explicitly explain how previous lessons relate to today’s writing session? Yes No
3. Activate prior knowledge relevant to today’s writing session? Yes No
Skills/Strategies Instruction/Practice
(___ minutes) __N/A
Does the Teacher:
4. Provide direct skill/strategy instruction? Yes No
5. Provide indirect skill/strategy instruction? Yes No
6. Provide an example of writing related to the instruction and/or practice? Yes No
7. Model the process of using focus skills/strategies? Yes No
8. Explain how or why skills/strategies/processes will help students as writers (the why of
instruction)? Yes No
Date ________________________ District ________________ School _________________
Teacher _____________________ Grade _________________ Observer _______________
Topic ______________________________
Length of Lesson: ___ Minutes ___ New Lesson ___ Continued Lesson
Page 46
DEVELOPING A WRITING OBSERVATION PROTOCOL 46
9. Ask open-ended and/or probing questions? Yes No
10. Engage the students in whole class discussion? Yes No
11. Engage the students in small group discussion? Yes No
12. Provide students with time, in class, to practice the focus skills/strategies? Yes No
13. Adjust whole class and/or small group skill/strategy instruction and/or practice based on
observations of perceived student needs? Yes No
14. What are the students doing during skill/strategy instruction/practice?
____ writing single words or phrases
____ writing single sentences (not connected)
____ writing more than one sentence (connected text)
____ marking text
____ oral responses
15. As part of the skills/strategies instruction and/or practice, students are working:
(Check all that apply)
___alone ___with a partner ___with a small group ___with the class
___ on paper ___ on computer ___ other (explain):________________________
16. What does the teacher do while students are working? (Check all that apply.)
___ circulates around the room
___ works with individuals
___ works with small groups
___ addresses whole class
___ other (explain): _____________________________________________________________
Page 47
DEVELOPING A WRITING OBSERVATION PROTOCOL 47
17. Does the teacher give students feedback on their writing or oral responses as part of the
skills/strategies instruction and/or practice? Yes No
If yes (check all that apply):
Format Types of Feedback
___ individually ___ comments ___ written ___ verbal
___ small group(s) ___ grades
___ whole class ___ rubrics ___ other (explain): ________________
Composing
(____minutes) ___N/A
Does the Teacher:
18. Provide an example of writing related to the lesson on composition? Yes No __ N/A
19. Model the relevant process of writing? Yes No __ N/A
20. Explain how or why skills/strategies/processes will help students as writers (the why of
instruction)? Yes No __ N/A
21. Adjust whole class and/or small group instruction based on observations of perceived student
needs? Yes No __ N/A
22. Have students working (Check all that apply)
___alone ___with a partner ___with small group ___with class
___ on paper ___ on computer ____other (explain): ______________________
23. In what part of the composing process are students engaged? (Check all that apply)
___planning/organizing ___drafting ___revising ___editing
___providing and/or receiving feedback ___ completing final copy
___ writing to a topic ___ freewriting ___ reflecting on what-why-how of lesson
Page 48
DEVELOPING A WRITING OBSERVATION PROTOCOL 48
24. What does the teacher do while students are composing?
___ circulates around the room ___ works with individuals
___ works with small groups ___ addresses the whole class
___ other (explain): _____________________________________________
25. Does the teacher give students feedback on their writing? Yes No
If yes (check all that apply:
Format Types of Feedback
___ individually ___ comments ___ written ___ verbal
___ small group ___ grades
___ whole class ___ rubrics ___ other (explain):
__________________________________
26. Does the teacher have students share their writing? Yes No
If yes (check all that apply):
____with a partner
____with small group
____whole class
27. What genre of writing are students working on today?
___ imaginative/literary type: _______________________________________
___ informational/expository type: _______________________________________
Miscellaneous
28. Does the teacher provide closure to the lesson? Yes No
Page 49
DEVELOPING A WRITING OBSERVATION PROTOCOL 49
29. Does the teacher assign homework that involves writing? Yes No
If yes, the purpose of the homework is
___ to practice the focus of targeted instruction
___ to engage in other kinds of writing
30. What are other adults in the room besides the teacher doing during the lesson?
______________________ __________________________________________________________
Title
______________________ __________________________________________________________
Title
______________________ __________________________________________________________
Title
Additional comments:
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
Page 50
DEVELOPING A WRITING OBSERVATION PROTOCOL 50
31. Overall, how would you rate the effectiveness of this writing session? 1 2 3 4
Categories* 0=Not
Observed
1=Not
Effective
2=Somewhat
Effective
3=Effective
4=Very
Effective
N/A
Lesson Objective 0 1 2 3 4
Activation/Use of Prior
Knowledge
0 1 2 3 4
Direct Skill/Strategy
Instruction
0 1 2 3 4
Models and/or
Modeling
0 1 2 3 4
Guided Practice 0 1 2 3 4
Feedback 0 1 2 3 4
Academic Challenge 0 1 2 3 4
Lesson Integrity 0 1 2 3 4
Classroom Climate 0 1 2 3 4
* These categories were developed primarily from our observation protocol. Additional guidance was
derived from:
Disston, J. (1997). Seeing teaching in videos: Using an interpretive video framework to broaden pre-
service teacher development. (Master’s project) MACSME Program, University of California,
Berkeley.
Page 51
DEVELOPING A WRITING OBSERVATION PROTOCOL 51
Howard, M., Shoemaker, B., Peach, H., & Tomlinson, D. (2007. PEP observation protocol.
Appalachian Mathematics and Science Partnership, Partnership Enhancement Program. Retrieved
from http://www2.research.uky.edu/amsp/pub/
Sharepoint%20Toolkit%20Documents/OTOP%20adapted%20for%20PEP%202.doc
Sapien-Melchor, R. (2006, January). English language development observation tool. San Diego
County Office of Education. Retrieved from http://www.sdcoe.net/lret2/eld/pdf/
ELDobservationtool.pdf
Wainwright, C. L., Flick, L., & Morrel, P. (2003). The development of instruments for assessment of
instructional practices in standards-based teaching. Journal of Mathematics and Science:
Collaborative Explorations, 6, 21–46.