The Development and Implementation of Local Assessment ...Assessment Portfolio (MAP) and Local Assessment Development (LAD), both keyed to the content areas and standards of the Learning

Maine Education Policy Research Institute

College of Education and Human Development The University of Maine

5766 Shibles Hall, Orono, ME 04469-5766 207/581-2493 • FAX 207/581-9510

September 2005

The Development and Implementation of Local Assessment

Systems in Maine Schools:

A Progress Report

The Development and Implementation

of Local Assessment Systems

in Maine Schools:

A Progress Report

Prepared by the Maine Education Policy Research Institute

Co-Evaluators: Janet Fairman Research Team: Debra Allen

Walter J. Harris Barbara Bales Amy Cates Lori Smith

Lisa Tyson Local Assessment System Implementation Study II Advisory Committee:

Gary MacDonald, MSAD #72 Superintendent Patrick Phillips, Deputy Commissioner of Education

Pamela G. Rolfe, Local Assessment Coordinator, Maine Department of Education

Jill M. Rosenblum, Project Coordinator, Maine Mathematics and Science Alliance

David Silvernail, Co-Director, Maine Education Policy Research Institute

For additional information, contact Dr. Walter Harris, Co-Director, Maine Education Policy Research Institute, The University of Maine, (207) 581-2467 or [email protected].

Table of Contents Introduction .............................................................................................................................................. 1 Research Questions.................................................................................................................................. 1 Overview of Methodology ...................................................................................................................... 2 Organization of this Report ..................................................................................................................... 3 Part I: Summary of Findings .................................................................................................................. 4 Alternative Education.......................................................................................................... 7 Special Education ................................................................................................................ 8 English Language Learners (ELL) ..................................................................................... 8 Career and Technical Education (CTE) ............................................................................. 9 Conclusions .............................................................................................................................................. 13 Evidence of Progress Made........................................................................................................... 13 Mixed Support for LAS................................................................................................................. 13 Time as a Barrier............................................................................................................................ 13 Financial Costs............................................................................................................................... 14 Human Capacity ............................................................................................................................ 14 Managing Social Change .............................................................................................................. 14 Communications and Public Relations......................................................................................... 14 Technical Assistance ..................................................................................................................... 15 Readiness to Certify Students’ Achievement of the Learning Results ...................................... 15 Professional Development ............................................................................................................ 15 Expectations for Student Achievement ........................................................................................ 15 Use of LAS Data............................................................................................................................ 16 Part II: The Status of LAS Development and Implementation: A Statewide Survey, Spring 2005 .................................................................................................................................... 17 Demographics ................................................................................................................................ 17 Leadership of LAS Implementation and Design ......................................................................... 19 Current Status of LAS Development and Implementation ......................................................... 21 English Language Arts ........................................................................................................ 21 Mathematics ......................................................................................................................... 23 Overall Progress in Other Content Areas........................................................................... 24 Number of Certification Assessments Planned ........................................................................... 26 Distribution of Assessments by Type........................................................................................... 26 Current Use of Local Assessment System Data .......................................................................... 27 Status of Selected Implementation Activities .............................................................................. 28 Providing Supplemental Instruction Prior to Replacement Assessments ........................ 30 Developing a Means of Determining the Value of Prior Education Experiences for Transfers......................................................................................................................... 30 Collaboration Among SAUs ......................................................................................................... 31 Level and Type of Help Needed ................................................................................................... 31 Confidence ..................................................................................................................................... 33 Percent of Graduates Estimated to Meet Learning Results......................................................... 33 Understanding the Requirements.................................................................................................. 34 Changes to Support the Class of 2008 Meeting the Standards ................................................... 34 Positive and Negative Impacts of LAS Implementation............................................................. 34 Boosters and Barriers to LAS Implementation............................................................................ 35 Supplementary Analysis .......................................................................................................................... 35 Distribution of Advanced and Less Advanced SAUs by Region ............................................... 36 Leadership in Advanced and Less Advanced SAUs ................................................................... 36 Other Characteristics of Advanced and Less Advanced SAUs .................................................. 37

Help Needed by Advanced and Less Advanced SAUs ............................................................... 37 Use of LAS data by Advanced and Less Advanced SAUs......................................................... 38 Comparison of 2004 and 2005 Survey Items ......................................................................................... 39 Progress in English Language Arts .............................................................................................. 39 Progress in Mathematics ............................................................................................................... 43 Summary................................................................................................................................................... 46 Part III: Interviews and Focus Groups: A Qualitative Study of LAS Progress in Fourteen Representative School Administrative Units (SAUs)............................................................................ 50 Demographics for SAU Sample ................................................................................................... 50 Data Collection for SAU Sample ................................................................................................. 51 Data Analysis ................................................................................................................................. 52 SAU Organization for LAS Development and Implementation................................................. 52 Personnel Resources for LAS Work............................................................................................. 53 Support and Leadership for LAS.................................................................................................. 53 Superintendents.................................................................................................................... 54 Principals.............................................................................................................................. 55 Teachers ............................................................................................................................... 56 Progress Toward LAS Development and Implementation ......................................................... 57 Curriculum Alignment ........................................................................................................ 57 Selecting Performance Indicators ....................................................................................... 57 Administering Assessments ................................................................................................ 59 Developing Rubrics ............................................................................................................. 60 Single Scoring Assessments ............................................................................................... 60 Double Scoring Assessments.............................................................................................. 60 Establishing Reliability ....................................................................................................... 61 Professional Development to Support LAS Implementation ........................................... 62 Time Spent on LAS Work............................................................................................................. 63 Assessment of Students in Alternative Education Programs ...................................................... 63 Assessment of Students in Special Education Programs ............................................................ 65 Assessment of English Language Learners (ELL) ...................................................................... 68 Assessment of Career and Technical Education (CTE) Students............................................... 69 LAS Data Management ................................................................................................................. 71 LAS Data Reporting ...................................................................................................................... 71 LAS Policy Development.............................................................................................................. 72 Use of Assessment Data ................................................................................................................ 72 Impacts of LAS.............................................................................................................................. 73 Impacts on Curriculum........................................................................................................ 73 Impacts on Instruction and Assessment ............................................................................. 74 Impacts on Teachers ............................................................................................................ 75 Impacts on Students............................................................................................................. 77 Impacts on Parents............................................................................................................... 78 Certification of Student Achievement .......................................................................................... 78 Supports and Barriers for LAS Development and Implementation ........................................... 79 Supports................................................................................................................................ 79 Barriers ................................................................................................................................. 79 Technical Assistance and Other Resource Needs........................................................................ 81 Summary .................................................................................................................................................. 82 Part IV: Appendices................................................................................................................................ 85 Appendix A: 2005 Survey............................................................................................................ 86 Appendix B: LASIS II Site Visit Interview Protocols ............................................................... 93 Appendix C: Previsit Information Tables ................................................................................... 100

Appendix D: Focus Group Protocols .......................................................................................... 102 Appendix E: Changes to Support the Class of 2008 .................................................................. 103 Appendix F: Boosters and Barriers ............................................................................................. 109

1

The Development and Implementation of Local Assessment Systems in Maine Schools:

A Progress Report

Introduction

In 1996 the Maine Legislature established the Maine Learning Results, a descriptive statement of the expectations held for what all Maine students are expected to know and be able to do as a result of their public school experience. Accompanying these expectations is the requirement that students’ achievement of the Learning Results be assessed with a combination of the Maine Education Assessment (MEA) and additional assessments that are locally developed. The locally developed assessments or Local Assessment Systems (LASs) are to be valid and reliable and ultimately provide an accurate representation of students’ achievement of the Learning Results. Technical and methodological resources were provided by the Maine Department of Education (MDOE) to assist school administrative units (SAUs) with the complex tasks of developing LASs. These resources included Measured Measurers: Technical Considerations for Developing a Local Assessment System, LAS Guide: Principles and Criteria for the Adoption of Local Assessment Systems, Considering Consistency: Conceptual and Procedural Guidance for Reliability in a Local Assessment System, and LAS Guide with Embedded Components for Accountability and Alternate Assessment. In addition, MDOE provided regional workshops, consultation on request, and two web-based inventories of assessment tasks, the Maine Assessment Portfolio (MAP) and Local Assessment Development (LAD), both keyed to the content areas and standards of the Learning Results. This report summarizes data collected in the spring of 2005 by the Maine Education Policy Research Institute (MEPRI) to assess the current status and progress of Maine SAUs in designing and implementing LASs. It is the third consecutive annual report on this topic and is written for MDOE personnel, school personnel, and policymakers to inform their efforts to assist schools in the creation of valid and reliable assessment systems that contribute to teaching and learning, hold SAUs accountable, and certify students’ achievement of the Learning Results.

Research Questions Research questions were designed by MEPRI staff and MDOE personnel to guide this study. These questions were developed in a context that included the data from previous studies, the perspectives that MDOE personnel had gained from their work in SAUs on topics related to the design and implementation of LASs, and the needs for information of MDOE personnel as they prepared to design and provide technical assistance to SAUs. The research questions to be addressed by the study included the following:

1. What progress has been made in the design and implementation of Local Assessment Systems (LASs) in Maine SAUs since June 2004 (when the results of the 2004 statewide survey were reported)?

2

2. What is the current status and technical quality of SAUs’ efforts to select and score

assessments in individual content areas and grade spans? 3. What progress has been made, what best practices exist, and what issues need to be

resolved in SAUs’ efforts to develop LASs that will assess the learning of students in programs in Alternative Education, Special Education, Career and Technical Education and programs for English Language Learners?

4. How is LAS data being collected, managed, reported, and communicated? 5. What policies relevant to LASs have been created or modified by SAUs? 6. How are SAUs responding to the systemic use of LAS data for formative assessment

of curriculum and instruction? 7. What progress are SAUs making toward certification of student achievement of the

Learning Results, and how is LAS data informing this progress? 8. What technical assistance or other resources are needed? 9. What factors, such as leadership structures and professional development

opportunities, have been helpful in creating and implementing LASs? 10. How have SAUs used time and other resources to meet both the state and local

requirements of implementing LASs?

Overview of Methodology Three methods were used to gather sufficient data to address all 10 research questions:

1. A survey was mailed to all SAUs in May 2005 with a request for it to be completed by the individual in each SAU who was most familiar with the status of LAS development (Appendix A). During the time that respondents were completing the survey potential policy changes effecting LASs were being discussed in the MDOE and the Education Committee of the Legislature and were reported in the news media. These discussions may have impacted responses to the survey. The overall return rate for the survey was 83% with a range of 52% to 97% across the nine superintendent regions. A detailed analysis of the survey results constitutes Part II of this report.

2. Interviews of LAS leadership personnel, and teachers who were not directly involved

in LAS leadership were conducted in 14 SAUs that represented all geographic regions (except the unorganized territories) and types of administrative structures. This sample also represented both large and small SAUs and SAUs with both high and low percentages of students in free-and-reduced-lunch programs. A total of 185 interviews were conducted with administrators and teachers. Survey protocols are

3

included in Appendix B. Like the survey, these interviews were conducted during a time in which policy changes were being discussed by MDOE personnel, the Education Committee of the Legislature and reported in the news media. These discussions may have affected responses to interview questions. Focus groups were conducted with representatives of Career and Technical Education, Alternative Education and Special Education programs. Due to scheduling difficulties, a focus group was not conducted with representatives of English Language Learner (ELL) programs, but three Directors of ELL programs were interviewed, and ELL was a focus of interviews in all 14 SAUs. The interview protocols for focus groups can be found in Appendix D.

3. Documents, policies, assessment templates and checklists of LAS tasks completed

were requested of all fourteen SAUs. Analysis of these documents informed the study. A detailed analysis of data gathered through interviews and focus groups constitutes Part III of this report.

Organization of this Report

This report is divided into four parts:

• Part I draws on data from the survey, interviews, and focus groups and provides a summary of the findings related to each research question and overall conclusions.

• Part II describes and analyzes the data resulting from the survey and includes a

Supplementary Analysis that divides responding SAUs into an Advanced Group and a Less Advanced Group for comparison on several key variables. Part II also includes a comparison of the 2005 survey with a similar survey conducted in spring 2004.

• Part III summarizes the interview and focus group data collected in 14 representative

SAUs. • Part IV, Appendices, includes the survey, summaries of narrative responses to survey

questions, and protocols used in interviews.

4

Part I: Summary of Findings 1. What progress has been made in the design and implementation of Local Assessment Systems

(LASs) in Maine SAUs since June 2004 (when the results of the 2004 statewide survey were reported)?

SAUs have continued to make progress in developing and implementing their LASs. While most SAUs were developing their LASs in spring 2004, (e.g., selecting performance indicators and selecting assessments), in spring 2005 SAUs were beginning to implement their LASs (e.g., administering and scoring assessments). SAUs have spent most of this year piloting and revising assessments, rubrics, and LAS templates. The piloting of assessments has revealed the need to revise assessment tasks and LAS templates. In some SAUs, educators are revisiting decisions about the number and type of assessments to be administered and the schedule of assessments. Some SAUs had selected a number of assessments and performance indicators in excess of the number recommended by MDOE, and are reducing these numbers. In other cases, educators realized that selected assessments did not work well, and they needed to select different assessments or revise curriculum and instruction to allow for better integration of curriculum, instruction and assessment. Most SAUs have single scored the assessments administered this year, and were just beginning to double score some of these assessments this past spring. Interview data indicate that many SAUs made the scoring of assessments in English Language Arts and Mathematics in grades 1, 5 and 9 a priority. Most SAUs have provided training for teachers on developing and scoring assessments but more work needs to be done. Continued progress is also indicated by a comparison of the results of the LASIS surveys conducted in 2004 and 2005 on four tasks related to assessment in English Language Arts and Mathematics (Tables 37-42). In English Language Arts, progress has been made in all grade spans in developing curriculum documents aligned with the Learning Results, aligning instruction and curriculum with the Learning Results, and providing professional development in common scoring for teachers. Progress has also been made in selecting performance indicators for assessment in the K - 4 grade span, while less progress was made on this task in the 5 - 8 and 9 - 12 grade spans. In Mathematics, progress has been made in all grade spans in developing curriculum documents aligned with the Learning Results, aligning instruction and curriculum with the Learning Results, and providing professional development in common scoring for teachers. Less progress was made in selecting performance indicators for assessment in all grade spans. Analysis of the 2005 survey data indicates that the level of progress made by SAUs could not be predicted by any of the following variables: superintendent region, the job role responsible for LAS coordination, attending enrollment, percent of students receiving free-and-reduced lunch, per pupil expenditures, and student/teacher ratios (Tables 31-34). Analysis of interview data from the sample of 14 SAUs indicates that good progress was made in six SAUs, while 6 SAUs showed some progress and 2 showed limited progress. Those SAUs

5

showing good progress can be characterized as having strong, stable administrative leadership for curriculum and assessment work, strong levels of agreement with the concept of LAS, and support for LAS work expressed by both administrators and teachers. These SAUs had also scheduled more time to do LAS work than SAUs that had made limited progress. Reviewing all sources of data, it appears that progress has been made on the tasks required to build and implement LASs. A considerable amount of revising curriculum, instruction, and assessment is occurring and adjustments are being made as LASs are being implemented. Most educators have indicated a strong need to continue the development of their expertise and capacity to do this work. However, SAUs are finding that this work is requiring more time to complete than expected because the changes needed to implement LASs impact a wide spectrum of constituents, belief systems, and expectations which have historical roots, and instructional practices that are firmly established. Respondents to both surveys and interviews were asked to describe boosters and barriers that have facilitated or impeded the process of implementing a LAS. The following categories of responses were apparent in these data. A more detailed list of boosters and barriers can be found in Appendix F. Boosters:

• Leadership structures for LAS work that were previously created for curriculum work • A specified coordinator for LAS work (curriculum coordinator or other person) • MDOE workshops and MDOE documents • Additional time for LAS work by restructuring school day or calendar • Teacher stipends for non contractual time and summer work • Collaboration and partnerships that result in sharing ideas and assessment models • Dedicated, professional staff who are open and willing to learn • New data management tools and processes • Support, including consultation from MDOE, financial and moral support from local

administrators and school committees and support from consulting organizations • Professional development

Barriers:

• Insufficient time (in particular, to score assessments and reflect on results) • Incomplete work on curriculum alignment • Poor communication among multiple schools within an SAU, parents, and community. • Late and inconsistent information from MDOE • Size and scope of LAS requirements and tasks (particularly for small school districts) • Lack of compensation or release time needed to accomplish tasks • Reluctance to give up instructional units that were deemed to be successful • Lack of adequate data systems for reporting and analyzing data • Lack of available assessments that are reliable and valid • Lack of direction for assessing students in special education, career and technical

education, and others who are struggling

6

2. What is the current status and technical quality of SAUs’ efforts to select and score

assessments in individual content areas and grade spans? Most SAUs were just beginning to double score assessments in the spring of 2005, and therefore it was too soon to determine the technical quality of the scoring process. Survey data indicate that 28% of SAUs reported that work was complete for double scoring common assessments in English Language Arts in grade span K - 4, 25% in grade span 5 - 8, and 33% in grade span 9 - 12 (Tables 8-10). In Mathematics, 37% reported that work was complete for double scoring common assessments in grade span K - 4, 34% in grade span 5 - 8, and 39% in grade span 9 - 12 (Tables 11-13). These data indicate that more SAUs had completed double scoring in Mathematics in all grade spans than had completed double scoring in English Language Arts. They also indicate that a higher percentage of SAUs reported work complete on double scoring in grade spans 9 - 12 in both English Language Arts and Mathematics than in grade spans 5 - 8 and K - 4. The smallest percent of SAUs reported work complete in double scoring in both English Language Arts and Mathematics for grade span 5 - 8. Survey data indicate that 28% of SAUs reported that work was complete on the task of establishing and documenting reliability among scorers in English Language Arts in grade span K - 4; 25% in grade span 5 - 8; and 33% in grade span 9 - 12 (Tables 8-10). In Mathematics, 28% of SAUs reported work complete on this task in grade span K - 4; 23% in grade span 5 - 8, and 33% in grade span 9 - 12 (Tables 11-13). These data indicate that the highest percent of SAUs reported work complete in the 9 - 12 grade span and smallest percentage in grade span 5 - 8, a pattern similar to the results reported above for the task of double scoring common assessments. Interviews in the 14 SAUs revealed that teachers often described the processes of reaching consensus over benchmarks and scoring of students’ work as producing conflict and stress within work groups. These elements sometimes slowed progress on LAS tasks. Some SAUs assigned administrators or other educators to facilitate work groups to overcome the divisiveness and to keep these groups moving forward. Respondents indicated that MDOE’s LAS documents were used to guide the technical aspects of the scoring process; however, some departed from the suggested protocols to some degree. Both the survey and interview data confirm that most SAUs believe that the technical aspects of double scoring assessments and establishing reliability are within their capability and are not tasks for which a high level of technical assistance is needed. Instead, time to score assessments and skilled group leaders for some work groups were cited as important needs for completing this task. 3. What progress has been made, what best practices exist, and what issues need to be resolved

in SAUs’ efforts to develop LASs that will assess the learning of students in programs in Alternative Education, Special Education, English Language Learners (ELL), and Career and Technical Education (CTE)?

7

The overall focus on assessment and accountability has brought to light significant issues related to specifying common standards for all students. Educators have become acutely aware of the challenge of aligning the goals and philosophies of these educational programs and Maine’s Learning Results. Survey data reveal that only 21% of SAUs indicated that work was complete on creating assessments for students with 504 plans. Nineteen percent indicated that work was complete for students requiring accommodations, and 25% of the SAUs reported that work was complete for creating alternative assessments (Table 24). Although 54% of SAUs reported that no action had been taken on creating assessments for students in programs for English Language Learners, this may be a misleading result. It cannot be determined if SAUs responding to this item had ELL programs for their students. Regarding CTE programs, 46% of the SAUs indicated that no action had been taken yet on developing LASs to assess students in this group, and 40% reported that no action had been taken yet related to alternative education programs (Table 25). Interview data confirm that little work has been completed in developing assessments for students in alternative education, special education, ELL, and CTE programs. Many educators in regular education programs reported that the design and implementation of LASs has increased their focus on educational outcomes and have generally increased the expectations they hold for students with disabilities. However, many educators in special education, alternative education, CTE, and ELL programs expressed skepticism about the appropriateness of the Learning Results standards for their respective students. Educators in alternative education, special education, and ELL programs said the Learning Results grade level performance indicators and LAS assessments were often too challenging for their students. CTE representatives said that very few of the Learning Results performance indicators apply to the content of their programs, which are organized around national standards for trade organizations. Best practices related to LAS development and implementation are not apparent at this time, as educators are struggling with prerequisite issues such as curriculum alignment and aspects of LAS assessments, scoring and reporting. However, researchers generally found that where CTE and alternative education programs were co-located in or adjacent to a regular education facility, more collaboration and coordination occurred between regular education programs and special programs. Although administrators in some SAUs made the involvement of educators in special programs a priority, most educators in these programs felt they had not been included at both state and local levels in the design and development of LASs. Alternative Education. In both a focus group and in on-site interviews, alternative education representatives described the wide range of needs of students in their programs. Some students required minimal support from alternative education (e.g., an altered school day schedule or altered program of study) while other students attended an alternative education program on a full-time basis that may be located at some distance from the regular school. Respondents described the difficulty of collaborating with regular education teachers and programs about the unique needs of their students and participating in the work of designing LAS assessments appropriate for their students given the small size of most alternative education programs.

8

Survey data indicate that among K - 12 and 9 - 12 SAUs, 65% indicated that help in building the capacity to assess students in alternative education programs was among their highest priority needs for assistance (Tables 26 and 27). In alternative education, unresolved issues related to LAS development and implementation included the following:

• the incongruity of holding alternative education students accountable to standards and assessments developed within and for regular education programs

• the requirement to administer assessments that are beyond students’ capabilities, forcing repeated failure before students can take replacement assessments

• remediation and replacement assessments have not yet been developed and/or do not meet the needs of students in alternative education

• ambiguity about what high school diploma options will be available to alternative education students who cannot meet Learning Results standards as measured by a LAS

Special Education. Assessment and accountability in special education have been part of the educational context for many years. Having specific learning objectives and specifying the means to achieve and measure those objectives have been part of students’ IEPs. The LAS requirements have added a complicating layer, which has had both positive and negative impacts. On the positive side, some special educators reported that the Learning Results have provided an organizational structure and new learning goals for students with disabilities. However, common assessments did not take into account students’ unique learning needs and the difficulty of assessing students’ progress. In special education, unresolved issues include all of the points listed above for alternative education. In addition, special educators reflected confusion about the role of the IEP and whether below-grade-level assessments could be administered to special education students. On the survey, 20% of the Advanced SAUs said that help developing alternate assessments (PAAPs) was one of the top five priority needs (Table 36). English Language Learners (ELL). In most SAUs in which interviews were conducted, issues related to the assessment of ELL students were not viewed as a priority because these SAUs had few or no students in this subgroup, and because most were engaged in more fundamental tasks related to developing their LASs. For larger SAUs with sizable ELL populations, and for SAUs that were in a more advanced stage of LAS development and implementation, issues related to assessing ELL students were identified as a priority area. On the survey, 25% of the Advanced SAUs indicated that their need for assistance in developing the capacity to assess ELL students was among the top five areas of help needed (Table 35). ELL teachers and coordinators reported that ELL students had great difficulty taking common assessments not only because of their limited English skills, but because many of them are also illiterate in their native languages. ELL students’ academic knowledge and skills were often significantly below grade-level expectations due to the lack of formal schooling experience in their native countries.

9

Career and Technical Education (CTE). The central issue related to the assessment of CTE students centers on the lack of congruence between the Learning Results and CTE programs that are guided by trade and industry standards. CTE representatives describe their programs as attempting to focus on the Learning Results but finding little overlap with national trade and industry standards. CTE representatives also expressed the concern that CTE students may become frustrated when their LAS assessments do not take into consideration their different learning styles, which tend to be more hands on and applied. CTE representatives suggested that the LAS model might prompt some CTE students to drop out of high school. The LAS requirements have created some unique problems for CTE programs. With few exceptions, CTE programs are located at some distance from regular high schools and serve multiple SAUs, each with their own LAS. Different school calendars within a CTE region are a barrier to coordination of LAS assessment schedules in some SAUs. Respondents also indicated that SAUs rarely count assessments of students’ work in CTE courses as evidence of achievement that is valid in LASs. Building capacity to assess CTE students was among the top five priority needs for help of SAUs that were Advanced in LAS work (Table 35). 4. How is LAS data being collected, managed, reported and communicated? On the survey, building capacity for developing systems to analyze LAS data and developing systems to record LAS data were identified as being among the top priority needs of SAUs. Interviews in the 14 SAUs indicated that although many SAUs had put new software systems in place, many were transitioning to these systems, upgrading their systems, and in the early stages of training teachers and entering assessment scores. A few SAUs had not yet selected a software system and were waiting to determine what systems might be compatible with the Maine Educational Data Management System (MEDMS). Even in SAUs that were implementing new data management systems, it was common for administrators to collect and report assessment data using simple spreadsheets. Analysis of these data for any purpose is limited. Respondents in the 14 SAUs reported that assessment scores were typically translated into letter grades for the purpose of reporting student achievement on report cards. Although some SAUs are beginning to introduce standards-based reporting in some grades, most student achievement was reported using traditional report cards. On the survey, 39% of the SAUs indicated that work on the task of developing a report card describing progress toward meeting content standards at the K - 4 level was either partially complete or complete. The same degrees of work completion were indicated by 32% of SAUs at the 5 - 8 level, and 24% at the 9 - 12 level (Tables 21-23). Interview data indicate that administrators and teachers were providing some general information to parents about LAS requirements through parent meetings and newsletters. Parent meetings were described as infrequent and often not well attended. More effort had been made to meet with parents of eighth- and ninth-grade students. Administrators indicated that they were somewhat apprehensive about communicating with parents about assessment and graduation guidelines because LAS requirements had been frequently revised, and they were concerned that additional revisions were possible.

10

On the survey, the percentage of SAUs that indicated work complete or partially complete on the task of Developed a procedure for public reporting of scores ranged from 42% to 50% across the grade spans, with the highest percentage in the 9 - 12 grade span (Tables 21, 22, and 23). Only 22% of SAUs indicated that LAS data is used to inform the community about students’ progress (Table 20). It appears that SAUs were in the early stages of developing ways to report assessment results, but the data were not yet ready for reporting. Several administrators and teachers suggested that a more intense information campaign related to standards-based education be mounted at the state level to increase public awareness about state educational standards and to support local efforts to measure and report students’ achievement of the standards. As evidence of this need, on a 4-point scale ranging from Do not understand at all to Completely understand, only 28% of SAUs indicated parents’ level of understanding to be in the top half of this scale. On the same item, 54% of SAUs indicated that freshmen had the same level of understanding (Table 30). 5. What policies relevant to LASs have been created or modified by SAUs? Survey results indicate that 40% of Less Advanced SAUs perceived that help in developing policies related to assessment was among the top five areas of assistance needed (Table 35). On the task of developing a policy for replacement assessments, 45% of SAUs reported work was partially complete or complete in the K - 4 and 5 - 8 grade spans, while 55% of SAUs reported the same level of completed work in the 9 - 12 grade span (Tables 21-23). This finding is consistent with interview data, which indicate that most SAUs were in the early stages of planning and drafting policies for remediation and replacement assessments. Many SAUs were waiting for policy models from the Maine School Management Association at the time of the site visits. 6. How are SAUs responding to the systemic use of LAS data for formative assessment of

curriculum and instruction? Survey data indicate that in 83% of SAUs teachers were using LAS data in an informal way to make decisions about curriculum and instruction. This finding is consistent with interview data. Somewhat less frequent was the use of LAS data to identify students’ needs for remediation (60%), the revision of curriculum (59%), the identification of curriculum strengths and areas of need (51%), and to inform priorities for professional development (48%) (Table 20). The lack of data management systems and lack of time to score all assessments and enter them into new data software prevented educators from using data in more formal ways. Educators interviewed in the 14 SAUs also reported that the lack of time to reflect on assessment results was a significant barrier to using data to inform curriculum and instruction. 7. What progress are SAUs making toward certification of student achievement of the Learning

Results and how is LAS data informing this progress? Respondents were asked how confident they were that their SAU would have assessments in place to certify that graduates of 2008 would meet the Learning Results in English Language

11

Arts and Mathematics. In addition, they were asked what percent of their graduates they estimated would meet the Learning Results standards in 2008. Survey results indicate that on a scale of 1, Not at all confident, to 4, Very confident, 75% of SAUs indicated confidence levels of 3 (32%) and 4 (43%) that assessments will be in place to certify that 2008 graduates will meet the Learning Results in 2008 in English Language Arts. Seventy-eight percent of SAUs indicated a confidence level of 3 (38%) and 4 (40%) in Mathematics (Table 28). When asked to project what percent of 2008 graduates would achieve the Learning Results in English Language Arts, 79% of SAUs reported that between 51% and 100% of their graduates would meet this criterion. Sixty-nine percent of SAUs indicated that between 51% and 100% of graduates would achieve the Learning Results in Mathematics (Table 29). These two questions required that respondents select answers based on their best guess about future developments in the implementation of LASs in Maine. The reader is cautioned in the use and interpretation of these data. During the time period in which the survey was being conducted discussions about the future of LASs were occurring among MDOE personnel and the Education Committee of the Legislature. Responses to these questions, which required speculation, may also have been influenced by reports of these discussions in the media. 8. What technical assistance or other resources are needed? Survey results indicate that K - 5 and K - 8 grade spans were most in need of assistance in building capacity to develop systems for analyzing LAS data (77%), to use data to inform curriculum sequence (70%), to use data to inform curriculum content (68%), and to develop systems for recording LAS data (68%). K - 12 and 9 - 12 SAUs indicated that their highest priority needs for assistance include help building capacity to assess CTE students (70%), to develop systems to analyze LAS data (69%), to assess students in alternative education programs (65%), and to assess students in programs for English Language Learners (62%) (Tables 26 and 27). Data from interviews and focus groups highlighted other areas of need that were more global. Time, staff, and data management resources were nearly universal needs. Financial resources were identified as a need in some SAUs. Many other comments by respondents can be characterized as a need for knowledgeable leadership. These respondents were frequently critical of SAUs decisions to send a small number of representatives to MDOE workshops about LAS implementation who were then expected to return to the SAU and train others. These respondents indicated that the designated trainers could not answer questions at more than a surface level and/or they were spread too thinly throughout the SAU. Respondents also commented that, although MDOE workshops were informative, they often did not match the needs of SAUs that were at different levels of LAS development and implementation. Personnel from more Advanced SAUs often attended MODE workshops that appeared to be appropriate for their needs only to find that the level of presentation and discussion was at a more rudimentary level.

12

A need for leadership toward an integrated curriculum also emerged from the interviews. Some respondents indicated that when curriculum was integrated, assessment tasks were more aligned with curriculum and could serve multiple content areas. While this condition existed in some SAUs, many respondents in other SAUs viewed it as a goal that required leadership that was not available. Finally, many respondents voiced a need for clear guidance from MDOE personnel and LAS documents on how to accomplish LAS tasks. Educators working with special education students, alternative education students and English Language Learners had specific questions about assessing their students and expressed the need to have MDOE personnel visit their schools. 9. What factors, such as leadership structures and professional development opportunities have

been helpful in creating and implementing LASs? It is clear from interview data that SAUs that assigned LAS leadership and coordination responsibilities to teams that already existed for curriculum alignment with the Learning Results were advanced in LAS work. More Advanced districts viewed LAS work as an extension of earlier work on the Learning Results. Less Advanced districts were those that started more slowly, had changes in leadership, or formed totally new committees or teams to do LAS work. Interview data indicate that superintendents played an important role in more Advanced SAUs through supporting the concept of LASs, providing resources, stimulating policy development and communicating with the public. However, curriculum coordinators and building principals were more significant leaders in getting the work done. Professional development opportunities were frequently listed as a booster to progress and one of the positive impacts of LAS work. LAS work was attributed with encouraging high-quality focused professional development with specific outcomes or products. For example, one superintendent, while concerned about the resources demanded by LAS implementation, indicated that LAS had caused “the best professional development my teachers have had in years.” On the negative side, topics related to LAS development were viewed by some respondents as excluding other important topics related to instruction. Further analyses of these data reveal no clear difference between Advanced and Less Advanced SAUs in the job role identified as having primary responsibility for LAS work (Table 32). 10. How have SAUs used time and other resources to meet both the state and local requirements

of implementing LASs? SAUs varied widely in the amount of time dedicated to LAS work including professional development. In SAUs in which interviews were conducted the total number of days scheduled for LAS work varied from .5 to 7 full days. Of these, the number of days scheduled for scoring assessments varied from 0 to 4 days. These figures do not include the personal time many teachers devoted to LAS tasks. Overall, SAUs making the most progress used professional development days and early release days for LAS development and scoring. Some SAUs hired

13

substitute teachers to allow content or grade-level teams to meet for full days. Many respondents reported that using brief periods of time for LAS work, such as after school hours, and before and after school meetings, was a frustrating and unproductive approach.

Conclusions

Evidence of Progress Made Survey results indicate that progress is being made on the tasks of developing and implementing LASs. Throughout 2004-2005 most SAUs were refining their assessment tasks and schedules after administering some assessment tasks and discovering that the assessment plans created earlier were in need of adjustment. Few SAUs were involved in refining the scoring process, reporting data, or using data to inform curriculum and instruction. More SAUs reported a higher level of work completion in English Language Arts and Mathematics. The content areas of Science and Technology, Social Studies, and Health and Physical Education were rated as partially complete or work complete by more than half of the SAUs responding to the survey with the largest percent in grade span 9-12. Few SAUs reported work to be partially complete or complete in the areas of Modern and Classical Languages, Visual and Performing Arts and Career Preparation. Mixed Support for LAS Support for the concepts of standards based education, accountability, and high expectations for students emerges from the interviews of teachers and administrators in regular education though most expressed concerns about the timelines for implementing LASs and the scope of the overall changes required. Many respondents indicated that LAS work has provided highly valuable professional development experiences and stimulated collaborative planning among teachers. As a result, curriculum has been more clearly defined in each grade span and there is more consistency, coherence, and common practices in curriculum and assessment across grade spans and throughout K-12. This is not true of educators who are responsible for programs in special education, career and technical education, alternative education, or programs for English Language Learners. The issues raised by these special groups are related to the unique characteristics of the students they serve, the concept of holding the same standards for all students regardless of their capabilities, the means of assessing achievement through LAS, and the criteria for receiving a high school diploma. The data suggest that these issues should be examined if the Learning Results and the LAS model are to apply to all students. Interview data also suggest that some teachers are resistant to LAS implementation because it may require that they eliminate, modify or reschedule units of instruction that have been developed and refined, sometimes over many years. Time as a Barrier The primary impediment to the development and implementation of LASs in most SAUs is the lack of time to do the work, particularly the scoring of assessments, processing the scores in data management systems and analyzing and reflecting on the results. Large blocks of time (full days or 2 days together) are viewed as most productive. More successful SAUs have found additional time by hiring substitutes,

14

scheduling common planning periods for teachers, using early release days, and paying stipends to teachers for the additional workload inherent in LAS planning and scoring assessments. Not all SAUs had the financial flexibility or flexibility with the school calendar to use these strategies. Financial Costs There are significant concerns about the cost of developing, implementing, and sustaining LASs among SAU administrators. Frequently mentioned costs include the creation of additional time for LAS work, stipends for teachers and teacher leaders, providing remedial services and developing and managing data systems. Methods of funding these activities within the Essential Programs and Services model are of particular interest. Human Capacity The need to develop the human capacity to fully implement LASs emerges from the qualitative data in several different forms. Interview data suggest that the training model used by many SAUs in which a very small number of representatives attend MDOE workshops and then return to train other SAU personnel is minimally effective. Teachers report that their representatives often find themselves inadequately prepared to respond to substantive questions upon returning to their schools. Alternative ways of developing this capacity in SAUs should be explored. Survey and interview data indicate that one knowledgeable person, typically a Curriculum Coordinator, is critical to successful LAS work in most SAUs. Some SAUs may need financial support or funding to create or support this position. Professional development is needed to build SAUs capacity to support strategies such as interdisciplinary approaches and the seamless integration of instruction and assessment. Managing Social Change Interview data confirm that the role of teacher is changing. Teachers are being asked to change curriculum, instruction and assessment practices and work in a collaborative mode with other teachers. Leadership personnel with group facilitation skills and time are needed to support major changes that are required in both beliefs and behaviors. Communications and Public Relations Interview and survey data indicate that there is a need for increased communication about LAS, among and between schools and between SAUs and MDOE. SAUs are in need of public support from beyond their boundaries as they develop LASs. Interview data indicate that some school leadership personnel believe that increased publicity about standards-based education and performance-based assessment sponsored by MDOE would assist SAUs in communicating the overall goals of these concepts to teachers, parents, and students.

15

LAS goals and requirements should be communicated effectively and consistently by MDOE. This is especially needed for the assessment of students in special education, alternative education, career and technical education programs and programs for English Language Learners. Technical Assistance Both survey and qualitative data indicate a need to see models of LASs that have been fully implemented and used to inform curriculum and instruction. Components of particular interest include data management systems, replacement assessments, alternative assessments, policies, and remediation programs. Assistance is also needed in K-12 and 9-12 SAUs in building capacity to assess CTE students, students in Alternative Education programs and for English Language Learners.

Technology can assist SAUs in recording, managing, analyzing, and using LAS data. However, few SAUs have devised or purchased the perfect system and many are waiting for MEDMS to be functional before they invest in technology for assessment. Leadership from the MDOE is needed to help school districts make choices that will meet their LAS data needs. Readiness to Certify Students’ Achievement of the Learning Results Thirty percent of the SAUs responding to the survey indicate that they anticipate that less than 50% of their students will meet the Learning Results in 2008. Although this data is based on respondents’ speculation, it may suggest the need to examine the opportunities that students have to gain the knowledge and skills encompassed by the Learning Results and to identify and explore barriers that may impede learning. Professional Development Survey and interview data indicate that the development and implementation of LASs have been a very positive professional development experience for teachers. Both leadership personnel and teachers described that more time was being devoted to discussions of students, curriculum and assessment and that LAS work was requiring more professional discussion among teachers. Both teachers and administrators reported that their knowledge of basic measurement concepts such as validity and reliability, and of assessment concepts and practices such as rubrics, benchmarks, and double scoring had increased.

Professional development topics related to LASs were valued by both teachers and administrators. However, many expressed the concern that other important topics related to instruction were not being considered due to LAS deadlines. Expectations for Student Achievement Some teachers and administrators commented that the process of developing and implementing a LAS had raised teachers’ expectations for students performance and that students were meeting those expectations.

16

Use of LAS Data Survey and interview data confirm that LAS data is being used informally by teachers in discussions about students and curriculum. Few SAUs have been able to use data in more formal ways (e.g. identifying implications of assessment data for systemic changes in curriculum and instruction) due to lack of time for scoring, and lack of technology for recording, analyzing and reporting LAS data.

17

Part II: The Status of LAS Development and Implementation: A Statewide Survey, Spring 2005

A survey was conducted to assess the status of LAS implementation in May 2005. In order to gain information about the progress made by all SAUs in developing and implementing LASs, some items that had been used in a similar survey in 2004 were included. The 2005 survey may be found in Appendix A. The survey was designed to obtain information from the entire population of Maine SAUs and, therefore, no sampling strategy was used. However, it was necessary to determine the maximum number of LASs that might be created in the State since a LAS might be designed and implemented for more than one SAU. This information was gained through e-mail and telephone dialog with representatives of school unions, CSDs, and other SAUs that share a superintendent. As a result of these communications, it was determined that the total population of 225 SAUs would generate a maximum of 159 LASs. Demographics The survey was mailed to all 225 SAUs in Maine in early May of 2005 with a request that it be returned by May 20. The tables below represent the responses of 187 SAUs representing a return rate of 83%. One hundred and thirty-five different LASs are represented in the returned surveys (some LASs are implemented in more than one SAU). Based on the assumption that a maximum of 159 LASs are expected to be designed and implemented in Maine, 85% of all LASs are represented in the data. Table 1indicates the number of SAUs that returned surveys from each region and the percent of the total number of SAUs, in each region that responded. Since some LASs might apply to more than one SAU these data might not represent precisely the number and percent of LASs actually being implemented. Table 1. Percent of SAUs Responding in Each Region

16 66.7%

19 95.0%

29 93.5%

29 93.5%

15 55.6%

32 96.9%

6 100.0%

17 81.0%

12 52.1%

12 75.0%

Aroostook County

Cumberland County

Hancock County

Kennebec Valley

Midcoast

Penquis

Unorganized Territories

Washington

Western Maine

York

n %

18

Table 1 indicates that all regions were well represented in the data. The Western Maine and Midcoast regions were least represented at 52.1% and 55.6% response rates. The highest response rates were from the Penquis region (96.9%), Cumberland County (95%), Hancock County (93.5%), and Kennebec Valley (93.5%). Table 2 indicates the number and percent of school administrative structures represented in the data, and Table 3 displays the number of individuals who completed or assisted in completing the survey. As indicated by the data in Table 3, more than one person participated in responding to the survey in many SAUs. Table 2. Administrative Structures, Sample and State

Table 3. Roles of Individuals Completing the Survey

Data were collected regarding the grade configuration and enrollment of each of the responding SAUs. Tables 4 and 5 present these data. Table 5 provides a breakdown of responding SAUs by grade configuration and enrollment category.

35 18.7%

100 53.5%

3 1.6%

0 .0%

28 15.0%

31 16.6%

7 3.7%

13 7.0%

Superintendent

Curriculum Coordinator

Assistant Principal

Headmaster

Assistant Superintendent

Principal

Teacher Leader

Other job role

n %

68 36.4 87 37.8

38 20.3 45 19.6

58 31.0 71 30.9

10 5.3 14 6.1

6 3.2 6 2.6

7 3.7 7 3.0

Union

Town with Individual Supervision

MSAD

CSD

Unorganized Territory*

Other

Sample

Sample

Percent

Districts

with Schools

in State State Percent

* Unorganized territiories was considered 6 SAUs.

19

Table 4. Grade Configuration, Sample, and State

Table 5. Enrollment Category, Sample and State

The data in Tables 4 and 5 indicate that over 90% of the SAUs that responded to the survey were K - 12 and K - 8 SAUs with approximately a third of all responding SAUs enrolling fewer than 200 students and slightly more than a third enrolling more than 851 students. Leadership of LAS Implementation and Design Respondents were asked to indicate which job roles have the primary responsibility for leading the process of developing a LAS in their SAU. Respondents were asked to limit their selection to two role titles. Table 6 displays these data.

6 3.2 6 2.6

91 48.7 111 48.3

4 2.1 4 1.7

86 46.0 109 47.4

6-12 or 9-12

K-12

K-5

K-8

Sample

Sample

Percent

Districts

with Schools


67 35.8 82 35.7

25 13.4 31 13.5

26 13.9 39 17.0

35 18.7 42 18.3

34 18.2 36 15.7

Fewer than 200

201 - 400

401 - 850

851 - 2000

More than 2000

Sample

Sample

Percent

Districts

with Schools


20

Table 6. Job Roles with Primary Responsibility of Leading LAS Development

Table 6 indicates that that the role of Curriculum Coordinator was most frequently indicated as a role with leadership responsibility for designing the LAS followed by principals, teacher leaders, and superintendents. These data were further analyzed by categories of district size (attending enrollment), and results are displayed in Table 7.

Table 7. Job Roles with Primary Responsibility of Leading LAS Development by SAU Size

155 82.9% 32 17.1%

84 44.9% 103 55.1%

186 99.5% 1 .5%

187 100.0%

159 85.0% 28 15.0%

142 75.9% 45 24.1%

154 82.4% 33 17.6%

156 83.4% 31 16.6%

Superintendent leading

development of LAS

Curriculum Coordinator

leading development of LAS

Assistant Principal leading

development of LAS

Headmaster leading

development of LAS

Assistant Superintendent

leading development of LAS

Principal leading

development of LAS

Teacher Leader leading

development of LAS

Other job role leading

development of LAS

n %

Not checked

n %

Checked

13 19.4 3 12.0 4 15.4 8 22.9 4 11.8

34 50.7 10 40.0 16 61.5 19 54.3 24 70.6

1 4.0

8 11.9 5 20.0 2 7.7 3 8.6 10 29.4

20 29.9 5 20.0 4 15.4 14 40.0 2 5.9

11 16.4 3 12.0 8 30.8 6 17.1 5 14.7

9 13.4 4 16.0 6 23.1 5 14.3 7 20.6

Superintendent leading development of

LAS

Curriculum Coordinator leading

development of LAS

Assistant Principal leading development

of LAS

Headmaster leading development of LAS

Assistant Superintendent leading

development of LAS

Principal leading development of LAS

Teacher Leader leading development of

LAS

Other job role leading development of

LAS

n %

Fewer than 200

n %

201 - 400

n %

401 - 850

n %

851 - 2000

n %

More than 2000

21

Curriculum Coordinator was the role most frequently selected in all size categories. Principals were frequently indicated as leaders in the larger 851-2,000 enrollment category of SAUs while teacher leaders were selected in the 401-850 category. Current Status of LAS Development and Implementation Respondents were asked to rate the progress of LAS work on nine different activities related to designing and implementing a LAS in two content areas, English Language Arts and Mathematics. A scale of 1 to 4 was employed where 1 signified No action on this yet, 2 indicated Planning in progress, 3 signified Partially complete,and 4 indicated Work complete. Although respondents were asked to rate “progress,” their ratings on the scale resulted in a representation of the current status of their work. Progress in LAS development and implementation is described in Tables 37-42 where data from 2004 and 2005 are compared. English Language Arts. Data representing the status of tasks related to LAS development and implementation in English Language Arts for each grade span is presented in Tables 8, 9, and 10 below.

Table 8. Status of LAS Tasks in English/Language Arts K - 4

1 .5% 10 5.4% 86 46.7% 87 47.3%

1 .5% 10 5.5% 116 63.4% 56 30.6%

7 3.8% 7 3.8% 58 31.2% 114 61.3%

4 2.2% 4 2.2% 59 31.9% 118 63.8%

5 2.7% 6 3.3% 62 33.7% 111 60.3%

5 2.7% 16 8.6% 61 33.0% 103 55.7%

11 5.9% 15 8.1% 83 44.9% 76 41.1%

11 6.0% 25 13.6% 96 52.2% 52 28.3%

32 17.3% 62 33.5% 65 35.1% 26 14.1%

Developed curriculum documents aligned with

MLR

Aligned instruction with curriculum and MLR

Selected a sample of performance indicators for

assessment

Used scoring rubrics for each assessment

Administered common assessments

Provided professional development in common

scoring for teachers

Double scored common assessments

Established and documented reliability among

scorers

Used LAS data to inform the design of curriculum

and instruction

n %

No action taken

on this yet

n %

Planning in

progress

n %

Partially

complete

n %

Work complete

22

Table 9. Status of LAS Tasks in English/Language Arts 5 - 8

Table 10. Status of LAS Tasks in English/Language Arts 9 - 12

The data in Tables 8, 9, and 10 indicate that the work of developing curriculum documents for English Language Arts that are aligned with the Learning Results was complete in nearly 50% of responding SAUs in all grade spans. However, the alignment of instruction with the Learning Results was less complete in grade spans K - 4 (30.6%) and 5 - 8 (30.1%), and more complete in the 9 - 12 grade span (42.0%). It appears that although curriculum has been aligned in document format in nearly 50% of SAUs, changing instructional practices has proceeded at a slower pace. More than 50% of the respondents in all grade levels reported that work was complete in the selection of assessments, use of scoring rubrics, administration of assessments and providing

1 .5% 14 7.6% 85 46.2% 84 45.7%

1 .5% 17 9.3% 110 60.1% 55 30.1%

7 3.8% 8 4.3% 60 32.4% 110 59.5%

3 1.6% 4 2.2% 64 34.4% 115 61.8%

4 2.2% 8 4.4% 68 37.2% 103 56.3%

5 2.7% 14 7.5% 67 36.0% 100 53.8%

12 6.5% 14 7.6% 88 47.8% 70 38.0%

10 5.4% 22 11.9% 106 57.3% 47 25.4%

33 18.0% 63 34.4% 69 37.7% 18 9.8%

Developed curriculum documents aligned with MLR



assessment







scorers

Used LAS data to inform the design of curriculum

and instruction

n %

No action taken

on this yet

n %

Planning in

progress

n %

Partially

complete

n %

Work complete

0 .0% 3 2.6% 53 46.1% 59 51.3%

0 .0% 6 5.4% 59 52.7% 47 42.0%

5 4.3% 4 3.4% 23 19.8% 84 72.4%

0 .0% 2 1.7% 29 25.2% 84 73.0%

0 .0% 3 2.6% 43 37.4% 69 60.0%

2 1.7% 6 5.2% 42 36.5% 65 56.5%

2 1.7% 12 10.3% 54 46.6% 48 41.4%

5 4.3% 17 14.8% 55 47.8% 38 33.0%

25 21.2% 32 27.1% 45 38.1% 16 13.6%




assessment






Established and documented reliability among scorers

Used LAS data to inform the design of curriculum and

instruction

n %

No action taken

on this yet

n %

Planning in

progress

n %

Partially

complete

n %

Work complete

23

professional development for teachers in common scoring. A higher percentage (60% to 72%) of SAUs reported that work was complete on this task in the 9 - 12 grade span. Use of LAS data to inform the design of curriculum and instruction was most frequently rated as Planning in progress or No action taken yet by more than 50% of all SAUs. Mathematics. Data representing the status of tasks related to LAS development and implementation in Mathematics for each grade span is presented in Tables 11, 12, and 13.

Table 11. Status of LAS Tasks in Mathematics K - 4

Table 12. Status of LAS Tasks in Mathematics 5 - 8

2 1.1% 7 3.8% 78 42.4% 97 52.7%

1 .5% 13 7.1% 101 54.9% 69 37.5%

7 3.8% 9 4.9% 59 32.2% 108 59.0%

4 2.2% 6 3.3% 52 28.3% 122 66.3%

4 2.2% 12 6.6% 60 33.0% 106 58.2%

8 4.4% 18 9.9% 60 33.1% 95 52.5%

18 9.9% 20 11.0% 77 42.3% 67 36.8%

13 7.1% 26 14.2% 93 50.8% 51 27.9%

32 17.6% 59 32.4% 59 32.4% 32 17.6%




assessment








instruction

n %

No action taken

on this yet

n %

Planning in

progress

n %

Partially

complete

n %

Work complete

2 1.1% 7 3.8% 77 41.8% 98 53.3%

1 .5% 16 8.7% 100 54.3% 67 36.4%

7 3.8% 8 4.4% 66 36.1% 102 55.7%

3 1.6% 5 2.7% 65 35.3% 111 60.3%

3 1.7% 11 6.1% 62 34.3% 105 58.0%

8 4.4% 17 9.4% 71 39.4% 84 46.7%

19 10.5% 16 8.8% 85 47.0% 61 33.7%

14 7.7% 22 12.1% 104 57.1% 42 23.1%

34 18.8% 58 32.0% 72 39.8% 17 9.4%



Selected a sample of performance indicators for assessment



Provided professional development in common scoring for

teachers




instruction

n %

No action taken

on this yet

n %

Planning in

progress

n %

Partially

complete

n %

Work complete

24

Table 13. Status of LAS Tasks in Mathematics 9 - 12

The data in Tables 11, 12, and 13 reflect a pattern of work completion for Mathematics similar to that of English Language Arts. In each grade span, a higher percentage of SAUs rated the development of curriculum documents aligned with the Learning Results as Work complete than rated the alignment of instruction as Work complete. Again, it appears that the work of creating aligned documents has moved more quickly than the alignment of instruction. It may be that the alignment of documents was the work responsibility of a few, while aligning instruction with curriculum and Maine Learning Results required participation of many. More than 50% of the respondents in all grade levels reported that the selection of assessments, the use of scoring rubrics, and administration of assessments as Work complete. A higher percentage (60% to 72%) of SAUs reported that work was complete on this task in the 9 - 12 grade span. In all grade spans, the use of LAS data to inform the design of curriculum and instruction was indicated as Work complete by the smallest percentage of respondents indicating an area of high need for technical assistance in this or other prerequisite LAS tasks. Overall Progress in Other Content Areas. English Language Arts and Mathematics were the content areas that were the primary focus of the 2005 survey. However, the survey also requested that respondents indicate the overall status of LAS development and implementation in the additional content areas of Science and Technology, Social Studies, Health and Physical Education, Modern and Classical Languages, Visual and Performing Arts, and Career Preparation. Respondents’ overall status of work in these content areas is displayed in Tables 14, 15, and 16.

1 .9% 3 2.6% 39 33.9% 72 62.6%

0 .0% 5 4.3% 57 49.6% 53 46.1%

5 4.3% 2 1.7% 20 17.4% 88 76.5%

0 .0% 2 1.7% 27 23.5% 86 74.8%

0 .0% 2 1.8% 35 31.0% 76 67.3%

1 .9% 12 10.7% 37 33.0% 62 55.4%

4 3.5% 12 10.5% 54 47.4% 44 38.6%

7 6.1% 16 13.9% 54 47.0% 38 33.0%

22 19.5% 31 27.4% 46 40.7% 14 12.4%

Developed curriculum documents aligned with

MLR



assessment







scorers

Used LAS data to inform the design of

curriculum and instruction

n %

No action taken

on this yet

n %

Planning in

progress

n %

Partially

complete

n %

Work complete

25

Table 14. Overall Status of LAS Development and Implementation K - 4

Table 15. Overall Status of LAS Development and Implementation 5 - 8

Table 16. Overall Status of LAS Development and Implementation 9 - 12

The data in Tables 14, 15, and 16, which pertain to content areas other than English Language Arts and Mathematics, indicate that the largest percentage of SAUs rated the status of their work as complete in the content areas of Science and Technology, Social Studies, and Health and Physical Education in all grade levels but especially in the 9 - 12 grade span. Modern and Classical Languages, Visual and Performing Arts, and Career Preparation reflected a higher level of completion in grade spans 9 - 12 and 5 - 8 and a lower level of completion in grades K - 4.

5 2.7% 24 13.1% 103 56.3% 51 27.9%

13 7.1% 41 22.3% 91 49.5% 39 21.2%

5 2.7% 41 22.3% 91 49.5% 47 25.5%

102 56.4% 63 34.8% 16 8.8% 0 .0%

58 31.7% 89 48.6% 35 19.1% 1 .5%

116 63.4% 58 31.7% 9 4.9% 0 .0%

Sci Tech

Soc Studies

Health/PE

Modern & Class Lang

Visual Performing Arts

Career Prep

n %

No action taken on this

yet

n %

Planning in progress

n %

Partially complete

n %

Work complete

0 .0% 3 2.6% 62 53.9% 50 43.5%

0 .0% 4 3.5% 73 63.5% 38 33.0%

0 .0% 3 2.6% 65 56.5% 47 40.9%

36 31.9% 50 44.2% 23 20.4% 4 3.5%

25 21.9% 61 53.5% 28 24.6% 0 .0%

55 48.2% 37 32.5% 21 18.4% 1 .9%

Sci Tech

Soc Studies

Health/PE

Modern & Class Lang


Career Prep

n %


yet

n %


n %

Partially complete

n %

Work complete

2 1.1% 15 8.4% 108 60.3% 54 30.2%

7 3.9% 30 16.7% 96 53.3% 47 26.1%

2 1.1% 32 17.8% 95 52.8% 51 28.3%

83 46.6% 73 41.0% 21 11.8% 1 .6%

54 30.2% 91 50.8% 33 18.4% 1 .6%

104 58.1% 60 33.5% 15 8.4% 0 .0%

Sci Tech

Soc Studies

Health/PE

Modern & Class Lang


Career Prep

n %


yet

n %


n %

Partially complete

n %

Work complete

26

Number of Certification Assessments Planned Respondents were asked to indicate the total number of certification assessments that were planned for each grade span in English Language Arts and Mathematics. If more than 12 assessments were planned for any grade span, respondents were asked to provide a rationale. Tables 17 and 18 indicate the number and percent of SAUs that planned fewer than 8, 8 to 12, and more than 12 assessments in English Language Arts and Mathematics respectively. Table 17. Number of Assessments in English Language Arts By Grade Span

Table 18. Number of Assessments in Mathematics by Grade Span

The majority of respondents indicated that they planned to administer 8 to 12 assessments in both English Language Arts and Mathematics, a range suggested by MDOE in the LAS Guide, Principles and Criteria for the Adoption of Local Assessment Systems. Respondents were also asked to provide a narrative rationale for planning more than 12 assessments. Although responses varied, most SAUs indicated that additional assessments were needed to fully assess all standards and ensure fairness and sufficiency. Some SAUs indicated they needed additional assessments for their early literacy initiatives. Other SAUs reported a decision to assess performance indicators singly rather than multiply, which increased the total number of assessments. It was also common for SAUs to indicate that they are now in the process of refining their assessment plans to reduce the number of assessments given. Distribution of Assessments by Type Respondents were asked to indicate the type and number of certification assessment tasks planned for each grade span in five content areas. These data are represented in Table 19.

27 15.3% 19 10.8% 9 8.2%

126 71.6% 134 76.1% 93 84.5%

23 13.1% 23 13.1% 8 7.3%

Fewer than 8

8 - 12

More than 12

Number of Assessments:

English/Language Arts

n %

K - 4

n %

5 - 8

n %

9 - 12

24 14.5% 18 10.8% 7 6.4%

108 65.1% 126 75.9% 71 64.5%

34 20.5% 22 13.3% 32 29.1%

Fewer than 8

8 - 12

More than 12

Number of

Assessments: Math

n %

K - 4

n %

5 - 8

n %

9 - 12

27

Table 19. Percent of Assessment Tasks by Type

The data in Table 19 indicate that in the K - 4 grade span 45.5% of assessment tasks planned by SAUs for English Language Arts and 38.4% of assessment tasks planned for Mathematics were locally developed (Local). LAD assessment tasks predominate for Mathematics, Science, Social Studies, and Health and Physical Education. This predominance of LAD tasks may be due to elementary teachers’ need to develop assessments for multiple content areas. In grade span 5 - 8 the majority of assessment tasks planned for English Language Arts were locally developed while LAD tasks were the predominate choices for Science, Social Studies, and Health and Physical Education. In grade span 9 - 12 locally developed assessment tasks were the predominant forms of assessment tasks planned for all content areas except Health and Physical Education for which LAD assessment tasks were planned. Except for K - 4 assessments of Health and Physical Education, MAP assessment tasks were less frequently planned for use in all grade levels and content areas. Current Use of Local Assessment System Data Respondents were asked to indicate the ways in which assessment data were being used in their SAUs. Seven choices were offered and respondents were asked to “check all that apply” and offered the option of briefly describing “Other.” These data are displayed in Table 20.

ELA Math Science SS Health

MAP 16.29% 19.37% 18.88% 13.76% 19.98%

LAD 38.17% 42.20% 61.69% 63.69% 65.32%

Local 45.54% 38.43% 19.44% 22.54% 14.70%

MAP 21.74% 20.24% 25.10% 8.37% 19.30%

LAD 38.21% 36.00% 48.31% 61.53% 60.10%

Local 40.05% 43.76% 26.59% 30.09% 20.60%

MAP 18.89% 17.27% 14.92% 8.50% 21.18%

LAD 30.20% 27.80% 38.23% 39.15% 51.68%

Local 50.91% 54.93% 46.85% 52.35% 27.14%

K - 4

5 - 8

9 - 12

28

Table 20. Use of Local Assessment System Data

Data in Table 20 indicate that LAS data were most frequently used in an informal way by teachers to make decisions about curriculum and instruction (82.9%). Somewhat less frequent was the use of LAS data to identify students’ needs for remediation (60.4%), the revision of curriculum (59.4%), the identification of curriculum strengths and areas of need (51.3%), and to inform priorities for professional development (47.6%). The least frequently checked uses of LAS data included formal analysis across grade levels and groups of students (30.5%). This level of analysis of LAS data across grade levels is essential if SAUs are to create a coherent curriculum based in the Learning Results. Twenty-two and a half percent of SAUs indicated that LAS data was being used to inform the community about students’ progress toward achieving the Learning Results. This may reflect the public discussion and ambiguity about changing diploma criteria that existed during the period of time the survey was distributed. Status of Selected Implementation Activities Respondents were asked to rate the progress of work on specific activities related to scoring and reporting data and developing procedures and policies related to LASs. In addition, they were asked to provide brief descriptions of strategies they had developed to provide supplemental instruction prior to replacement assessments, and to determine the value of prior educational experience for transfers. Respondents’ ratings of the status of work in their SAUs are described by grade span in Tables 21, 22, and 23 below. A summary of strategies follows the Tables.

32 17.1% 155 82.9%

130 69.5% 57 30.5%

91 48.7% 96 51.3%

76 40.6% 111 59.4%

98 52.4% 89 47.6%

74 39.6% 113 60.4%

145 77.5% 42 22.5%

172 92.0% 15 8.0%

LAS data used informally in

teachers' decisions

LAS data formally analyzed

across grade levels

LAS data used to identify

curriculum strengths and

areas of need

LAS data used to revise

curriculum

LAS data used to inform

priorities for professional

development

LAS data used to identify

students' needs for

remediation

LAS data used to inform the

community about students'

progress

LAS data (Other)

n %

Not checked

n %

Checked

29

Table 21. Status of Implementation Activities K - 4

Table 22. Status of Implementation Activities 5 - 8

Table 23. Status of Implementation Activities 9 - 12

31 16.9% 28 15.3% 49 26.8% 75 41.0%

42 22.8% 65 35.3% 63 34.2% 14 7.6%

46 25.1% 55 30.1% 54 29.5% 28 15.3%

50 27.2% 85 46.2% 39 21.2% 10 5.4%

133 72.3% 37 20.1% 10 5.4% 4 2.2%

43 23.9% 66 36.7% 45 25.0% 26 14.4%

Selected "percent of points" or "pattern of

performance" for performance level

Developed procedure for public reporting of scores

Developed a policy for replacement assessments

Developed a means of providing supplemental

instruction prior to replacement assessments

Developed means of determining the value of prior

educational experience for transfers

Developed a report card describing progress toward

meeting content standards

n %

No action taken

on this yet

n %

Planning in

progress

n %

Partially

complete

n %

Work complete

29 16.2% 25 14.0% 48 26.8% 77 43.0%

39 21.7% 63 35.0% 65 36.1% 13 7.2%

42 23.5% 57 31.8% 52 29.1% 28 15.6%

41 22.8% 93 51.7% 36 20.0% 10 5.6%

130 72.2% 36 20.0% 10 5.6% 4 2.2%

46 26.4% 73 42.0% 42 24.1% 13 7.5%

Selected "percent of points" or "pattern of performance"

for performance level



Developed a means of providing supplemental instruction

prior to replacement assessments





n %

No action taken

on this yet

n %

Planning in

progress

n %

Partially

complete

n %

Work complete

12 10.5% 12 10.5% 27 23.7% 63 55.3%

22 19.1% 35 30.4% 44 38.3% 14 12.2%

19 16.7% 33 28.9% 37 32.5% 25 21.9%

14 12.2% 44 38.3% 46 40.0% 11 9.6%

81 70.4% 17 14.8% 11 9.6% 6 5.2%

33 30.0% 51 46.4% 15 13.6% 11 10.0%

Selected "percent of points" or "pattern of performance"

for performance level



Developed a means of providing supplemental

instruction prior to replacement assessments





n %

No action taken

on this yet

n %

Planning in

progress

n %

Partially

complete

n %

Work complete

30

The data in Tables 21, 22, and 23 indicate that work was more complete in all activities in grade span 9 - 12 with the exception of the development of report cards describing progress toward meeting standards. More SAUs in the K - 4 grade span indicated Work complete on this activity. Ninety-eight SAUs reported that they were using traditional report cards while they were developing or introducing a standards based report card for a grade span. Providing Supplemental Instruction Prior to Replacement Assessments. Most SAUs described providing before school, in-school and after-school remedial instruction to students who do not meet standards. Title I programs were frequently used in grade spans K - 4 and 5 - 8. Learning Labs are frequently mentioned as strategies in grade span 9 - 12. The use of Ed Techs and part-time in-school tutors with pull-out programs were additional strategies mentioned. One high school described the use of scheduled Academic Support Periods which were taught by faculty during the school day or after school and require that students attend for several days a week over a minimum 3-week period. Regional summer programs were also being piloted. Developing a Means of Determining the Value of Prior Educational Experience for Transfers. It is clear from the responses to this item that SAUs were very concerned about accepting reports of LAS proficiencies transfer students may have demonstrated in other SAUs either within Maine or in other states. A few SAUs indicated they plan to give transfer students placement tests. Some hope to review data through MEDMS. Others indicated the need for more guidance in this area. Most SAUs were just beginning to explore options and develop policy regarding students who transfer. Typical responses indicate that the traditional strategies of having the guidance counselor or principal make a judgment were unchanged. On a related survey item that asked SAUs if they had a policy or procedure to address how AFS or exchange students would meet the standards, only five SAUs indicated they had a policy or procedure in place. Respondents were asked to rate the progress made by their SAUs in developing a LAS that meets technical standards for assessing students with specific needs. Data summarizing these responses are found in Table 24. Table 24. Status of Work Developing LASs for Students with Specific Needs

These data indicate that work was more complete in providing accommodations and alternative assessments and meeting the assessment needs of students with 504 plans than it was for assessing students who were English Language Learners (ELL). This finding must be read with caution as 16 respondents (8.6%) did not rate progress on ELL perhaps indicating that they have

16 8.6% 102 54.5% 24 12.8% 37 19.8% 8 4.3%

10 5.3% 36 19.3% 51 27.3% 51 27.3% 39 20.9%

7 3.7% 9 4.8% 54 28.9% 81 43.3% 36 19.3%

5 2.7% 14 7.5% 48 25.7% 74 39.6% 46 24.6%

English Language Learners

504 Plans

Require Accommodations

Alternative Assessments

n %

Missing

n %

No action taken

on this yet

n %

Planning in

progress

n %

Partially

complete

n %

Work complete

31

no ELL students. It cannot be determined from the survey data whether the 55.5% of respondents that indicated “No action taken…” provided this rating because they had no ELL students, or in fact, they had ELL students but had not taken action on this task. Respondents were asked to rate the progress made by their SAUs in developing a LAS that meets technical standards for students in Career and Technical Education and Alternative Education programs. Responses were sorted to include only SAUs with K - 12 and 9 - 12 programs. These data are included in Table 25. Table 25. Status of Work Developing LASs to Assess Students in CTE and Alternative Education

Table 25 indicates that the majority of districts had either taken no action on the task of developing assessments for students in Career and Technical Education and Alternative Education programs or had planning in progress.

Collaboration Among SAUs Respondents were asked if their SAUs were currently collaborating with one or more SAUs for the purpose of developing a LAS. Fifty-five percent (54.7%) indicated that they were collaborating, and 45.3% indicated that they were not collaborating. SAUs that indicated they were collaborating were asked to describe how collaboration has been helpful. Most responses to this item provided the name of the collaboration or partnership. These included the Maine Center for Educational Services, Penobscot River Education Partnership, Southern Maine Partnership, Western Maine Partnership, Down East Education Partnership, Casco Bay Education Alliance, and the Per Pupil Professional Development Collaborative. Professional development was the most common purpose listed although a few SAUs indicated that they were sharing assessment and curriculum ideas.

Level and Type of Help Needed Respondents were asked to indicate the level of help needed by their SAU to build the capacity to accomplish specific tasks related to implementing a LAS. A scale was provided that ranged from 1 to 5 with 1 indicating No help needed, 3 indicating Moderate help needed, and 5 indicating A lot of help needed. The data displayed in Tables 26 and 27 summarizes these data for K - 5 and K - 8 districts (Table 26) and K - 12 and 9 - 12 districts (Table 27).

3 3.1% 45 46.4% 34 35.1% 10 10.3% 5 5.2%

9 9.3% 39 40.2% 23 23.7% 20 20.6% 6 6.2%

Career and Tech Ed

Alternative Ed

n %

Missing

n %

No action taken

on this yet

n %

Planning in

progress

n %

Partially

complete

n %

Work complete

Only K - 12 and 9 - 12 districts were included in this table.

32

Table 26. Level of Help Needed K - 5 and K - 8 Districts

Table 27. Level of Help Needed K - 12 and 9 - 12 Districts

2 2.2% 30 33.3% 19 21.1% 32 35.6% 5 5.6% 2 2.2%

8 8.9% 20 22.2% 38 42.2% 13 14.4% 8 8.9% 3 3.3%

2 2.2% 33 36.7% 28 31.1% 13 14.4% 12 13.3% 2 2.2%

2 2.2% 32 35.6% 42 46.7% 3 3.3% 4 4.4% 7 7.8%

4 4.4% 12 13.3% 13 14.4% 15 16.7% 15 16.7% 31 34.4%

4 4.4% 8 8.9% 9 10.0% 19 21.1% 15 16.7% 35 38.9%

2 2.2% 9 10.0% 18 20.0% 33 36.7% 12 13.3% 16 17.8%

2 2.2% 15 16.7% 10 11.1% 32 35.6% 15 16.7% 16 17.8%

4 4.4% 12 13.3% 23 25.6% 19 21.1% 21 23.3% 11 12.2%

2 2.2% 11 12.2% 22 24.4% 40 44.4% 5 5.6% 10 11.1%

3 3.3% 13 14.4% 22 24.4% 28 31.1% 12 13.3% 12 13.3%

40 44.4% 7 7.8% 2 2.2% 21 23.3% 11 12.2% 9 10.0%

35 38.9% 6 6.7% 2 2.2% 30 33.3% 8 8.9% 9 10.0%

23 25.6% 17 18.9% 11 12.2% 21 23.3% 3 3.3% 15 16.7%

3 3.3% 18 20.0% 17 18.9% 27 30.0% 2 2.2% 23 25.6%

Help select assessments

Help establish reliability

Help administer asessments

Help train teachers

Help dev system to record LAS

Help dev system to analyze LAS

Help use LAS data for curr content

Help use LAS data for curr

sequence

Help dev policies related to

assessment

Help dev accom to assess progress

Help dev alt assess (PAAPS)

Help assess CTE students

Help asess Alt Ed programs

Help assess ELL students

Help develop report card

n %

Missing

n %

No help needed

n %

2.00

n %

Moderate help

needed

n %

4.00

n %

A lot of help

needed

3 3.1% 45 46.4% 15 15.5% 25 25.8% 7 7.2% 2 2.1%

2 2.1% 34 35.1% 24 24.7% 25 25.8% 8 8.2% 4 4.1%

2 2.1% 60 61.9% 17 17.5% 14 14.4% 2 2.1% 2 2.1%

2 2.1% 39 40.2% 20 20.6% 28 28.9% 7 7.2% 1 1.0%

1 1.0% 18 18.6% 19 19.6% 22 22.7% 14 14.4% 23 23.7%

2 2.1% 13 13.4% 15 15.5% 17 17.5% 25 25.8% 25 25.8%

1 1.0% 23 23.7% 28 28.9% 17 17.5% 20 20.6% 8 8.2%

1 1.0% 25 25.8% 21 21.6% 22 22.7% 22 22.7% 6 6.2%

4 4.1% 22 22.7% 18 18.6% 27 27.8% 13 13.4% 13 13.4%

3 3.1% 26 26.8% 19 19.6% 36 37.1% 8 8.2% 5 5.2%

4 4.1% 23 23.7% 25 25.8% 20 20.6% 20 20.6% 5 5.2%

7 7.2% 11 11.3% 11 11.3% 31 32.0% 22 22.7% 15 15.5%

10 10.3% 15 15.5% 9 9.3% 33 34.0% 18 18.6% 12 12.4%

11 11.3% 17 17.5% 9 9.3% 26 26.8% 21 21.6% 13 13.4%

2 2.1% 24 24.7% 18 18.6% 24 24.7% 12 12.4% 17 17.5%

Help select assessments

Help establish reliability

Help administer asessments

Help train teachers

Help dev system to record LAS

Help dev system to analyze LAS


content


sequence

Help dev policies related to

assessment

Help dev accom to assess

progress

Help dev alt assess (PAAPS)

Help assess CTE students

Help asess Alt Ed programs

Help assess ELL students

Help develop report card

n %

Missing

n %

No help needed

n %

2.00

n %

Moderate help

needed

n %

4.00

n %

A lot of help

needed

33

When the percent of SAUs indicating Moderate help needed to A lot of help needed are totaled in Tables 26 and 27, it appears that the needs for help differed somewhat by grade span. K - 5 and K - 8 SAUs indicated the highest needs were for assistance in developing a system for analyzing LAS data (76.7%), using LAS data to inform curriculum sequence (70.1%), using LAS data to inform curriculum content (67.8%), and developing a system for recording LAS data (67.8%). K - 12 and 9 - 12 SAUs indicated that their highest need for assistance includes help assessing CTE students (70.2%), developing systems to analyze LAS data (69.1%), assessing students in alternative education programs (65.0%), and assessing students in programs for English Language Learners (61.8%). Confidence Respondents were asked to indicate how confident they are that their SAU will have assessments in place to certify that graduates will meet the Learning Results in English Language Arts and Mathematics in 2008. Confidence ratings ranged from 1, Not at all confident to 4, Very confident. Responses to these items are displayed in Table 28. Table 28. Confidence in Graduates’ Meeting Learning Results

Table 28 indicates that somewhat more SAUs were highly confident that certification assessments will be in place in the area of English Language Arts in 2008 than in Mathematics. This question required that respondents speculate about the future and readers are cautioned about interpretation of these results. Percent of Graduates Estimated to Meet Learning Results Respondents were asked to estimate what percent of their graduates would meet the Learning Results in English Language Arts and Mathematics in 2008. Percentage ranges were provided as indicated in Table 29. Table 29. Percent of Graduates Estimated to Meet Learning Results

12 9.5% 20 15.9% 40 31.7% 54 42.9%

10 7.9% 18 14.3% 48 38.1% 50 39.7%

Confidence SAU will have assessments in place to

certify that 2008 graduates meet MLR in ELA

Confidence SAU will have assessments in place to

certify that 2008 graduates meet MLR in Math

n %

1: Not at all

confident

n %

2

n %

3

n %

4: Very

Confident

7 5.9% 19 16.1% 56 47.5% 36 30.5%

9 7.8% 26 22.4% 48 41.4% 33 28.4%

Estimated percent of 2008 high school graduates

achieving LR in English/Language Arts

Estimated percent of 2008 high school graduates

achieving LR in Mathematics

n %

Less than 25%

n %

26% - 50%

n %

51% - 75%

n %

76% - 100%

34

Seventy eight percent respondents indicated that more than 51% of their graduates would meet the Learning Results in English/Language Arts in 2008. Slightly fewer (69.8%) indicated the same percentage range for Mathematics. This question required respondents to speculate about the future. Readers are therefore cautioned about interpretation of these results.

Understanding the Requirements Respondents were asked to rate parents’ and students’ understanding of the graduation requirements held for the graduates of 2008. A scale of 1 to 5 was used for this item with 1 defined as Do not understand at all, and 5 meaning Completely understand. Table 30 represents these data. Table 30. Parent and Student Levels of Understanding

Table 30 indicates that more SAUs viewed freshman as having a greater understanding of the graduation requirements that will apply in 2008 than parents. However, only 5.3% of SAUs viewed freshmen as fully understanding these requirements. Changes to Support the Class of 2008 Meeting the Standards SAUs indicated that they have made a wide variety of changes to support 2008 graduates in meeting the standards (see Appendix E). Some areas of change included: curriculum and coursework; school schedules; advisory programs; after-school remediation programs; tutoring programs; learning labs/learning centers; emphasis on literacy and/or mathematics; and staffing. Positive and Negative Impacts of LAS Implementation SAUs were asked to describe positive and negative impacts of LAS implementation. Responses to this item were quite numerous, with the number of positive written comments (334) almost equal to the number of negative comments (359). Responses were organized by themes or topics. Positive impacts cited most frequently included the following: improved accountability; a better understanding of assessment among educators; increased/ improved communication about teaching and learning; increased/ improved teacher collaboration; more focused/ aligned curriculum with Learning Results; and better coherency/ consistency in curriculum and goals within SAUs and across state. Other positive impacts cited somewhat less frequently were: using data to inform instruction; improved instruction; higher standards or increased expectations/ rigor for student learning, and professional development.

20 15.2% 74 56.1% 35 26.5% 3 2.3%

5 3.8% 55 42.0% 64 48.9% 7 5.3%

Parent understanding of 2008 grad

requirements

Freshman understanding of 2008 grad

requirements

n %

Do not

understand at all

n %

2

n %

3

n %

Completely

understand

35

Negative impacts cited most frequently included the following: assessments not fully embedded or integrated with instruction; too much focus on or too many assessments; teacher stress, anxiety or frustration/ lower teacher morale; less time for some curricular or instructional units; teachers out of classroom more; and less time for other topics of professional development. Less frequently cited negative impacts were: confusion over NCLB and LAS requirements and confusion due to changes in LAS rules; drain on SAU resources/ increased teacher workload; and problems communicating with parents. Boosters and Barriers to LAS Implementation SAUs were asked to describe boosters and barriers other than the resources of time and money that have facilitated or impeded progress (see Appendix F). Some of the statements made by SAUs pertained more to positive and negative impacts, rather than boosters and barriers. These statements were included in the analysis for the survey item on impacts. Responses were organized by themes or topics. Chief boosters cited included: resources, guidance, and professional development provided by the MDOE (such as LAD and MAP tasks); professional development provided by educational organizations; having an organizational structure within the SAU to do LAS work; committed staff; principal and teacher leadership; and new staff positions. Chief barriers cited were: the scope of required LAS work and number of assessments; lack of good assessment models in some areas; unclear communication and frequent changes in LAS requirements; negative media coverage of LAS requirements; lack of data management systems for LAS; resistance and anxiety among educators; increased workload without compensation; and the lack of a curriculum coordinator position within some SAUs.

Supplementary Analysis Two progress groups were identified to determine whether there are characteristics that distinguish SAUs that are further along in developing and implementing their LAS (Advanced) from those that are less developed (Less Advanced). Advanced SAUs were defined as those that indicated Partially complete or Work complete on all of the following tasks for both English/Language Arts and Mathematics on the 2005 survey:

• Developed curriculum documents aligned with Maine Learning Results, • Aligned instruction with curriculum and Maine Learning Results, • Selected a sample of performance indicators to assess using Maine’s Balance of

Representation or some other method of prioritization, • Used scoring rubrics for each assessment, • Administered common assessments, • Provided professional development in common scoring for teachers, • Double scored common assessments, • Established and documented reliability among scorers, and • Used data to inform the design of curriculum and instruction in both English and

Mathematics at all relevant grade levels.

36

Fifty-five SAUs were identified that met the definition of Advanced. The remaining 132 districts were characterized as Less Advanced. Distribution of Advanced and Less Advanced SAUs by Region Table 31. Progress Groups by Superintendent Region

Table 31 indicates that the highest percentages of SAUs that were Less Advanced in the design and implementation of LASs were in the Midcoast region, Western Maine, and Kennebec County. The highest percentages of SAUs that were Advanced in LAS development were in the Unorganized Territories, Hancock County, Cumberland County, and Washington County. Overall, it appears that superintendent region was not a predictor of progress status. Leadership in Advanced and Less Advanced SAUs Table 32. Job Roles Responsible for LAS Development by Progress Group

The data in Table 32 address the issue of leadership. The role of Curriculum Coordinator was viewed as the primary leadership role for developing a LAS. However, the data indicate that there is little difference between the percent of Advanced and Less Advanced SAUs that designated the role of Curriculum Coordinator as having this responsibility. The data also

n % n %

Aroostook 12 75.00% 4 25.00%

Cumberland 12 63.16% 7 36.84%

Hancock 18 62.07% 11 37.93%

Kennebec 24 82.76% 5 17.24%

Midcoast 14 93.33% 1 6.67%

Penquis 23 71.88% 9 28.13%

Unorganized Territories 0 0.00% 6 100.00%

Washington 11 64.71% 6 35.29%

Western Maine 10 83.33% 2 16.67%

York 8 66.67% 4 33.33%

Less Advanced Advanced

n % n %

Superintendent 22 16.67% 10 18.18%

Curriculum Coordinator 72 54.55% 31 56.36%

Assistant Principal 1 0.76% 0 0.00%

Headmaster 0 0.00% 0 0.00%

Assistant Superintendent 21 15.91% 7 12.73%

Principal 29 21.97% 16 29.09%

Teacher Leader 29 21.97% 4 7.27%

Other 20 15.15% 11 20.00%


37

indicate that Advanced SAUs more frequently reported the role of Principal as the key leader while Less Advanced SAUs report Teacher Leaders in this role. Table 33. Number of Leaders of LAS Development by Progress Group

Respondents were requested to select up to two professional roles as having primary responsibility for leadership of LAS development and implementation. Table 33 indicates that there was little difference between Advanced and Less Advanced SAUs in the number of roles assigned this responsibility. Other Characteristics of Advanced and Less Advanced SAUs The 2005 survey was coded so that SAUs could be identified by the research team. This enabled survey data to be analyzed with other available data. Table 34 compares Advanced and Less Advanced SAUs on enrollment, percent of free-and-reduced lunch, elementary and secondary per-pupil expenses (2003-2004 data), and students per teacher. T-tests revealed no significant differences between Advanced and Less Advanced SAUs. Table 34. District Characteristics by Progress Group*

*No statistically significant differences were found. Help Needed by Advanced and Less Advanced SAUs The 2005 survey asked respondents to indicate the level of help needed by their SAU to build the capacity to accomplish 15 different tasks related to the development and implementation of a LAS. Responses were requested on a 5-point scale where 1 indicated No help needed and 5, indicated A lot of help needed. Ratings of “4” and “5” were totaled and these totals were analyzed separately for the Advanced and Less Advanced groups. The results of this analysis are displayed in Table 35.

n Mean Median n Mean Median

Enrollment 132 947 426 55 893 395

Free/Reduced Lunch 130 36.22% 36.05% 52 39.92% 44.74%

Elementary Per-Pupil Expenses (03-04) 132 $6,630 $6,002 55 $6,055 $5,714

Secondary Per-Pupil Expenses (03-04) 132 $7,355 $7,016 55 $7,245 $6,801

Students per Teacher (03-04) 132 12.68 13.68 54 13.05 13.2


n % n %

Two or more leaders 61 46.21% 25 45.45%

One leader 71 53.79% 30 54.55%


38

Table 35. Top Five Areas of Help Needed*

*Help needed was defined as a response of 4 or 5 (“a lot of help needed”). Table 35 reveals that both Advanced and Less Advanced SAUs indicate that their priorities for help included the development of data management systems to analyze LAS data and the development of a data management system for recording LAS data. Less Advanced SAUs needed help in developing policies related to LAS, and in using LAS data to inform curriculum sequence and content. More Advanced SAUs indicated that priority needs for help were in developing the capacity to assess students who are English Language Learners, those in Career and Technical Education programs, and those who need alternative assessments. Use of LAS data by Advanced and Less Advanced SAUs Respondents were asked to indicate how LAS data was being used in their SAUs. Seven choices were provided and respondents were asked to check all that applied. Table 36 allows comparison of the use of data by Advanced and Less Advanced SAUs.

Table 36. Use of LAS Data by Progress Groups

Table 36 reveals that both Advanced and Less Advanced SAUs most frequently indicated the use LAS data informally in teachers’ discussions about curriculum and instruction, although Advanced SAUs indicated this use more frequently. A higher percentage of Advanced SAUs indicated the use of LAS data to inform priorities for professional development than do Less

Area

% of

Districts Area

% of

Districts

Help develop a data management system to analyze

LAS 62.12%

Help develop a data management system to analyze

LAS 32.70%

Help develop a data management system to record

LAS 52.27%

Help develop a data management system to record

LAS 25.45%

Help develop policies related to assessment 39.39% Help assess ELL students 25.45%

Help use LAS data to inform curriculum sequence 37.12% Help assess CTE students 23.64%

Help use LAS data to inform curriculum content 35.61% Help develop alternative assessments (PAAPS) 20.00%


n % of Districts n % of Districts

LAS data used informally in teachers' discussions about curr. and inst. 106 80.30% 49 89.09%

LAS data formally analyzed across grade levels and groups of students 41 31.06% 16 29.09%

LAS data used to identify curriculum strengths and areas of need 68 51.52% 28 50.91%

LAS data used to revise curriculum 77 58.33% 34 61.82%

LAS data used to inform priorities for professional development 59 44.70% 30 54.55%

LAS data used to identify students' needs for remediation 79 59.85% 34 61.82%

LAS data used to inform the community about students' progress 27 20.45% 15 27.27%

LAS data (Other) 13 9.85% 2 3.64%


39

Advanced SAUs. More Advanced SAUs also used LAS data to inform the community about students’ progress than do Less Advanced SAUs.

Comparison of 2004 and 2005 Survey Items Some items on the 2004 and 2005 surveys remained identical and permit comparison. However, some caution is urged due to differences in the distribution of the surveys. In 2004 the survey was mailed to all superintendents with instructions to duplicate the survey if more than one LAS was being developed under their supervision. Although the return rate was 67%, it was unclear how many individual LASs were represented since individual SAU codes were not included on the survey. The 2005 survey was mailed to all superintendents with schools under their supervision with SAU identification codes. Superintendents who supervised more than one SAU were contacted to determine how many surveys they would require and reminders were sent when surveys were not returned. This resulted in a return rate of 83%. The difference in return rates, 67% in 2004 and 83% in 2005, and the distribution across regions must be considered in comparing 2004 and 2005 data. Progress in English Language Arts Four items related to progress in English Language Arts remained the same on the 2004 and 2005 surveys. These are included in Tables 31, 32, and 33 below which allow comparison of the percent of SAUs indicating the level to which work was complete on four tasks related to designing and implementing a LAS. The survey question asked respondents to indicate the level of progress made on each task on a scale that included 1, no action taken; 2, Planning in progress; 3, Partially complete; and 4, Work complete. These ratings resulted in indications of the current status of work at two specific points in time, June 2004 and May 2005. Comparison of these ratings can be considered as an assessment of progress. In Tables 37, 38, and 39 the number and percent of SAUs that indicated each level of progress is represented. In addition, shaded areas have been added to highlight composite ratings. For each task, the number and percentage of SAUs indicating No action taken and Planning in progress has been totaled to form a composite percentage. The same calculation and shading has been done for Partially complete and Work complete. This strategy allows a more wholistic examination of the data.

40

Table 37. Progress in English/Language Arts K - 4

2 1.7% 1 .5%

8 6.7% 10 5.4%

10 8.4% 11 5.9%

30 25.2% 86 46.5%

78 65.5% 87 47.0%

108 90.7% 173 93.5%

1 .5%

11 9.3% 10 5.4%

11 9.3% 11 5.9%

65 55.1% 116 63.0%

41 34.7% 56 30.4%

106 89.8% 172 93.4%

2 1.7% 7 3.7%

8 6.7% 7 3.7%

10 8.4% 14 7.4%

31 26.1% 58 31.0%

77 64.7% 114 61.0%

108 90.8% 172 92.0%

4 3.5% 5 2.7%

23 20.0% 16 8.6%

27 23.5% 21 11.3%

49 42.6% 61 32.8%

38 33.0% 103 55.4%

87 75.6% 164 88.2%

No action taken on this yet


No action taken/Planning in progress

Partially complete

Work complete

Partially complete/Work complete

Developed curriculum documents

aligned with MLR




Partially complete

Work complete


Aligned instruction with

curriculum and MLR




Partially complete

Work complete


Selected a sample of performance

indicators for assessment




Partially complete

Work complete


Provided professional

development in common scoring

for teachers

n %

2004

n %

2005

41

Table 38. Progress in English/Language Arts 5 - 8

2 1.7% 1 .5%

9 7.7% 14 7.6%

11 9.4% 15 8.2%

28 23.9% 85 45.9%

77 65.8% 84 45.4%

105 89.7% 169 91.3%

1 .5%

14 12.1% 17 9.2%

14 12.1% 18 9.7%

62 53.4% 110 59.8%

39 33.6% 55 29.9%

101 87.0% 165 89.7%

2 1.7% 7 3.8%

5 4.3% 8 4.3%

7 6.0% 15 8.1%

32 27.6% 60 32.3%

76 65.5% 110 59.1%

108 93.1% 170 91.4%

5 4.5% 5 2.7%

21 19.1% 14 7.5%

26 23.6% 19 10.2%

49 44.5% 67 35.8%

34 30.9% 100 53.5%

83 75.4% 167 89.3%




Partially complete

Work complete


Developed curriculum

documents aligned with

MLR




Partially complete

Work complete



curriculum and MLR




Partially complete

Work complete


Selected a sample of

performance indicators for

assessment




Partially complete

Work complete



development in common


n %

2004

n %

2005

42

Table 39. Progress in English/Language Arts 9 - 12

Table 37 indicates that overall, there has been progress on all four tasks related to English Language Arts between 2004 and 2005 even though a smaller percentage of SAUs are reporting their work on these tasks as Work complete. Fewer SAUs are rating their work as No action taken and Planning in progress. Table 38 shows a similar pattern for grade span 5 - 8 except for the task, “Selected a sample of performance indicators for assessment.” These data indicate that a slightly smaller percentage of schools are indicating Work complete for this task, and slightly more SAUs are indicating No action taken yet. Table 39 indicates overall increases in the percentage of SAUs reporting Partially complete or Complete on all tasks related to English Language Arts except the task, “Selected a sample of performance indicators for assessment.” Although a high percentage of SAUs reported their work on this task to be Complete, a slightly higher percentage indicated No action taken yet. This may be a result of a somewhat larger sample in 2005.

1 1.0%

8 8.3% 3 2.6%

9 9.3% 3 2.6%

21 21.9% 53 45.7%

65 67.7% 59 50.9%

86 89.6% 112 96.6%

14 14.7% 6 5.3%

14 14.7% 6 5.3%

45 47.4% 59 52.2%

35 36.8% 47 41.6%

80 84.2% 106 93.8%

1 1.1% 5 4.3%

3 3.2% 4 3.4%

4 4.3% 9 7.7%

24 25.3% 23 19.7%

66 69.5% 84 71.8%

90 94.8% 107 91.5%

5 5.4% 2 1.7%

24 26.1% 6 5.2%

29 31.5% 8 6.9%

35 38.0% 42 36.2%

27 29.3% 65 56.0%

62 67.3% 107 92.2%




Partially complete

Work complete




MLR



Partially complete

Work complete



curriculum and MLR




Partially complete

Work complete




assessment




Partially complete

Work complete





n %

2004

n %

2005

43

In summary, the data pertaining to English Language Arts indicate overall progress had been made between 2004 and 2005 for all grade spans in developing curriculum aligned with the Learning Results, aligning instruction with curriculum and the Maine Learning Results, and providing professional development in common scoring for teachers. Progress had also been made in the task, “Selecting a sample of performance indicators for assessment” in grade span K - 4, but not in grade span 5 - 8. A slightly higher percentage of SAUs were reporting Work complete on this item in 2005, but there was also an increase in the percent of SAUs reporting No action taken and Planning in progress. Progress in Mathematics Assessing progress in the design and implementation of LASs in Mathematics followed the same pattern as the previous analysis of progress in English Language Arts. Tables 40, 41, and 42 display the data used in this analysis. Table 40. Progress in Mathematics K - 4

2 1.7% 2 1.1%

7 5.9% 7 3.8%

9 7.6% 9 4.9%

25 21.0% 78 42.2%

84 70.6% 97 52.4%

109 91.6% 175 94.6%

1 .8% 1 .5%

12 10.2% 13 7.0%

13 11.0% 14 7.5%

51 43.2% 101 54.6%

53 44.9% 69 37.3%

104 88.1% 170 91.9%

2 1.7% 7 3.8%

7 5.9% 9 4.9%

9 7.6% 16 8.7%

29 24.6% 59 32.1%

79 66.9% 108 58.7%

108 91.5% 167 90.8%

7 6.0% 8 4.4%

30 25.9% 18 9.9%

37 31.9% 26 14.3%

52 44.8% 60 33.0%

26 22.4% 95 52.2%

78 67.2% 155 85.2%




Partially complete

Work complete




MLR




Partially complete

Work complete



curriculum and MLR




Partially complete

Work complete




assessment




Partially complete

Work complete





n %

2004

n %

2005

44

Table 41. Progress in Mathematics 5 - 8

2 1.7% 2 1.1%

9 7.8% 7 3.8%

11 9.5% 9 4.9%

22 19.1% 77 41.6%

81 70.4% 98 53.0%

103 89.5% 175 94.6%

1 .9% 1 .5%

14 12.3% 16 8.6%

15 13.2% 17 9.1%

50 43.9% 100 54.1%

48 42.1% 67 36.2%

98 86.0% 167 90.3%

2 1.8% 7 3.8%

4 3.5% 8 4.3%

6 5.3% 15 8.1%

29 25.7% 66 35.9%

77 68.1% 102 55.4%

106 93.8% 168 91.3%

6 5.4% 8 4.4%

31 27.7% 17 9.4%

37 33.1% 25 13.8%

48 42.9% 71 39.2%

26 23.2% 84 46.4%

74 66.1% 155 85.6%




Partially complete

Work complete




MLR




Partially complete

Work complete



curriculum and MLR




Partially complete

Work complete



performance indicators

for assessment




Partially complete

Work complete





n %

2004

n %

2005

45

Table 42. Progress in Mathematics 9 - 12

An analysis of the data displayed in Tables 40 (grade span K - 4) and 41 (grade span 5 - 8) reveals the same pattern of progress indicators. On the tasks of “Developing curriculum aligned with the Maine Learning Results” and “Aligning instruction with curriculum and Maine Learning Results” a smaller percentage of SAUs indicates No action taken and Planning in progress in 2005 than in 2004. Over the same time period a larger percentage of SAUs indicate that their work is Partially complete. Although fewer SAUs rate these tasks as Work complete, the higher percentage of SAUs indicating Partially complete demonstrates progress. In both grade spans, K - 4 and 5 - 8, fewer SAUs indicate that their work on the task, “Selected a sample of performance indicators for assessment,” is Complete in 2005 than in 2004. However, there was an increase in the percent indicating Partially complete on this task indicating that the work has progressed but is not yet complete. Table 42, representing the grade span 9 - 12 indicates overall progress in Mathematics between 2004 and 2005 on all tasks except “Selected a sample of performance indicators for assessment.”

1 .9%

8 8.3% 3 2.6%

8 8.3% 4 3.5%

18 18.8% 39 33.6%

69 71.9% 72 62.1%

87 90.7% 111 95.7%

1 1.1%

12 12.6% 5 4.3%

13 13.7% 5 4.3%

39 41.1% 57 49.1%

42 44.2% 53 45.7%

81 85.3% 110 94.8%

2 2.1% 5 4.3%

1 1.1% 2 1.7%

3 3.2% 7 6.0%

20 21.3% 20 17.2%

70 74.5% 88 75.9%

90 95.8% 108 93.1%

5 5.4% 1 .9%

31 33.3% 12 10.6%

36 38.7% 13 11.5%

32 34.4% 37 32.7%

24 25.8% 62 54.9%

56 60.2% 99 87.6%




Partially complete

Work complete




MLR




Partially complete

Work complete



curriculum and MLR




Partially complete

Work complete




assessment




Partially complete

Work complete





n %

2004

n %

2005

46

Although the data reflect a very slight increase in ratings of Work complete, it also indicates a larger increase in the SAUs rating this task as No action taken and Planning in progress. In summary, overall progress between 2003 and 2004 in Mathematics has been made in three of the four tasks compared. Little progress has been made in the task “Selected a sample of performance indicators for assessment” by all grade spans indicating an area of need.

Summary Part II of this report presented data from a statewide survey conducted in May 2005, along with supplemental analysis of that data. Broadly, the survey asked SAUs to indicate their progress on LAS tasks, to estimate what percentage of high school graduates might meet standards in 2008, to describe impacts of LAS implementation, and to describe boosters, barriers, and needs. With a return rate of 83% from 187 SAUs, this survey provides comprehensive and current information about the status of LAS work across the state. Major findings are summarized briefly below. A synthesis of findings and implications from both the statewide survey and the qualitative study is presented in Part I of this report. A comparison of SAU responses on the 2004 and 2005 statewide surveys for four survey items revealed that SAUs have made steady progress across all grade spans in English Language Arts and Mathematics on developing curriculum documents aligned with the Learning Results, aligning instruction and curriculum with the Learning Results, and providing professional development in common scoring. Less progress was also made in selecting performance indicators for assessment, with more progress being made in English Language Arts in grades K-4. Across grade spans, between 91% and 98% of the SAUs indicated on the 2005 survey that they have partially completed or completed work in administering assessments in English Language Arts and Mathematics. Between 79% and 88% of the SAUs indicated work was partially complete or complete in double scoring assessments. Between 73% and 83% of the SAUs indicated they had partially completed or completed work in establishing and documenting reliability among scorers. Between 65% and 72% of the SAUs indicated they had planning in progress or had partially completed work needed to use LAS data to inform the design of curriculum and instruction. For most LAS tasks, a higher percentage of SAUs indicated that work was complete in the 9-12 grade span. For the tasks of establishing reliability and using data to inform decisions, a higher percentage of SAUs indicated work complete in the 5-8 grade span. SAUs also indicated progress in other content areas, though not to the same extent as in English Language Arts and Mathematics. The largest percentage of SAUs indicated that work was complete in the areas of Science and Technology, Social Studies, and Health and Physical Education, particularly in the 9-12 grade span. In the areas of Modern and Classical Languages, Visual and Performing Arts, and Career Preparation, a larger percentage of SAUs indicated that planning was in progress or partially complete in the 5-8 and 9-12 grade spans. These findings indicate that SAUs have placed priority on completing work in English Language Arts and Mathematics and in the 9-12 grade span. This strategy appears to be a response to the state’s graduation requirements and the need to get LASs in place and students ready to meet standards, particularly in English Language Arts and Mathematics.

47

Analysis of the 2005 survey data indicates that the level of progress made by SAUs could not be predicted by any of the following variables: superintendent region, the job role responsible for LAS coordination, attending enrollment, percent of students receiving free and reduced lunch, per pupil expenditures, and student/ teacher ratios. SAUs indicated they use both locally developed assessments and the LAD and MAP tasks disseminated by the state. Locally developed tasks predominate in English Language Arts and Mathematics, and Social Studies in the 9-12 grade span. LAD tasks predominate in the content areas of Science and Technology, Social Studies, and Health and Physical Education in the K-4 and 5-8 grade spans, and in Health and Physical Education in the 9-12 grade span. Across the grade spans, between 7% and 13% of the SAUs indicated they have planned more than 12 certification assessments for English Language Arts, while between 13% and 29% of the SAUs have planned more than 12 assessments for Mathematics. While most SAUs indicated they felt the additional assessments were needed to fully assess the standards and to ensure fairness and sufficiency, other SAUs indicated they were in the process of refining their assessment plans and reducing the number of assessments. SAUs indicated more progress had been made in the 9-12 grade span on the following implementation tasks: selecting “percent of points” or “pattern of performance” for determining students’ performance level; developing a procedure for public reporting of scores, developing a policy of replacement assessments; developing a means of providing supplemental instruction prior to replacement assessments; and developing a means of determining the value of prior educational experience for transfers. More progress had been made in the K-4 grade span on developing a report card describing progress toward meeting the content standards. Survey responses indicated that work was underway in most of these areas, and in the early stages of planning for report card changes. Ninety-eight SAUs reported that they were using traditional report cards while they were developing or introducing a standards-based report card in a grade span. Most SAUs had not begun to make plans for determining the value of prior educational experience for transfers. On a related survey item, only five SAUs indicated they had a policy or procedure to address how AFS or exchange students would meet the requirements of a LAS. SAUs were asked to indicate progress on developing LASs for students with special needs. Sixty-four percent of the SAUs indicated work was partially complete or complete for alternate assessments; 63% of the SAUs indicated this level of progress for students needing accommodations; and 48% of the SAUs indicated this level of progress for students with 504 plans. While 55% of the SAUs indicated no action yet on developing assessments for English language learners, some of these SAUs may not have any students in this category. In the area of Career and Technical Education and Alternative Education, most SAUs (82% for CTE and 64% for alternative education) indicated either no action had been taken or that planning was in progress for developing LASs for students in these programs. Fifty-five percent of the SAUs indicated they were collaborating with other SAUs for the purpose of developing a LAS. SAUs collaborated with a variety of educational organizations and regional collaboratives or partnerships, primarily on professional development. A few SAUs collaborated on sharing assessment and curriculum ideas.

48

SAUs indicated the areas where they most need help. The type of help needed varied by grade span and the SAU’s organizational structure. K-5 and K-8 SAUs indicated priority needs for help in developing a system for analyzing LAS data, using data to inform curriculum sequence and content, and developing a system for recording LAS data. K-12 and 9-12 SAUs indicated priority needs for help in assessing students in Career and Technical Education (CTE) programs, alternative education programs, and English Language Learner (ELL) programs, and developing systems to analyze LAS data. Eighty-three percent of the SAUs indicated that LAS data are being used informally in teachers’ discussions about curriculum and instruction. Sixty percent of the SAUs reported that data was being used to identify students’ needs for remediation. Fifty-nine percent of the SAUs said data are being used to revise curriculum. Fewer SAUs indicated that data are being used in more formal ways or to inform the community about student progress. Clearly, SAUs are concerned about the need to collect, analyze, manage, and report LAS data. SAUs offered an almost equal number of positive and negative impacts resulting from LAS implementation. Most frequently cited positive impacts included: improved accountability; a better understanding of assessment among educators; increased and improved communication about teaching and learning; increased and improved teacher collaboration; more focused and aligned curriculum with Learning Results; and better coherency and consistency in curriculum and goals within SAUs and across the state. Most frequently cited negative impacts included: assessments not fully embedded or integrated with instruction; too much focus on assessment or too many assessments; teacher stress, anxiety or frustration and lower teacher morale; less time for some curricular or instructional units; teachers out of classroom more; and less time for other topics of professional development. Forty-three percent of SAUs indicated they were very confident that they would have assessments in place to certify that their 2008 high school graduates would meet standards in English Language Arts, while 40% of the SAUs indicated this level of confidence in the area of Mathematics. Thus, a majority of SAUs were less confident that they would have assessments in place to certify student achievement by 2008. Further, a large percentage of SAUs (almost half) estimated that between 51-75% of their 2008 graduates would meet standards in English Language Arts or Mathematics, while almost a third of the SAUs estimated that between 76-100% of their graduates would meet the standards. While these findings are difficult to interpret given the speculative nature of the questions, it appears that SAUs do not expect to have their LASs fully in place by 2008, and they anticipate a large percentage of students who will need additional interventions in order to meet the graduation requirements by the current deadline of 2010. Once the assessments, remediation services, and replacement assessments are in place, a higher percentage of students might be expected to meet standards. SAUs indicated that they have implemented a wide variety of changes to support 2008 graduates in meeting the standards (see Appendix E). Some areas of change included: curriculum and coursework; school schedules; advisory programs; after-school remediation programs; tutoring; learning labs and learning centers; emphasis on literacy and mathematics; and staffing. SAUs were asked to describe boosters and barriers other than the resources of time and money (see Appendix F). Chief boosters were: resources, guidance, and professional development provided by the MDOE (such as LAD and MAP tasks); professional development provided by

49

educational organizations; having an organizational structure within the SAU to do LAS work; committed staff; principal and teacher leadership; and new staff positions. Chief barriers were: the scope of required LAS work and number of assessments; lack of good assessment models in some areas; unclear communication and frequent changes in LAS requirements; negative media coverage of LAS requirements; lack of data management systems for LAS; resistance and anxiety among educators; increased workload without compensation; and the lack of a curriculum coordinator position within some SAUs. SAUs were asked which job role had the primary responsibility to lead LAS work. The largest percentage of SAUs indicated that the curriculum coordinator led this work, followed by principals, teacher leaders, and superintendents. The job role leading LAS work differed somewhat by SAU enrollment size.

50

Part III: Progress and Issues Related to LAS Development: A Qualitative Study of 14

Representative SAUs A qualitative study was conducted to understand the current status and progress on LAS, and to explore areas that could not be examined in depth by the statewide survey, including: educators’ views about LAS; the role of leadership, staffing, and organizational structures to support LAS work; strategies for creating time and for mobilizing other resources to support LAS work; impacts of LAS implementation; and identification of supports, barriers, and needs. The study included focus group interviews and intensive fieldwork in 14 representative SAUs. Focus group interviews were conducted in February 2005 with statewide representatives of special education, alternative education, and career and technical education (CTE). Representatives of English language learner programs (ELL) from three large SAUs were interviewed by phone as a focus group interview could not be scheduled. Fieldwork in the 14 SAUs was conducted from February through June 2005 and included individual interviews, document collection and analysis, and observation of LAS scoring sessions in two SAUs. These interviews also included educators from special education, alternative education, and ELL programs. Phone interviews were conducted with CTE representatives serving some of the 14 SAUs.

Demographics for SAU Sample A sample of 14 SAUs was selected for study in the spring of 2005. The process for selecting the sample began with nominations by MDOE personnel familiar with SAU progress on developing LASs. This list was sorted by geographic region, enrollment size, free-and-reduced-lunch participation rates, and SAU administrative structure (independent, SAD, or union). Fourteen SAUs were selected to represent all geographic regions (with the exception of the Unorganized Territories), and other variables. Table 43 shows the demographics for this sample. SAU names, enrollment, and free-and-reduced-lunch participation rates do not appear in this report, in order to maintain confidentiality for the SAUs participating in this study.

51

Table 43. Demographics for 14 SAUs

Admin. Structure Grades in SAU SAU Enrollment

Free-and-Reduced- Lunch Participation

Superintendent Region

SAD K-12 Large high Western Maine Independent K-12 Large low York Independent K-12 Large high Western Maine Union K-8 Small low Midcoast Union K-8 Large low Hancock SAD K-12 Small high Aroostook SAD K-8 Small high Penquis Union K-12 Large high Western Maine Independent K-12 Small high Western Maine SAD K-12 Large low Kennebec SAD K-12 Small high Western Maine SAD K-12 Small high Washington Union K-12 Small high Washington Independent K-12 Large low Cumberland

Note: SAU enrollment categories are: small SAD or independent (under 2000); large SAD or independent (over 2000); small union (under 900); and large union (over 900). Participation in free-and-reduced lunch was coded as low if less than 30% and high if above 30%. Data Collection for SAU Sample Qualitative data were collected from the 14 SAUs through phone interviews and on-site interviews conducted over a period of 1-2 days (typically 2 days) in spring 2005. Each of four researchers was responsible for collecting data for two or four SAUs. Interviews were conducted with educators in all grade spans and included those who were involved in the leadership of LAS work and those who were not directly involved. Across the 14 SAUs, a total of 185 interviews were conducted with the following educators: 13 superintendents; 3 assistant superintendents; 9 curriculum coordinators; 37 principals; 78 teachers; 25 special education teachers and directors; 8 alternative education teachers; 6 English language learner (ELL) coordinators; and 5 career and technical education (CTE) program directors. Researchers used a standardized protocol (see Appendix B) to conduct an initial phone interview with the primary coordinator for LAS to obtain general information about how the SAU had organized to develop and implement a LAS and about what progress had been made to date. Interviews that could not be conducted during site visits were done by phone shortly after the visit. All interviews followed a standardized protocol for the appropriate job role or educational program, although questions were sometimes asked in a different order. All interviews were taped, unless a subject objected to taping. Documents related to the development of LASs were also collected, such as: LAS templates; copies of district policies on LAS; scoring guidelines and materials; and schedules of planned professional development and scoring sessions for LAS.

52

Data Analysis Focus group interviews were taped and transcribed. Data were organized into themes and patterns of similar concerns or issues were identified across the three focus groups. Results of this analysis were shared with shared with the LASIS II Advisory Committee. Interviews in the 14 SAUs were taped. Each researcher listened to the taped interviews from his/her site visits and typed narrative summaries of each interview which included direct quotations. Researchers also summarized all interview data within each SAU by completing a questionnaire that included ratings for progress on different tasks, frequency and degree of communication and collaboration about LAS, frequency of professional development, degree of support and leadership for the LAS, degree of inclusion of special education and alternative education students in the administration of local assessments, and the most frequently mentioned supports and barriers. The research team shared and reviewed all data and developed tables for the purpose of comparing progress and other variables across all 14 SAUs. SAUs were sorted into three groups according to their overall progress on major LAS tasks. This sorting resulted in six SAUs showing evidence of “good progress,” six SAUs showing “some progress,” and two SAUs showing “limited progress.” Early results of the data analyses were shared with the LASIS II Advisory Committee in the form of a narrative summary. A full discussion of the results is presented in this report. SAU Organization for LAS Development and Implementation The 14 SAUs varied somewhat in the way they organized to develop and implement LASs, but predominantly used district-wide vertical teams with representatives from each grade span to coordinate work on the LAS. Ten SAUs used vertical teams (five SAUs had vertical teams for each content area), while three SAUs used horizontal teams (separate district-wide teams for each grade span). One SAU used a combination of both organizational structures. Many of the SAUs (including those showing the most progress), used organizational structures that already existed from previous work on curriculum revision. These committees were renamed to reflect the additional responsibility for assessment. The district-wide teams met periodically to ensure coherency across grades and to assess district-wide progress on LAS tasks. The teams made broad decisions in several areas: planning district-wide teacher professional development on LAS and scoring sessions; determining the number and type of assessments to be used; deciding which assessments must be double scored and how scores should be reported, collected, and managed; and developing district policies related to the LAS (e.g., remediation services and replacement assessments). To select, revise, and score assessments, the 14 SAUs predominantly used grade-level and content teams for elementary grades and content teams in the middle and secondary grades. In some SAUs, principals, allied arts teachers, special education teachers, and educational technicians participated in scoring sessions. These respondents felt the direct experience in scoring had deepened their knowledge about assessments, scoring, district curricula, and the Maine Learning Results. Two SAUs were planning to certify teachers for scoring assessments.

53

In the six SAUs showing the most progress overall, assessments were scored during full-day, inservice sessions or early release days. In SAUs showing less progress, assessments were scored during teacher planning periods, after school, or during teachers’ personal time. Two SAUs did not schedule any time for scoring assessments this year. Personnel Resources for LAS Work All but 3 of the 14 SAUs had a full-time curriculum coordinator or another person who filled this role. In three SAUs, an assistant superintendent assumed the role and responsibilities for curriculum and LAS coordination. The largest district had two additional curriculum coordinators, one for elementary and one for secondary grades. In one SAU, an assistant principal took on this role. Two other SAUs which lacked full-time curriculum coordinators had a high school librarian who was a former teacher leader or a team consisting of a principal and three teacher leaders to coordinate LAS work. In the two SAUs making only limited progress overall, one SAU had a full-time curriculum coordinator (which was a newly created position), while the other SAU did not have a full time curriculum coordinator and used a principal and teacher team to coordinate LAS work. In 5 of the 14 SAUs, the person serving as curriculum coordinator was newly hired in the 2004-2005 school year. In one SAU showing the least progress, a new superintendent had made the part-time position of curriculum coordinator a full-time position this year, and a new person was hired for this position. Having the same person in this leadership role did not appear to be a factor determining SAU progress on LAS work. Three of the newly hired curriculum coordinators were among the SAUs showing the most progress. However, respondents did mention that a change in leadership—whether superintendent, curriculum coordinator, or principal—often included a change in views and priorities regarding the LAS. Overall, most districts relied on the position of curriculum coordinator to coordinate all aspects of LAS work. Grade-level teams of teachers, sometimes led by principals, guided the implementation of the LAS. Key LAS coordinators were generally not provided with a stipend but paid their regular salary, and they generally did not have a reduction in their job responsibilities to compensate for their work on LAS. Only one district compensated a teacher leader who was assisting the assistant superintendent with district-wide LAS coordination. Nine of the 14 SAUs provided stipends for teachers to work on the selection, development, or scoring of assessments. The availability of stipends was helpful but was not a determining factor in the districts’ overall progress. Support and Leadership for LAS While most of the people responsible for coordinating district work on LAS indicated they strongly supported the idea and state goals for LASs, some administrators and teacher leaders indicated weaker support. Reasons for weaker support included: skepticism about the district’s ability to develop truly valid and reliable assessments; skepticism about the ability to compare results or certify students when LASs across the state are not the same; and concerns about district and teachers’ capacity to find the time to maintain a system that requires many assessments in each content area and double scoring of each assessment. Support and leadership were also weaker in SAUs that had a high turnover in superintendent and principal positions.

54

Data from several different interview questions were used to determine individual levels of support and leadership regarding the LAS. These included: a question asking how superintendents or principals communicated their view of LAS with teachers; a question about the purpose of LAS; and a question about supports that have been helpful. Respondents also made many voluntary comments about support and leadership. Eleven of the 14 SAUs had strong superintendent support for LAS, while three had more moderate support. Nine SAUs had strong principal support, while five SAUs had more moderate or variable levels of principal support. Seven SAUs showed evidence of strong teacher support for LAS, while an equal number of SAUs had more mixed levels of teacher support. Five of the six SAUs showing the most progress on LAS had generally strong support from the superintendent, principals, and teachers. SAUs making less progress had more moderate and mixed levels of support, particularly from principals and teachers. Superintendents. Superintendents generally agreed with the goals of LAS, and felt the requirements supported the educational reforms underway in their districts. They said completing work on the LAS was the top priority. Several superintendents said they would continue their work on assessments even if the state pulls back on its requirements. Superintendents were concerned about the impact on district resources, sustainability, teacher morale, and changes in LAS policies or rules. Some comments included:

The standards are good, the philosophy of LAS is excellent, and I agree with it. I think the purposes are exactly right—to collect data, to make instructional decisions based on data, and for accountability. We just don’t have time to reflect on the results. (District 10 interview) It comes down to good instruction for kids and how this assessment system really informs teachers so they can get kids to the Learning Results standards. As educators, we need to go easy until our assessments are valid and reliable. (District 14 interview) I hope the state doesn’t back off this initiative. We’re still going to go forward if they do, but if they back off, it will undermine what we’re trying to do. (District 2 interview)

Overall, superintendent support for LAS was not a critical factor in SAU capacity to make good progress, but it was very helpful. Superintendents in all 14 SAUs provided resources of time and money for LAS work regardless of their level of support or agreement with the LAS requirements. But some superintendents were credited with showing stronger leadership on curriculum and assessment reform. Some superintendents identified external sources of funding (e.g., grants, NCLB funding, or Title V funding) and hired outside consultants and professional organizations to guide curriculum and assessment development and train teachers. Two superintendents were credited with getting administrators and teachers focused on curriculum and assessment reform and holding them accountable for those goals. These superintendents hired or named curriculum coordinators to facilitate this work, and reorganized the leadership structure to coordinate LAS work. In one district, the busing schedule was changed from two runs to one run so that all schools could have a common starting and ending time. This allowed teachers from the different grade spans to meet together after school more easily.

55

Two superintendents developed collaborative efforts with neighboring districts to share assessment models and/or to provide a remediation program for secondary students. Evidence of superintendent and school board leadership was found in the provision of additional personnel (curriculum coordinators), time, and funding for professional development days devoted solely to LAS work. The cost of additional personnel, professional days, teacher stipends, substitutes, and sending personnel to training outside the district was a significant investment for many districts, and one that was more affordable for higher wealth SAUs. Principals. Principal and teacher support were stronger in the six districts making the most progress than in districts making less progress. These variables may have been more critical than superintendent support in determining the amount of progress made, as principals and teachers were more directly involved in leading and conducting the work of developing, implementing, and scoring assessments. Principals voiced fairly strong support for LAS goals, and acknowledged the need to support curriculum and assessment reform in their schools. Principals’ concerns centered on the impacts on administrators’ and teachers’ time, teacher morale, impacts on students, and their schools’ readiness to communicate with parents and report assessment results. Principals’ views are reflected in the following representative comments:

This local assessment work is the best thing to happen to education since I’ve been in it. Many people and groups are complaining about the work and stress, but that is because they never did the work on aligning their curriculum and assessment since the time that the Learning Results came out. Now the deadlines are getting close. . . . That whole idea of being held accountable is pretty scary. (District 10 middle school principal interview) It’s been tough, but the LAS has been beneficial. We needed to make changes in the high school curriculum and instruction, and LAS has helped to jumpstart that. (District 10 high school principal interview) I think teachers recognize its value. They talk about benefits to their curriculum and instruction, but they’re tired. Some are kicking and screaming, but most, I believe, will get there. (District 2 middle school principal interview) Those who are teacher leaders understand, but they are pursuing leadership programs so they get the larger picture. (District 2 elementary school principal interview) We’re much more accountable for every kid in our school whether or not they have a disability. It’s expanded our scope of ability. But the concept that all kids will come up to the same level of ability is foolish. (District 14 elementary school principal interview)

Principals have fewer options to provide financial resources or to change leadership structures than do superintendents, but several principals did use their leadership to provide more time for teachers to work on curriculum and assessments. Principals sometimes reorganized the school schedule so that grade-level teams could have a common planning period, or so that teachers across teams could work together on content teams. One middle school principal created an additional planning period twice a month for her teachers by using arts specialists to cover classroom instruction during this time. Another middle school principal devoted four times the

56

number of inservice days for work on assessment in his school than did principals in other schools in his SAU. This principal had participated in MDOE training on assessment over the past several years, and felt confident to lead the training and scoring sessions with his teachers. Teachers. Teachers’ support for the LAS requirements was mixed, but was most strong in SAUs that had a history of strong administrative leadership, and that had provided more opportunities for teachers to collaborate on LAS work. While teachers agreed with the idea of setting standards and holding students and teachers accountable, some teachers worried that the bar had been raised too high for many students. Teachers also recognized that assessment data should be used to inform curriculum and instruction. However, the lack of data management systems and the backlog in scoring, analyzing, and reporting back results left teachers wondering for what purpose they were administering assessments. A lack of time to reflect on and make use of the data being generated was the most common concern. Teachers supported the MDOE’s efforts to review and streamline the Maine Learning Results, the MEA, and LAS requirements and deadlines. Some typical comments included the following:

The purpose of a LAS is to ensure that each school, grade is preparing kids well enough to perform well. More consistency—teachers not all doing their own thing. LAS should be the plan for teachers. (District 11 elementary teacher interview) LAS holds me to a higher standard of teaching. (District 14 elementary teacher interview) It’s made us look at what we do, why we teach what we do. (District 5 secondary teacher interview) I’m more reflective on my teaching and what strategies I use. (district 8 secondary teacher interview) It’s given me good direction. Sometimes I get off track. I need to narrow down and focus more on certain performance indicators. (District 11 middle level teacher interview) The work needs to slow down, because the work isn’t good. . . . Teachers want to do their best work. (District 3 middle level teacher interview)

Teacher leadership on LAS took the form of serving on district-wide curriculum and assessment committees and leading grade-level teams. In SAUs without fulltime curriculum coordinators, teacher leadership was critical. Teachers helped to coordinate and monitor all aspects of LAS work district wide, planned professional development and scoring days, developed training and scoring guides, and led training for their peers. In some districts, one teacher leader provided the key leadership and coordination for LAS work. SAU capacity to maintain progress could easily be disrupted if this teacher leader were to leave or retire. Teacher resistance to LAS work limited the ability of some SAUs to make good progress. Some principals said they felt compelled to lead the work in their schools because they couldn’t get enough teachers to agree to take on leadership roles for their grade levels. Some teacher leaders complained that their peers did not help work on developing assessments and rubrics, and did not administer or score assessments as planned. While many teachers were willing to score assessments during planning periods or in their own time, some teachers said they would only do this work during scheduled inservice days. In at least three districts, concern about teacher

57

workload was becoming a contractual issue. One superintendent said he could not assign teachers to serve on curriculum and assessment committees or teams, but had to ask for volunteers. Thus, superintendents and principals could use their leadership to set priorities and to mobilize resources for LAS work, but progress was slow if only a few teachers were willing to help lead and conduct this work. Progress Toward LAS Development and Implementation Prior to a site visit by researchers for this study, curriculum coordinators (or other persons responsible for coordinating LAS work) in the 14 SAUs were asked to complete two tables indicating the current status of work for each grade span on several activities related to LAS development and implementation (see Appendix C). The first table asked SAUs to summarize progress in aligning curriculum with the Maine Learning Results, by grade span and content area. The second table summarized progress by grade span and content area in selecting performance indicators to be assessed, selecting assessments, administering assessments, developing rubrics, holding scoring sessions, and documenting how reliability in scoring was established. Data from these tables, along with interview data, were analyzed to determine the overall level of progress on LAS tasks in each SAU. A summary for each task follows. Curriculum Alignment. Many of the SAUs in the sample worked on aligning their curricula with the Maine Learning Results during the 2003-2004 school year. Some SAUs had been working on this task during the past 5 years, and a few SAUs worked on this task as recently as summer 2004. In some SAUs that had worked on alignment for several years, respondents said their work was mostly at the level of the content standards, and not at the more specific level of the performance indicators. In SAUs making the most progress, respondents said curricula were aligned or mostly aligned and in place this school year for five content areas (English Language Arts, Mathematics, Science, Social Studies, and Health/PE). Curricula were partly aligned for Career Preparation, Modern and Classical Languages, and Visual and Performing Arts, with more progress in these three content areas in grades 5-12. In SAUs making the least progress, respondents said curricula were aligned or mostly aligned for English Language Arts and Mathematics, and partly aligned in Science, Social Studies, and Health and Physical Education, and Visual and Performing Arts. Curricula for Career Preparation and Modern and Classical Languages were not aligned for grades K - 4 and were partially aligned for grades 5 - 12. These SAUs were often in the process of revising/rewriting curricula this year or had just revised curricula during the past year. Selecting Performance Indicators. SAUs showing the most progress have completed work on selecting performance indicators to assess, while other SAUs were mostly to partially complete on this task. Content areas that are less far along are: Career Preparation, Modern and Classical Languages, and Visual/Performing Arts. Respondents in the 14 SAUs described taking different approaches to selecting performance indicators for assessment. About half the SAUs started by selecting high priority performance indicators and then identifying appropriate assessments to cover these indicators. The other half began by looking at what performance indicators were already covered by the existing curricula

58

and assessments in the district. A few SAUs referred to the second approach as “backward planning,” and wanted to preserve the rich curriculum units and assessments they had been developing over the past few years. These SAUs filled in remaining gaps by selecting LADs or MAPs. About three SAUs used a combination of both approaches. All but one of the six SAUs showing the most progress began by selecting high priority performance indicators. Most SAUs used grade level or content teams to select priority performance indicators and then assessments. Some SAUs used grade-span teams for this work. Most of this work took place during the 2003-04 school year and summer of 2004. SAUs referred to the MDOE’s LAS guidelines and documents (e.g., Balance of Representation Report) to assist with this effort. None of the respondents described using a formal process to select and rank indicators, but instead used a more informal process of discussion and consensus. The development of assessments also varied across the SAUs. Some SAUs had spent a good deal of time in the past few years developing common assessments, where all teachers in a grade-level team participated in the development and refinement of assessments. Some of these assessments were retained for the LAS. Other SAUs selected classroom assessments which all teachers in a particular grade would agree to administer. These assessments had not gone through the same level of scrutiny and refinement, but were chosen to meet the deadlines for completing the LAS templates. One SAU almost exclusively used common assessments that had originally been classroom assessments. Teachers continued to score these assessments individually, and did not double score them. This approach was also more common in grades 9 - 12, where many courses are typically taught by one teacher. Individual teachers developed classroom assessments for the purpose of LAS and single scored them. Collaboration to develop common assessments and double scoring was rarer at the secondary level than at the elementary level. All SAUs provided a sample of their LAS templates for the purpose of this study. Most respondents said they understood the state’s LAS requirements, although many respondents complained that the “rules constantly change,” and they were sometimes uncertain about how many indicators they needed to assess and how many assessments were required. For example, some SAUs had as many as 10-15 measures per content cluster (most often for the English Language Arts cluster of writing and speaking in grades K - 4), rather than the MDOE suggested minimum of five measures per cluster. Most SAUs selected roughly the MDOE suggested number of 8-12 assessments per content area and grade span, although the exact number was sometimes slightly below or above this number. SAUs made an attempt to evenly distribute the number of assessments across each grade within grade spans, but overall planned fewer assessments for kindergarten and grades 11 - 12. At the secondary level, SAUs planned more assessments in grades 9 - 10, in order to leave students with ample time for remediation and replacement assessments. Several SAUs had discovered they had selected more assessments than they needed and were in the process of reducing the number of planned assessments. A review of the assessment templates indicated that SAUs used a mix of different types of assessments, including: selected and constructed response type local assessments, projects, portfolios, writing assessments, LADs, and MAPs. Most SAUs, even those showing the most progress, predominantly selected LADs, followed by MAPs, and then local assessments. Respondents indicated that their selection of LADs and MAPs was prompted by the need to meet

59

state deadlines, and concerns that their locally developed assessments might not be considered sufficiently valid and reliable. Respondents in most SAUs voiced concerns about using data from assessments that are in the piloting or revision stages, in order to make high-stakes decisions for students, such as certification for high school diplomas. Respondents were not confident that all assessments specified for their LASs were sufficiently valid or reliable. A few SAUs have included the MEA in their LAS template, but most have not yet decided whether or how they will include this assessment data for the purpose of certifying student attainment of the Maine Learning Results. Respondents indicated that MEA scores would only be used to help students show achievement of standards. Of the 14 SAUs, only 3 SAUs had submitted waivers to the state. These SAUs differed in their level of progress and time spent on LAS. One SAU spent 7 full days solely on LAS development and scoring this year, while two SAUs spent between 1 half day and 2 days on LAS work. The primary reason given for submitting a waiver was the concern about being able to collect and report out scores for all assessments, due to the lack of a data management system and manager. In one SAU showing limited progress, a new curriculum coordinator was struggling to motivate teachers to catch up on completing tasks that had not been addressed during the previous year when the district lacked strong leadership. This district will work on scoring and building a data management system in the 2005-2006 school year. Administering Assessments. While some SAUs began piloting their selected assessments last year (or had administered their locally developed assessments for more than a year), most of the SAUs were just beginning to pilot the LADs and MAPs this school year. In most cases, all teachers were piloting the selected assessments, while in some districts only a few teachers piloted each assessment. In SAUs that showed the most progress, teachers were administering all or most of the selected assessments. In SAUs showing less progress, teachers were administering some or most assessments. By late winter, many SAUs (particularly those less far along) had followed the MDOE’s suggestion to focus on administering and scoring assessments in English Language Arts and Mathematics. SAUs also made it a priority to administer ninth-grade assessments. Many respondents commented that the large number of school closings due to snow prevented the administration of some scheduled assessments. Another problem that surfaced was coordinating the timing of assessment administration. While SAUs attempted to administer common assessments at uniform times during the year, this often did not happen, as some teachers felt their students were not ready to take the assessments at the planned time. Thus, a tension existed between the goal of administering assessments at uniform times and administering the assessments when students were ready. For teachers in the early elementary grades, when students’ reading and writing skills develop rapidly over the course of a school year, having students take assessments at different times of the year presented problems in selecting benchmark papers and achieving reliability in scoring. Teachers piloting LADs and MAPs sometimes felt that the timing of assessments did not fit with a logical instructional sequence. After piloting these assessments, grade level and content teams would discuss the need to spend more time preparing students for the assessment, delaying the assessment, or revising the assessment and rubric. Some teachers also felt the LADs and MAPs did not fit well with their curriculum, or were too challenging. One of teachers’ top complaints

60

was the amount of preteaching time they felt the assessments required. Social studies and science teachers in particular disliked spending several weeks to teach and assess on specific historical events or scientific facts that they deemed to be more minor in relative importance. The lack of a good fit between assessments and curriculum was a result of the rapid process of selecting assessments, and the lack of ownership teachers had for non-locally developed assessments. In SAUs where teachers had been given time to discuss and revise the assessments, the LADs and MAPs had became more “local” assessments and more embedded into curriculum and instruction. Both administrators and teachers voiced concern over the number of assessment papers and scoring documents they were accumulating, and wondered where they would find room to store these documents and how long they should keep them. They also wondered if teachers would have time to review their students’ work from the previous grade. Developing Rubrics. Rubrics were all or mostly developed in the 14 SAUs, given the fact that SAUs were primarily using LADs and MAPs which include rubrics. However, grade-level and content teams spent much of this school year revising rubrics both before and after they administered assessments. In some cases, teachers felt the language of the assessments and rubrics would not be accessible to most students. In other cases, teachers decided to alter an assessment or remove some tasks, which required revision of the rubric. Some teams worked on translating their 5- or 6-point scaled rubrics into 4-point scaled rubrics to better align with LAS guidelines. In SAUs that were furthest along, there was an effort to produce a set of uniform instructions for administering assessments, explanations for the purpose of assessments, and uniform rubrics. Teachers commented that one of the biggest benefits of the time they had spent working on assessments was that they deepened their understanding of how rubrics work, an appreciation for the value of rubrics, and a common language to discuss assessment and rubrics. A few teachers said the word and concept of a “rubric” was new to them. Single Scoring Assessments. SAUs making the most progress had single scored all assessments administered this year, while SAUs less far along had single scored most assessments. One SAU had not done any single scoring yet. Teachers primarily single scored assessments during planned inservice or early release days. In some SAUs, there were few district-wide days planned for this work, so teachers single scored assessments during their planning periods, after school, or in their own time. Teachers remarked that English Language Arts assessments, particularly for middle and secondary grades, took a good deal of time to score. Double Scoring Assessments. SAUs were just beginning to double score assessments this school year. Most SAUs double scored some of the assessments that were administered this school year, placing priority on double scoring for English Language Arts and Mathematics in grades 1, 5, and 9. One SAU had double scored most assessments, and one had not done any double scoring. Double scoring and revising rubrics and assessments were the main focus of professional development time this year.

61

SAUs referred to MDOE’s LAS guidelines and documents as a basis for their double scoring procedures. Some SAUs reproduced these documents for teachers to use in grade level and content team scoring sessions, while other SAUs wrote their own guidelines based on the MDOE’s guidelines. Double scoring was new to most teachers, with the exception of teachers who had scored district-wide writing assessments or Developmental Reading Assessments (DRAs). This experience proved to be very useful in helping teachers to build the capacity for double scoring other local assessments. English Language Arts and K - 8 teachers had more experience in double scoring. Teachers in some grades and content areas reported that they have not done any double scoring yet. The primary obstacle to double scoring mentioned by all respondents was the lack of scheduled time for scoring sessions. Districts making the most progress had scheduled full inservice days or early release days for this work. Most districts scheduled very few days/hours for this work. The result was that teachers and administrators felt overwhelmed by the large number of assessments that needed to be double scored, and concerned about the lack of time scheduled to get through the backlog of assessments. Elementary teachers had responsibility for scoring and revising many more assessments than did middle or secondary school teachers who specialized in one subject. Thus, the burden of assessment work has fallen more heavily on elementary grades. Another obstacle to double scoring was the small number of teachers in some content areas, such as Visual and Performing Arts and Health and Physical Education where there might be only one teacher per content area. In at least one SAU, the physical education teacher collaborated with his/her counterpart in a neighboring district to develop assessments. Establishing Reliability. Documenting reliability in scoring was one task that all SAUs had only partially completed. While some SAUs required teachers to complete standardized scoring sheets and turn all sheets into the central office, these sheets sometimes only contained teachers’ scores and comments and the resulting level of reliability across scorers, but did not describe the process the team had used to establish reliability. A researcher observed several different teams during scoring sessions in two SAUs. In some SAUs, the curriculum coordinator reported that teachers felt the scoring sessions were sometimes stressful, as teachers were forced to compare their students’ work with that of their colleagues’ students. Curriculum coordinators reported that teachers would not be comfortable having an outside observer. Thus, the research team was not able to observe scoring sessions in all 14 districts. Both formal interviews and informal conversations with scoring teams revealed variation in the process of establishing reliability. For example, some teams did not interrupt scoring to recalibrate, while others did. Most teams began by selecting benchmark papers and agreeing on the rubric. Then teachers randomly selected 25% of the papers and proceeded to double score papers from other teachers’ classrooms. The team then computed the percentage of agreement. If they agreed 70% of the time, then they single scored the remaining papers. If they did not have a high level of agreement, then teachers selected another 25% of the papers for double scoring. In at least one SAU, teachers used a combination of other sources of evidence to supplement assessment scores (e.g. teacher observation, GLE or MEA scores) when they could not achieve 70% agreement. In some SAUs, teachers primarily single scored and then informally compared

62

their scores to work toward consistency in scoring. Differences in the amount of time scheduled for scoring may partly explain differences in scoring procedures. Most teachers had a fairly good understanding of the process for establishing scoring reliability, but sometimes needed to check with a teacher leader or curriculum coordinator on questions that arose. Some teacher leaders indicated that they felt underprepared to answer questions about this. Many teachers said they would like more training on double scoring and reliability. It is difficult to determine precisely the technical quality of the scoring process, due to the lack of opportunities to observe double scoring in all 14 SAUs and to collect detailed information about the procedures each team used to double score and establish reliability. While SAUs generally made sincere efforts to follow the procedures recommended in MDOE’s documents (the Measured Measures and Considering Consistency documents), there was some variation in how the process was carried out across teacher teams and grade spans. Where some teachers had not done any double scoring yet, they had not begun to document reliability. Since most districts were only beginning to double score, and only for a limited number of assessments and grade levels, it is too early to report on the technical quality of this effort. Professional Development to Support LAS Implementation. The predominant approach to professional development in the 14 SAUs was to send coordinators and a few teacher leaders out of the district for training on LAS requirements, LAS development, and scoring. These leaders returned to train other teacher leaders in the district during brief half day or less sessions, and then teacher leaders were expected to train their colleagues during the scheduled sessions for LAS development or scoring. Most teachers felt the amount of training they had received within the district was insufficient to lead and conduct this work. In particular, teachers voiced the opinion that they had experienced a rather steep learning curve during the past year or two on assessment topics. SAUs that had made the most progress had offered more frequent opportunities for professional development on LAS requirements and scoring. These SAUs also offered frequent opportunities for teachers to collaborate across schools within the district, and special education teachers were frequently included in professional development or scoring sessions. District administrators frequently communicated with teachers about their LAS through inservice, newsletters, or by participating or leading grade level or content teams. By contrast, SAUs showing less progress held fewer opportunities for professional development within the district, fewer opportunities for teachers to collaborate across schools within the district, and only included special educators in professional development or scoring some of the time. Teachers rarely attended professional development outside the district, and only rarely had training on scoring. District administrators communicated less frequently with teachers about LAS and were less involved in leading grade level or content teams. Across all SAUS, most or all of the time spent on professional development during the past 2 years has been focused on developing and implementing LASs. A few SAUs tried to implement other initiatives, such as literacy instruction and assessment, alongside the LAS effort. Multiple initiatives at the same time stressed teachers, and SAUs found it difficult to motivate teachers to spend additional time on professional development. While respondents felt they needed this time to accomplish the required tasks within the specified deadlines, there was a strong concern

63

about the lack of time and attention to other important topics of professional development. Both administrators and teachers cited the need for professional development in the areas of content and instructional methods, differentiation of instruction, and inclusion. Teachers explained that they needed time to build other areas of knowledge in order to remain excited and motivated in their profession. Respondents said that an almost exclusive focus on the LAS during the past 2 years had diminished teacher morale. Concerns about teacher workload have become a contractual issue in some SAUs. While some SAUs were able to find funding for stipends or substitutes, other SAUs had more limited fiscal resources to motivate and compensate teachers. Thus, there is a tension between the need for more time to work on LAS tasks and the need to include other topics of professional development. A few SAUs indicated they would not spend as much time on LAS work next year as they did this year, acknowledging the concerns about teacher workload and broader professional development needs. Time Spent on LAS Work Districts varied widely in the amount of time they scheduled for professional development and work on their LASs, and for scoring assessments specifically. The total number of days SAUs scheduled to focus on LASs varied from 0.5 to 7 full days. Of these, the number of full days spent on scoring assessments varied from 0 to 4 days. These figures do not include the time teachers spent in their personal time to work on assessment development and scoring. Overall, SAUs making the most progress scheduled more days for work on LAS than did other SAUs. While SAUs predominantly used inservice and early release days for this work, some SAUs scheduled work during teachers’ planning periods or 1 hour per week. Respondents reported that having a full day or 2 days consecutively to work on LAS development, revision, and scoring was very beneficial and helped them make good progress. Other respondents, who had only an hour or 2 to work on the LAS, found this approach frustrating and a barrier to completing tasks. Assessment of Students in Alternative Education Programs Researchers held a focus group with 13 representatives of alternative education programs statewide in February 2005. Administrators and teachers from public and private alternative education programs described issues and challenges related to assessing students in alternative education programs within the LAS. Data from the focus group interview were used to guide interview questions for the site visits. Curriculum coordinators, high school educators, and alternative education teachers in the 14 SAUs were asked about the assessment of students in alternative education programs. Half of the SAUs did not have formal alternative education programs, while half did. A total of eight alternative education teachers were interviewed in SAUs that operated these programs. Respondents said that students in alternative education programs generally take the regular form of assessments (on-grade level) without accommodations or modifications, unless they are also identified for special education services. Some alternative education teachers said they allow students extra time to take the assessments. Some respondents said they feel students should have individualized assessments. Seven of the 14 SAUs visited have alternative education students. Of the seven, six SAUs administer all or most of the LAS assessments to this group of students, while one SAU does not administer these assessments to this group.

64

Some alternative education programs have been working for some time on aligning their curriculum with the Maine Learning Results standards. Respondents said that their curriculum and instructional methods cover the standards in a way that is different from regular education programs. Respondents said their programs try to make learning more relevant to students’ everyday lives, and engage students in “hands-on”, applied learning. Respondents indicated their belief that regular secondary education programs tend to focus on the needs of college-bound students, while many alternative education students will not attend college but will seek employment after high school. Thus, alternative and regular education programs generally approach instruction and assessment in different ways, and students in these programs may have different needs. Alternative education representatives held the view that their students need to focus on a smaller number of essential performance indicators, and that these students should not be expected to master the same curriculum and standards as regular education students. Respondents indicated that the requirement for alternative education programs to use the same common assessments used in regular education programs was having some negative impacts on their programs and students. One impact was that alternative education teachers felt they had to put aside the curriculum work they had been doing in order to administer assessments selected by regular education teachers to fit regular education curricula. This has had the impact of reducing teachers’ flexibility and creativity in their efforts to help alternative education students meet the standards. Impacts on students in alternative education programs were sometimes described more negatively. Several alternative education representatives felt their students would not be able to meet all the required standards for graduation. Teachers said students were becoming frustrated by the assessments and that failing to obtain a score of “3” (meets the standards) had the effect of diminishing students’ self-esteem and motivation to continue their schooling. One alternative education program director remarked: “We’ve developed systems that will reinforce their failure” (focus group interview). Another director commented: “Kids taking these tests feel scared and more stupid. It evaporates all the progress we’ve made with them” (phone interview). By contrast, other respondents said their students were able to “meet” or “partially meet” standards on their local assessments, and that the students have not fully realized that they must meet standards in order to receive a high school diploma. Other issues identified as being barriers to assessment related to the staffing of alternative education programs and communication with SAUs. Some program representatives said they have a small staff, and it is difficult to incorporate all the assessments that are required by the LAS, and difficult to follow the requirements for double scoring for reliability. For programs serving one district, communication with the SAU was reported to be fairly strong, although alternative education representatives still reported being excluded from SAU work on the LAS. Programs operating within the middle or high school building reported the strongest collaboration with regular education teachers. One alternative education teacher working in a high school said the communication between his staff and the regular education teachers was good, and that his SAU plans to develop a personal learning plan for each student, and to monitor students’ progress in meeting standards for graduation. For regional programs that serve several SAUs, communication and coordination on assessment is difficult to impossible. Alternative education programs cannot realistically administer all the different assessments selected by multiple SAUs.

65

Alternative education representatives said that their efforts to share ideas within a consortium have been helpful, and they support further collaboration. These educators indicated that more guidance is needed to outline how alternative education programs will be involved in the LASs and how students should be assessed. They also expressed the need to have their programs included in training and planning for LASs, at both the state and local levels. Alternative education teachers need more training on how to develop replacement assessments and score assessments. Respondents predict a large number of students will need remediation in order to pass the local assessments. Most SAUs have not yet developed remediation programs or replacement assessments. Alternative education representatives said the needs of both at-risk youth and adult learners need further consideration. Assessment of Students in Special Education Programs Researchers held a focus group with 16 representatives of special education programs statewide in February 2005. These program directors discussed the issues and challenges in assessing special education students within the LAS requirements. Data from the focus group interview informed the development of interview questions for site visits with the 14 SAUs. A total of 25 special education directors and teachers were interviewed during the site visits throughout the spring of 2005. Special education directors and teachers identified areas of concern related to LAS requirements, the administration of assessments, communication between regular and special education teachers, and the involvement of special education staff in LAS work. Respondents also identified areas of need related to staffing, training, guidance, and time. Each of these topics is discussed in turn. Perhaps the issue of most concern was the requirement to administer common assessments to special education students. Respondents expressed the view that special education students should be assessed to measure their progress on individual learning goals that align to critical skills in the Maine Learning Results, rather than being assessed for their attainment of standard grade-level expectations that may be unrealistic. Requiring a student to take and retake assessments that may be well above the student’s instructional level was viewed as “setting up the student for repeated failure,” and as having a detrimental impact on the student. While some special education students were reportedly taking the regular assessments without difficulty, sometimes with accommodations, other students were struggling and were frustrated. One special education teacher described special education students’ reactions to assessments:

We’ve had a couple [students] blow up, slam down their pencil and storm out. We’ve had a few who just fill in the bubbles, just to get it done. It’s amazing how hard most of them try. (District 3 interview)

Respondents indicated that SAUs are generally administering most or all of the planned, on-grade level assessments to special education students within the regular classroom. Of the 14 SAUs visited, 8 administered all or most of the LAS assessments, and 6 administered only some of the assessments to special education students. All but 2 of the 14 SAUs primarily administered on-grade level assessments. Ten SAUs assessed special education students in the regular classroom, while 4 SAUs primarily assess this group in the resource room. The six SAUs making the most progress overall in their LAS work predominantly administered all or most of their

66

assessments and on-grade level assessments to their special education students, and assessed these students in the regular classrooms, with accommodations being specified through a formal process. Respondents voiced concerns about how assessments were administered to special education students. Special education staff were concerned that special education students’ needs may not be met when regular education teachers administer assessments without the involvement of special education staff or a review of the student’s Individual Educational Plan (IEP). In one SAU where special education personnel were involved in the LAS work, teachers collaborated in modifying the language used in classroom instruction and assessment to make instructional content more accessible to special education and at-risk students. Respondents were concerned about the lack of replacement assessments, policies, and remediation services. Regular education teachers held concerns about having students take assessments in the resource room, which meant students sometimes missed important classroom instruction. Regular education teachers commented that they sometimes aren’t sure whose work they are scoring when special education staff write students’ dictated responses on assessments. Respondents in most SAUs said they have a formal process for determining what accommodations or modifications may be made in assessing special education students, by reviewing the student’s IEP and through Pupil Evaluation Team (PET) meetings. In 5 of the 14 SAUs visited, the process was more informal, with classroom teachers frequently making decisions about how to assess special education students in their classroom without the assistance of special education staff. Some of the accommodations teachers described making included: giving students more time on the assessment, reading the instructions to students, pointing to sections of the assessment to keep students focused on tasks, having the student assessed in a resource room, and writing responses for the student with the student dictating their response. Communication between regular and special education teachers was cited as an issue in many SAUs. Special education personnel reported it was often difficult to find out when assessments were going to be administered and to obtain current copies of the assessments in time to plan appropriate accommodations or modifications for special education students. Some respondents said communication and coordination with private providers of special education services was difficult. In a few SAUs, curriculum coordinators and principals made it a priority to include special education staff in all aspects of SAU development, implementation, professional development, and scoring sessions. This leadership facilitated good communication and coordination between regular and special education programs and staff in a few SAUs. Most special education personnel described feeling “out of the loop” on district and state communications on LAS. Special education personnel identified their exclusion from the work being done on LASs as a significant barrier. In most SAUs, special education personnel have either been excluded from the professional development and assessment committees or have been included only in recent months. In the 14 SAUs visited, 5 SAUs frequently included special education teachers in LAS workshops, 8 SAUs included these teachers only some of the time, and one SAU rarely included these teachers in workshops. An elementary special education teacher said: “It was a decision to leave the special education staff out of the training. The work started 5 years ago, and we were told we didn’t have to attend” (District 4 interview). A secondary special education teacher

67

commented: “People are so involved in developing the system and getting their own work done, there’s no time to integrate with special education. We’re not part of the process or even thought of as part of the process” (District 10 interview). The failure to fully involve special education personnel in LAS work has meant that assessments are often developed without regard to the wide variation in students’ abilities and needs, and that special education personnel are lagging behind regular education teachers in their capacity to understand LAS assessment issues and to develop good assessments. Special education personnel identified needs for capacity building including both staffing and training. While respondents in some SAUs described their special education staffing as adequate, other respondents said their SAUs do not have enough special education teachers or educational technicians to ensure classroom support when assessments are administered to special education students. This puts regular education teachers in the position of making decisions about accommodations or modifications to assessments. Limited staffing for special education also means that special education teachers cannot easily participate in various grade level or grade-span meetings when they take place at the same time or on the same day. Special education personnel were in the early stages of receiving professional development on LAS requirements, and only a few had attended training outside the SAU. Their training needs encompassed all aspects of LAS assessment including: assessment development, rubrics, issues of validity and reliability, scoring, the process for determining what accommodations or modifications are appropriate, and development of alternate assessments. A few SAUs have included special education teachers in assessment development and scoring sessions with regular education teachers, but typically special education teachers are expected to work separately, in isolation of the grade-level teams working on LAS. Across the 14 SAUs, special education directors and teachers varied in their level of familiarity with the MDOE’s LAS guidelines. Some respondents had seen the MDOE document LAS Guide with Embedded Components for Accommodations and Alternate Assessment (November 2004), but many respondents had not seen or fully read the document. In one large SAU, a special education supervisor said she would not give the document to her staff as they were too overwhelmed and busy trying to keep up with the administration of assessments. In some SAUs, the document was used to guide professional development and work sessions on assessment development. Special educators repeatedly said they need more guidance in many aspects of assessing special education students. One area where there is significant uncertainty is the question of whether special education students can be given below-grade-level assessments (at their instructional level), instead of severely modifying grade-level assessments. Another question that was raised was whether special education students are required to take all assessments that are planned for a grade level, even when students are still working several grades below that grade level. A few respondents wondered whether students who are suspended from school still need to take all the planned assessments and how this would happen. Several respondents wondered how to report scores for students taking alternate assessments and how scores should be aggregated. The need for time to communicate with regular education teachers, to prepare for assessment administration, modify assessments, and develop alternate assessments was a big concern for all special education personnel. The work on assessments has been added to teachers’ existing responsibilities, and they feel overwhelmed. Most SAUs have not scheduled many days for LAS

68

work to take place, and special education personnel wonder when they will find the time to work with all the assessments that are required for a LAS. Some respondents said they had developed a few alternate assessments, but work on these was in the preliminary stages or had not been started in many SAUs. Respondents likened the process for developing alternate assessments with the process for developing Personalized Alternate Assessment Portfolios (PAAPs), and said it was enormously time consuming. Most respondents said they did much of this work in their personal time, as there was not sufficient time provided within the workday. In one SAU, respondents said they had been given one release day specifically to work on alternate assessments. Respondents acknowledged that one of the positive impacts of the LAS requirements was that they have increased teachers’ academic expectations for all students including special education students. One special education director said: “Students with disabilities are being taken much more seriously as they are now clearly expected to achieve and be assessed too” (District 7 interview). A special education teacher said:

It’s raising expectations. It’s made [special education] teachers very aware of what the standards are and what the general curriculum is . . . They [teachers] got into the habit of just remediate, remediate, instead of what every student is expected to know. We’ve had conversations that you wouldn’t have had 2-3 years ago. (District 3 interview)

Special education instruction is changing to focus more on the Maine Learning Results and academic skills, rather than focusing more narrowly on life skills and basic skills. Still, respondents believed that many special education students will not be able to meet standards required for a certified diploma, and said that another diploma option needs to be available to encourage these students to finish school. An equity issue raised by many respondents from special education, alternative education, and regular education was the question of fairness in providing certified diplomas to special education students who have taken modified or alternate assessments, but not to low-performing students who take regular assessments and do not qualify for special education services. Assessment of English Language Learners (ELL) A focus group interview could not be conducted with representatives for ELL, due to scheduling difficulties. Therefore, researchers conducted phone interviews with ELL coordinators for three large SAUs in Maine. These systems had ELL enrollments of approximately 40 and 1,100 students. In addition, questions regarding the assessment of ELL were included in the interview protocols for the 14 district sample. Eight SAUs had no ELL students, and three SAUs had very few ELL students. Three SAUs had several ELL students, ranging from approximately 30 to 100 students. Respondents said that most ELL students take the regular, on-grade level assessments. Some SAUs administer assessments with accommodations or modifications to ELL students, and some SAUs administer below grade level assessments to ELL students. Respondents described similar issues and challenges in assessing ELL students. The primary challenge is students’ limited language skills, which makes it difficult for the students to read and write well enough to do the assessments. The students come from a variety of countries and

69

represent many languages and ethnicities. Students are included in regular classrooms, but receive additional services to support their learning of English. Teachers said that most students have recently arrived in the country and speak little or no English. Another challenge is that many ELL students have not had any formal schooling previously, so their language skills and other academic skills are seriously delayed compared with other students in their grade level. Teachers said these gaps make it difficult to administer assessments for the LAS, even with accommodations or modifications. ELL teachers held the view that for most of the ELL students, the LAS assessments are irrelevant. Students feel frustrated when taking the assessments and sometimes cannot put anything down on paper, given their limited ability to read and write in English. Teachers said much of students’ work cannot be scored. Some teachers held the view that these students should be exempted from the LAS requirements, as they take the World Class Instructional Development and Assessment (WIDA) and MEA. Teachers said that more effort should be made to develop appropriate accommodations or modifications for these students. Teachers cautioned that scores should only be used for formative purposes, as they questioned the validity of the assessments for these students. A lack of communication between regular education teachers and ELL teachers was also cited as an issue in some SAUs. With limited staffing, it is difficult for ELL teachers to find out what assessments are going to be administered and to obtain the latest version of the assessment in enough time to make the appropriate accommodations or modifications if necessary. ELL teachers reported feeling “out of the loop” in SAU work on LASs, and reported they are spending a lot of time trying to modify the assessments for students. In other SAUs, communication and collaboration was reported to be fairly good. The small number of ELL students in the vast majority of SAUs across the state serves to make this group of students somewhat invisible. ELL teachers suggested that LAS documents should specifically refer to this group of students by name, to remind educators of the special needs of the group. More guidance is needed on how to assess these students. Assessment of Career and Technical Education (CTE) Students Researchers held a focus group with representatives of six CTE programs from different regions in the state in February 2005. This provided an opportunity for CTE directors to describe issues and challenges related to assessing CTE students. Data from the focus group interview informed fieldwork in the 14 SAUs, where curriculum coordinators and high school educators were asked about the assessment of CTE students. In addition, directors for five CTE programs (two had participated in the focus group interview) were interviewed by phone. Researchers asked about communication and collaboration between CTE programs and the SAUs they serve, and to what extent students’ work in the CTE programs is being used to certify achievement of the Maine Learning Results standards for LASs. Almost all CTE students take assessments for certification within their regular high school courses. Respondents in two of the 14 SAUs said CTE students take some of their assessments within their CTE courses. Two of the 14 SAUs did not have any students attending CTE programs.

70

CTE program directors agreed that there was little progress yet in coordinating with their sending SAUs on meeting LAS requirements. One of the reported barriers to communication and collaboration is the physical distance between SAUs and the CTE centers. CTE centers that were physically attached to a secondary school generally, but not always, reported closer coordination with those SAUs than with more distant sending SAUs. CTE directors meet frequently with superintendents from the sending SAUs. One director reported “We are all on the same page” (phone interview). A significant barrier cited is that each SAU is developing its own unique LAS, so students attending CTE programs take different assessments in their regular secondary schools. In order for the CTE courses to count toward certification, sending SAUs would have to collaborate and agree on which CTE courses and assessments they would include as part of their LASs. Getting teachers or curriculum coordinators from the different schools together to collaborate is difficult. Some CTE directors reported that teachers in secondary schools were resistant to working with CTE faculty in developing assessments, while a few directors described some positive efforts in collaboration between the two faculties. One CTE director reported strong collaboration with a sending SAU on integrating instruction in the areas of health occupation, auto technology, building construction, and history. Two CTE teachers were collaborating with secondary school teachers in creating assessments. A CTE curriculum coordinator planned to conduct professional development on the Learning Results standards for CTE teachers, so that CTE courses can build on the Learning Results standards. The CTE director said the collaboration between the two educational systems has mutual benefits: “It’s helping to make the CTE courses more rigorous and it’s helping the high school show kids how the standards are relevant to them. Working together is the key” (phone interview). CTE directors agreed that the Maine Learning Results performance indicators are largely unrelated to the course content and learning goals of CTE programs. Directors also said that many of the CTE students don’t feel that their regular secondary school coursework and assessments are relevant to students’ career goals. Directors generally expressed the view that the Learning Results are aimed more at students planning to attend a 4-year college, and that secondary school courses are geared toward college preparation and academic skills, while the CTE courses are aimed at students needing more technical or vocational training. CTE programs follow national industry standards specified by the different trades. Directors said that measuring student achievement of concepts and skills takes different forms in CTE and secondary school programs: CTE courses use hands-on, applied, performance-based assessment, while most secondary schools continue to use more traditional, paper-and-pencil type assessment. Thus, coordinating assessment between the two types of programs is difficult. One CTE program hired a mechanical engineer to identify performance indicators covered by the CTE courses. The review found that most programs of study related to only five or six performance indicators. Courses in Mathematics, science, and communications covered the most indicators. CTE directors statewide have reviewed their programs for alignment with the Learning Results. CTE directors said they felt sidelined from the process of developing the Maine Learning Results and LAS work, both at the state and local levels, and they would welcome more involvement. CTE directors said they hope that in time SAUs will begin to count students’ work in the CTE courses as part of the evidence for meeting standards for graduation. Directors indicated that students’ work in CTE courses should be valued, in the same way their work in more academic

71

courses is valued. Some SAUs are discussing the possibility of using CTE course assessments to supplement the assessment in regular secondary courses, often as replacement or alternate assessments. LAS Data Management Most SAUs (10) were transitioning to a new data management software system. Five SAUs had selected Power School, while other SAUs had chosen School Master, Pinnacle, Grade Quick, and NPV. These SAUs had either just selected the software system or were in the early stages of implementing the system. Very little teacher training had occurred yet, and SAUs were in the process of collecting scores to enter into systems later this summer or during the next school year. Respondents were optimistic that the systems they had chosen would be able to give them useful information about how students are doing at the level of the performance indicators. In some SAUs, teachers were being asked to enter their lesson plans and assessments and to note which performance indicators were covered. Teachers will also enter attendance, grade, and assessment data. Four SAUs had not yet decided on purchasing a new system, and were waiting to use MEDMS. Two big concerns regarding data management systems were the time and personnel needed to manage a system. Although SAUs had selected a system, few knew who would manage the system. In most SAUs, a teacher had developed a simple spreadsheet to keep track of assessment scores, until scores could be entered into a more sophisticated software system. It appeared that there would be a severe backlog of scores to be entered from the current school year. Administrators voiced concern about the readiness of the MEDMS, and felt that the need to purchase additional software systems and hire data entry people posed a significant financial impact that should be off-set by funding through the Essential Programs and Services (EPS) funding model. LAS Data Reporting Because SAUs had not scored all assessments, and scores had not been entered into a data management system, assessment results have only been reported to students and parents on a limited basis. Teachers received basic information about how students were performing on assessments in their grade level, and reviewed this information during their grade-level or content-team sessions. Teachers informed students in their classrooms about their performance on assessments, but generally translated the rubric scores into a letter or point grade for the purpose of report cards. The lack of a fully operational data management system is one of the primary barriers to the use of assessment data to inform curriculum and instruction. Most SAUs were using traditional report cards, and said they were waiting to see how the LAS requirements evolved before deciding on how to alter their report cards. Four SAUs were beginning to use standards based reporting, typically alongside traditional report cards, for one grade or grade span (i.e., for kindergarten, K - 4/5, and 9 - 12). One SAU included portfolios in their reporting system. A few SAUs had begun reviewing models for standards-based report cards, and knew that they would be moving in this direction in the future. Respondents in all SAUs felt that parents were generally not well informed about the Learning Results, LAS requirements, and graduation requirements. While some SAUs reported that

72

parents were not resistant to the use of standards-based report cards in the elementary grades, they did report some resistance at the middle and secondary grades. One concern was whether post-secondary educational systems would accept high school transcripts if they did not include letter or point grades and student rankings. Administrators and teachers indicated that it would be critical for the MDOE to play a strong role in helping to inform and educate the public in these areas, and to create support for the shift toward standards-based reporting. Respondents in all SAUs commented on the need to inform the public about standards-based reporting, and to build support for this reporting approach. Administrators and teachers were in agreement that this was one area where the state could provide assistance. SAU communication with parents about LAS and graduation requirements was relatively infrequent, and occurred primarily during parent nights and eighth-grade or ninth-grade meetings. More communication occurred at the school level, through principal commentaries in school newsletters, parent nights, and conferences with parents, and PET meetings with parents of special education students. Administrators said they were reluctant to communicate anything about LAS rules and deadlines when they felt there had been frequent changes in the requirements. This put administrators and teachers in the awkward position of telling parents their children would have to meet certain requirements, when those requirements were later reduced or eliminated. Administrators generally took a very cautious approach to communicating with parents about the LAS, and appear to be waiting for the state policies on standards and assessment to reach a more stable state. LAS Policy Development SAUs were in the early stages of planning for policies and programs for remediation. Of the six SAUs showing good progress, three had drafted policies on remediation. Three other districts showing some progress had drafted policies. Respondents said they were reviewing the model policies that were prepared by Maine School Management Association and were disseminated by the MDOE in the spring of 2005. Two SAUs were planning to implement remediation services this summer for secondary students that had not met standards on their assessments. One SAU collaborated with neighboring districts to develop and deliver the remediation program. Other SAUs hoped to pilot remediation programs for secondary students later next school year, but had not fully developed these programs yet. Teachers voiced concerns about that lack of planning for remediation for elementary and middle school students not meeting standards on assessments. Administrators cited the huge impact on districts’ fiscal resources of providing additional tutors and remediation programs for all students who will need these services in the coming years. They felt this was another cost that needed to be addressed through the EPS funding model. Most SAUs were in the early stages of discussing the need for policies for replacement assessments. Three SAUs had policies in place, and two other SAUs had drafted policies. Most SAUs had not created replacement assessments, and there was a great deal of uncertainty about how this should be done to ensure validity and reliability. Teachers were also concerned about the time needed to create replacement assessments. Use of Assessment Data In SAUs that provided time for teachers to score assessments together and discuss the results, teachers were beginning to use assessment results informally to make decisions about curriculum, instruction, and assessment. Teachers discussed differences in how they introduced

73

topics, the amount of time they spent on topics, how they administered the assessments, and how students performed across classrooms, and were planning changes to their instruction and assessment for the next school year. Thus, assessment data was being used informally on a limited basis. The lack of a data management system, and lack of time to score all assessments and enter them into new data software, prevented assessment data from being used in a more formal way to inform curriculum and instruction. Teachers generally did not have data in a disaggregated form to look at differences in student performance. Teachers were frustrated that they would not receive detailed results and analysis for this year’s assessments until later next year. An elementary teacher commented: “I’m wondering what we’re going to do with the information they will get, and I want it to be worth something. I’m wondering how much will it mean to look at scores for students from a prior year” (District 4 interview). Most respondents said that the purpose of a LAS was to inform curricular and instructional decisions, while a few respondents focused primarily on the idea of using assessment data to see if students were meeting standards. Impacts of the LAS Impacts on Curriculum. Six districts were primarily using LADs and MAPs for their LASs, five districts used many LADs and MAPs, and three districts used only a few LADs and MAPs. Respondents had mixed views about the use of LADs and MAPs. Some teachers felt these assessments fit their curriculum, and could be smoothly integrated into the instructional sequence. Other teachers felt the selection of LADs and MAPs had been forced upon them in the rush to meet state deadlines. Many teachers felt the LADs and MAPs and other local assessments had become the de facto curriculum, reducing instructional time. A frequent comment was: “We don’t have a curriculum, we have assessments.” Yet, in SAUs where the curriculum had “drifted” and alignment and revision was still underway, teachers were grateful to have the assessments to help focus curriculum and guide instruction. Generally, in SAUs where teachers had been given time to work on selecting assessments, revising or “tweaking” them, teachers had more positive views about the assessments, and were trying to weave them into their existing curricula. In SAUs that had scheduled little time for teachers to work on curriculum and assessments, the LADs and MAPs were treated as add-ons that did not fit smoothly with curriculum and instruction. However, there was a good deal of variation across teachers, grade spans, and departments (in secondary grades) within the SAUs. Thus, teachers’ views about “teaching to the test” were related to how well the assessments were integrated into curriculum and instruction. Where assessments were becoming more embedded, teaching to the test was viewed more positively. One of teachers’ chief complaints was the amount of time needed to prepare students for assessments and to administer assessments. The additional amount of time needed for assessment, and additional number of assessments throughout the school year meant that teachers felt they could not get to topics or units they had covered in the past. For some teachers, who had developed rich units of instruction, this was a real loss that reduced their enthusiasm for teaching. Teachers articulated two contrasting views about the role of the state in determining curriculum and assessment. In SAUs where teachers had spent more time developing curriculum units and

74

assessments and teachers had more ownership for their work, teachers wanted to maintain local control over what is taught, and said they needed time to continue their work and good progress. In SAUs where teachers had been less engaged in curriculum and assessment work, teachers were reluctant to lead this work and said they would be happy if the MDOE would just “tell us what they want—just give us the curriculum, give us the assessments.” One of the primary benefits of the work on LASs was the improvement in coherency and consistency in curriculum and instruction across classrooms and grade spans within districts. In SAUs that had just recently begun to work on curriculum alignment and revision, the state mandate and deadlines for developing LASs were viewed as powerful catalysts that stimulated and supported district goals for curriculum and instructional improvement. Administrators and teachers said that the LAS work had raised educational standards and teachers’ expectations for all students, and that the grade-span and grade-level discussions on curriculum and assessment had created more coherency, consistency, and accountability. Concerns about helping low performing students meet standards were having positive impacts and changing curriculum. In one SAU, secondary mathematics teachers noticed that many students were failing to pass a required semester course in algebra. They developed a two-semester course that uses a hands-on and applied instructional approach, with a reduced class size. The superintendent reported a significant drop in course failure resulting from this change, and said the district is planning similar courses in English and physical science. The superintendent said that higher expectations for all students to succeed was stimulating more discussion about different approaches to motivate and engage students, prompting pedagogical shifts. “There’s more interest in experimenting with different teaching approaches” (District 10 interview). While the Maine Learning Results have strongly guided curriculum work at the local level, the impact of the LASs on curriculum has been limited so far, as SAUs are in the early stages of piloting assessments this year. Without sufficient time to administer and score all assessments, and to reflect on the results, data cannot be used to inform curricular and instructional change. Impacts on Instruction and Assessment. As noted in the section above, there were differences in the extent to which teachers had integrated assessments into their curriculum and instruction, and differences in their views about “teaching to the test.” Where teachers felt the assessments fit their curriculum, or had built instruction around the assessments, teachers felt there was a logical coherence in the instructional sequence. These teachers did not feel they were spending more time on assessment than instruction. Where teachers had added additional assessments without other curricular changes, assessments sometimes interrupted their instructional sequence. These teachers reported feeling they were spending too much time on assessment, and not enough on instruction. This problem was partly a result of the lack of fit between curriculum and assessment, but was also a result of the use of more assessments than were recommended by the state. In a few cases, teachers reported they were assessing twice—once to meet the LAS requirements to generate rubric scores, and once to generate letter grades for report cards. Teachers in early primary grades reported that many of their assessments must be administered individually or one-on-one with students (e.g. reading and mathematics assessments). The requirement to do more assessing means that teachers have less time to direct whole class activities. One second grade teacher commented:

75

Administering assessments takes time away from instruction. It’s frustrating. You wish you could be in front of your class teaching, instead of over in the corner assessing. (District 10 interview)

In all SAUs, respondents said that curriculum and instruction is much more focused around the Maine Learning Results than in the past. While some SAUs had been working on aligning their curriculum for several years, they were primarily aligning at the level of the content standards, and not with particular performance indicators. The LAS requirements forced these SAUs to go back to their work on alignment and to focus on performance indicators. In some SAUs, teachers are being asked to include the performance indicators in their lesson and assessment plans. Thus, instructional plans and assessment results are focused more at the level of performance indicators. While teachers generally had positive views of the Maine Learning Results, many had concerns about which skills and concepts were given priority and the degree of specificity in the writing of the performance indicators. In general, teachers advocated for fewer performance indicators and more broad language in specifying the most critical skills and concepts for students within each content area. Teachers also supported the idea of specifying standards at each grade level, rather than for grade spans. For example, SAUs across the state offer topics in science and social studies at different grade levels within the middle and secondary grades. This reduces statewide consistency in what and when topics are taught and assessed, and presents problems for students transferring from one SAU to another. Most teachers did not report that they were teaching differently, but said the topics they taught might have changed to correspond with the topics of the assessments, such as LADs and MAPs. Teachers reported a change in how they thought about and conducted assessment. Teachers felt the time they had spent working on assessments and rubrics and discussing different instructional approaches with their peers will improve their instructional and assessment practices. Teachers reported they were beginning to use a wider variety of assessment types and using rubrics more often. As it is still very early in the implementation of the LASs, it may be too early to see dramatic changes in instructional practice. Some respondents reported that substitute teachers are being used more often to cover classes to release teachers for training or work on the LASs. While administrators and teachers felt the time was needed for these efforts, they were concerned that having teachers out of the classroom had the impact of diminishing instructional quality during release time. Impacts on Teachers. The most immediate impact on teachers was that of time spent on revising, administering, and scoring assessments. Most respondents felt the pace of change required was too much too fast. They described feeling overwhelmed by the amount of work to do in so little time, and said this had had a negative impact on morale. Some teachers said they or their colleagues were leaving the teaching profession or retiring because of the effects of recent changes in educational policy at the state and federal levels. This appeared to be more of a problem among veteran teachers. There was a prevailing sentiment that work done too quickly without sufficient time would not be of good quality. Rather, respondents said they would appreciate relief in the number of content areas to be assessed, number of assessments required, double scoring requirements, and deadlines so that the work they do will be of good quality.

76

Some site visits took place after the MDOE revised LAS requirements and deadlines in the spring of 2005. While educators welcomed the relief that these adjustments will bring, many were concerned that the state might retreat further from LAS requirements and that this would stall the momentum for educational improvement. For SAUs that have spent a considerable amount of time working on LAS, teachers want to know that the work they’ve done has been for a good purpose. A second-grade teacher said: “The worst thing that could happen is if the state would back off or totally stop this. This was a vision, and you have to make this into something meaningful (district 4 interview). Teacher collaboration was both a positive and negative impact. In some cases, discussion and decision making about curriculum and assessment did not always go smoothly. Teachers sometimes felt uncomfortable or defensive when forced to share their work and their students’ assessment results with their peers. This created a feeling of tension and stress when teachers worked together on assessments. One teacher said: “There has been a great deal more conversation about assessment and more collaboration than ever before, but the negative side of this is that there has been much more conflict within the staff and much more interpersonal stress” (District 14 interview). In one SAU, a team facilitator left a scoring session in tears and quit her leadership position in frustration over the divisiveness within the team. In this SAU, language arts teachers appeared to have made less progress on assessment work and were having more difficulty reaching consensus over rubrics and scoring than teams in other content areas. An elementary teacher from this SAU commented: “Teachers sometimes get protective, if they don’t feel they’re doing as well as others, they can take it personally instead of seeing it as formative. If you look at it as a formative process, you can relax” (district 4 interview). An elementary principal from the SAU said:

Putting the work on the table—you can’t get any better staff development than that. You’re forced to come out of your classroom and put your work up against somebody else’s, and you’re going to change your teaching as a result of that. You see there’s more than one way to get to an answer. (District 4 interview)

In other SAUs, some teachers were resistant to making the required changes and predicted that “this will all go away.” These teachers came to scoring sessions without having administered or scored assessments as agreed to by the team. At least two SAUs had organized their grade-level teams so that a principal or curriculum leader from outside the group could facilitate the group to help avoid the stalemates that might otherwise occur. Administrators and teachers said that it is important for facilitators to have strong leadership skills and training in conflict management to effectively move their groups forward and resolve problems. Overall, respondents said that one of the most important benefits of the LAS work was the positive impact on teachers’ conversations about teaching and assessing—both the professional level of conversation and the degree of collaboration had improved. One superintendent noted: “The most impressive impact is the conversation that educators are having about kids. Sometimes the questions posed in the development of a local assessment system really stretch people’s experiences. And as a result of stretching, they start thinking differently” (district 10 phone interview). In many SAUs, teachers are spending more time than in the past discussing curriculum and assessment, sharing ideas about best practices, and looking at the relationship between instruction and student achievement with their peers. Respondents noted that

77

collaboration within grade levels and across grade spans has improved. One superintendent said: “It’s breaking down barriers between grade spans and buildings in the district. It’s helped propel change and movement toward the standards.” (District 11 phone interview) Administrators noted that teachers are beginning to see that their professional responsibilities extend to more active participation in developing curriculum and assessment and using assessment data to inform these areas. One superintendent said: “That’s really changed the standard of professionalism to say ‘I am in charge of the outcomes for the students,’ and that validates a lot of things they’re trying to get across in their classroom” (District 11 phone interview). In this district, teachers designed and delivered a summer remediation program to help secondary students meet standards. Another important benefit was the positive impact on educators’ knowledge of assessment. Administrators and teachers have deepened their knowledge of assessment, rubrics, and scoring and are beginning to share a common language to discuss these topics. For some educators, the learning curve was steeper than for others. A typical comment was that “Sometimes growth comes from frustration.” Respondents reported that their work on LASs has given them a better understanding of essential components of a good assessment. Some educators had also attended training outside their SAUs on how to analyze and use assessment data to inform curriculum and instruction, although they had limited opportunities to review assessment results. Overall, educators reported that the educational expectations for students have become more concrete and explicit, providing more direction for teachers. Impacts on Students. Respondents said that students are becoming more aware of the LAS requirements, but are still not fully informed. Secondary students are most aware, given the requirement to meet standards in order to receive a high school diploma. The level of awareness is expected to increase as SAUs continue to fully implement their planned assessments and begin to report out the results to students and parents. All students are beginning to use the language of assessment in daily conversation, sometimes asking: “Is this another assessment?” Teachers said that rubrics and the focus on the Learning Results have helped students to understand academic expectations, and that as expectations have increased student performance has also increased. Respondents reported that some students were stressed by the focus on assessment and meeting standards. Some assessments were too challenging for some students (e.g., kindergarten and first grade assessments requiring written responses). Many teachers said that the reactions of students depended in large part on the way teachers presented the assessments. Children were confused when assessments interrupted the instructional sequence and didn’t seem connected to the curriculum. On the other end of the continuum, teachers in one district said their students tended to be apathetic and did not take the assessments seriously, thus “throwing the results” off. Most teachers said they tried to present assessment in a low-key manner, as part of learning and instruction. While teachers said that some assessments had worked better than others, most teachers indicated that students had performed reasonably well on the assessments that were piloted this year. Where teachers had used certification assessments to replace other assessments and had worked on making curriculum and assessment more seamless, teachers reported fewer negative reactions from students. In at least two SAUs that were in the midst of curriculum revision and alignment work, teachers said that students had been doing mostly creative writing and were not experienced in writing

78

expository pieces or taking constructed response type assessments. Teachers were beginning to emphasize different writing styles to better prepare students for the LAD and MAP assessments. In some SAUs, administrators and teachers noted that secondary students often had poor reading and writing skills, which lowered their performance on assessments. Some districts were implementing early literacy initiatives as well as thinking about ways to help struggling secondary students. Teachers expected their students’ performance across content areas to improve over time as they became more proficient readers and writers. Students who demonstrate low performance but do not qualify for special education are of particular concern to educators, along with students in alternative education and special education programs. While educators may administer assessments with accommodations or modifications, or administer an alternate assessment to students who qualify for special education, they do not have this leeway with students not qualifying for special education. Thus, many educators worry that it will be difficult for a large group of low-achieving students to meet standards on the assessments and that these students will become, and have already become, frustrated and unmotivated to continue their schooling. Several teachers expressed the view that it would be unfair to award a certified diploma to a special education student who has taken modified assessments testing more basic skills, while denying a certified diploma to low-performing students who must take all of the regular assessments. School systems anticipated a growing need for tutoring and remediation programs, which will require significant investments of time and money to develop and operate. Superintendents were not sure how they will provide the remediation that will be needed at all grade levels. Impacts on Parents. Respondents in all SAUs said parents were not fully aware of the LAS requirements. Parents of high school students and students with disabilities were more aware and were reported to be more concerned about their children’s ability to pass required assessments and receive a high school diploma. One middle school principal said a parent was so concerned about her special education student being able to pass assessments that she had asked the principal if she should consider moving to another state. Special education directors and teachers said they anticipated a higher level of parental concern and lawsuits as more parents begin to understand that their children may not be able to meet standards or receive a certified diploma. A special education director said: “We’re having these difficult conversations with parents of fourth graders. They are recognizing that their kids need to meet the standards” (District 7 interview). While some parents were reported to be accepting of the shift toward standards-based reporting in the elementary grades, parents of secondary students were reported to be less accepting of this idea. One SAU said it probably will not pursue this change at the secondary level due to resistance among parents. A few other SAUs were beginning to introduce these changes to parents, and were trying to inform parents about the Maine Learning Results standards and LAS requirements. These efforts appeared to be minimal, however, and not sufficient to fully inform the public about state and federal educational requirements. Certification of Student Achievement SAUs were beginning to double score assessments for certification this year, and most are focused on making sure ninth-grade assessments are ready for administration and are being scored. SAUs were discussing the need for policies on remediation and replacement assessments,

79

but only a few have actually drafted policies. Two SAUs were planning to implement remediation programs for secondary students this summer and coming school year. SAUs have generally not begun to develop remediation services below the secondary level, and have not developed replacement assessments. Administrators and teachers are concerned about the availability of time and fiscal resources to meet the challenges of helping all students meet standards. They are also concerned about the impact on students who fail to meet standards, and who will not qualify for a diploma that certifies standards have been met. Educators worry that many students, particularly those in alternative education, special education, or career and technical education may choose to drop out of school. Supports and Barriers for LAS Development and Implementation Supports. Respondents were asked what resources or supports had helped with LAS work. Supports mentioned in the focus group interviews are discussed earlier in the report. Across the 14 SAUs visited, supports believed to be the most important were:

• Curriculum coordinator • MDOE workshops and documents on LAS • Administrator support • Administrator provision of more time in the school day, calendar to work on LAS • Teacher stipends • Assistance and training from external organizations • Regional partnerships and collaboration

Having one person to coordinate LAS activities, monitor progress, and disseminate information to teachers was an important resource. In SAUs that did not have a full-time curriculum coordinator, teachers and administrators were compelled to take on this work on top of their regular duties. Administrator support took the form of setting priorities in the SAU, adding additional professional days, reorganizing the calendar to create more time for teachers to work together, and providing funding for teacher stipends. Ten of the 14 SAUs created additional time by adding professional days, reorganizing the daily schedule for common planning periods, or hiring substitutes for release time. Teacher stipends facilitated work on curriculum development and assessments during the summer and after school, and for teacher leaders coordinating district-wide and grade-span work. Nine of the 14 SAUs offered teacher stipends for LAS work. Respondents said the MDOE workshops they had attended were helpful and that MDOE personnel were very responsive to questions. External organizations also provided important guidance and training in many SAUs. Three SAUs used external organizations extensively, and nine SAUs used them moderately. Seven SAUs used consultants to help with curriculum and assessment development. Collaboration with neighboring districts was a support in 9 of the 14 SAUs, with 3 SAUs planning or discussing the joint development of assessments with other SAUs. Barriers. Respondents were asked what barriers impeded LAS work in their SAUs. Barriers mentioned in the focus group interviews are discussed earlier in the report. Across the14 SAUs, the key barriers mentioned were:

• Insufficient time for teachers to develop their own assessments, to score assessments

80

• Fast pace of work and deadlines • Insufficient time to reflect on assessment results and plan for curriculum and instructional

changes • Curriculum alignment work that was not completed in previous years • Lack of a data management system to analyze and report back results • Changes in state assessment policy and requirements • Insufficient state funding for LAS work • Lack of SAU policies/programs for remediation, replacement assessments • Variety and number of assessments required by the state and NCLB • Resistance from some teachers • Small staff size in some schools, SAUs • Inclusion of grade 5 in the grade 5 - 8 grade span • Insufficient knowledge about how to develop assessments, analyze assessment data

The lack of sufficient time to work on all aspects of LAS work was the most important barrier mentioned by all respondents. Many administrators and teachers said they would like to see 5-10 days added to the school calendar to provide time for teachers to evaluate assessment results and to make curriculum and instructional changes for the next school year. In addition to the local assessments, schools are administering several other assessments to satisfy state and NCLB requirements. Respondents said the cost for additional professional days, assessment coordinators, data managers, and remediation programs should be shared by the state and funded through the EPS funding model. Administrators and teachers were frustrated over the delays in getting a fully operational data management system. A few SAUs were waiting until MEDMS is fully functional, but most SAUs (10) had gone ahead and purchased software from private vendors. These systems are in the early stages of implementation, and assessment scores from the 2004-2005 school year have not been fully entered. Changes in state assessment policy were viewed as barriers that serve to support teacher skepticism and resistance. Teacher resistance was more of a problem in SAUs that had not had strong and stable administrative leadership and among more veteran teachers. When some teachers did not fully participate in the LAS work, the burden fell to other teachers. Respondents said they would welcome some relief in the deadlines and requirements, but they strongly urged the state to steer a clear course and hold that course. Some teachers felt that grade 5 should be included in the K - 4 grade span, rather than in the grade 5 - 8 grade span, to better conform to the configuration of most elementary schools in the state. In some SAUs, it was difficult for fifth grade teachers to travel between schools to work with middle level teachers on the LAS. Small school staffing and low capacity to lead curriculum and assessment change in some SAUs was cited as a barrier. The need for training and guidance was a common theme heard in all SAUs, but was a greater need for some SAUs. SAU policies and programs for remediation and replacement assessments are in the early stages of development. Teachers are uncertain what supports will be available to help students who do not meet or partially meet standards. While some SAUs have begun to plan for supports at the

81

secondary level, little work has been done to meet the expected need at the elementary and middle levels. SAUs are not certain how they will meet this cost. Technical Assistance and Other Resource Needs Researchers asked respondents what technical assistance or other supports would be helpful in their LAS work in both the focus group interviews and site visit interviews. Some respondents made specific suggestions, while others mentioned the barriers listed above, which imply different needs. Responses fell into several categories:

• time resources • financial resources • staffing resources • leadership resources • human capacity—knowledge • guidance from state policy and MDOE personnel • data management systems • communication about LAS

Time, financial, staffing, and data management needs are discussed above, under the section on barriers. Leadership needs cited by respondents related to the need for stronger direction and leadership by district administrators and development of leadership skills for teacher leaders to skillfully facilitate teacher work sessions on LAS. In one SAU, a secondary department chairperson said that district administrators had communicated that double scoring and establishing reliability were not priorities. The teacher said the SAU needed stronger leadership. In the same SAU, special education teachers complained that their special education director had not provided necessary leadership to involve them in LAS activities and help to move work along. In several SAUs, respondents mentioned the difficulty of getting grade-level teams to work productively, when groups became divided. Having an administrator or outside consulting teacher facilitate these groups was one strategy that helped to keep groups on task. Other respondents said that teachers needed more training in leadership and conflict management to be effective facilitators. Administrators and teachers said their involvement in the LAS work has had a huge impact in their knowledge of the Learning Results, and developing and scoring assessments. Respondents felt they needed more training in establishing validity and reliability for locally developed assessments, double scoring, and developing alternate and replacement assessments. A few teacher leaders had opportunities to attend external training in these areas while most other teachers did not. In larger SAUs, not all teachers could participate on assessment teams or committees at the same time. Thus, some teachers were further ahead in developing their knowledge of assessment while other teachers needed to be brought up to speed. Differences in knowledge made it difficult for teachers to communicate and collaborate. The need for solid and sustained training for all teachers was evident, but training needs to be offered at different levels of sophistication. Teachers who had larger chunks of time to focus on training or LAS work felt more prepared to do the LAS work and felt they were making good progress on LAS tasks. Training on topics that lasted only an hour or two was insufficient to prepare teachers to do this work.

82

Respondents frequently said they would like more clear guidance from the LAS documents and MDOE personnel on how to accomplish required tasks. Educators working with special education students, alternative education students, and ELL had specific questions about assessing these groups of students, and wanted more guidance. Respondents in general said it would be helpful to have an MDOE representative visit their SAU to answer questions they have about scoring, reporting of results, etc. Some respondents said they were not sure how NCLB and LAS reporting requirements meshed, and they would like more succinct guidelines on that topic. Respondents said they hoped the MDOE review of the Learning Results, the MEA, and LAS guidelines would result in clearer, streamlined, and realistic educational goals. Educators would like relief in the number and types of assessments that must be administered during the school year, and less duplication in assessment for LAS and NCLB purposes. Many teachers said they would like to see more consistency statewide in curriculum and assessment, in terms of when courses and topics are offered in middle and secondary grades. Teachers also said that expectations should be specified for each grade and not by grade span to promote greater consistency. Greater consistency would reduce the problems experienced by students who transfer between SAUs. Administrators and teachers generally felt that parents were not familiar with the Maine Learning Results and LAS requirements. They said SAUs will need considerable assistance from the state in helping to educate and inform the public about state educational standards and assessments, and that the shift to standards-based reporting could only take place with more public understanding and support.

Summary

Part III of this report presented data from a qualitative study that explored progress and issues related to the development and implementation of LAS. The study produced rich data that serve to deepen our understanding of the complexity of the tasks at hand, and areas needing further examination, clarification, or support. Major findings from this study related to the research questions are summarized briefly here. A synthesis of findings and implications from both the statewide survey and the qualitative study is presented in Part I of this report. There is evidence that SAUs are making progress on required LAS tasks. Generally, more progress has been made in developing, administering, and scoring assessments in English language arts (ELA) and in mathematics than in other content areas, with a priority placed on completing work in grades 1,5, and 9. Most SAUs focused on piloting assessments during the 2004-05 school year, and were beginning to score a few of them. Most SAUs had selected and purchased software to manage assessment data, but they were in the early stages of transition to these new systems. Assessment scores had not been fully entered into the new systems as of June 2005, and analysis of data was minimal. This limited SAUs’ ability to inform students and parents about results and limited teachers’ ability to use data results to inform curriculum and instructional decisions. SAU policies on assessment were in the planning stage or were being drafted. SAUs were considering different models for these policies. Replacement assessments and alternate

83

assessments generally did not exist yet. A few SAUs were making plans to provide remediation to secondary students, but educators generally voiced concerns about the lack of replacement assessments and remediation programs for students at all grade levels. Six SAUs that made the most progress overall on LAS tasks had completed or mostly completed work on aligning their curriculum with the Maine Learning Results, had strong administrative leadership for LAS, had strong support for LAS across all administrator and teacher levels, demonstrated leadership by formally scheduling more time to work on LAS (typically full days) and by mobilizing other resources to support the work; and used existing organizational structures to work on assessment. SAUs making less progress were struggling to work on curriculum alignment and assessment development at the same time, had weaker support and leadership for LAS, mobilized fewer resources to support LAS work, and spent less time working on LAS tasks. In addition to leadership and support for LAS and time scheduled for LAS work, other supports mentioned by respondents included: SAU provision of teacher stipends; MDOE workshops and documents on LAS; assistance and training from external organizations; and regional partnerships and collaboration among SAUs. Barriers included: incomplete work on aligning curriculum with the Maine Learning Results; lack of time for all required tasks; lack of a data management system and analysis of results; changes in LAS policy and requirements and confusion over these; scope of work required by state and the No Child Left Behind legislation; lack of sufficient training in some areas of LAS development; resistance from some teachers; small staff size in some SAUs, and the inclusion of grade 5 in the 5-8 grade span. Respondents said they needed more time to complete work on LAS in all content areas, technical assistance with their data management systems, and assistance in communicating with the public about standards-based education and imminent changes in the way student achievement is reported. Some SAUs said they need financial support for several areas of LAS work, while most SAUs were more concerned about funding additional inservice days for LAS, data management systems, and remediation services for K-12. Teachers generally held the view that they needed time at the end of each school year to reflect on student assessment results as well as time at the beginning of each school year to plan changes in curriculum, instruction, and assessment. Administrators and teachers said that LAS implementation has had many positive impacts, including the following: serving as a catalyst for and supporting educational reform in SAUs; focusing curriculum, instruction, and assessment on the Maine Learning Results performance indicators; increasing coherency and consistency in curriculum and instruction within and across SAUs; increasing educators’ academic expectations for students (particularly for students in special education); increasing accountability for schools, teachers, and students; increasing educators’ knowledge and skill in creating good assessments and rubrics; and encouraging greater collaboration among teachers and conversations focused on teaching and learning. Other impacts that were a concern for some educators included: the stress teachers felt from workload issues related to the LAS timelines and scope of required work; the stress some teachers felt sharing their students’ work with peers; the shift toward “teaching to the test” where assessments did not fit well with the SAUs curriculum or were not yet fully integrated with instruction; the reduction in time for some instructional activities; the stress some students felt

84

when taking or failing assessments; and the lack of time to develop other professional development needs. Issues related to the assessment of students in special education, alternative education, English language learner (ELL) programs, and career and technical education (CTE) centered on the lack of involvement of educators in these programs in the LAS development and implementation, as well as the lack of alignment in educational goals and philosophies between these educational programs and regular education. Across all programs, respondents said that better communication and coordination needs to take place between regular education programs and these special programs. In addition, educators in these programs to not agree that their students should be held to all of the same requirements as regular education students, or that they should be expected to demonstrate their knowledge in the same ways as regular education students. Rather, educators in these programs said that it is more important to monitor students’ individual progress and to challenge and support each student’s learning in appropriate ways. Representatives for CTE programs said that very few of the Maine Learning Results performance indicators relate to the content of their programs. These findings indicate a need for further examination and clarification in the way the Maine Learning Results and LAS requirements will apply to these educational programs.

85

PART IV: APPENDICES

86

Appendix A

2005 Survey

87

SCHOOL ADMINISTRATIVE UNIT PROGRESS ON THE IMPLEMENTATION OF A LOCAL ASSESSMENT SYSTEM May 2005

Note: Your responses are totally confidential. No individual or individual SAU will be identified in any report or discussion based on the data collected with this survey.

1. Please indicate the role of the person completing this survey. Superintendent Assistant Superintendent Curriculum Coordinator Principal Assistant Principal Teacher Leader Headmaster Other (specify): ___________________________

2. In your SAU, which job role has had the primary responsibility leading the process of developing the local assessments system? Please check no more than two of the roles below.

Superintendent Assistant Superintendent Curriculum Coordinator Principal Assistant Principal Teacher Leader Headmaster Other (specify): ___________________________

3. Rate your SAU’s progress on each of the following activities. Your rating should represent the progress being made in English/Language Arts across each grade span.

ENGLISH/LANGUAGE ARTS

1 = No action taken on this yet 2 = Planning in progress 3 = Partially complete 4 = Work complete

K - 4 5 - 8 9 - 12 Our SAU has . . . 1 2 3 4 1 2 3 4 1 2 3 4 a. developed curriculum documents aligned with

Maine Learning Results.

b. aligned instruction with curriculum and Maine Learning Results.

c. selected a sample of performance indicators to assess using Maine’s Balance of Representation or some other method of prioritization.

d. used scoring rubrics for each assessment e. administered common assessments f. provided professional development in common


g. double scored common assessments h. established and documented reliability among

scorers

i. used LAS data to inform the design of curriculum and instruction

j. Please indicate the total number of certification assessments that have been planned for each grade span for English/Language Arts.

The information contained in this survey pertains to the Local Assessment System for the following SAUs:

88

k. If you have created more than 12 assessments for K - 12 in English/Language Arts, please provide a brief rationale below.

4. Rate your SAU’s progress on each of the following activities. Your rating should

represent the progress being made in Mathematics across each grade span.

MATHEMATICS


K - 4 5 - 8 9 - 12 Our SAU has . . . 1 2 3 4 1 2 3 4 1 2 3 4 a. developed curriculum documents aligned with

Maine Learning Results.

b. aligned instruction with curriculum and Maine Learning Results.

c. selected a sample of performance indicators to assess using Maine’s Balance of Representation or some other method of prioritization.

d. used scoring rubrics for each assessment e. administered common assessments f. provided professional development in common


g. double scored common assessments h. established and documented reliability among

scorers

i. used LAS data to inform the design of curriculum and instruction

j. Please indicate the total number of certification assessments that have been planned for each grade span for Mathematics.

k. If you have created more than 12 assessments for K - 12 in Mathematics, please

provide a brief rationale below. 5. Please indicate the number of certification assessments in your SAU’s LAS that

are MAP tasks, LAD tasks, and locally developed tasks. Count modified MAP and LAD tasks as locally developed.

English/LA Math Science & Tech Soc. Studies Health & PE

TASKS K - 4 5 - 8 9 - 12

K - 4 5 - 8 9 - 12

K - 4 5 - 8 9 - 12

K - 4 5 - 8 9 - 12

K - 4 5 - 8 9 - 12

MAP

LAD

Locally Developed

89

6. Indicate the progress your SAU has

made toward developing an LAS.


K-4 5-8 9-12

1 2 3 4 1 2 3 4 1 2 3 4

Science and Technology Social Studies

Health and Physical Education

Modern and Classical Languages

Visual and Performing Arts

Career Preparation

7. Indicate the progress your SAU has made on each of the following activities toward implementing an LAS.


K-4 5-8 9-12

1 2 3 4 1 2 3 4 1 2 3 4

a. selected either the “percent of points earned” or the “pattern of performance” method for determining a student’s performance level in content areas and across assessments

b. developed a procedure for public reporting of scores at the required levels for certification.

c. developed a policy for replacement assessments

*d. developed a means of providing supplemental instruction prior to replacement assessments

**e. for students who transfer to your SAU, developed a means of determining the value of the prior educational experience toward meeting the standards of the LAS.

***f. developed a report card that describes students’ progress toward meeting content standards

• If you indicated 2, 3, or 4 to d. above in any grade span, please provide a

brief description of how this will be done. ** If you indicated 2, 3, or 4 to e. above in any grade span, please provide a

brief description of how this will be done. ***If you indicated 2, 3, or 4 to f. above in any grade Yes span, are you also providing a “traditional” report card? No

90

8. Is your SAU currently collaborating with one or more Yes

SAUs for the purpose of developing an LAS? No

8a. If “Yes” to item 8, briefly describe the nature of this collaboration and how it has been helpful.

9. Please indicate the progress that your SAU has made in developing an LAS that

meets technical standards to assesses the progress of the following groups of students:


Planning in

progress Partially complete

Work complete

English Language Learners

Students in Career and Technical Education Programs

Students in Alternative Education Programs

Students with 504 plans

Students with disabilities who require accommodations

Students with disabilities who require alternate assessments (PAAPs)

10. Indicate the level of help your SAU needs to build the capacity to accomplish the following tasks.

Level of help needed in my SAU to build the capacity to . . .

No help needed

Moderate help needed

A lot of help needed

1 2 3 4 5

select, develop or adapt assessments

establish reliability among scorers

administer assessments

train teachers to score assessments

develop a data management system for recording LAS data

develop a data management system for analyzing LAS data

use LAS data to inform curriculum content

use LAS data to inform curriculum sequence

develop policies related to assessment(e.g., graduation, replacements, accommodations)

91

Level of help needed in my SAU to build the capacity to . . .

No help needed

Moderate help needed

A lot of help needed

1 2 3 4 5 develop accommodations for assessing the progress of students with disabilities

develop alternate assessments (PAAPs) for students with disabilities

assess students enrolled in Career and Technical Education programs

assess students in Alternative Education programs

assess students who are English Language Learners

develop a report card that reports students’ progress toward meeting standards in each content area

Not at all confident

Very confident

1 2 3 4 11. How confident are you that your SAU will have

assessments in place to certify that high school graduates in 2008 will have met the Learning Results in English/Language Arts?

12. How confident are you that your SAU will have assessments in place to certify that high school graduates in 2008 will have met the Learning Results in Mathematics?

13. In your opinion, what percent of the high school graduates in your SAU in 2008 will have achieved the Learning Results in . . .

English/Language Arts less than 25% 26%—50% 51%—75% 76%—100% Mathematics less than 25% 26%—50% 51%—75% 76%—100% Do not

understand at all

Completely understand

1 2 3 4

14. In your opinion, how well do the parents of current ninth grade students (Class of 2008) understand the graduation requirements that will apply to their children?

15. In your opinion, how well do the current ninth grade students (Class of 2008) understand the graduation requirements that will apply to them?

16. Has your SAU developed a policy or procedure that addresses how students who are

planning an AFS or other student-exchange program will meet the requirements of the LAS?

Yes No If yes, please describe briefly below.

92

17. List below three positive and three negative impacts of the implementation of Local Assessment Systems

Positive Impacts Negative Impacts

1.__________________________________ 1.__________________________________ 2.__________________________________ 2.__________________________________

3.__________________________________ 3.__________________________________

18. We know that “time” and “money” can be both boosters and barriers to the

development/implementation of an LAS. Please help us identify other boosters and barriers to the LAS process that have facilitated or impeded your progress.

Boosters Barriers

__________________________________ __________________________________ __________________________________ __________________________________

__________________________________ __________________________________ __________________________________ __________________________________

19. Which of the following describe the current use of Local Assessment System data

in your SAU? (check all that apply) LAS data . . .

are used informally in teachers’ discussions about curriculum and instruction are formally analyzed across grade levels and groups of students are used to identify curriculum strengths and areas of need are used to revise curriculum are used to inform priorities for professional development are used to identify students’ needs for remediation are used to inform the community about students’ progress toward achieving

the Learning Results other (please describe) _________________________________

20. Below, please describe any changes (e.g., new courses, programs, schedules)

that have been implemented in your SAU to support the class of 2008 to meet the requirements of the Learning Results.

93

Appendix B

LASIS II Site Visit Interview Protocols Initial questions by phone for Curriculum Coordinator or other key person who is very knowledgeable about district-wide work on curriculum and LAS and the specifics: 1. How long have you been working in this school system/district? 2. Describe the structure for implementing the LAS work. (Who is leading it for different grade

spans/content areas? How is it structured? How many people involved? How are teachers involved?)

3. Have you created templates to show how assessments are distributed across content areas and grade

spans? (We’d like to get copies of the templates for English Language Arts, Mathematics, Science, Social Studies for each grade span during our site visit. We’ll also mail you a table to fill out to indicate what work has been done in each area.)

4. Are you planning any scoring sessions that we might be able to observe? (We’d like to see how you

are scoring, calibrating the assessments.) 5. Has your district developed district-wide policies on LAS, for example, on replacement

assessments, remediation, scoring, and how to integrate MEA and local assessment? (We’d like to get copies of policy documents during our visit.)

6. We’re also interested in district communications with parents, students, and the public about the

LAS, changes in report cards, etc. (We’d like to get copies of these types of communication during our visit.)

7. Does your district have an Alternative Education program? Who is the coordinator? Does your district have students in a Career and Technical Education program? Who is the

coordinator? Does your district have an ELL Designate (coordinator)? Who is the designate?

Questions for site visit interview with district Curriculum Coordinator (or other key person who is knowledgeable about district curriculum and LAS work)

Part 1: Status of Work HOW DISTRICT ORGANIZED 1. How did your system organize itself to develop and implement a LAS? (Is it different for

different grade spans or content areas?) 1a. Please describe your role in the LAS? PROGRESS ON SPECIFIC COMPONENTS OF LAS 2. Looking at a template for assessment in English Language Arts in one grade span, please

describe how your district selected which performance indicators to assess? (same process for other content areas?)

3. To what extent does the LAS use common versus classroom assessments?

94

SCORING OF LAS 4. How are the assessments being scored this year? (How were staff selected/organized to do

work? Who involved? What professional development for scorers? How are assessments scored—single or double scored? By certified scorers?)

5. How has your district determined the reliability of the scoring of assessments? (How was it documented?)

6. Do you anticipate doing any other work on scoring prior to the start of the next school year? COMPARABILITY

7. Will your district use MEA scores as part of the evidence for certifying that students have met standards? How much will the MEA count? (Any district policy on this?)

REPLACEMENT ASSESSMENTS/REMEDIATION 8. Has your district decided how remediation will be handled if students fail an assessment?

How will replacement tasks be developed?

DATA MANAGEMENT 9. How does the district plan to manage data for the LAS? (current practice and plans for change)

Who handles this? 10. How is assessment data being used at this time? (by admin., teachers, etc.) DATA REPORTING 11. How is your district communicating with parents, students, the community about the LAS?

(how, when, frequency—collect sample documents) 12. Has your district changed the format of report cards in response to the LAS? (get sample) 13. Does your district compare student performance on the MEA and the LAS? Will you be

reporting this comparison publicly? SPECIAL STUDENT SUBGROUPS

14. Please describe any issues your district has identified in the following areas and how your district is resolving these issues:

• Assessing/certifying students in special education? (number of students?

number/percentage taking regular assessment? assessments with accommodations? alternative assessments?) To what extent do these students have opportunities to learn skills required by the Learning Results?

• Assessing/certifying students in career and technical ed? (number of students? efforts to

coordinate assessment between the district and the program?) • Assessing/certifying students in alternative ed programs? (number of students? Describe

program? Do they take regular form of local assessments?) • Assessing/certifying English Language Learners? (number of students? Do ELL students

receive direct support beyond regular classroom instruction? How? Do they take the regular form of local assessments?)

• Assessing/certifying transfer students?

95

Part II: Views, perspectives on time, impacts, resources, supports, barriers—

TIME SPENT ON LAS 15. When is work on the LAS being done? (for development, training, scoring—e.g., inservice,

release time, summer stipends, etc)

PURPOSE OF LAS 16. How do you view the purpose of the LAS? (Have your views changed over time?)

IMPACTS/SUSTAINABILITY 17. How has the LAS impacted your district’s curriculum? (e.g., focus of curriculum, time for

subjects) 18. How has the LAS initiative impacted classroom instruction in your district? 19. How has the LAS impacted collaboration among teachers? 20. How has the LAS impacted teacher learning? 21. How has the LAS impacted students? 22. How has the LAS impacted district communication with parents? (in both directions) 23. How has the LAS impacted district resources? (time, staffing, budget, etc.) 24. How has the LAS impacted other district priorities or initiatives? 25. Any other impacts I haven’t asked about? 26. What concerns do you have about your district’s ability to maintain the LAS in future years?

How might you use LAS data in the future?

OTHER SUPPORTS/BARRIERS 27. What resources or supports have been helpful to your district in

developing/scoring/implementing local assessments? (e.g., documents—LAS Guide, Measured Measures, Considering Consistency, Balance of Representation Report, or LAD/MAP assessments, MDOE personnel, teachers or others in district with expertise, support from other districts, reallocation of financial or staffing resources)

28. Has your district participated in the MDOE scoring sessions (e.g., LAD sessions, piloting, field-testing, or regional sessions on LAS?

29. What technical or other resource supports would be helpful for developing/scoring/implementing local assessments?

30. What are some of the barriers to developing/implementing the LAS?

Questions for Superintendent

PURPOSE/VISION OF LAS 1. How do you view the purpose of the LAS? (Have your views changed over time?) 2. How do you communicate your vision about the LAS to your principals and teachers? 3. How do you communicate about the LAS with parents and the community? IMPACTS/SUSTAINABILITY 4. How has the LAS impacted district resources? 5. How has the LAS impacted other district priorities or initiatives? 6. How has the LAS impacted teacher 7. How has the LAS impacted students? 8. What other impacts has the LAS had in your district? 9. What concerns do you have about your district’s ability to maintain the LAS in future years?

96

OTHER SUPPORTS/BARRIERS 10. What resources or supports have been most helpful to your district in developing/implementing

the LAS? 11. What technical or other resource supports would be helpful for developing/implementing the

LAS? 12. What are some of the barriers to developing/implementing the LAS?

Questions for teacher leaders, principals, or others who are ACTIVELY INVOLVED IN LEADING work on the LAS will be derived from the questions drafted for the Curriculum Coordinator, based on what we learn about the district’s organization for this work from our initial phone conversation with the Curriculum Coordinator. What questions we need to ask will depend on the district’s structure and what we learn during the on-site interview with the Curriculum Coordinator. It will be a smaller subset, including both questions on progress made within particular grade spans, content areas and questions on views. T= teacher question, P=principal question Suggested topics/questions: HOW DISTRICT ORGANIZED 1. Please describe your involvement in developing/implementing the LAS in your district? (How

selected? How involved? What kind of leadership role?) 2. How have other teachers been involved in selecting performance indicators to be assessed? In

developing local assessments? In scoring assessments? 3. When is the work on developing, scoring assessments being done? How are teachers or teacher

leaders compensated? PROGRESS/SCORING 4. To what extent are the local assessments planned for your grade span/content area being

administered this school year? 5. How are these assessments being scored? 6. What training, calibration sessions have been held? How was this done? 7. How has reliability of scoring been determined? How documented? 8. Do you anticipate any other work on scoring before the start of the next school year?

DATA MANAGEMENT/DATA REPORTING 9. How do you use data from local assessments? 10. Has your district changed report cards in response to the LAS? 11. How do you communicate with parents and students about the LAS?

PURPOSE OF LAS 12. How do you view the purpose of the LAS? 13. Have your views changed over time? T: 14. To what extent do other teachers in your grade span share your view? 15. Describe your superintendent’s view about the LAS. T: 16. Describe your principal’s view about the LAS.

97

IMPACTS/SUSTAINABILITY 17. How has the LAS impacted your curriculum? T: 18. How has the LAS impacted your teaching? P: 19. How has the LAS impacted teaching in your grade span? 20. How has the LAS impacted collaboration among teachers in your grade span? 21. How has the LAS impacted your students? 22. How has the LAS impacted your communications with parents? 23. How has the LAS impacted other district priorities or initiatives? 24. Other impacts? (on resources?) 25. What concerns do you have about your district’s ability to maintain the LAS in future years?

(distinguish between current effort and future years) SUPPORTS/BARRIERS 26. What resources or supports have been helpful to your district in developing/implementing the

LAS? 27. What technical or other resources and supports would be helpful in this work? 28. What are some of the barriers to developing/implementing the LAS? Questions for teachers (T), Department Heads (T), or principals (P) who are NOT leading LAS work--(We’ll interview one principal and one teacher from each grade span.) STATUS OF WORK/SCORING T: 1. To what extent are the local assessments planned for your grade level/content area being

administered this year? T & P: 2. When is the work on developing LAS and scoring assessments being done? T & P: 3. Who is involved in this work? To what extent are teachers involved in this work? How

are they selected? Are they compensated for the time spent on this work? T: 4. How are these assessments being scored? T & P: 5. What training on scoring has taken place so far? How has this been done?

SPECIAL STUDENT GROUPS T: 6. Do you have any special education students in your classroom? Are they taking the

regular assessments in the LAS? How many (or approximate percentage) take these assessments with accommodations? How many (or approximate percentage) take alternate assessments?

T: 7. Do you have any LEP students in your classroom? Are they taking the regular assessments in the LAS? How many (or approximate percentage) take these assessments with accommodations? How many (or approximate percentage) take alternate assessments?

USE/REPORTING OF DATA T & P: 8. How are you using assessment data? T: 9. How are you communicating with parents, students about the LAS?

PURPOSE OF LAS T & P: 10. How do you view the purpose of the LAS? (Have your views changed over time?) P: 11. How do you communicate your vision about the LAS with your teachers? With parents?

98

IMPACTS/SUSTAINABILITY T & P: 12. How has the LAS impacted your curriculum? T: 13. How has the LAS impacted your teaching? T & P: 14. How has the LAS impacted collaboration among teachers? T & P: 15. How has the LAS impacted your students? T & P: 16. How has the LAS impacted your communications with parents? T & P: 17. How has the LAS impacted other district priorities or initiatives? T & P: 18: Other impacts? T & P: 19. What concerns do you have about your district’s ability to maintain the LAS in future

years? SUPPORTS/BARRIERS T & P: 20. What resources or supports have been helpful to your district in developing/implementing

the LAS? T & P: 21. What technical or other resources and supports would be helpful in this work? T & P: 22. What are some of the barriers to developing/implementing the LAS? T & P: 23. Other comments about the LAS?

Questions for Special Education Director and special education teachers (K-8 and 9 - 12) Issues/Impacts: 1. Please describe any issues your district has identified with the assessment of students in special

education? 2. How has your district resolved these issues? 3. Are there other ways the LAS has impacted special education students? Form of assessment: 4. How many (or approx. what percentage) of the special education students take the regular

assessments in the LAS? 5. How many (or approx what percentage) take these assessments with accommodations? 6. How many (or approx what percentage) take alternate assessments? 7. What progress has been made to date in developing alternate assessments for students in special

education? (by grade span, content area) How was this work done? Who involved? 8. Have you developed a “template” that indicates when/how individual students in special

education will be assessed? How was the template developed? 9. Is there an implementation schedule for alternate assessments? Sustainability/Supports/Barriers: 10. What concerns do you have about your district’s ability to maintain the LAS in future years? 11. What resources or supports have been helpful in developing/implementing alternative

assessments for students in special education? (MDOE supports?) 12. What technical assistance or other resource supports would be helpful? 13. What are some of the barriers to developing/implementing assessment for special education

students?

99

Questions for CTE Coordinator (or provider) 1. Please describe the nature of the Career and Technology Education program? 2. Please describe the students attending this program? 3. What percentage of these students takes the local assessments in the regular form? 4. Please describe any issues your district has identified with the assessment of students in Career and

Technology Education (CTE) programs? 5. How has your district resolved these issues? 6. In what other ways has the LAS impacted these students? 7. To what extent do the school or district and the CTE program coordinate student assessment? 8. What technical supports or resources would be helpful in this effort?

Qs for Alternative Education Coordinator (or provider) 1. Please describe the nature of the Alternative Education program? (adolescent vs. adult program?

location of services? focus of curriculum?) 2. Please describe the students attending this program? 3. What percentage of these students takes the local assessments in the regular form? 4. Please describe any issues your district has identified with the assessment of students in the

alternative education program? 5. How has your district resolved these issues? 6. In what other ways has the LAS impacted these students? 7. To what extent do the school or district and the alternative education program coordinate student

assessment? 8. What technical supports or resources would be helpful in this effort? Qs for ELL Designate or Coordinator 1. How many students in each grade span (PK - 4, 5 - 8, 9 - 12) are designated LEP (participate in the

English Language Learners—ELL program)? 2. Please describe what type of direct support is provided to LEP students in addition to regular

classroom instruction? 3. What percentage of these students takes the local assessments in the regular form? 4. Approx. what percentage of these students takes the assessments (LAS) with accommodations? 5. Approx. what percentage of these students takes alternate assessments (LAS)? 6. Please describe any issues your district has identified with the assessment of LEP students? 7. How has your district resolved these issues? 8. In what other ways has the LAS impacted these students?

100

Appendix C

Previsit Information Tables

Q.1. What is the status of curriculum alignment with the Maine Learning Results in your district at this time? (i.e., To what extent do you have curriculum in place that prepares students to meet the Maine Learning Results?)

Please use one of the following marks to complete each box: “X” = curriculum is aligned and in place “M” = curriculum is mostly aligned/in place “P” = curriculum is partially aligned/in place “N” = curriculum alignment has not yet been addressed

Content Area PreK-2 3-4 span 5 - 8 span 9 - 12 span

English Language Arts Mathematics Science Soc Science Health/PE Career Prep Modern Languages Visual/Performing Arts

101

Q.2. What is the status of work on your district’s local assessment system at this time? Please use one of the following marks to complete each box: “X” = work is complete; “M” = work is mostly complete, or complete for most assessments; “P” = work is partially complete or complete for some assessments; “N” = not yet addressed

Content Area/ Grade Span

Selected MLR performance indicators to be assessed

Identified which assessments will be used

Administering these assessments this school year

Developed scoring rubrics for each assessment

Held scoring sessions to train scorers, and score assessments

Documented how reliability was determined for each assessment

ELA PK - 4 ELA 5 - 8 ELA 9 - 12 Math PK - 4 Math 5 - 8 Math 9 - 12 Science PK - 4 Science 5 - 8 Science 9 - 12 Soc PK - 4 Soc 5 - 8 Soc 9 - 12 Health/PE PK - 4 Health/PE 5 - 8 Health/PE 9 - 12 Career PK - 4 Career 5 - 8 Career 9 - 12 Languages PK - 4 Languages 5 - 8 Languages 9 - 12 Vis/P Arts PK - 4 Vis/P Arts 5 - 8 Vis/P Arts 9 - 12

102

Appendix D

Focus Group Protocols

Below is the protocol used for a focus group with Directors of Career and Technical Education programs. The same questions and probes were used in focus groups for special educators, alternative educators and telephone interviews with Directors of programs for English Language Learners. Nature of Program Describe the nature of the CTE Education program. Number of students, grade levels. Location, focus of curriculum. Describe the characteristics of students who attend. Issues/Impacts What issues arise related to students in the CTE program and Local Assessment Systems? How have these issues been resolved? Which issues remain unresolved? How has LAS impacted students in CTE programs? Impact on parents? …on teacher collaboration within CTE programs? …on collaboration between CTE teachers and teachers in sending high schools? …on general ed teachers’ perceptions of students in CTE programs? …on teaching in the CTE program? …on CTE curriculum? …on you? Form of Assessment/Scoring How many CTE students take the regular LAS assessments? How many take the LAS assessments with accommodations? Has there been collaboration between CTE teachers and high school teachers in the design of LAS assessments? In what content areas? Have CTE teachers designed replacement assessments that have been acceptable by sending schools? Have CTE instructors been involved in scoring LAS assessments? How has scoring of LAS assessments been done for students in CTE programs? View of the Purpose of LAS What is your view of purpose of LAS? Have your views changed over time? How are views communicated to principals and teachers? How communicated with parents and communities? Examples/samples? What role do national trade standards play in the assessment of CTE students? Use of Assessment Data How are assessment data being used? How are the results of assessments being communicated to students, parents, other teachers of your students? Sustainability/Support/Barriers What resources or supports have been helpful in developing LAS assessments for CTE students? What technical or other resource supports would be helpful for developing/implementing LAS for students in CTE programs? What are some of the barriers to developing/implementing LAS for these students?

103

Appendix E

Changes to Support the Class of 2008 (Survey item #20)

This is a list of changes that have been implemented which support the class of 2008 meeting the requirements of the Learning Results. Adjustments

• Adjustments in K-12 Science to introduce more Earth/Space content. • Adjustments in K-12 Social Studies to introduce more economics content. • The high school has adjusted the math sequence.

Advisory Programs

• Anticipated: Teaming at grade 9 with advisor/advisee component. • Changes in advisor-advisee program. • New advisory program. • Players /Coaches for grade 9 (Advisor/Advisee). • We are beginning advisor /advisee. • We have also implemented an advisory program and student led conferences to support the

class of 2008 to meet requirements and classes of the future. After-School Programs

• After school help is available. • 21st Century After School Program allows us to provide some extended learning

opportunities • After school and study hall tutors hired. • Block 5 - after school at the high school. • Implemented after school study session.

Alignments

• A strong focus on developing a comprehensive math assessment and K-12 alignment that is K-16 usable, (i.e., college ready for every student.

• Adult Education options being aligned as an option. • All curricular grades K-12 are aligned. • Electives are being aligned to help fill in individual gaps and provide options for students.

Block and Nonblock Scheduling

• 4x4 block schedule. • 4x4 block schedule. Fewer courses, better concentration, and more time. • Somewhat of a block schedule. Monday - periods 1-8. Tuesday & Thursday periods 1, 3, 5,

7 (double). Wednesday & Friday periods 2, 4, 6, 8 (double). • We have changed morning schedule from block to non-block in order to assure that all

freshmen can take Algebra I.

104

Coursework

• A chemistry/physics course in the 3rd or 4th high school year to teach and assess these standards in science.

• Accelerated English 9 class. • Algebra I as the first high school math course. • Career prep course for all high school students. • Changes in high school science program of study; an integral science freshman course. • Eliminating courses that do not contribute to high school expectations. • Everyday math (everyday algebra) • Foreign language 1-6 implemented. • Grades 9 - 12 have 9 advance placement courses. • High School science for freshman now covers a broad curriculum. Integrated science 1

&2. • Implementation of career prep course for juniors. • Implementation of full year of health at grades 9 - 12. • Integrated grade nine math curriculum (Core Plus). Full year Chemistry/Physics course for

junior year. Aligned curriculum with Maine Learning Results. • Integrated science. • Math/English Language Arts Essentials course. • New freshmen structure and course changes to facilitate access to assessments by all

students. • New health curriculum. • New high school course structures so that students meet LRs by sophomore or junior year. • Reading 100. • Social studies and science have courses students can take to work on common assessments. • Standards have been embedded in current course offerings with varying degrees of quality.

We’ll need to reduce tracking and tighten the core path in math and science to ensure students can meet on common assessments. Tough discussions, not happening yet.

• Tech math - low functioning. • There is a pre-algebra class.

Curriculum

• 9-10 grade schedules front-loaded to allow time at other end for remediation. Students can fulfill math requirement by grade 11, math sequence changed and condensed.

• An applied algebra course has been added to the schedule to provide students in the class of 2008 with remediation as a result of 8th grade assessments.

• Change in 9 - 12 science schedule and course offerings. • High school math sequence revised to Algebra 1, Geometry, Algebra 2 (math lab in

freshman year as second math course). Science sequence required, Intro, Biology, Chemistry (3 science credits needed for graduation). Vocational student must take Algebra 2 & Chemistry in junior year at our high school.

• K-8 curriculum changes in order to use MAP/LAS assessments. • K-8 system has implemented new math program, written and revised curricula. Work

schedules to include 1 hour for math and 1 hour for English Language Arts - K-8. • Math Department is reviewing NSF Math programs to increase math achievement. • Moved curriculum to insure standards met by 10th grade.

105

• Realigned math curriculum to prepare students for 9 - 12 MLRs. • Revised science courses and sequence of courses. • Revised science sequence. • Schedule change. • Some h.s. curriculum revisions to courses. • We added American Sign Language as a foreign language option.

Late Starts and Extended Days

• Late start at h.s. 1 day per week to allow for work on all aspects of LAS/instruction. • Longer teacher day initiated to provide for daily team meeting time to focus on curriculum,

instruction and differentiating instruction to meet specific learning needs. • We have 3 late starts so teachers can score student work from 7:30-9:30.

Learning Labs

• A learning lab was developed for our high school students. • Learning lab at middle school. • Learning labs. • Learning labs in English Language Arts and math. • Modified “learning labs” - math. • We have implemented computer aided instruction in reading and math, learning centers.

Learning Centers

• Establishment of 9 - 12 learning center being initiated in 05/06 to support students who are not meeting standards.

• Restructured Learning Center to allow for Ed. Techs to be in the classroom. • We have a Learning Center open every day and two days a week after school from 2:30 -

4:00. Literacy

• Increased literacy instruction at the JR High. • Some 9th grade students have extra block for literacy support. • Stress on literacy strategies. • Students with low reading scores on 8th grade MEA and NWEA and teacher

recommendation are scheduled into a READ 180 lab at SH.S. Math Labs

• Implementing a Math lab FY 06. • Math lab. • Math lab to support remediation. • Math labs at high school. • Remedial labs in math & English Language Arts.

106

New Courses

• Added a remedial course in English Language Arts & math. • Addition of career prep grades 8-10. • All of the above - new courses, programs and schedules have been implemented to support

class of 08. • Development and implementation of a literacy course for at risk English Language Arts

students. • High school math department established a new course 2 years ago called LR math for

those students entering high school who are in danger of not being able to get through Algebra, Geometry, and Advanced Algebra, the courses where LR assessments are given.

• High School: English Language Arts - Implemented an assessment skills development course to parallel the grade 9 English survey course. The assessment course is designed for those who are having difficulty or displayed weak skills upon entering grade 9. The assessment courses taken along with the English survey course. This extends to grade 10. Math - same format will be set up in math for the 05-06 year.

• Implementation of a pre-standard literacy course in English Language Arts (Math in 2005- 2006).

• Math Booster class in middle school. • New classes - Creative Writing, Integrated Math, Math labs and writing labs in place of

study halls. • New classes instituted for credit in earth science, economics, geography and geometry. • New literacy program (freshmen required two blocks of English). • New math program - higher level thinking skills. • New music program at 5 - 8. • New visual and performing arts course that all students will enroll in.

New Hires

• Additional 3/5 position at K - 4 level. • Additional of an Ed. Tech. • At the H.S. we are also exploring hiring an Ed Tech III to work with struggling students in

English Language Arts and math as prevention. • Have had to add one new teacher for Health, not feasible to do 5/6 assessments in one

semester, needed full year of health. • Hiring of literacy specialists (K-8). • K-12 gifted and talented teacher for 05-06. • Possibly an adolescent literary coordinator. • We hired a curriculum coordinator.

Plato

• Implementation of PLATO labs at three of our schools. • Plato Learning system. • We’ve implemented PLATO for students to have additional practice.

107

Remediation

• Additional /different instruction by teacher with students. • Math and English Language Arts workshops and remediation help classes. • Remediation program at High School. • Work for students who have failed a course and course-work for incomplete standards

work. Requirements

• 4 years of Social Studies required (yet the 4th is an elective and therefore not subject to LR because not all students will take the same core).

• Added two new credits to graduation requirements (Sci & SS) and required specific level of math courses for grad requirement. Added a Title IA component to our high school.

• Additional course requirements in math, science, and social studies. • All requirements in place for all students (IEP exceptions) for courses and common

assessments four years ago and working very well. • All students will be taking college ready course. • Increased graduation requirements. • Increasing the number of credits needed. • New high school requirements for classes, more math (1 more), more science (1 more). • Over the years, we’ve increased the number of required credits . • Senior Exhibitions required for graduation. • 3rd-year requirement in Mathematics for graduation.

Structure of Learning Environment

• Formation of curriculum teams, assessment facilitators and cross high school content area teachers. Restructuring of an academic team at middle school.

• The high school is currently undergoing a restructuring of their course and schedules. • We are redoing high school structures (class time and course descriptions and performance

levels to support learners needs). • We are planning our new schools structure around how we feel we can best provide the

most in-depth and compressive educational experience. Some of the data we have received from common assessments this year is helping to drive some of those decisions.

Review Curriculum and Requirements

• Examined course of study at high school. • May need to add more geography and economics at a later date to high school curriculum. • More analysis and discussion of test scores, classroom performances to set school wide

goals. • SAT process at all 3 buildings now includes discussing and documenting accommodations

that are necessary for students. • We have discussed ideas for students who do not meet LAS criteria to be certified as to

how we will re-teach and administer replacement assessments. We do not feel we have a reliable LAS system to certify 2008 graduates at this time. With the current changes proposed for a timeline for implementation and certification. We will take this time to

108

implement and evaluate our current system. Year 2005-2006 will allow us time to make necessary revisions, changes, and staff training to develop a system we have faith in the measures student performance to a standard.

Study Halls

• Academic study for at-risk grade 9 students. • MLR support grade 9 & 10 where content teachers in math and English available during

designated study halls for tutorial or replacement. • New support study hall. • On the H.S. level content teachers have study periods with students struggling in their

subject area. Ex, English Language Arts teacher works with struggling English Language Arts kids.

• PARK guided study halls, manned by a teacher who works to help students meet requirements.

Summer School

• Positive - Summer programming with credits to standards. • Re-examination of summer school to have standards based component. • Regional summer school. • Summer academy enrichment opportunities to offer more access to some courses. • Summer academy remedial support - English Language Arts & Math. • Summer Academy. • Summer programming for students in middle school. This would be those children who

need additional instructional time as demonstrated by low achievement. • Summer school. • Summer school will provide a two-tiered approach. • Summer sessions. • We are exploring ways to provided summer support, after school support, etc. • We are implementing our remedial programs beginning this summer. • We held summer school last summer and will do so again this year. • We provide summer school and courses before and after the school day.

Tutoring

• After school tutoring. • Half time tutor at high school for student intervention. • Have a tutorial as part of block schedule to support student achievement of MLRs. • Math tutor. • Positive - tutorial at HS in all content areas. • Tutorial classes.

Writing Lab

• English/writing labs (work one on one with an English teacher). • Writing lab. • Writing labs (9 - 12).

109

•

Appendix F

Boosters and Barriers (Survey Item #20) Respondents were asked to list boosters and barriers to the LAS process that have facilitated or impeded progress. A large quantity of data was generated by this request. The text of this report lists the major categories of boosters and barriers. This section expands the categories slightly and provides a small sample of the actual responses. Assessment Boosters:

• Assessment committees at each building. • Assessments have forced us to look at the curriculum we are teaching and how we are

teaching. • Availability of LAD and MAP tasks has helped and provided a common pool of

assessments.

Barriers: • A lack of good assessments in some content areas. • Focus on assessment, not instruction. • Large number of assessments to complete.

Communication Boosters:

• Collegial interaction • Created more discussion among staff about curriculum. • More public communications regarding LAS from the State.

Barriers:

• Lack of communication from the state to schools and schools to the community. [Affects our] credibility.

• Mixed messages regarding NCLB and Maine Learning Results. • Negative articles and coverage in the press.

Impacts Barriers:

• Credibility. Our credibility has been diminished with the constant revision of the LAS plan. • Hidden costs in terms of finances and resources. • Unfavorable press coverage.

110

Data Management Boosters:

• A pre-made data and reporting system. • Some teachers seeing the relationship between curriculum, instruction, assessment, and

redefining what they teach based on data. • Understanding how useful the data is to increase achievement.

Barriers:

• No final reporting mechanism is defined. • No real plan for keeping students’ products safe for 16 years. • No system to deal with data collection.

Dedication and Professionalism Boosters:

• Committed staff (did scoring training on their own time). • Staff that has a willingness to do work and be positive with the many changes. • We all want to do a good job for kids.

Frustrations Barriers:

• Frustration among staff members concerned about special needs children and decreased instructional time.

• Staff grumbling about changing targets. • We are extremely tired and frustrated with trying to figure this out on our own. We are at a

point where DOE needs and must provide new LR, and even the assessments we are supposed to use.

Leadership Boosters:

• Leadership of principals and teacher leaders. • One person facilitating the progress. • Ability to have classroom teachers assume leadership roles.

Maine Department of Education Boosters:

• Guidelines from DOE. • Individual support: Tom Keller and Pam Rolfe • Monthly county curriculum leader’s meeting with Diana Doiron [MDOE English Language

Arts Specialist].

111

Barriers: • Failure of DOE to meet timelines on MEDMS, LAS Guide, etc. • Lack of DOE leadership and consultants. • Unclear or changing messages from the State or others working on LAS.

Funding Boosters:

• Grants provide money and coaches /consultants. • CSRD grant to pay teachers for their work and to provide professional development. • Extra money and time for training area leaders.

Barriers:

• Low salaries for increased load. • Money. • Stipend to lead curriculum workgroup

Moving Target /Still Evolving Barriers:

• Changes in requirements at the state level. • Changes in timelines and deadlines. • The moving target (DOE) served to frustrate and irritate those who’ve worked hard to stay

on target. Professional Development Boosters:

• Creating professional learning communities. • Professional development from MDOE and Maine Assessment Cooperative. • Quality professional development.

Barriers:

• All time has to be spent on LAS. Other important professional development is suffering. • Amount of professional time devoted to assessments. • Limited professional development.

Shared Goals Boosters:

• Common language and understanding of work to be done. • K-12 common goals for teaching and learning. • Sticking with the big picture (for the most part).

112

Staff Boosters:

• Caring staff. • Open and willing staff. • Staff skills and knowledge regarding assessment.

Staff Positions Boosters:

• Addition of Dean of Curriculum position. • Assessment coordinator • Curriculum coordinator.

Barriers:

• Lack of curriculum coordinator. • No curriculum director position.

Student Learning Groups Barriers:

• Even with proposed modifications there is a lack of consideration for at-risk student population.

• Lack of state directions about special groups (alternative education, special education). • Trying to provide opportunities for career-tech. students.

Support Boosters:

• A superintendent who champions and campaigns for the completion of the work. • Direct support through supported grade level and content area meetings. • School board support of early release days for scoring assessments.

Barriers:

• Lack of state level support for local change with the policies. • Parental support.

Time /Timelines Boosters:

• Given extra time. • Some early release time added. • Timelines.

Barriers:

• Pressure from unrealistic timelines. • School year is not long enough. • Time. Our teachers did LAS in English Language Arts, Math, Science, and Social Studies.