October 5, 2012 · National Center for Education Evaluation and Regional Assistance . Ruth Curran Neild, Ph.D., NCEE Commissioner . In September, NCEE released the State and District

National Board for Education Sciences Meeting Summary: October 5, 2012 1

October 5, 2012 Washington, DC

Location Institute of Education Sciences (IES) Board Room

80 F Street, NW

Washington, DC 20001

Participants National Board for Education Sciences (NBES) Members Present Bridget Terry Long, Ph.D., Chair

Kris D. Gutierrez, Ph.D., Vice Chair

Deborah Loewenberg Ball, Ph.D.

Anthony S. Bryk, Ed.D. (by phone)

David Chard, Ph.D.

Adam Gamoran, Ph.D.

Robert Granger, Ed.D.

Larry V. Hedges, Ph.D.

Susanna Loeb, Ph.D.

Margaret (Peggy) R. McLeod, Ed.D.

Judith Singer, Ph.D.

Robert A. Underwood, Ed.D.

Hirokazu Yoshikawa, Ph.D.

Ex-Officio Members Present John Q. Easton, Ph.D., Director, IES, U.S. Department of Education (ED)

Elizabeth Albro, Ph.D., Acting Commissioner, National Center for Education Research (NCER)

Ruth Curran Neild, Ph.D., Commissioner, National Center for Education Evaluation and Regional

Assistance (NCEE)

Joan Ferrini-Mundy, Ph.D., Assistant Director, National Science Foundation (NSF), Directorate for

Education and Human Resources


Peggy McCardle, Ph.D., Branch Chief, Child Development & Behavior Branch, Center for

Research on Mothers and Children, Eunice Kennedy Shriver National Institute of Child Health

and Human Development (NICHD), National Institutes of Health (NIH)

Chuck Pierret, Bureau of Labor Statistics

Marilyn Seastrom, Ph.D., Chief Statistician and Program Director, National Center for Education

Statistics (NCES) (for Sean P. “Jack” Buckley, Ph.D., Commissioner)

Deborah Speece, Ph.D., Commissioner, National Center for Special Education Research (NCSER)

NBES Staff Ellie Pelaez, Designated Federal Official (DFO)

ED/IES Staff Karen Akins

Corinne Alfeld

Lisa Bridges

Doris Dixon

Swati Mehta

Allen Ruby

Kim Sprague

Invited Presenters Sue Betka, Deputy Director for Administration and Policy, IES

Steve Fleischman, M.A., Chief Executive Officer, Education Northwest

Barbara Means, Ph.D., Director, Center for Technology in Learning, SRI International

Anne Ricciuti, Ph.D., IES Deputy Director for Science

Thomas Smith, Ph.D., Director and Principal Investigator, National Center on Scaling Up Effective

Schools and Associate Professor of Public Policy and Education, Department of Leadership,

Policy, and Organizations, Vanderbilt University


Members of the Public Nora Boretti, U.S. Government Accountability Office (GAO)

Megan Foster, Society for Research in Child Development

Jean Gossman, Education Daily

Stephanie Gregg, Washington Partners, LLC

Monica Herk, Ph.D.

Allie Huyghe, Strategic Education Research Partnership (SERP) Institute

Kim Hymes, Council for Exceptional Children

Sarah Grady, National Opinion Research Center at the University of Chicago

Carla Jacobs, Lewis-Burke Associates

Jim Kohlmoos, National Association of State Boards of Education

Myrna Mandlowitz, Southwest Educational Development Laboratory

Michele McLaughlin, Knowledge Alliance

Jeff Mervis, Science Magazine

Kristen Neishi, National Opinion Research Center at the University of Chicago

Juliana Paré-Blagoev, SERP Institute

Sarah Sparks, Education Week

Scott Spicer, GAO

Erika Thompson, American Association of State Colleges and Universities


Meeting Summary

Call to Order, Chair’s Remarks Bridget Terry Long, Ph.D., NBES Chair

Dr. Long called the meeting to order at 8:34 a.m., and Ms. Pelaez, DFO, called the roll. NBES

members unanimously approved the minutes of the June 20, 2012, NBES meeting. Dr. Long

welcomed two new Board members, Drs. Hedges and Loeb, noting that the Board has almost

reached its full complement. Dr. Long thanked Dr. Herk, former Executive Director of NBES, for her

assistance organizing the meeting and putting together the agenda. She hoped Dr. Herk would

continue to be involved with the Board. Dr. Easton said that Dr. Herk will be missed, but the Board

will have assistance; he said efforts are under way to assure that the Board is supported.

Dr. Long reviewed the agenda in the context of three areas on which the Board generally

focuses:

• IES research funding

• Impact through disseminating and scaling up promising practices

• Advocating for support and use of research in policymaking

Swearing-In of New Members John Q. Easton, Ph.D., IES Director

Dr. Easton swore in Drs. Hedges and Loeb. He noted that the Board now has 13 members.

Update: Recent Developments at IES John Q. Easton, Ph.D., IES Director

Dr. Easton welcomed the Board members and Ms. Pelaez. In August, IES convened a meeting of

outside experts on the use of value-added measures and student growth measures to assess

teacher effectiveness. The attendees represented a wide variety of fields. Despite a huge amount

of new research in the past few years, some in the field are not fully aware of what others are

doing. The IES meeting sought to gather the experts, facilitate conversation, and learn where they


agree or differ. That information will not only inform IES research grant-making and evaluation

considerations but also serve as a basis for advising ED on policy issues. A detailed summary of

the meeting, as well as the short summaries of key research prepared by the participants in

advance, will be available online shortly, said Dr. Easton. No specific follow-up effort has been

planned.

Commissioner Updates National Center for Education Evaluation and Regional Assistance

Ruth Curran Neild, Ph.D., NCEE Commissioner

In September, NCEE released the State and District Receipt of Recovery Act Funds: A Report from

Charting the Progress of Education Reform--An Evaluation of the Recovery Act’s Role, which

documents how funding was spent and includes the characteristics of funded schools and

districts, amounts, etc. It is part of a larger study of major Federal funding efforts and reflects an

NCEE effort to get interim reports out to the public more quickly. Dr. Neild said the What Works

Clearinghouse recently released a new practice guide, Teaching Elementary School Students To

Be Effective Writers, which she said is her favorite so far, because it offers a good framework and

nice examples. It also demonstrates NCEE’s interest in providing practice guides that are more

narrowly focused and more useful to classroom teachers than previous guides. The What Works

Clearinghouse also recently released the Reporting Guide for Study Authors to help researchers,

who are submitting materials to the What Works Clearinghouse, better understand the

components of research it includes.

Nine of the 10 Regional Education Laboratories (RELs) are up and running (one award is still being

contested). The nine RELs have linked with about 70 research alliances with whom they work on a

variety of topics. In the first year, the RELs and the alliances focus on building relationships through

small technical assistance projects that create community and support shared knowledge. In the

absence of a contract for a cross-REL coordinating task (which is under a stop-work order as a

result of the protest), IES staff are doing their best to ensure coordination among the RELs and to

encourage them to take the lead in areas in which they excel. REL representatives have taken

part in Society for Research on Educational Effectiveness (SREE) working groups and panels, and

attendees at a recent SREE conference expressed a lot of interest in the RELs’ early progress. Dr.

Neild noted that NCEE wants to ensure that RELs pay attention to the audiences they serve and


produce products that are not only technically sound but also readable, relevant, and useful.

Thus, review of products will take those factors into account, and the review criteria developed

can be shared across RELs and with IES.

NCEE has held three What Works Clearinghouse reviewer training sessions in the past 3 months.

More certified reviewers are needed to meet the demand to speed up evaluation of projects

and publish evidence to support scaling up programs. The training sessions also serve as

mechanisms to disseminate more research to the field (e.g., to those who attend the training but

do not seek reviewer certification). Dr. Neild said NCEE is considering whether to hold more

training sessions that would engage more broadly people representing State- and district-level

organizations.

In early August, it came to light that some of the older records in the Education Resources

Information Center (ERIC) included personally identifiable information. Therefore, NCEE prevented

access to those records temporarily. As they are revised, access to the full text of those records

will be restored, and the effort is moving along, said Dr. Neild.

Among the notable new NCEE contract awards is one for analytic technical assistance and

development that allows a lot of flexibility, so that NCEE can respond to issues that raise concerns

for the whole field as they arise. Among the topics to be addressed as part of this contract is a

guidance document on seizing opportunities for low-cost, quick-turnaround, policy-relevant

impact studies. Dr. Neild pointed out that several NCEE contract awards address postsecondary

education, such as evaluations of promising strategies for Federal college access programs and

experiments with Pell grant expansions. A new small business contract will focus on assessments of

postsecondary research studies for the What Works Clearinghouse.

Discussion

In response to the perception that the previous RELs lacked a coherent approach and seemed to

address research issues in a random manner, NCEE instituted a framework for the new RELs that

requires them to develop research alliances around certain topics. NCEE emphasizes that

collaboration with the research alliances should be driven by actions that practitioners want to

accomplish, which can be broad at the outset, such as reducing the dropout rate. NCEE expects


RELs to develop cohesive research agendas. Participants expressed interest in the evaluation

contract to support professional development of math teachers, and Dr. Neild said she could

provide more details about such efforts later.

National Center for Education Research (NCER) Elizabeth Albro, Ph.D., NCER Acting Commissioner

Applications for all of NCER’s 2013 awards have been submitted, and review of applications

begins soon. In July, NCER announced the last of its 2012 grants awards. They represent a wide

variety of projects, from eye-tracking studies to understand how children process graphical

information to using art to improve science education. Several awards support efficacy studies,

such as evaluation of the long-term outcomes of “doubling up on algebra” policies for students

behind in mathematics. A lot of work supported by NCER reflects the relationships over time of

different projects; for example, research found that using technology to teach adolescents how

to write is effective but complicated, so a new study is looking at feasibility of implementing that

intervention at a larger scale, said Dr. Albro.

NCER announced three new postdoctoral training awards and two new research and

development (R&D) centers (one on State and local policy and one on adult literacy). It also

announced four new awards in statistical and research methodology and three new awards

supporting evaluation of State and local policies and programs. Of the latter, one grantee will

evaluate Florida’s college and career readiness initiative for 11th graders who are not likely to

meet college acceptance criteria. (The initiative evolved from a voluntary to a mandatory

program, and researchers will assess its impact on college access and persistence.) Another

grantee is evaluating the long-term effects of retention under New York City’s student promotion

policy.

Dr. Albro said that an NCER-funded statistician from the University of California, Los Angeles (Li Cai),

received the Presidential Early Award for Scientists and Engineers. She was pleased that ED had a

large pool of candidates from which to make nominations for the award. She also announced

that six NCER staff members have taken the What Works Clearinghouse reviewer training and

spoke highly of it; one has already obtained certification.


Efforts have begun to support more public access to research conducted by NCER-funded

researchers, which is challenging, because these publications are not vetted and published by

IES but rather are published in scientific peer-reviewed journals. In one attempt to capture the

range and volume of publications produced by NCER-funded research grants, NCER publishes a

booklet listing all such publications; last year, it included more than 2,000 research reports and

articles. During FY 2013, NCER is identifying which research is available through ERIC and how it

can encourage researchers to submit more of their findings so that they are publicly accessible.

Beginning in 2012, researchers are required to submit their final accepted manuscripts to ERIC for

public dissemination, but NCER is seeking ways to make the results of more NCER-funded

research available to the public.

Discussion

Dr. Albro noted that about 10 percent of reviewed grant applications across the two research

centers were funded in 2012, and the rate has been steady for some years. Dr. Long said she will

ensure that Board members receive an email that includes the link to the NCER’s awards (which

include further links to the abstracts for each grant, said Dr. Albro). Dr. Long suggested the Board

regularly receive an email with a link to the NCER awards announcement after each round of

awards.

National Center for Special Education Research (NCSER) Deborah Speece, Ph.D., NCSER Commissioner

NCSER funded 49 grants across many initiatives in fiscal year (FY) 2012, reflecting a steady

increase over the past several years. Dr. Speece reported that the increasing number of

applications and awards demonstrates that NCSER, the youngest IES center, is maturing and

developing its presence.

Traditionally, NCER and NCSER have both had more awards supporting IES Goal 2, Development

and Innovation, than Goal 3, Efficacy and Replication. For new awards made in 2012, about 26

percent of NCSER’s awards focused on Goal 2, while about 60 percent addressed Goal 3. Dr.

Speece said the shift indicates that more interventions are being tested for efficacy which is a

positive development.


NCSER announced two postdoctoral training grant awards, both in early childhood education. It

also funded two R&D centers to undertake large, 5-year efforts—Georgia State University’s Special

Education Research and Development Center on Reading Instruction for Deaf and Hard of

Hearing Students and the University of North Carolina’s Center on Secondary Education for

Students with Autism Spectrum Disorders. Dr. Speece said she was pleased with the quality of the

applications, including their concentration on secondary-school students.

This past summer, NCSER sponsored the second Summer Research Training Institute on Single-

Case Design Research at the University of Wisconsin Madison to attract more investigators who

want to explore this approach for special education research, which is similar to, but broader

than, single-subject design research. The Institute allows investigators to keep pace with fast-

changing methodology. Single-case design can be applied to the field of education more

broadly. NCSER received 80 applications for the institute, and 40 people—ranging from early- to

late-career investigators—attended. NCSER is also working with the What Works Clearinghouse on

communicating the goals and vision around using single-case design research.

In November, NCSER and the National Institute on Disability and Rehabilitation Research and the

Office of Special Education Programs, both part of ED’s Office of Special Education and

Rehabilitative Services, will convene a technical work group of experts and practitioners in the

area of education of adolescents with disabilities to begin developing research models. The initial

meeting will provide a context in which to think about a successful school-based model for

addressing the problems faced by adolescents with disabilities. Dr. Speece hoped the initiative

would build over time.

Finally, NCSER updated its Projects and Programs booklet to include all of its programs and

funded projects through 2012. It includes online links to each program or project for more details.

Discussion

When asked whether long-term English-language learners (ELLs) will be considered by the

technical working group addressing adolescents with disabilities, Dr. Speece said the

categorization is difficult, partly because difficulty learning English does not necessarily represent a

learning disability among these adolescents. She said that NCSER requests for applications note


areas such as this one that require more input from the field. In response to another participant,

Dr. Speece acknowledged that continued growth in the number of NCSER awards during a

period of funding cuts is unlikely; maintaining a steady state would be a victory, she noted.

National Center for Education Statistics (NCES) Marilyn Seastrom, Ph.D., Chief Statistician and Program Director, National Center for Education

Statistics, NCES

NCES recently released results on writing and science assessments from the National Assessment

of Educational Progress (NAEP) that involved interactive computerized assessments. In early

December, NCES will release results from the Trends in International Mathematics and Science

Study (TIMSS) and the Progress in International Reading Literacy Study (PISA).

NCES redesigned the National Household Education Survey from a telephone-based to a mail

survey and contracted the U.S. Census Bureau to conduct the survey; results will be available in

the spring. The NCES website includes a link to the 2012 revised statistical standards, and Dr.

Seastrom said comments are welcome throughout the month.

The Schools and Staffing Survey (SASS) has been conducted every 4 years since 1988. In

conjunction with the Office for Civil Rights, the survey is being redesigned to become a nationally

representative survey that will collect data every 2 years. The results will no longer include State-

level detail, but the redesign allows more flexibility to gather information on subjects of topical

interest.

Once NCES puts in place its Longitudinal Study of Early Adolescence (for students in grades 6–8), it

will have data on a continuous cohort of students from kindergarten through 12th grade. Efforts are

underway to link NAEP and TIMSS data. NCES is working to improve data quality and increase

productivity of data collection for K–12 research. Several activities focus on improving data about

educational training—for example, on conditional postsecondary credentials that change in the

workplace through different types of job training. The NAEP is also assessing technical and

engineering literacy.


For years, said Dr. Seastrom, NCES has collected data at the school district level; to increase

transparency, NCES is now extending its data collection to the established boundaries for school

service areas. On the program side, it is improving access to program data and exploring ways to

make data public while protecting sensitive information. NCES is revising its licensing agreements

to make data available to researchers for acceptable research uses.

IES’s Peer Review Process Anne Ricciuti, Ph.D., IES Deputy Director for Science

In response to suggestions raised by the Board at the June NBES meeting, IES implemented the

following changes:

• To enhance transparency, the names of members of standing panels who reviewed the 2012

grants (i.e., last year’s grant applications) are available online, and the chairs of panels are

noted. In order to preserve confidentiality, the members who served on small or specialty

panels were listed in a combined list.

• To increase the prestige associated with serving on a review panel, IES is considering how to

make the Standards and Review Office more prominent on the IES webpage.

• To better understand the diversity of panel members, the paperwork for reviewers now

includes an optional section in which reviewers can indicate their gender and race/ethnicity.

The information will be used internally to track the makeup of the panels.

• To better support panel chairs, the Standards and Review Office gathered relevant information

already published by IES and others in various other formats and organized it into a

supplement to the reviewers’ handbook specifically for chairs. It includes IES expectations and

procedures and general tips on meeting management, using input from experienced chairs.

The panel chairs for the upcoming review cycle have been asked for input and feedback on

the usefulness of the supplement.

• To assist panel reviewers, the reviewers’ handbook has been revised to incorporate more

general information (borrowed from IES procedural documents) about the types of panels and

reviewers.

• To assess whether the order in which applications are reviewed has a significant effect on

review outcomes, IES has randomly assigned the order of review for panels reviewing the main

competitions for NCER and NCSER grants.


• To better inform applicants about progress, IES is implementing the Applicant Notification

System, which allows applicants to log in online and see the status of the review process and

funding. The system is modeled on those of NSF and NIH.

Dr. Ricciuti welcomed suggestions from Board members regarding potential new reviewers. Dr.

Ricciuti will develop a short list of areas of expertise in which there is a particular need for reviewers

(e.g., ELLs, ELLs with disabilities) and send it to the Board members. She will also provide the online

link to the list of funded projects and the list of panel members.

Discussion

Numerous Board members thanked Dr. Ricciuti for her responsiveness. She clarified that the

names of reviewers for some competitions were combined into a single list because some panels

are so small, so specialized, or meet so infrequently (e.g., one-time competitions for R&D centers)

that it would be too easy for an applicant to identify the reviewers on a specific panel and even

the reviewers who evaluated that applicant’s submission. Dr. McCardle noted that the new IES

reviewers’ handbook indicates that reviewers are welcome to conduct a post-review debriefing;

she suggested that IES stipulate in writing that debriefings should address the review process and

that a DFO should be in the room during the debriefing. No further discussion of individual

proposals should be permitted at debriefings. Dr. Ricciuti agreed, and indicated that only general

issues should be discussed during the debriefings.

Dr. Granger asked for Dr. Ricciuti’s assessment of the review procedures for reports that carry the

IES imprimatur and are published (and therefore reviewed) by IES. Dr. Ricciuti responded that the

report review process generally works well, although she has tried to institute improvements over

the years. IES is working more collaboratively with the centers than it had in the past. The review

process is modeled on that of academic journal peer review. Dr. Granger suggested collecting

feedback about the process from authors who have been through it. He also suggested

collecting such information from panel reviewers. Dr. Ricciuti said reviewers complete evaluation

forms to identify areas for IES improvement, and the forms have recently been revised to include

specific questions about training, review materials, etc. She said IES would consider taking a

systematic approach to evaluating the report review process.


Dr. Easton noted that he and Dr. Ricciuti have spent a lot of time over the past 2 years on a

handbook that sets expectations for centers submitting reports and for reviewers; that draft

handbook is on Dr. Easton’s desk for review. He suggested putting the issue of the report review

process on a future Board agenda for discussion.

Dr. Ricciuti said the staff of the Standards and Review Office has been working to ensure that

applicants get constructive feedback on their submissions. Ms. Pelaez has assisted in managing

the logistics contract and online system to improve how well IES tracks resubmissions. It is fairly

easy to distinguish a first-time submission from a resubmission, said Dr. Ricciuti, but the system

must be revised to identify a second resubmission, third resubmission, etc. She emphasized that

IES does not limit the number of resubmissions, but it is watching the NIH process (which does limit

resubmissions) closely and considering the approach.

Communication and Dissemination Ruth Curran Neild, Ph.D., NCEE Commissioner

Dr. Neild hoped to gather ideas and reactions from the Board about the dissemination strategy

for the What Works Clearinghouse. To better assess the question, “Does IES have a good

dissemination strategy for the What Works Clearinghouse?” Dr. Neild broke the question down into

the following components:

• What are the intended outcomes of the strategy? What are the proximal and distal outcomes

intended?

• Who is the target audience(s)? Who are the sub-audiences of the target audience?

• What information does the audience need?

How does the audience usually get information (i.e., through what channels)?

• What gives the information credibility?

• How can information be packaged so that the audience can use it?

• What do we want the audience to do with the information?

Dr. Neild hoped these questions would spur a more sophisticated analysis of what constitutes

good dissemination. She proposed answers to some of the questions but said she welcomes help

from the Board in thinking through the questions.


The key audiences for the What Works Clearinghouse are education practitioners—people who

make decisions about what happens in schools (teachers, policymakers, etc.)—and education

researchers, who use the What Works Clearinghouse to see what others are doing in the field. Dr.

Neild believes that education practitioners are the main target; IES wants them to use the What

Works Clearinghouse findings to make decisions. In practice, however, the What Works

Clearinghouse has more robust strategies for reaching researchers than practitioners. Now that

the What Works Clearinghouse has gathered sufficient content and established its credibility, NCEE

should focus more on reaching practitioners.

Next, the What Works Clearinghouse must find additional ways to increase demand while also

increasing the supply of resources and products. IES has already made some improvements

toward this end:

• Made Quick Reviews quick. In the past, when a study was mentioned in the press, the What

Works Clearinghouse took 3–4 months to assess the quality of the study; now that process

takes 10–12 days. In one situation, a study was mentioned by a small media outlet, and the

What Works Clearinghouse was able to produce a Quick Review so rapidly that by the time

the study was covered by a major news outlet, the story was linked to the Quick Review.

• Initiated announcements about new products on Facebook.

• Revised product templates to emphasize important information and increase clarity.

• Shortened the online news flashes and focused on using clearer wording.

• Revised the home page to make it less text-heavy and less dense and improve navigability

and clarity. The site now includes a rotating header and more prominent links to the

companion site, Doing What Works.

Dr. Neild said the What Works Clearinghouse is seeking ways to get more out of its existing

products—for example, by developing more practice guides and using other media to leverage

them. At the June NBES meeting, Dr. Singer said the What Works Clearinghouse website was too

text heavy and could incorporate more video, so Dr. Neild and her colleagues are working with

the website contractor to examine the feasibility of producing video “teasers,” a Podcast, and

other abridged products to spark interest. In addition, IES could develop more products similar to

the practice guides but shorter, such as guides that focus on a single practice or policy. Even the

current narrowly-focused guides have about five recommendations each, and many educators


request more specifics. Shorter, more focused guides are less costly and easier to produce than

full practice guides, said Dr. Neild.

The What Works Clearinghouse search function is useful but time-consuming and requires some

knowledge to manage successfully. Dr. Neild said the results of the most common searches

could be compiled, so that users can quickly obtain a simple, ready-made summary that

responds to common questions.

Dr. Neild said most efforts have been focused on the demand side, but the What Works

Clearinghouse could provide guidance to practitioners about how to use findings in the form of

DVDs or print materials, for example. Products could address broad issues, such as how to

interpret null findings.

Effective dissemination involves networks that convey the message. The What Works

Clearinghouse and the RELs have worked together for years, and the RELs are particularly good at

dissemination. The RELs could help the What Works Clearinghouse better connect with the

networks that educators find credible, gather feedback on products, and develop other products

that support the What Works Clearinghouse’s products.

Finally, Dr. Neild asked for input on framing the What Works Clearinghouse agenda. She noted

that she can track numbers of downloads and Web hits, but wondered what questions should be

asked to better inform the What Works Clearinghouse products and strategies.

Observations on the What Works Clearinghouse Steve Fleischman, Chief Executive Officer, Education Northwest

Mr. Fleischman currently directs the 2012-2017 REL Northwest at Education Northwest and has

spent many years focused on the production and promotion of evidence for improving

education, including some time as the communication director for the What Works Clearinghouse

when it began. The What Works Clearinghouse is a work in progress, and he offered several

comments intended to support and improve it.


If the What Works Clearinghouse is intended to be a decision-making tool and not only a

repository of knowledge, there is a long way to go. Mr. Fleischman contended that education

practitioners are the primary audience for such a tool, while researchers are secondary.

Researchers produce the work that goes into the What Works Clearinghouse and “consume” the

information it provides, but they are not consumers of the prime purpose of the products—that is,

to make better decisions about challenges in education. Within the arena of education

practitioners, said Mr. Fleischman, teachers are not likely to be among the primary audience,

unless they are organized into decision-making bodies. He recommended narrowing the

definition of “target audience” to focus on decision-makers and looking more closely about who

they are, what they need, and how they can use the What Works Clearinghouse as a resource.

Instead of focusing on one-way dissemination of information, Mr. Fleischman suggested

considering how to communicate results and foster exchange of information. The dissemination

approach is not very engaging.

To increase demand for products, it may be necessary to change perceptions about the What

Works Clearinghouse “brand” through new tools, resources, or mechanisms to engage people

who know of it but still do not use it and to attract new users. More thought should be given to

increasing the supply of products that help users figure out what to do with research findings,

either by improving existing products (e.g., the practice guides) or developing new ones.

With the latest changes, the What Works Clearinghouse website is moving closer to the kind of

online resource that users want, said Mr. Fleischman, but he recommended thinking about how

we use other consumer reporting sites (e.g., Consumer Reports, Yelp, TripAdvisor, and CNET) to

make decisions. While it is difficult for government-sponsored entities to break out of entrenched

patterns for presenting information, Mr. Fleischman suggested the What Works Clearinghouse site

become even bolder.

Mr. Fleischman praised the new products proposed by Dr. Neild. He suggested looking at the

Institute for Healthcare Improvement (IHI) for additional ideas, because it presents information in

numerous formats and various media and also because it is focused on improving practices.


The Board should consider what level of experience or knowledge constitutes expertise and

whether IES would be comfortable with products that feature nonacademic experts who have

developed expertise as practitioners. In some cases, such individuals can enhance the

acceptance of products and recommendations.

Mr. Fleischman said the RELs can be an effective sales force to promote the What Works

Clearinghouse. The RELs bring to bear their experience understanding context and relationships.

For example, when REL Northwest at Education Northwest held a webinar for the Bureau of Indian

Education Schools about preventing dropouts, it made sure to involve a key person in that

community who was recognized in the field—but not necessarily a research expert.

Bear in mind that the What Works Clearinghouse has competition, said Mr. Fleischman. Educators

typically get their research information from vendors who supply professional development along

with direct support. Another source is the Association for Supervision and Curriculum Development

ASCD Education Leadership series, which provides research summaries. IES should learn from the

competition. Finally, said Mr. Fleischman, more work is still needed to ensure that all the relevant

offices and programs across ED—not just in IES—are working together.

Discussion

Dr. McLeod appreciated that IES is attempting to improve communication, because there has

traditionally been a disconnect between Federal and district efforts. She emphasized the

importance of reaching policymakers not just at the national level but at the State and school

district levels as well, particularly because vendors are already playing a significant role in

providing them information. Dr. McLeod said it would be interesting for the Board to hear from a

panel of users—whom she believes are typically central office staff, not teachers or principals—

about their perceptions of the What Works Clearinghouse. She pointed out that parents need

information as well, and the What Works Clearinghouse should consider how to reach them.

Finally, she said she was excited about the prospect of engaging the RELs as a way to reach

districts and States.


Dr. Underwood emphasized the need to better understand where decision-makers currently get

their information so that we do not make false assumptions. Further, the What Works

Clearinghouse should consider how it gets feedback and in particular whether users get some sort

of reward for providing feedback. It may be appropriate to publicize current tracking measures,

which could spur others to use the What Works Clearinghouse more.

Because the research agenda for the What Works Clearinghouse is potentially vast, said Dr.

Yoshikawa, consideration should be given to the range of interventions for which the What Works

Clearinghouse information could be used. The first step may be to focus on the types of

interventions that you most want to know about. Because some of the interventions are targeted

at a network level, testing and evaluating those interventions could set up a research agenda. Dr.

Yoshikawa also recommended a State- or district-level saturation strategy. For continuous

improvement efforts, Dr. Yoshikawa suggested a strategy broader than assessing a single piece of

information; for example, the What Works Clearinghouse could evaluate the effects of sustained

engagement efforts around a broad question over the course of a year. Finally, he called for

more clarity around the measures for evaluation, particularly whether they assess immediate or

long-term results.

Dr. Hedges suggested that IES take a closer look at how the intended audience uses and

interprets the What Works Clearinghouse products. Dr. Singer said publishers routinely require

authors to identify the intended audience, the branding needed to reach the intended

audience, the means of dissemination, and the ongoing promotion of products before

producing the product, while it appears that the What Works Clearinghouse was built without

having detailed enough information about the intended audience. She pointed out that IES

needs an overall media strategy, including one for the What Works Clearinghouse.

Asked to comment on the What Works Clearinghouse’s media presence, Ms. Sparks of Education

Week, confirmed that it is not easy to get information about what products have been updated

or revised. She asked whether districts who are interested in becoming reviewers or acquiring

training on single-case studies could do so through the What Works Clearinghouse. Dr. Neild

responded that connecting individuals from districts and States to reviewer training opportunities

was not a consideration until the What Works Clearinghouse received some requests about it.


Dr. Singer suggested that the What Works Clearinghouse reach out to specialized media outlets.

She added that it should take advantage of social media outlets—but cautioned that if it is not

able to keep its social media presence up to date, it should shut down that aspect, because an

out-of-date profile is worse than none. Dr. Singer offered a number of specific suggestions for

improving the reach of the What Works Clearinghouse that she hoped would spur IES to

brainstorm about other potential mechanisms:

• Partner with organizations that reach the target audiences, such as the Education Writers

Association (which sponsors seminars to help its members better understand research).

• Look for innovative ways to communicate, such as briefings.

• Identify a “public face”—a champion or spokesperson—for the What Works Clearinghouse.

• Build a community around the What Works Clearinghouse, perhaps with assistance from the

RELs.

• Promote the journal club approach used among medical practitioners (a self-guided, group

discussion of published research).

• Consider providing grants to local school systems to promote the use of the What Works

Clearinghouse.

Dr. Long echoed the importance of enhancing two-way communication, not just dissemination.

Researchers in general need to know how to make their research more effective—that is, how to

make it useful to decision-makers. Dr. Long suggested that IES engage with some of the largest

school districts in the country and ask for feedback about how those districts—from the

policymakers at the top to the teachers in the classroom—use the What Works Clearinghouse

products. The results could boost credibility, which could then be used to promote the What Works

Clearinghouse further. Regarding competitors, such as vendors, Dr. Long stressed the importance

of communicating what does not work so that practitioners and decision-makers do not waste

time and resources on ineffective products.

Dr. Granger suggested that the What Works Clearinghouse would benefit from better information

on its users, the decisions those users need to make in their work, the tools they need to improve

their decisions, and their current sources of information about research. He suggested that

research on using evidence to make decisions should be part of the overall IES mission—not just a

function of the What Works Clearinghouse.


Also, Dr. Granger noted, identifying what works may not be enough. Other considerations, such as

the cost and ease of implementation of interventions are also important. Potential users want to

know, for example, whether effectiveness was determined by research conducted in settings like

theirs. Dr. Granger pointed out that asking researchers to include such implementation, cost, and

contextual information in evaluation reports might be an important way to improve the relevance

of the Clearinghouse’s information.

Discussion turned to the strengths and weaknesses of the practice guides. Dr. Neild said she

particularly liked Teaching Elementary School Students To Be Effective Writers because the

recommendations are clear, focused, and useful. The guide also give specific examples that

teachers will understand and that may whet their appetites to learn more. Dr. Neild said there is

consideration of developing products that elaborate on the examples in the guides. Dr. Ball said

she preferred the tone of the practice guide Developing Effective Fractions Instruction for

Kindergarten Through 8th Grade, because it piques readers’ interests with ideas that fly in the

face of typical practice. The writing guide, in contrast, lacks excitement, and the quality of

evidence to support it is not high. She suggested that further discussion may be needed on the

relationship of research to practice, particularly if the recommendations of a practice guide are

not limited to the highest levels of evidence.

Dr. Loeb supported the idea that the What Works Clearinghouse should provide more guidance

on how to interpret the evidence for effective interventions, recognizing that the research is not

always clear. Dr. Neild responded that, in the spirit of transparency, there is consideration of

posting more information online about how review panels reach their conclusions. She is

interested in pursuing mechanisms that describe the approaches a user can take to apply the

evidence to decision-making.

Dr. Loeb noted that the What Works Clearinghouse focuses on internal and not external validity.

She suggested providing more details about the outcome measures and other methods of

evaluation used to assess effectiveness. Dr. Neild agreed with the assessment and requested

more input from Board members on how to gauge external validity and other important issues like

impact on subgroups.


Dr. Chard noted that some States and districts have customized the research into approaches

they can use to put theory into action, which IES should explore. He added that the What Works

Clearinghouse should not overlook the possibility of partnering with the vendors of educational

materials, who provide billions of dollars’ worth of educational tools every year. Also, a better

understanding is needed of what is happening in teacher education that drives users to the What

Works Clearinghouse for information.

Dr. Gamoran said that when the What Works Clearinghouse began, there was little valid research

available. Now, research is trying to catch up with the need for evidence that is useful to identify

what works, for whom, and in what circumstances. Efforts are moving forward with iterative

research and implementation research as well as in large-scale effectiveness studies, which can

assess the heterogeneity of treatment effects. Dr. Gamoran suggested that the What Works

Clearinghouse should link more closely with the Doing What Works website. Clearly, there is a need

to link advice about implementation with information about programs deemed effective.

Toward that end, Dr. Singer suggested considering the user’s perspective, for example, by asking,

“What would I have to change to implement these recommendations?” The answer may vary

depending on the user’s position; thus, it may be appropriate to consider offering the same

materials in different formats for different audiences. Dr. Singer pointed out that the practice

guide Improving Mathematical Problem Solving in Grades 4 Through 8 appears to be written by

a researcher interested in theorems and proofs; most users are more likely to be interested in

learning about alternative ways of implementing the approaches. Much of the supporting

information could be moved to an appendix, she said. In addition, the practice guides should be

written in an engaging, active voice and include, for example, links to short videos showing a

teacher applying the intervention. Dr. Singer suggested using focus groups, which could provide

valuable insight that would improve the What Works Clearinghouse materials.

Dr. Gutierrez echoed Dr. Singer’s concept, suggesting that products answer the question, “What

do I need to carry out these recommendations?” Implementing the recommended practices

may require expertise that the user lacks. Insights from practitioners who have implemented the

practices is needed to ensure that the products are useful.


Dr. Long suggested the Board discuss dissemination in more depth at a future meeting. Dr. Neild

thanked the Board members for their insights and hoped to return to the Board in the future with

improvements to the What Works Clearinghouse as a result of the discussion.

Lunch

Participants adjourned for lunch at noon, and the meeting resumed at 1 p.m.

Design-Based Implementation Research (DBIR) Barbara Means, Ph.D., Director, Center for Technology in Learning, SRI International

Thomas Smith, Ph.D., Director and Principal Investigator, National Center on Scaling Up Effective

Schools and Associate Professor of Public Policy and Education, Department of Leadership, Policy,

and Organizations, Vanderbilt University

Dr. Means pointed out two very different strategies to conducting research to improve education.

One method is to identify and disseminate interventions that work—the original intent of the What

Works Clearinghouse. Another is DBIR, which embeds research into the process of policymaking

and practice. DBIR is not a specific methodology but rather a perspective that draws on

methodologies such as implementation, design, and improvement research. Dr. Means

described the underlying principles of DBIR:

• Focus on a persistent problem of practice from the point of view of multiple stakeholders. The

focus of research is not defined solely by the investigators or the institution; rather, the goal of

the research is negotiated among those involved.

• Commit to an iterative, collaborative design. Collaboration aims to combine insights from

prior research and practice to create a design that is meaningful and useful to all. Outcomes

of collaboration are more likely to be implemented.

• Pay attention to both classroom learning and improvement.

• Develop a systemic capacity for sustaining improvement. DBIR seeks to increase the capacity

for systems to evaluate, research, and implement interventions across various real-world

settings.


Dr. Means explained that DBIR combines three types of research goals: building knowledge,

designing and developing programs, and place-based improvement. She gave examples of

projects that incorporate most, but not all, of the principles of DBIR. The Center for Learning

Technologies in Urban Schools (LeTUS) funded by the NSF, for example, brought urban school

districts together with university researchers around a mutually-agreed-on goal and an

improvement target. The Project-Based Inquiry Science curriculum began as a randomized

controlled trial but evolved through collaboration with the schools that revealed a need to support

teachers in learning a science curriculum that did not synchronize neatly with the State standards.

In addition to the importance of negotiating mutual goals, collaborating, and promoting co-

design with opportunities for refinement, DBIR emphasizes that developing knowledge about

implementation, such as cost and ease of implementation, is as vital as evidence of

effectiveness. Some knowledge can be gained quickly, through experience or focus groups, such

as how long it takes to cover a topic in the classroom. Dr. Means emphasized that DBIR requires

long-term partnerships.

DBIR faces funding challenges, Dr. Means noted. To support a true partnership, the funding

structure cannot put one stakeholder in a dominant position over others. It should encourage

negotiations among partners with equal say as well as interdependence. However, in such

situations, accountability becomes tricky. DBIR requires an investment in creating a common

language among stakeholders to better determine research and improvement goals, but

because the effort evolves over time, it is difficult to describe the outcome to a potential funder.

Successful DBIR requires long-term funding to build a sustainable partnership. Finally, Dr. Means

noted that it is difficult to build a body of knowledge for the field when the research efforts are all

place-based.

Dr. Means noted that DBIR does not replace other kinds of research. Over time, she said, it may

be possible to conduct meta-analyses of DBIR projects to determine how the approach as a

whole improves practice and increases capacity. Dr. Means concluded that we should consider

whether the quality of education research can be evaluated, at least in part, by improvements in

outcomes.


Dr. Smith said the National Center on Scaling Up Effective Schools, funded by IES, began its efforts

focusing on capacity-building. Researchers aimed to identify achievement gaps, then work with

school districts to build the capacity to identify effective practices (including existing efforts),

adapt them to their own schools, and implement interventions in other schools to close the gaps.

The Center’s efforts to work within school districts brought to light the challenges of comprehensive

school reform that districts face: lack of teacher buy-in, mismatch of the intervention and the

context, and the lack of alignment between the goals of researchers and the needs of schools.

Dr. Smith said the first step was to identify schools that were beating the odds among low-

performing subgroups. The Center partnered with districts in Florida and Texas because those

States had the strongest data systems. The initial goal was to learn from schools at the higher end

of the distribution for value-added gains in student performance and transfer their practices to

schools at the lower end. Having completed the intensive process of in-depth, on-site review of

schools and combining the findings with the results of survey data, the Center is beginning Year 3

of its effort, which looks like DBIR.

Currently, efforts are underway to facilitate partnerships between districts and the design team.

The design team will share what it has learned from the first 2 years and what evidence suggests is

effective, the schools will discuss their needs, and the partners will work together to design

appropriate implementation strategies for interventions. Dr. Smith said there was no clear

consensus on whether to focus on a single intervention or multiple interventions at the outset, and

so the decision has been postponed until the initial design is proposed. However, there will be

pilot implementations, and the results will feed into future iterations, eventually resulting in full

implementation that has been tailored to meet the district’s unique circumstances. Eventually, the

Center will withdraw and, ideally, the district will have the capacity to continue the approach.

Collaboration is at the core of the approach, and each partner has equal weight in the

partnership. The Center focuses on the essential components of effective schools, and from this

perspective, seeks to understand not just what successful schools are doing but how they are

doing it. Dr. Smith pointed out that the differences in performance across schools was not related

to, for example, the lack of a rigorous, aligned curriculum (because all the districts had them and

teachers followed them closely) but rather the systemic approach that individual schools used to

address the issue.


To illustrate, Dr. Smith said that those schools within the Broward County Public Schools system in

Florida that achieved high performance focused on personalization of social and academic

learning by creating systematic structures that promote strong relationships between students and

adults, an atmosphere of caring and trust, instructional activities that draw on students’

experience and interest, and strong links with parents. In the design phase, instead of generalizing

the findings from the literature, the partners sought examples of how the high-performing schools

implemented these systems—as well as what the low-performing schools failed to do. Dr. Smith

pointed out that school districts are more likely to buy in to the approach when they see the

success of other schools within the same district.

The Center has two agendas: building team capacity for innovation, design, transfer,

implementation, and scale-up of interventions, and deepening understanding of the practices

that distinguish effective schools. In Phase 3 of the Center’s efforts, the teams will assess the

implementation of the designs, build capacity to implement the design innovations in more

schools, monitor the outcomes, and develop tools to help others transfer effective practices

across their districts. Dr. Smith emphasized that once a design is successful, the next step is to

implement it in other contexts to identify the areas of overlap between effective practices and

various contexts.

Discussion

Dr. Bryk said the Carnegie Foundation for the Advancement of Teaching supports network

improvement communities, which are similar to DBIR in some ways but focus on the capacity to

accelerate learning from practices to improve interventions rapidly. He noted that the

Foundation’s efforts consciously use different language; for example, cognitive scaling focuses on

implementing an intervention with fidelity, but Dr. Bryk says the real goal is efficacy with reliability at

scale.

So much inquiry looks at the product, said Dr. Bryk, but the knowledge about how to implement

interventions is lacking, and building capacity means learning how to improve in practice. Good

starting points are needed, combined with an understanding of how to integrate an intervention

practically within an organizational context at scale—sometimes called “adaptive integration.”


A key principle of adaptive integration, said Dr. Bryk, is that it is problem-centered and demands

accountability for making progress toward a measurable target. Demonstrating the worth of an

intervention is important, but the intervention also has to move the target outcomes and move

them reliably, he said. He gave the analogy of a hospital system that recognized the need to

address mortality from sepsis by implementing practices that bring down sepsis rates in all of its

hospital facilities. The system had to develop the how-to knowledge that extends beyond place,

and that is where the Foundation’s efforts and those of DBIR proponents and the National Center

on Scaling Up Effective Schools can make an important contribution to the field, Dr. Bryk

concluded.

Dr. Gutierrez expressed enthusiasm for the attention to capacity-building, the use of adaptive

practices, and a problem-centered focus. She noted that data collected from observation can

be fed back to practitioners so they can see what they are doing. However, she wondered

whether we have to train researchers differently to consider how data can be used to make

decisions about changing practices. She also wondered why there is not yet strong evidence

about the direct impacts of interventions on teaching and learning.

Dr. Means said that DBIR requires both the knowledge acquired from academic training and a

willingness to make improvement the top priority. Dr. Smith added that college and graduate

students are learning the new approach by participating in collaborative teams in the field. He

added that the initial design of the Center’s work provided feedback to districts about how to

close the gap between their schools’ real-world practices and the theory behind the practices.

However, that approach has yielded little improvement in instruction practices in class, so the

Center is shifting gears to emphasize co-design and identify changes that could be made.

Dr. Gamoran appreciated the focus on implementation and collaboration with people on the

ground, an essential component of developing programs that are actually used. However,

evaluation of the impact of interventions remains an important component. The conclusion of Dr.

Smith’s project will identify some important hypotheses about what works, but they should be seen

as the beginning of a much longer study of the impact on student performance.


Dr. Ball questioned how DBIR is distinct from practice-based research. She asked how the NSF sees

the push toward DBIR in relation to its own investments in the Statewide Systemic Initiatives and

other programs. She called for a deeper look at what it means to “contribute” to practice. Dr.

Means felt that a meaningful contribution could be one that led people to implement a change

that resulted in improvement. She also said that the principles of DBIR need to be distinguished so

that funding partners will better understand the ideas, which Dr. Ferrini-Mundy confirmed.

Dr. Means said the NSF laid the groundwork for DBIR with programs that brought researchers and

practitioners together in a systemic context. Dr. Ferrini-Mundy agreed, noting that NSF supports

several partnerships to foster cross-boundary collaborations that benefit from and inform this kind

of research.

Dr. Yoshikawa said he is an enthusiastic user of this model, most recently in a teacher

development program for preschool teachers in Chile. He said his program follows the IHI model

(although not intentionally) and involves getting local buy-in and ownership and day-to-day

tracking of progress on measures determined by the practitioners, among other components.

Dr. Loeb noted that IES research is geared toward creating a body of generalized knowledge

about what works and how to implement those interventions, whereas the work of DBIR

proponents and others seems to mirror that of a consultant whose job is to help systems develop

processes to make better decisions and achieve better outcomes. Dr. Means said she failed to

see the distinction, because one part of DBIR is using the lessons learned in the field to build

knowledge about improving implementation at scale. She added that DBIR attempts to give all

the partners an equal stake in setting the agenda, which distinguishes it from the work of

consultants. Dr. Means said the biggest difference between DBIR and the Carnegie Foundation’s

adaptive integration approach is that the latter puts rapid-cycle analysis at the center of its efforts.

Dr. Smith added that the aim of the Center’s efforts is to build the capacity of a district to identify,

implement, and evaluate interventions on its own. In addition, it seeks to develop tools and

knowledge on how to measure effectiveness that will be widely shared with the field (not kept in

house, as a consultant would).


Dr. Singer raised concerns about the generalizability of findings from this novel approach to

research, such as the small sample sizes, the (essentially) retrospective case-control design, and

potential selection bias of schools that choose to adopt innovation. Dr. Smith agreed that findings

may not be generalizable, but he hoped these efforts would foster better understanding of the

process that brings people together to study the evidence, design strategies collaboratively,

gather feedback, and redesign efforts.

Dr. Bryk noted that the Carnegie Foundation’s approach reflects a very different discipline than

academic research. The first principle is learning quickly to improve. It involves disciplined inquiry,

a common language and common measures, with frequent tracking over time that is practical,

feasible, and provides concrete evidence about what works. Unlike academic research, there is

less focus on, for example, internal consistency and more attention to the practical aspects of an

intervention.

Dr. Bryk added that the IHI promotes two models: the breakthrough collaborative, in which an

intervention seems to be effective so efforts are made to implement it in more settings to

determine if the effects can be reproduced reliably, and innovative collaboration, in which some

hypotheses are put into practice on a small scale and those that show promise move into

different contexts to determine effectiveness and reliability.

Finally, Dr. Bryk reiterated the point made by Drs. Mean and Smith that the goal of DBIR and

related practices is capacity-building. Ideally, these efforts will change the culture of work so that

everybody involved takes ownership of the improvement goals and everybody has a say in

achieving those goals.

Dr. Granger raised the concern that the claims of effectiveness that result from this approach

may not be based on sufficiently rigorous information to be trustworthy. He wondered whether

experiments are needed, or whether each district’s approach should be seen as single-case

study that contributes to the larger body of evidence through replication. On the one hand, said

Dr. Granger, the effects of such limited studies would have to be large in order to attribute them

to the continuous improvement intervention; on the other hand, people like Dr. Bryk seem to be

saying that replication across sites is a reasonable standard for identifying effects and they may

be right.


Dr. McCardle pointed out that the National Center on Scaling Up Effective Schools received a 5-

year grant of $13 million. The NIH provides grants for community-based participatory research in

which the co-principal investigators in the community receive some of the funding. Dr. Means

reiterated the need for a funding structure that supports equal partnership.

A New IES Research Program John Q. Easton, Ph.D., IES Director

Dr. Easton said that IES posted a proposal online for comment about a new IES research topic in

continuous improvement research in education. It is strongly influenced by the philosophies

behind DBIR, the IHI, and The Checklist Manifesto, as well as Dr. Easton’s own background. He said

knowing what works plays an important role in school improvement, but it is not enough. We also

need to build the capacity to implement what works through measurement and adaptation. Dr.

Easton posed three questions to the Board to elucidate their perceptions of the role of IES within

the explosion of activity around implementation.

1. How should IES refine the conceptual framework for the proposed topic so that potential grant

applicants will have a clear understanding of the purpose of the topic?

2. Are the three continuous improvement strategies described as examples (creating a safe,

orderly, and supportive learning climate for students from preschool through high school;

improving students’ transition to high school; and increasing access to college and

postsecondary training) sufficient?

3. What standards of evidence should be built into the topic? What types of studies would

provide adequate evidence of promise, efficacy, or effectiveness?

Discussion

Dr. Long asked for more elaboration on the perceived need to add such a resource topic. Dr.

Easton responded that a funding partnership may be the best way to support the development of

new models of continuous improvement that address systemic problems in practice and to

produce proof that these models can solve problems.


In response to Dr. Chard, Dr. Easton said the proposal included the three continuous

improvement strategies as a means of narrowing the field and establishing some broad criteria

for evaluating applications. He added that the three strategies address pressing, almost

intractable issues.

Dr. Chard wondered whether the proposal seeks to recast the IES goal structure; Dr. Easton

responded that IES has been thinking about the design and development side of the goal

structure and at what point the field needs sufficient evidence of impact.

Dr. Gamoran suggested that this proposal could be the first of IES’s programs to implement a

process for renewed or additional funding on the basis of promising initial results. Dr. McCardle

said NICHD grantees receive 5-year awards that are eligible for renewal, but the proposal review

process is the same (e.g., it includes peer review). In such cases, previous performance is

considered. Dr. Gamoran saw the proposal as an opportunity to go beyond encouraging

researchers to move their research from Goal 2 into Goal 3 by making that an explicit

expectation.

Dr. Loeb said the duration of the grant should be considered carefully. The collaborations needed

to pursue this research take time to build and involve some risk; in some cases, the topic would

only be useful for entities that already have collaborations in place.

To narrow the range of focus of potential applications, Dr. Yoshikawa suggested the proposal

provide some examples of specific research questions or guidelines appropriate for building a

new field of research—for example, the researchers could look at the relationship of short-term

and long-term changes and carefully define the measures used to assess such changes. The

proposal should ask applicants to relate their research to the current IES goal structure in an

explicit way. Dr. Hedges said that it may be necessary for either the applicant or IES to define the

measures used to demonstrate improvement.

Dr. Singer felt the proposal does not clearly identify what IES seeks to encourage. If it aims to

affect what happens in schools, it seems too narrow, and many researchers would decline to

apply. It may be that the proposal seeks to foster the growth of DBIR and similar approaches or

that it aims to encourage more collaboration. Dr. Easton clarified that IES already established


another program to fund collaborations; he thought the proposal should foster the growth of DIBR

and related approaches. In that case, Dr. Ball recommended, the IES should look to NSF for

guidance. She added that it is not clear how DBIR builds a broad knowledge base.

Dr. Ball also said the three strategies identified are problematic, because they compartmentalize

research by grade bands. She said a lot of evidence shows the relationship between elementary

school and postsecondary performance, but most proposals default to preschool, secondary,

and postsecondary research. The proposal lacks discussion about improving the workforce and

teacher quality, she noted.

Dr. McLeod said that districts are grappling with the implementation of the Common Core

Standards, which could be a topic for research within this proposal. She asked what the

collaborations envisioned by this proposal would look like. Dr. Easton said many States and districts

already have research partnerships in place, some of which receive IES funds for evaluation, but

he would like them to focus more on problem-solving.

Dr. Granger suggested that thinking through how to conduct and learn from such DBIR efforts in

the next 3 years was a worthy goal.

Dr. Long felt IES should develop some guidelines, such as the definition of an existing

collaboration. Dr. Ferrini-Mundy said the NSF has been investing for a long time in partnerships.

She said there seems to be a new push for real-time evidence and continuous improvement that

could be introduced as a strategic approach in the areas identified by IES in the proposal. It may

be necessary to support capacity-building to help schools learn how to implement this approach.

Dr. Ferrini-Mundy said the proposed dollar amount seems low compared with previous

partnerships, especially considering that the proposal calls for a large, systemic approach.

Dr. Gamoran noted that many partnerships have been established over the past 10 years. The IES

proposal offers a model to guide those partnerships on how to proceed: identify a goal and

engage one of the continuous improvement strategies outlined in the proposal to address that

goal.


Dr. Loeb said that within the continuous improvement approach, two components seem to stand

out: developing real-time evidence and building a framework for reflective practice at the district

level. Collaborative partnerships can offer opportunities to experiment and ask questions in a low-

cost, relatively easy manner, and such efforts often produce better, richer findings, said Dr. Loeb.

However, she noted, the proposal initially seems to focus first on building a reflective practice but

then shifts away from that concept. Dr. Easton noted that a participant in a project of the National

Center on Scaling Up Effective Schools said that by working within a collaborative partnership, he

and his colleagues built up a resistance to the pressure to buy the next so-called silver bullet,

which seems to echo the benefits of reflective practice.

Dr. Bryk pointed out that to accelerate learning, a network of stakeholders should be aligned

around a specific set of problems. Selecting the target problems, however, raises tensions and

can lead to political dilemmas. These new approaches require the participants to learn as they

go; as such, they do not have clear strategic goals and timelines. In real life, said Dr. Bryk, you

want to be thoughtful about your first steps and have an overall goal in mind, but you want your

process to evolve as you progress.

Dr. Long concluded the conversation by noting that additional comments should be sent to Dr.

Easton. Dr. Easton said that IES hopes to have a well-written, clear request for applications posted

by the end of February 2013.

Policy and Legislative Update Sue Betka, Deputy Director for Administration and Policy, IES

Ms. Betka said there is bipartisan support in Congress for reauthorization of the Education Sciences

Reform Act (ESRA) and for IES. Although the Elementary and Secondary Education Act is the

administration’s priority, it is more likely that the ESRA will move more quickly through Congress.

However, Ms. Betka said it is unlikely that any progress will be made during the lame duck session;

she hoped that legislation would move forward in the next Congress. The U.S. House of

Representatives’ Education and the Workforce Committee asked the GAO to study IES,

presumably to gather background for a review, said Ms. Betka.


Ms. Nora Boretti, who is part of the GAO team working on the House Committee’s request to

evaluate IES, attended the meeting and shared additional insight on the subject. The process is in

the early stages of discussion about timeframes and methods, but the goal is to provide a report

for the House Committee in the spring as the basis for a discussion about ESRA reauthorization.

Ms. Boretti said that GAO would like to interview Board members and IES center directors. She said

she would provide the Board with more detailed information at a later date. The research

questions proposed (but not finalized) for the report are the following:

• How does IES compare to selected research organizations (to be determined) on cost,

relevance, and other outcome measures?

• What are the strengths and weaknesses of the four component IES centers?

• How have IES governance structure and standards for quality research supported or hindered

the Agency’s ability to fulfill its mission?

• To what extent does IES coordinate its activities internally and with other relevant entities,

including ED’s Comprehensive Centers programs?

Ms. Betka explained that ED has received very little specific guidance about sequestration (i.e.,

the requirement to cut the Federal budget across the board if Congress does not pass the

Budget Control Act to reduce the deficit). The White House released a sequester report in late

September; it could not be released until the 2013 appropriations were determined. The 2013

appropriation, however, is a continuing resolution that expires on March 27, 2013; it allows ED and

almost all other agencies to operate at the FY 2012 level and includes a 0.6 percent increase.

Using historical spending rates as a guide, Ms. Betka estimated that ED would spend about 31

percent of its total appropriation between now and March 27, 2013.

The White House determined that the ED sequester proportion would be 8.2 percent on the basis

of a full year at the FY 2012 level. President Obama continues to encourage Congress to prevent

the sequester. Ms. Betka said that no one really believed the sequestration would happen, but if

there is no deficit reduction package by January 2, 2013, it will go into effect. She did not know

whether the sequestration would apply to a half-year appropriation. Ms. Betka pointed out that

the calculations are very complicated, and because there is still time to avoid sequestration,

there seems to be little interest in working out the details.


The FY 2012 budget for IES was about $594 million, so the sequestration amount (8.2 percent)

would be about $49 million. If sequestration comes to pass, it would be applied at the program,

project, and activity levels.

Discussion

Dr. Gamoran pointed out that the NSF’s National Science Board plays a bigger role than NBES

does in interacting with Congress and other constituencies in support of education research. Dr.

Long said that the role of NBES in advocating for research funding is unclear. Dr. Long, Ms. Betka,

and former NBES Executive Director Dr. Herk met with House and Senate staff members on August

1, 2012, and provided them with an annual report and briefing materials. Dr. Long said the staff

members seemed to have read the annual report and knew a lot about IES and the Board. At

that meeting, Dr. Long emphasized that the Board is pleased with IES, and she provided some

good examples demonstrating the importance of education research.

Dr. Long noted that in recent years, the Board has written memos to Congress about issues of

concern (e.g., suggesting that States that received waivers to No Child Left Behind rules be

required to build research into their approaches). At the meeting in August, the staff members

were very interested in the Board’s comments about ESRA. They seemed focused on coming up

with ideas about ESRA and also had questions about the RELs.

Dr. Granger said that when he chaired the Board, it was a new entity, so he could not do much

more than present resolutions passed by the Board to Congress. He said the Board has not taken

an activist role; rather, it has offered support and encouragement. The previous Board chair, Jon

Baron, was more active in communicating with Congress, but he was more familiar with Congress

through his line of work. Dr. Long said she has talked with Mr. Baron, who has helped her make

connections on the Hill. She added that the most recent NBES annual report has more information

than previous reports, which gave her more talking points.


Dr. Granger noted that if the relationship between IES and Congress is good and there is support,

it may be prudent to focus on reauthorization before addressing other issues, such as the amount

of money IES receives. In general, he believed, the Board should follow the lead of the IES staff

unless there are qualms about the information staff provides. Dr. Long agreed that focusing on

reauthorization should be the Board’s priority.

Dr. Granger pointed out that Congressional staff members do not get briefings about work in

progress, as the Board does. He suggested inviting staff members to attend certain substantive

sessions at Board meetings might be a way to create and sustain interest in IES’s work. Ms. Dixon of

the Office of Legislative Affairs said that her Office does meet with Congress—most recently about

the What Works Clearinghouse and the RELs. Ms. Betka said IES provides briefings to Congress and

its staff on request and also offers briefings whenever a major NAEP report is released. Dr. Gutierrez

supported the idea of regular briefings, for example, once a year, for Hill staff members; Dr.

Hedges said briefings should be coordinated with the release of new products. Dr. Underwood

suggested that staff members be invited to Board meetings. Dr. Long concluded that the

consensus of the group seemed to be that the Board should take a more proactive stance in its

effort to affect policy and support for education research.

Board Leadership for 2013: Roundtable Discussion

Dr. Long explained that the terms of several Board members—including hers and that of Vice

Chair Dr. Gutierrez—expire in November. Also, she and Dr. Gutierrez were elected to serve in their

current positions for 1 year, so the Board must select new leadership. The Board appointment

process was simplified by elimination of the requirement for Senate confirmation, but it can still

take months. Dr. Easton believes that the reappointment process for all five members whose

terms end in November will be completed before those terms expire.

Dr. Granger said that, historically, chairs and vice chairs served for 2 years. He said Drs. Long and

Gutierrez have done a great job, and it would be useful for the Board to maintain some continuity

of leadership, especially if it may be without an executive director for some period. He proposed

that the Board request that Drs. Long and Gutierrez continue to serve in their leadership roles for

another year; if a problem arises around the reappointment process, the Board can meet by

teleconference to determine an alternative solution.


Dr. Easton noted that two new Board candidates are in the vetting process. It is likely that the next

Board meeting will include the full complement of 15 members.

Drs. Long and Gutierrez left the room while the Board deliberated on the proposal to extend their

terms as chair and vice chair, respectively. Discussion centered around alternatives and the

process for holding an urgent meeting to elect new leadership if the reappointment process is not

completed in time.

The Board members voted unanimously to extend the terms of the current chair and vice chair for

another year. Drs. Long and Gutierrez expressed their gratitude to the Board.

Closing Remarks and Adjournment John Q. Easton, Ph.D., IES Director, and Bridget Terry Long, Ph.D., NBES Chair

Dr. Long concluded that she was pleased with the meeting and with the way the IES staff is taking

the Board’s comments to heart. She hoped the Board would continue to build on the momentum

it has created. Dr. Easton thanked the Board members for an exciting and worthwhile discussion.

He reiterated his request that Board members send him additional comments on the new IES

research topic. Dr. Long thanked Ms. Pelaez and Dr. Herk for their hard work. She also thanked the

staff of AFYA, Inc., the meeting contractor that handled the logistics of this meeting. Dr. Long

adjourned the meeting at 4:02 p.m.

October 5, 2012 · National Center for Education Evaluation and Regional Assistance . Ruth Curran Neild, Ph.D., NCEE Commissioner . In September, NCEE released the State and District

Documents