Informing Reform: Changing The Relationships Between Research,
Policy and Practice in Context
Some Observations from the Work of the Consortium on Chicago School Research
Anthony S. BrykFounding Director
Reframing the relationship between . . .
Research
Practice Policy
A distinct mission —informing reform—anchored in a set of core values about the role of applied inquiryin social problem solving
Value Perspectives
• The relationship of social science to social problem solving in a democratic society is best viewed as an educational one. . . .In the broadest sense, we aim to educate the public about the improvement of public education.
Value Perspectives
• We eschew the role of “technical advisor to the prince.” We embrace a more democratic vision of policy making as a competition of ideas, where our role is to inform the competition with the best possible evidence and ways of thinking about that evidence.
Value Perspectives
• We embrace an ethical responsibility to seek out actively the diverse interests operative around school reform, and assure that a wide range of questions are continually addressed. The research agenda should not be beholden to any one group.
Value Perspectives
• A steadfast commitment to technical quality undergirds the conceptualization, design, and actual conduct of the research. While ideas from various stakeholders infuse the process; in the end we engage in disciplined inquiry.
Value Perspectives
• We are mindful that instrumental use of research findings is at best occasional. Distinctive contributions can also be made by reshaping the larger conceptualizations that focus subsequent policy debates—the importance of an empirically grounded, coherent theory of action.
Five Key Organizational Features
1. Extensive stakeholder consultation on research agenda, study design and interpretation of results
2. Distinctive model of funding and governance
3. Building analytic capacity for the long term
4. A commitment to intensive public informing
5. A distinctive researcher role within a network organizational structure
1. A Leadership Structure Designed For Consultation
Directors from multiple institutions• Responsible for day-to-day research
Steering Committee brings diverse perspectives• Establishes agenda of supported research
• Provides intellectual guidance
• Insures multiple perspectives
Constituent Advisory Board • Build a civic-elite constituency for the work
• Advance use of information
Ad hoc committees for particular studies
• Formally, Directors have extensive control over the work, but…
• Operate in an extraordinarily public and deliberately pluralistic environment
– role of Steering Committee in strategic resource allocation
– role of both Steering Committee and Constituent Advisory Board in shaping and reviewing work in progress
– all data made public
• General support from three major foundations (plus project specific support)
• Accountability
– on regular basis to Steering Committee and Constituent Advisory Board
– when requesting grant renewal
2. A Distinctive Model of Funding and Governance
3. Building Analytic Capacity for the Long Term
• Developing a base of researchers focused on reform in a place
• Enhancing the capacity of the system to understand its own operations
• Expanding the analytic resources for this work—a comprehensive longitudinal archive on students, schools and communities.
Multiple Methods and Measures
• Surveys
• Longitudinal case studies of schools
• Short-term case studies
• Analysis of administrative records
• Classroom observations
• Collection of student work
• Network analysis
• Longitudinal analysis of test scores
Data Source Level(s) Content Illinois State Report Card State, District, School Demographic characteristics,
achievement, truance, etc. Principal Survey (1992) School School reform, governance, program
participation, satisfaction Teacher Survey (1991) Teacher, School School reform, governance,
instructional practices, satisfaction Teacher Personnel Data (1993, 1997, 1998 forward)
Teacher Education, certification, tenure
Teacher Survey (1994) Teacher Instructional practices, teachers’ work lives
Student Survey (1994) Student Engagement, extracurricular activities, attendance, classroom experience
Principal, Teacher, and Student Surveys (1997, 1999, 2001.2003,2005)
Principals, Teachers, and Students
Five Essential Supports indicators and special reform topics.
Citywide Tests of Basic Skills (1987-2005)
Student Item responses and Rasch measures for reading and math
Administrative History Records Student Student mobility, high school course taking and failures, background characteristics
U.S. Census Data (1990 and 2000) Census Tract, Census Block Group
Mobility, language use, education levels, housing, etc.
Project on Human Development (1996, 2000 )
Community Institutional, social, and human resources.
Miscellaneous Other Community Data
Community (e.g. crime statistics, birth records, EPA sites, etc.)
A Comprehensive Data Archive
A Distinctive Research Process
• An interplay of theory, past research, and stakeholder concerns guide the questions we ask.
• The crafting of a research report typically begins as descriptive. As analyses proceed, a small set of focal ideas emerge.
• In directing attention to these ideas, we help to deepen public understanding, focus debate about policy concerns, and create opportunities to build common ground in a pluralist environment.
Media Output Audiences Stakeholder Input
Researchers, other urban school systems
Chicago public-at-large
Local school communities
Business leaders, Advocacy & Community
groups
Key Decision-makers
ConferencesReportsWeb site
Media
School reportsCable TV
Briefings Reports and summaries
Briefings Report summaries
Research Panels
Intermediary Groups (CALSC, Principals’ Assn..)
Constituent Advisory Board
Steering Committee
4. Intensive Public Informing
5. A Distinctive Researcher Role within a Network Organizational Structure
Sorting Through the Murky Terrain of Multiple Realities
• Individuals view schools through particular lens, according to interests, position and background
• Each brings insights that suggest questions and evidence to be gathered
• Extraordinary diversity of questions are possible
Researchers’ Distinctive Role in Pluralistic Environment
• Bring coherence and scientific discipline
• Keep research questions centered on fundamental issues that materially affect learning
• Reframes social and organizational working relationships
– Researchers have horizontal rather than vertical orientation
– Involvement of local researchers expands analytic capacity focused on schools
– Emphasis on mutual learning among researchers, policy makers and practitioners
• Stimulates broader processes of community education
• Advances conversation about school problems and how to proceed
Transforming the Researcher’s Role to Work within a Local Context
Challenges For Researchers
• Involvement of stakeholders complicates and lengthens the research process
• Exacerbates tension between timeliness and research quality
• Demands broader repertoire of skills among researchers
• Pressure on senior researchers to communicate frequently with multiple audiences
• Less flexibility for researchers in pursuing “their own research program”
Challenges Working with a School System
• Mutual trust• Collaboration with the research and evaluation
unit in the school system• Flexibility in adjusting to changes in leadership• Keeping top management informed: no surprises• Bring to light issues that need attention, avoiding
ad hominem criticism
• An inherent tension in “criticizing” the system as we seek to increase public understanding of what is required to expand opportunities
Necessary conditions: