Top Banner
DATA, DATA EVERYWHERE: PROGRESS, CHALLENGES, AND RECOMMENDATIONS FOR STATE DATA SYSTEMS CCSSO NCSA June 19, 2011 Presenters: Sunny Becker, Human Resources Research Organization (HumRRO) Shelby Dietz, Center on Education Policy (CEP) Jennifer Stegman, Oklahoma State Department of Education Kris Ellington, Florida Department of Education Discussant: Lauress Wise, HumRRO Moderator: Hilary Campbell, HumRRO
58

Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Apr 18, 2015

Download

Documents

The presentation by a number of states and agencies made at the CCSSO-sponsored National Conference on Student Assessment.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

DATA, DATA EVERYWHERE:

PROGRESS, CHALLENGES, AND

RECOMMENDATIONS

FOR STATE DATA SYSTEMS

CCSSO NCSA June 19, 2011

Presenters:

Sunny Becker, Human Resources Research Organization (HumRRO)

Shelby Dietz, Center on Education Policy (CEP)

Jennifer Stegman, Oklahoma State Department of Education

Kris Ellington, Florida Department of Education

Discussant: Lauress Wise, HumRRO

Moderator: Hilary Campbell, HumRRO

Page 2: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

2

Data, Data Everywhere: Progress, Challenges, and Recommendations for State Data Systems

Sunny Becker

Presented to:

CCSSO NCSA Conference

June 20, 2011

Page 3: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Demands on State Data Systems

3

Page 4: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Demands on State Data Systems

4

Page 5: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Demands on State Data Systems

5

Page 6: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Demands on State Data Systems

6

Page 7: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

7

CEP Investigation of Achievement Trends

• Center on Education Policy (CEP) conducted a

multi-year study of the impact of NCLB.

• The study used up-to-date state assessment data

for all 50 states and DC.

• All data were verified by states prior to inclusion.

Page 8: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Products of this Research

1. A rich repository of verified data

– Available on CEP’s website for free use by secondary

researchers

2. 50-51 individual state reports (“profiles”) each

year

3. Series of reports on national trends

– Took care to tally trends rather than merge dissimilar

data

8

Page 9: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

CEP Reports---A Sample

• Answering the Question That Matters Most: Has Student Achievement

Increased Since No Child Left Behind?

• State Test Score Trends Through 2007-08, Part 1: Is the Emphasis on

Proficiency Shortchanging Higher- and Lower-Achieving Students

• STSTT 2007-08, Part 2: Is There a Plateau Effect in Test Scores?

• STSTT 2007-08, Part 3: Are Achievement Gaps Closing and Is

Achievement Rising for All?

• STSTT 2007-08, Part 4: Has Progress Been Made in Raising Achievement

for Students with Disabilities?

• STSTT 2007-08, Part 5: Are There Differences in Achievement Between

Boys and Girls?

• STSTT 2007-08, Part 6: Has Progress Been Made in Raising Achievement

for English Language Learners?

• STSTT 2008-09, Part 1: Rising Scores on State Tests and NAEP

• STSTT 2008-09, Part 2: Slow and Uneven Progress in Narrowing Gaps

• STSTT 2008-09, Part 3: Student Achievement at 8th Grade

9

Page 10: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Data Collected

• Test data

– Percentage of students at each achievement level

• Overall and disaggregated by sex, race/ethnicity, income, Title I,

students with disabilities, ELL

• Grades 3-8 and high school

– Mean scale scores, standard deviations, and number of students

tested

• Overall and disaggregated by sex, race/ethnicity, income, Title I,

students with disabilities, ELL

• Grades 3-8 and high school

– Years 1998-99 through 2008-09

• Test characteristics

– What test is used at each grade level?

– What years are comparable?

• Not always a straightforward question

10

Page 11: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Tight Timeline to Report Current Results

Annual state assessment

results [summer-fall]

Collect, verify & analyze

data [winter]

Report trends [spring]

Next annual state assessment

results [summer-fall]

11

Page 12: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Data Collection Approach

12

HumRRO pulled information from state web sites

Materials were standardized by HumRRO

Materials sent to state

State revised as necessary

State returned verified materials

Page 13: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Considerations

• Overwhelmed and understaffed departments

• State staff changes

• Individual state staff members with:

– Data expertise

– Policy expertise

– “On the ground” expertise

• Uncommon analyses are “out of the blue” for state

staff

– We analyzed trends at basic-and-above as well as

proficient-and-above.

13

Page 14: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

“Out of the Blue” Questions

1. Relative scores on Reading for males and

females reversed in 2009 from previous years.

2. The number of tested 11th grade students varied

quite a bit from year to year.

3. NAEP Mapping Study reported trend lines for

each state. Some year spans disagreed with ours.

14

Page 15: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

So What Did We Learn?

Let’s look at a couple of indicators

• State capacity

• Data availability

15

Page 16: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Trends in State Capacity to Provide Data

First, some background….

• Our techniques improved over time. – Clarity regarding missing data and questions about trends

– Sample graphs to show basic trend lines and intended comparisons

– State familiarity with the requests --- in some cases

• We logged all communications with each state. – To keep track of where we stood

– To trigger a reminder to silent states

– To improve the process over time

16

Page 17: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Trends in State Capacity to Provide Data

After Year 1, the total # of

communications leveled out. Outliers persisted.

17

0

5

10

15

20

25

30

2006 2007 2008 2009

Year

Per-State Communications Trends

Average per State

Maximum per State

0

50

100

150

200

250

300

350

400

450

500

2006 2007 2008 2009

Year

Total Communications Logged

Page 18: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Trends in Data Availability Over Time

• Greater availability of disaggregated demographic

groups

– Students without disabilities

– Non-ELL students

– Title I and non-Title I

• Greater availability of metrics not required by NCLB

– mean scale scores and standard deviations

18

Page 19: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Recommended Data Collection Strategies

• Take time to research each state’s testing

programs and policies

• Plan for back-and-forth with state experts to

resolve anomalies

• Carefully construct “out of the blue” questions

19

Page 20: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Recommendations for State Data Repositories

• Documentation, documentation, documentation

• Clear, easy-to-locate instructions for data users

• List of caveats and policy changes

• Contact person/department for data questions

And we hope:

• PARCC and SBAC may yield more similar data

approaches (fingers crossed!)

20

Page 21: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

For Further Information

• Reports and data repository:

www.cep-dc.org

• Questions about the data collection and

standardization process:

Sunny Becker at [email protected]

21

Page 22: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

22

Data, Data Everywhere: Progress,

Challenges, and Recommendations for

State Data Systems

2011 National Conference on Student Assessment

June 19, 2011

Shelby Dietz, Research Associate

Center on Education Policy

Washington, D.C.

Page 23: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

23

CEP Research

• 10 years of data collection and reporting

on high school exit exams and other high

school assessments

• Data collected through state contacts,

state websites, and media reports

• Data verified through surveys, emails,

and phone calls

Page 24: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

24

Why State Data Systems are

Important

• State benefits for planning future

assessment policies or changes in current

policies (i.e. “What are other states

doing?”)

• Availability of data lessons the burden on

state contacts

Page 25: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

25

Progress

• In 2005, 8 of the 26 states with HSEE

could provide cumulative pass rates when

applicable

• In 2010, 12 of the same 26 states could

provide cumulative pass rates with 3

additional states able to provide them at a

later date

Page 26: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

26

Progress (cont.)

• Response timelines have improved

• Evidence of more advanced data systems

such as course-taking rates disaggregated

by student grade levels

Page 27: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

27

Challenges

• Reasons cited for not reporting cumulative pass

rates

-Data not collected

-No unique ID system in place

-Data not available

-Data not tracked on an annual basis

• Transparency of data

• Time constraints on department personnel and

turnover

Page 28: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

28

Recommendations

• Updated website/online data

• Include “last revision” date on website

• Enable department personnel to link to

additional online information

• Tell us what method of data collection

works best for your department

• When personnel are overloaded, connect

us with assistants who can verify data

Page 29: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

29

Shelby Dietz, Research Associate

Center on Education Policy

1001 Connecticut Avenue, NW, Suite 522

Washington D.C. 20036

P: 202.822.8065

F: 202.822.6008

Email: [email protected]

Web: www.cep-dc.org

Contact Information

Page 30: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Jennifer Stegman, Oklahoma State department of Education

Jennifer Stegman, Assistant State Superintendent Oklahoma State Department of Education

June 2011

Jennifer Stegman, Oklahoma State Department of Education 30

Page 31: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Jennifer Stegman, Oklahoma State Department of Education 31

Page 32: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

The Wave ◦ Created in 2004

◦ SIF compliant student information system

P20 Data Coordinating Council ◦ Created in 2009

◦ Oversight for the coordination of the creation of a unified, longitudinal student data system

Information Technology Consolidation and Coordination Act ◦ Created in 2011

Jennifer Stegman, Oklahoma State Department of Education 32

Page 33: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

◦ Maintain proper audit procedures to assure high standards of data quality and reliability.

◦ Ensure effective mechanisms to maintain confidentiality of student records and adherence to the Family Education Rights and Privacy Act (FERPA) requirements.

◦ Implement a student identifier number that can be used from pre-school through adults, connecting all three education agencies and the Oklahoma Employment Security Commission (OESC).

◦ Create linkages between and among data systems so data can be transferred across systems and among interested parties to address questions that cut across levels of the educational system and agencies.

Jennifer Stegman, Oklahoma State Department of Education 33

Page 34: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Further recommendations : ◦ Connect essential data elements relating to student-

level course work and course grades. ◦ Incorporate college-readiness measures into the

data system. ◦ Provide help to and enable appropriate access to the

unified longitudinal database by a wide range of stakeholders to serve a variety of purposes, including improving of teaching and learning, informing public policy, fostering a culture of evidence-based decision making, conducting research, evaluation system and program effectiveness, and providing reports to various stakeholder groups.

Jennifer Stegman, Oklahoma State Department of Education 34

Page 35: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Further recommendations for the P-20 Data System: ◦ Incorporate teacher preparation attributes (e.g.,

certification type, school of origin) into the data system.

◦ Incorporate analysis and business management tools into the system

◦ Implement greater interactive reporting capabilities to respond to a range of stakeholders.

◦ Include student groups not now included (e.g., home-schooled) in the data system

◦ Complete basic policies such as data use/access protocols, data quality standards and governance

Jennifer Stegman, Oklahoma State Department of Education 35

Page 36: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Jennifer Stegman, Oklahoma State Department of Education 36

Page 37: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Past ◦ Special Populations reported as Pass or Fail

◦ Reports may have lagged one to two years

Present ◦ Variety of disaggregations and performance

categories

◦ Beginning to standardize terms

Future ◦ On-demand reports

◦ Longitudinal and student connected data

Jennifer Stegman, Oklahoma State Department of Education 37

Page 38: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Jennifer Stegman, Oklahoma State Department of Education

I asked a statistician for his phone number ... and he gave me an estimate.

38

Page 39: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

The requests for data comes from a variety of stakeholders each with their own need for a different slice of data.

Legislature

Research Institutions and Research Companies

Graduate Students

Schools and Districts

Parents

USDE

Jennifer Stegman, Oklahoma State Department of Education 39

Page 40: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Moving towards a highly developed technology and information system ◦ Oklahoma has been moving towards on-demand

reporting but is still only providing “canned” reports.

Identifying and developing clearly defined variables and quality control procedures

Centralizing data systems to increase security and manage accessibility

Jennifer Stegman, Oklahoma State Department of Education

40

Page 41: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Jennifer Stegman Assistant State Superintendent

Office of Accountability and Assessment

Oklahoma State Department of Education

[email protected] 405-522-6250

Jennifer Stegman, Oklahoma State Department of Education 41

Page 42: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

In Florida

In

Data, Data, …

Kris Ellington, Deputy Commissioner

Florida Department of Education June 2011

Page 43: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

The Data Focus

Florida

◦ Data Systems

◦ Statewide Longitudinal Data Systems

Modernization Project

Partnership for Assessment of College

and Career Readiness (PARCC) Plans

◦ Interactive Data Tool

◦ Common Data Standards

June 2011 Slide 43 K. Ellington, Florida Department of Education

Page 44: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Florida’s Educational Data Systems

Data Quality Campaign: Meet all 10 Essential Elements; Meet 7 of 10 State Actions

Public pre-kindergarten through graduate school student level data for public schools, community colleges, career and technical education, adult education, and the state university system

Staff, facilities, finance, and financial aid

Post-school employment and non-education system program data

June 2011 Slide 44 K. Ellington, Florida Department of Education

Page 45: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Necessary Ingredients

State level imperatives, funding

System of common course numbers and directories

Statewide articulation agreements and oversight committee

A state level culture of data management and interagency sharing

A means to follow the records of individual students

across geographic areas, education sectors, and related

programmatic areas.

67 School Districts, 28 community colleges, 11 state universities

Very open public records laws

June 2011 Slide 45 K. Ellington, Florida Department of Education

Page 46: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

A Brief History… 1975 ◦ State Universities data system deployed

◦ Statewide K-12 student assessment data collected/reported

◦ Course numbering system for postsecondary

1983-84 ◦ K-12 Course Code Directory/common course numbers

◦ FETPIP began – links education data to employment, public assistance, military, etc.

1991 ◦ PK-12 data system added student-level enrollment,

demographic, transfers, transcripts, teacher identifier to match teachers to students

2003 ◦ Education Data Warehouse is deployed

June 2011 Slide 46 K. Ellington, Florida Department of Education

Page 47: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Florida’s SLDS Project

3 Grants; Total $38.5 Million

Statewide Longitudinal Data System, Round 3

July 2009 – June 2014

Statewide Longitudinal Data System, ARRA

July 2010 – June 2013

Race to the Top, Section C

October 2010 – June 2014

2 Foci

Accessing Data

Using Data

June 2011 Slide 47 K. Ellington, Florida Department of Education

Page 48: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

A Coherent System of Improvements

June 2011 Slide 48 K. Ellington, Florida Department of Education

Page 49: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Access to Florida’s Data Stakeholder Level of Access Access Type Sample of Accessible Data

Public Aggregate – information

available at the school, LEA, or

state level; information will

comply with FERPA regulations

Dashboards

Pre-defined Reports

Customizable Reports

Teacher data (certification, highly qualified)

Assessment data (interim and summative)

Demographic data (English Language Learners, Exceptional

Education)

School Indicators data (graduation rate, dropout rate)

Students/

Parents

Confidential – Individual student

information accessible to

students and their parents only

Plus access to aggregates

identified for “Public”

Single sign-on access to

applications

FACTS.org (college and career plans)

Teachers Confidential –

All students in teacher’s assigned

classroom

Plus access to aggregates

identified for “Public”

Single sign-on access to

applications

Assessment Tools (PMRN and others created by RTTT)

Confidential student- and staff-level data via LIIS

Standards Instructional Tool (CPALMS)

Exceptional Education Resource (PEER)

www.FloridaSchoolLeaders.org

Principals Confidential –

All students in school

Plus access to aggregates

identified for “Public”

Single sign-on access to

applications

Assessment Tools (PMRN and others created by RTTT)

Confidential student- and staff-level data via LIIS

Standards Instructional Tool (CPALMS)

Exceptional Education Resource (PEER)

June 2011 Slide 49 K. Ellington, Florida Department of Education

Page 50: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

PARCC States

June 2011 Slide 50 K. Ellington, Florida Department of Education

Participating States Governing Board States

Page 51: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

Goals

A pathway to college and career readiness

Assessments that enable cross-state

comparisons

Better assessments

Better use of technology in assessments

Match investments in testing with

investments in teaching

June 2011 Slide 51 K. Ellington, Florida Department of Education

Page 52: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

PARCC Management

Governing Board States AZ, AR, DC, FL, GA, IL, IN, LA,

MD, MA, NJ, NY, OK, RI, TN

Governing Board Chair Massachusetts Commissioner Mitchell Chester

Fiscal Agent State

Florida US Education Department award is to Florida on

behalf of 25 states to oversee budget, procurement,

and reporting functions

Project Management

Partner

Achieve Achieve is a bipartisan, non-profit organization that

helps states raise academic standards, improve

assessments, and strengthen accountability to

prepare all young people for postsecondary

education, work, and citizenship

June 2011 Slide 52 K. Ellington, Florida Department of Education

Page 53: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

PARCC Data System Standards

Establish a set of common identity

management and data technology

standards for member states.

Consult national experts to provide

guidance on platform design, the pros and

cons of development versus acquisition,

and how to capture and compare data

across states.

June 2011 Slide 53 K. Ellington, Florida Department of Education

Page 54: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

PARCC Administration and

Reporting Process

States provide vendor with student

demographic information

Vendor creates individualized student access

and assessment forms

After assessment administration, data

processed by vendor and made available

through the Interactive Data Tool

June 2011 Slide 54 K. Ellington, Florida Department of Education

Page 55: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

PARCC Interactive Data Tool

Immediate and flexible access to performance data for teachers, parents, students, administrators, policymakers and the public

An interactive data warehouse and a dynamic, user-friendly web portal

Data displays that lead to appropriate instructional and evaluative action

Allow users to browse assessment data for different indicators across different groups and create customizable reports

June 2011 Slide 55 K. Ellington, Florida Department of Education

Page 56: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

PARCC Interactive Data Tool

An open-source system

Professional development modules available

for training on the tool’s functioning,

capabilities, and appropriate uses of data

accessed via the tool.

Will be compliant with platform-aligned

data standards for easy accessibility and use

by current state systems

June 2011 Slide 56 K. Ellington, Florida Department of Education

Page 57: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

PARCC Administration & Reporting

Process

The vendor will use the Interactive Data

Tool to send student-level and summary

data for each assessment component to

each PARCC state’s longitudinal data

system.

Data available for state uses (additional

state reports, accountability, longitudinal

research and analysis, links to

postsecondary/employment data.)

June 2011 Slide 57 K. Ellington, Florida Department of Education

Page 58: Data Data Everywhere CCSSO Presentation at National Conference on Student Assessment

More Information Available

PARCC

◦ http://www.parcconline.org/

Florida Department of Education Data

Systems ([email protected])

◦ http://www.fldoe.org/arm/datasystems.asp

June 2011 Slide 58 K. Ellington, Florida Department of Education