Page 1
Copyright © 2015 by Educational Testing Service.
The Future of the NAEP Survey Questionnaires: Measuring what Matters
for Student Achievement in the U.S.
1
Jonas P. BertlingDirector, NAEP Survey Questionnaires
Educational Testing Service
NCSA 2015 Symposium “How NAEP is leading assessments into the future”June 23, 2015 | San Diego, CA
Page 2
Copyright © 2015 by Educational Testing Service.
Overview
• Future NAEP Survey Questionnaires will include additional noncognitive and opportunity-to-learn modules
• Why are noncognitive factors relevant?
• Implementation strategy: 3 key innovations
• What have we done so far?
• Importance of these innovations to ensure that NAEP stays relevant
2
Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation
Page 3
Copyright © 2015 by Educational Testing Service.
NAEP measures what students in the U.S. know and can do
Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation
Page 4
Copyright © 2015 by Educational Testing Service.
NAEP measures what students in the U.S. know and can do
4
How are students in the Nation doing in key subject-areas?
How do subgroups perform?
Why do students succeed?
What can we do to improve student learning outcomes?
How do students approach learning situations and utilize learning opportunities?
How do learning opportunities differ across the Nation?
How are students in the Nation doing in terms of other outcomes than achievement?
Noncognitive factors
Social Emotional Learning (SEL)
Character
PersonalityMindsets, Essential Skills and Habits
Inter- and Intrapersonal Skills
Soft Skills
Nonacademic skills
Take Action
Cognitive Assessment
Noncognitive Assessment
How can NAEP help stakeholders to take action?
Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation
Page 5
Copyright © 2015 by Educational Testing Service.
Survey Questionnaires data puts achievement results into context…
5
…and can add value to the Nation’s Report Card by capturing important additional
variables.
Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation
Page 6
Copyright © 2015 by Educational Testing Service.
Opportunity-to-Learn (OTL) Factors:
(COGNITIVE ASSESSMENT)(SURVEY QUESTIONNAIRES)
“Noncognitive” Student Factors:
Achievement
Cognitive Ability*
Domain-general factors
(“Core”)
Subject-specific factors
*not measured with the NAEP survey questionnaires
NAEP Contextual Variables
OTL At School
“Noncognitive” Student Factors
Students’ exposure to opportunities to acquire relevant
knowledge and skills, both at school and outside of school.
Skills, strategies, attitudes, and behaviors that are distinct from
content knowledge and academic skills (e.g., motivational and self-
regulatory variables)
OTL Outside of School
Basic Demographics
Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation
new areas of focus
Page 7
Copyright © 2015 by Educational Testing Service.
Why Noncognitive Factors?
• Related to important life outcomes (e.g., achievement, health, well-being, citizenship behavior)
• Noncognitive factors are about as predictive for many achievement outcomes as cognitive factors
• More malleable than cognitive factors
• Can have positive impact on cognitive development
• Important workforce readiness skills
See e.g., Farrington et al. (2012); Heckman & Kautz (2014), Kautz et al. (2015) for comprehensive reviews of the research literature on noncognitive factors
7
Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation
Page 8
Copyright © 2015 by Educational Testing Service.
3 Key Innovations for the NAEP Survey Questionnaires
Expand topics to include more contextual factors critical for student
achievement
Improve measurement and reporting of contextual factors
Improve questionnaire design for TBA to maximize validity while keeping student
burden low
I.
II.
III.
Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation
Page 9
Copyright © 2015 by Educational Testing Service.
Alignment with Implementation Recommendations from The Governing Board
Expand topics to include more contextual factors critical for student
achievement
Improve measurement and reporting of contextual factors
Improve questionnaire design for TBA to maximize validity while keeping student
burden low
I.
II.
III.
“NAEP reporting should be enriched by greater use of contextual data derived from background or non-cognitive questions asked of students, teachers, and schools”
-National Assessment Governing Board (2012, p. 2)
Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation
Page 10
Copyright © 2015 by Educational Testing Service.
Key Challenges & Implementation to Date
• Which factors to measure?
• How to measure them?
• How to keep student burden low?
0
5
10
15
20
25
30
35
NAEP PISA TIMSS/PIRLS
Minutes
Subject-Specific Core
Item DevelopmentPre-testing in Cognitive Labs
Large-scale Pilot Testing
Operational Administration
Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation
Page 11
Copyright © 2015 by Educational Testing Service.
11
Item DevelopmentPre-testing in Cognitive Labs
Large-scale Pilot Testing
Operational Administration
2014
Considerations for Selection of Areas for Development:1. Mandated reporting categories
2. Not intrusive
3. Clearly related to student achievement
4. Malleable, actionable
5. Can be measured with questionnaires
6. Clearly tied to (subject-specific) framework
7. Based on established/validated in research and/or other ILSAs
Implementation for NAEP 2017 Core, Reading, Writing, and Mathematics:
Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation
Page 12
Copyright © 2015 by Educational Testing Service.
12
Item DevelopmentPre-testing in Cognitive Labs
Large-scale Pilot Testing
Operational Administration
2014
1. Review of Literature (Research, Policy Frameworks, Other Media)
2. Consider different measurement approaches
3. Consultation and reviews with advisory groups and key stakeholders including:Questionnaire Standing Committee, Subject-area Standing Committees, NAEP Principals Panel, NAEP State Coordinators, NAEP Design and Analysis Committee
Implementation for NAEP 2017 Core, Reading, Writing, and Mathematics:
Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation
Page 13
Copyright © 2015 by Educational Testing Service.
Questionnaire Standing Committee: Current Members
Roger TourangeauWestat
Stanley PresserUniversity of Maryland
Angela DuckworthUniversity of Pennsylvania
Leslie RutkowskiIndiana University Bloomington
Hunter GehlbachHarvard University
Rob SantosUrban Institute
Gerunda HughesHoward University
Norbert SchwarzUniversity of Michigan
David Kaplan University of Wisconsin-Madison
Jonathon StoutLock Haven University
Henry LevinTeachers College, Columbia University
Akane ZushoFordham University
Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation
Page 14
Copyright © 2015 by Educational Testing Service.
14
Item DevelopmentPre-testing in Cognitive Labs
Large-scale Pilot Testing
Operational Administration
Noncogs
Domain-general (Core)
OTL Outside of School
OTL At School
“Noncognitive” Student Factors
Basic Demographics
Subject-specific
12 1
23
3
4
56
7
5
4
1. Grit2. Desire for learning &
Growth Mindset
1. Self-Efficacy/Confidence2. Achievement Goals3. Interest & Enjoyment
3. Socioeconomic status & family academic support
4. Technology Use5. School Climate
OTL At School
4. Out-of-school Activities5. Resources for Learning
and Instruction (incl. technology)
6. Organization of Instruction
7. Teacher Preparation (incl. noncognitive factors and familiarity with technology)
5 Core Modules Additional Subject-specific modules
OTL Outside of
School
Implementation for NAEP 2017 Core, Reading, Writing, and Mathematics:
2014
“Balanced” reporting approach based on indices
AND select single questions
Indices allow for more robust reporting based on
multiple questions on the same topicJonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation
Page 15
Copyright © 2015 by Educational Testing Service.
• All new and revised items go through rigorous pre-testing before administered in any pilot or operational assessment
• One-on-one cognitive interviews with trained interviewers
– Structured probing with both general and item-specific probes
– How do students understand items? What are difficulties? How can items be improved? How do alternative versions compare?
Item DevelopmentPre-testing in Cognitive Labs
Large-scale Pilot Testing
Operational Administration
2014 2014-2015
Implementation for NAEP 2017 Core, Reading, Writing, and Mathematics:
Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation
Page 16
Copyright © 2015 by Educational Testing Service.
• Coglabs provided basis for data-informed revisions of response options for more clarity and consistency across items
• Coglabs helped with language simplification
• Platform for test of new item formats (e.g. vignettes for potential future use)
• Respondents preferred items with more (e.g., 5 or 6) response options allowing for more differentiated choices than simple yes/no questions
• Eye tracking study showed that 4th grade students can answer and enjoy answering matrix questions
Item DevelopmentPre-testing in Cognitive Labs
Large-scale Pilot Testing
Operational Administration
2014 2014-2015
Key Lessons learned from cognitive interviews:
Implementation for NAEP 2017 Core, Reading, Writing, and Mathematics:
Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation
Page 17
Copyright © 2015 by Educational Testing Service.
Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation
Heat map from NAEP SQ Eye-Tracking Study
Page 18
Copyright © 2015 by Educational Testing Service.
• Large-scale digital based pilot will inform decision for operational
• Innovations:– “Spiraling” design to select best questions: Not every question
answered by every student to reduce burden
– Use of timing data in analysis to build final forms
– Advanced psychometric analyses to identify and validate indices
– Item version comparisons to select best measurement approach
– Variation of item sequences to control for potential biases
Item DevelopmentPre-testing in Cognitive Labs
Large-scale Pilot Testing
Operational Administration
2014 2014-2015
Implementation for NAEP 2017 Core, Reading, Writing, and Mathematics:
2016 2017
Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation
Page 19
Copyright © 2015 by Educational Testing Service.
Summary• Future NAEP Survey Questionnaires will include additional
noncognitive and opportunity-to-learn modules;
• Noncognitive factors are important achievement predictors, malleable, and potential additional outcomes;
• Implementation strategy focuses on (1) new topics, (2) better measurement, and (3) improved design;
• Item development and pre-testing for new 2017 modules successfully completed; large-scale pilot will follow in 2016.
19
These innovations will ensure that NAEP stays relevant as policy interests shift from monitoring student literacy in core subjects to promoting lifelong learners who are able and eager to face the demands and challenges of a truly global society.
Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation
Page 20
Copyright © 2015 by Educational Testing Service.
Questions? Comments?
20
ETS NAEP Survey Questionnaires Team Members
Jared Anthony
Thank you!
[email protected]
Debby Almonte Laurie StaubJan Alegre Ryan Whorton
Jonas P Bertling | The Future of the NAEP Survey Questionnaires | NCSA 2015 Presentation