Page 1
For Official Use EDU/PISA/GB(2012)28 Organisation de Coopération et de Développement Économiques Organisation for Economic Co-operation and Development 12-Oct-2012
___________________________________________________________________________________________
_____________ English - Or. English DIRECTORATE FOR EDUCATION
PROGRAMME FOR INTERNATIONAL STUDENT ASSESSMENT
Governing Board
PROGRESS REPORT ON QUESTIONNAIRE DEVELOPMENT AND DESIGN
34th meeting of the PISA Governing Board
29-31 October 2012
Madrid, Spain
Andreas Schleicher; E-mail: [email protected] ; Tel: +33.1.45.24.93.66
JT03328578
Complete document available on OLIS in its original format
This document and any map included herein are without prejudice to the status of or sovereignty over any territory, to the delimitation of
international frontiers and boundaries and to the name of any territory, city or area.
ED
U/P
ISA
/GB
(20
12
)28
Fo
r Officia
l Use
En
glish
- Or. E
ng
lish
Page 2
EDU/PISA/GB(2012)28
2
TABLE OF CONTENTS
1. Conceptual background ........................................................................................................................ 3 1.1 The PISA approach to large scale assessment .................................................................................... 4 1.2 Overarching conceptual framework for PISA 2015 ........................................................................... 5 1.3 Determining priorities for context assessment in PISA 2015............................................................. 8
2. Questionnaire Development from April to September, 2012 ............................................................. 10 Module 1: Teacher qualification and professional knowledge, and the new Teacher Questionnaire .... 11 Module 2: Science Teaching Practices ................................................................................................... 12 Module 3: School-level learning environments for science ................................................................... 12 Module 4: Science-related outcomes ..................................................................................................... 13 Module 7: Student SES, family and home background ......................................................................... 13 Module 9: Educational pathways in early childhood, Module 14: Parental involvement, and the Parent
Questionnaire option .............................................................................................................................. 13 Module 10: Domain-general student behaviour and attitudes ................................................................ 14 Module 12: Learning Time and Curriculum, and the Educational Career optional questionnaire ......... 15 Module 15: School leadership and management .................................................................................... 16 Module 19: Assessment, evaluation, and accountability....................................................................... 16
3. Questionnaire design and rotation in the Main Study StQ ................................................................. 17 3.1 Designing the PISA 2015 Field Trial ............................................................................................... 17 3.2 Time restrictions and design implications for the Main Study ......................................................... 18
REFERENCES .............................................................................................................................................. 20
APPENDIX: DRAFT OVERVIEW OF FIELD TRIAL CONTENT .......................................................... 22
Tables
Table 1. Questionnaire content used in .................................................................................................. 5 Table 2. Distribution of content ............................................................................................................. 9
Figures
Figure 1. Modular structure of the PISA 2015 context assessment design ............................................. 7
Page 3
EDU/PISA/GB(2012)28
3
PROGRESS REPORT ON QUESTIONNAIRE DEVELOPMENT AND DESIGN
Introduction
1. This document provides an update on work on the PISA 2015 context questionnaire framework
and questionnaire content, following feedback from the previous two PGB meetings and from further
discussions in the Questionnaire Expert Group. The document outlines the content for the 2015 core and
optional contextual questionnaires and discusses issues relating to design and sampling. In particular, the
document:
Describes the policy-driven overarching conceptual framework that drives questionnaire
development in PISA 2015 and outlines criteria for choosing the questions to be included in the
field trial and main study context questionnaires.
Outlines progress in questionnaire development for each of the policy modules and discusses
design implications (e.g. mode or sampling) for each of the core and optional questionnaires.
Closes with a discussion of the implications of design issues (e.g. timing and rotation of the
student questionnaire in the main study).
Lists (in an appendix) all of the constructs under consideration, by module and questionnaire
instrument.
2. The PGB is invited to:
REVIEW the draft frameworks and content for the PISA 2015 context questionnaires;
PROVIDE DIRECTIONS for the further development of the PISA 2015 core and optional
contextual questionnaires.
3. In particular, the PGB is asked to advise on:
The criteria set out in paragraph 24 for content reduction.
The recommendation in paragraph 73 to have one version of the Student Questionnaire in the
Main Survey, with no rotated content.
1. Conceptual background
4. The questionnaire design for PISA 2015 builds on the frameworks set up in earlier waves and
intends to serve policy needs by providing two types of knowledge: (1) indicators used to evaluate and
monitor educational systems, and (2) multi-level (student, school, and system) models that analyse
relationships between different components of the educational system, especially between context factors
and student outcomes.
5. The main innovation in the development of the PISA 2015 Questionnaire Framework and the
questionnaire instruments is a much closer relationship with policy interests. This was achieved by
focusing the development of new material on a series of modules addressing major policy areas, after
consultation with the PGB to determine their priorities in the development of this new material. As a result,
Page 4
EDU/PISA/GB(2012)28
4
questions on teaching and learning, including learning time allocation and assessment practices, will
receive more attention, as well as school policies (evaluation, leadership, school management, and school
curriculum), and non-cognitive outcomes (both general and science-related). At the same time, it was
proposed that all policy areas should receive as much coverage as possible. Also, comparing new PISA
results against previous cycles becomes increasingly important, as PISA is maturing as a data base for
long-term educational monitoring and trend analysis.
6. As a result, PISA 2015 will address a very broad range of topics - a mixture of new and well
established content -, some of them being presented in more detail than others. To cope with the amount of
material, the Main Survey design for the Student Questionnaire will need to be revisited, possibly adjusting
the administration time, once exact time estimates are available from the Field Trial.
7. Computer-based delivery will allow for new, interactive item formats and questions being
tailored to the respondent. These innovations will support enhanced monitoring of policy-relevant
indicators, and more sophisticated analyses of patterns and relationships both within and between
countries. Computer-based delivery will also enable research on the processes of filling out the
questionnaire, helping to enhance the validity of the PISA context assessment.
1.1 The PISA approach to large scale assessment
8. In the first assessment cycle, PISA mainly focused on the installation of procedures and the step-
by-step implementation of the three major domains of assessment: Reading in 2000, Mathematics in 2003,
and Science in 2006. Assessment frameworks had to be developed, negotiated with stakeholders, and
implemented. PISA has also been recognised for its detailed, theory-based assessment of family
background, socio-economic status, and immigration background, allowing for sophisticated analyses of
equity issues within and across countries. A lot of effort went into the definition and operationalisation of
individual student background indicators, finally leading to the establishment of a powerful, integrated
index of economic, social, and cultural status for each student (ESCS).
9. Furthermore, PISA has covered specific aspects of school environments and out-of-school
learning opportunities. With regard to science education – which will be the major domain of student
assessment in PISA 2015 - PISA 2006 covered science-related learning opportunities, teaching practices,
and school-level context. In addition to these measures of educational processes, PISA 2006 captured
various kinds of non-cognitive learning outcomes such as student motivation, student engagement, beliefs,
and aspirations. Thus, with PISA 2006, the so-called “Background Questionnaires” had become a mixture
of (a) general student background variables and (b) scales covering a variety of domain-specific processes
and outcomes. At the same time, analytical techniques and reporting improved substantially. The initial
report on PISA 2006, “Science Competencies for Tomorrow's World” (OECD, 2007), was a breakthrough
in that it established multi-level modelling estimates of school effects explaining student outcomes and the
degree of equity, both within and across countries.
10. The second wave of PISA cycles, starting with PISA 2009, established a coherent overarching
framework for context questionnaire development. The PISA 2009 Questionnaire Framework (OECD,
2009) was built on a strong and broad knowledge base grounded in educational effectiveness research. The
PISA 2009 report, organised into six volumes, was based on even more diverse and more sophisticated
analyses (see Table 1). In addition to student achievement, non-cognitive outcomes such as student
engagement were studied in detail, together with cognitive and meta-cognitive strategies. Moreover, the
impact of background variables, individual support factors, school level input, processes and policies, as
well as system level factors was reported – all assessed by student and school questionnaires. Enhanced
reporting on trends both for student achievement and for context measures constituted another important
improvement in PISA 2009.
Page 5
EDU/PISA/GB(2012)28
5
Table 1. Questionnaire content used in "PISA 2009 Results: 'What Students Know and Can Do'". (OECD 2010)
Volume I: Student Performance in Reading, Mathematics and Science: Student background: Gender Volume II: Overcoming Social Background: Equity in Learning Opportunities and Outcomes. Student background: Socio-economic background, gender, immigration status (language spoken at home, age of arrival, country of origin) Individual support: Parental support (at beginning of primary education/at age 15), pre-primary education (attendance, quality) Volume III: Learning to Learn: Student engagement, strategies and practice Student background: Socio-economic background, gender, immigration status, language spoken at home Outcomes: Enjoyment of reading, time and material used for reading, metacognition (awareness of strategies), self-reported use of reading strategies (memorisation, elaboration, control) Volume IV: What makes a school successful? Resources, policies and practices Student background: Socio-economic background, age of school entry, grade repetition Student-reported processes: learning time (previous education, learning time at school, enrichment/remedial education, after-school lessons), teacher-student relations, disciplinary climate, teacher’s stimulation of reading engagement School input, policies and processes (reported by the principal): type of school (public/private), number of programmes, class size, educational resources (e.g., ICT, library), school responsibility for assessment and
curriculum/for resource allocation, extra-curricular activities provided, school admittance/grouping/transfer policies, assessment practices/purposes, use of achievement data, school accountability, methods for monitoring teachers, teacher and student behaviour, parent involvement and expectations, leadership.
11. The Questionnaire Expert Group (QEG) for PISA 2012 set out to further develop and clarify the
goals, the policy needs, the opportunities as well as the challenges of PISA. The PISA 2012 Questionnaire
Framework (EDU/PISA/GB(2010)23) acknowledged that PISA, as a policy-oriented programme, has to
serve the needs of policy makers in participating countries.
1.2 Overarching conceptual framework for PISA 2015
12. The Core 6 proposal for Questionnaire Development in PISA 2015 identified four broad areas of
interest: i) Outcomes, ii) Student background, iii) Teaching and learning and, iv) School policies and
governance.
Outcomes
13. Educating a person means fostering his or her individual development as a unique, self-
determined, knowledgeable person who gradually gains in ability to participate in society. As each PISA
assessment is a cross-sectional study, PISA cannot capture developmental processes, but PISA serves as a
snapshot of development at the age of 15 years. This snapshot of course includes an assessment of literacy
and life skills, but in addition to these cognitive outcomes, factors such as motivation, self-regulation,
attitudes, and beliefs are important educational outcomes. Recent scientific research on education has
highlighted that non-cognitive factors present important skills for individual and social development
(Heckman, Stixrud and Urzua, 2006; Almlund, Duckworth, Heckman and Kautz, 2011). Therefore, PISA
questionnaires also address non-cognitive outcomes. In PISA 2015, with collaborative problem solving
being one of the cognitive domains, dispositions that are relevant for collaboration should be part of the
design.
Page 6
EDU/PISA/GB(2012)28
6
Student background
14. In addition to the snapshot of cognitive and non-cognitive outcomes, PISA gathers retrospective
and prospective information about educational pathways and careers across the lifespan. In recent years,
researchers as well as public debates in many countries have stressed the importance of early childhood
education (Barnett, 2008; Blau and Curie, 2006; Cunha, Heckman, Lochner and Masterov, 2006; Sammons
et al. 2011). Therefore, PISA aims at collecting retrospective information on early childhood education.
However, validity of such data gathered from 15-year old students is limited; therefore, the optional Parent
Questionnaire will be used for this purpose. In order to understand educational careers, family background
such as the socio-economic status and immigrant background should be taken into account. The
distribution of educational opportunities and outcomes can help describe the extent to which countries
succeed in granting equal opportunities to their students.
Teaching and Learning
15. School-based instruction is the core process of formal, systematic education. Therefore, policy
makers need information on teaching, learning, and organisation of schools. To increase explanatory power
of the study, assessment of teaching and learning tends to focus on the major domain of assessment, i.e.
science in PISA 2015. The knowledge base of school effectiveness research (Scheerens and Bosker, 1997;
Creemers and Kyriakides, 2008) allows for the identification of core factors, i.e. teachers’ qualifications
and professional activities, teaching practices, and generally learning opportunities provided both within
and outside of school. It is a challenge to address teacher and teaching-related factors in PISA, but student
and school questionnaires may provide at least some proximal indicators, and the optional Teacher
Questionnaire will significantly enhance the scope of the design. Both science-related and general
information on teaching are indispensable if PISA aims at covering educational factors that are strongly
related to student achievement.
School policies and governance
16. As policy makers have a limited direct influence on teaching and learning processes, information
on school-level factors which help improve schools, and thus indirectly improve student learning, shall
have high priority. As to teacher and teaching variables, school effectiveness research has built a strong
knowledge base showing that certain “essential supports” (Bryk, Sebring, Allensworth, Luppescu and
Easton, 2010) drive school improvement. These essentials are professional capacity, a well-organised
curriculum, parental and community involvement, school climate and leadership. All of these will be
addressed within the PISA 2015 questionnaire design. In addition, resources (budget, personnel) and their
distribution between schools are important in educational policy making.
17. To meet policy needs directly, PISA also needs to address issues related to governance on the
system level (Hanushek and Woeßmann 2011; Woeßmann, Lüdemann, Schütz and West, 2007). “Locus of
decision making” measures and accountability practices describe core aspects of governance, namely the
distribution of power and control between central and local institutions. Allocation, selection, choice, as
well as assessment and evaluation are the basic processes deployed by policy makers and/or school
administrators to steer school quality and school improvement.
18. These four areas were differentiated into 19 more fine-grained policy issues, which were
accepted by the PISA Governing Board (at its meeting in October 2011) as the building blocks of the PISA
2015 design for context assessment. Figure 1 provides a schematic overview of this modular structure. It
also shows (circles) how PISA covers all levels of activities related to education: the individual and
classroom level (teaching and learning), the school level, and the system (governance) level.
Page 7
EDU/PISA/GB(2012)28
7
Figure 1. Modular structure of the PISA 2015 context assessment design
19. Because PISA intends to support educational policy making, the framework can be compared
with one of the most comprehensive pieces of recent international research on education policy. Sykes,
Schneider and Plank (2009) provide 76 chapters with state-of-the-art reviews of educational policy
research. The section on “policy implications of educational resources, management, and organisation”
mentions eight policy issues, e.g. class size, equity policies, school choice, and resources. All of these are
covered by the modules provided by this framework. The section on “teaching and learning policy” covers
10 issues, such as opportunity-to-learn and teacher quality, which are also covered by the modules – with
the exception of demography. The sections on “educational access and differentiation” as well as “politics
and the policy process” each include 4 or 5 issues covered in our modules, such as early childhood
education, shadow (out-of-school) education, assessment policies and ethnicity. Still, a few differences can
be found between the context assessment framework and the Handbook of Educational Policy Research.
PISA does not relate to research on policy processes and specific actors/stakeholders (e.g., the role of
teacher unions). The assumption followed in this framework is that this is so because PISA is not meant to
provide data on decision-making processes. On the other hand, the Handbook of Educational Policy
Research does not include any chapter on non-cognitive outcomes, which nevertheless are important
reporting categories in PISA.
20. To sum up, the modular approach to context assessment in PISA 2015 allows for broad coverage
of policy issues and related research questions. However, there are strict limits to length of questionnaires
Page 8
EDU/PISA/GB(2012)28
8
and the time needed to fill them out. Therefore, in addition to breadth, PISA needs to focus on the most
important issues (modules, or constructs within modules).
1.3 Determining priorities for context assessment in PISA 2015
21. The PISA Governing Board was asked to indicate the top six priority modules for further
developmental work, based on policy relevance and the need for improvement from previous cycles. In one
way or another, all topics will be covered in the PISA 2015 questionnaires, but more emphasis will be
devoted to those priority areas identified by the PGB.
22. The following six areas received the most votes for high policy relevance and need of further
development work for PISA 2015 in order of popularity:
19. Assessment, evaluation and accountability
02. Science teaching practices
04. Science-related outcomes: motivation, attitudes, beliefs, strategies
12. Learning time and curriculum
10. Domain-general student behaviour and attitudes
15. Leadership and school management
In addition to these priorities, four modules are assigned medium priority:
01. Teacher qualification and professional knowledge
09. Educational pathways in early childhood
03. School-level learning environments for science
07. Student SES, family and home background
The following modules were rated of lowest priority. While these will be included to some extent in
the questionnaires, there has been less development in these areas:
13. School climate: Interpersonal relations, trust, expectations
08. Ethnicity and migration
11. Student dispositions related to collaborative problem solving
05. Out-of-school science experience
06. Science career
16. Resources
14. Parental involvement
18. Allocation, Selection and choice
17. Locus of decision making within the school system
23. Independently, the PISA 2015 Questionnaire Expert Group (QEG) had delivered a similar rating.
Taking into account the results of the PGB rating exercise as well as suggestions from the QEG,
Page 9
EDU/PISA/GB(2012)28
9
questionnaire development focused on the high priority and (to a lesser extent) the medium priority
modules.
24. Besides strengthening the alignment with policy needs, a clear focus on high and medium priority
modules helps limit the length of the questionnaires. Another method of keeping the size of material
manageable is distributing it across different questionnaires, such as Student and School Questionnaire,
optional Parent and Teacher Questionnaires, and short, optional add-ons to the Student Questionnaires
(Educational career, ICT familiarity), as shown in table 2. As a rule, information will be gathered from
those stakeholders who can provide the most valid perspective.
Table 2. Distribution of content
Module StQ ScQ PQ TQ ICT EC
1 Teacher education and professional knowledge
2 Teaching practices
3 School environment for science learning
4 Science-related non-cognitive outcomes
5 Out-of-school science learning experiences
6 Science career
7 Student SES, family and home background
8 Ethnicity and migration
9 Educational pathways in early childhood education
10 General non-cognitive outcomes
11 Dispositions to CPS
12 Learning time and curriculum
13 School climate
14 Parental involvement
15 Leadership and school management
16 Resources
17 Locus of decision making
18 Admission, selection and choice
19 Assessment, evaluation, and accountability
25. As a third step in content reduction, constructs are prioritised according to the following
principles:
Page 10
EDU/PISA/GB(2012)28
10
In as far as possible, questions and items with proven validity and measurement quality
(established measures) will be retained from previous PISA cycles, especially from PISA 2006,
to allow for trend reporting.
Questions that serve the needs of multiple modules are preferred for reasons of efficiency.
Short questions will be preferred to longer questions.
Questions that can be expected to work across cultures, and/or may help to describe
country/culture differences, will receive higher priority.
If quality is otherwise equal, constructs will be retained that are more strongly related to student
outcomes.
If there are two or more similar constructs, all but one will be dropped.
26. A reduced, but still relatively large set of questions will be submitted to the PISA 2015 Field
Trial which will take place in 2014. After the Field Trial, a final reduction will be conducted (again in
accordance with above described procedures).
2. Questionnaire Development from April to September, 2012
27. Two meetings of the Questionnaire Expert Group (QEG) took place in 2012, one in May 2012
and one in August 2012.
28. The May meeting focused on the research and policy background of high and medium priority
modules, discussing both conceptual and methodological innovations. In preparation for that meeting, The
PISA consortium (Core 6) staff contracted questionnaire developers and provided background papers for
10 modules and four international options. These papers reviewed relevant research and experience from
previous PISA cycles and laid out conceptual foundations for further development.
29. Based on recommendations from the May QEG meeting, PISA consortium (Core 6) staff and
contracted developers started to revise existing PISA measures and develop new questionnaire material,
once again with a clear focus on high priority modules. At this stage of work, no limitations regarding
breadth and depth were applied, because the goal was to procure a wide set of measures – sometimes even
including several alternative approaches to cover a specific construct. The QEG was thus to receive a wide
array of possible high-quality questions and items, which could then be discussed, reviewed, prioritised,
and – where needed - selected by the QEG. 16 working papers were submitted to the QEG meeting in
August, 2012, comprising some 368 measures across all 19 modules and the international questionnaire
options. At their August meeting, the QEG made a first selection. Following their advice and comments
received from the Secretariat, PISA consortium (Core 6) staff are currently engaged in a second wave of
questionnaire development, revising the proposed material and in some cases also developing new material
to meet requests from the QEG and/or the Secretariat. Results from this second wave of questionnaire
development will be discussed by the QEG in December, 2012 during a series of webinars. In January
2013, PISA consortium (Core 6) staff will open a web-based forum for NPMs, asking for overall
comments and discussion of specific issues. This forum will provide an opportunity for countries to
provide comments on cultural appropriateness and national feasibility of new measures. PGB members and
the Secretariat will receive the final instruments in March 2013 for discussion at the April, 2013, meeting
of the PGB, and for implementation, translation, and adaptation for the Field Trial.
Page 11
EDU/PISA/GB(2012)28
11
30. To provide a more detailed insight into this on-going process, the following section describes
conceptual foundations and questionnaire development for the high and medium priority modules. Also,
the innovations planned within related international options will be explained in this context.
31. A full overview of material currently discussed within the PISA consortium is presented in the
appendix.
Module 1: Teacher qualification and professional knowledge, and the new Teacher Questionnaire
32. Teachers are at the core of school instruction. Many studies have demonstrated a clear influence
of teaching practices and other teacher-related factors on student learning and outcomes. In recent years,
this relation has received growing interest. Many stakeholders are valuing the central role teachers play in
education, also regarding their role as school leaders and key sources of innovation. The PISA 2015
Questionnaire Framework particularly pursues this issue in two Modules: While Module 2, “Teaching
practices”, deals with professional behaviour within the classroom, Module 1, “Teacher qualification and
professional knowledge”, deals with background variables, individual qualification and competence, as
well as professional practices – such as collaboration and professional development - on the school level.
33. This layout of teacher-related factors in the PISA 2015 Questionnaire Framework is in line with
OECD’s Teaching and Learning International Survey (TALIS), which distinguishes between the teachers’
tasks in terms of classroom practices on the one hand and teachers’ professional activities on the other
hand.
34. In PISA, teacher-related information so far has been gathered from school administrators and
students only. The School Questionnaire asked about initial education of recruited staff and participation in
professional development. These measures will be addressed in PISA 2015 as well, and information on
policies regarding both issues will be added. Also, a scale on students’ perception of teacher engagement
has been adapted from the research literature. Thus, there will be some coverage of Module 1 issues in the
standard questionnaires. In addition, PISA 2015 will include an optional Teacher Questionnaire (TQ) to
tackle this high priority policy module in more detail.
35. The implementation of Module 1 constructs will become much more feasible with the
introduction of this International Option. Teachers themselves will provide much more valid information
on their background, initial education, professional career and professional development activities than
school administrators could do. It will thus be possible to address policy issues of high relevance:
recruitment, selection, and allocation of teachers to schools; career changers; teachers teaching science
classes without any formal training for science education; fluctuation among the teaching staff.
36. An important feature of the Teacher Questionnaire (TQ) in PISA will be the split of the sample:
Up to 10 science teachers will be sampled (TQ-Science), as well as 10 or more teachers teaching 15-year
old students in other subjects (TQ-General). Both groups will receive specific questionnaires, with an
overlap for background variables (TQ-Core). Thus, science teachers can be asked about science-specific
aspects, such as the relative coverage of content knowledge, pedagogical content knowledge (i.e.
knowledge about student learning, typical misconceptions etc.) and general pedagogical knowledge (e.g.,
classroom management techniques) – information that will be gathered through an interactive question
which allows respondents manipulate a pie chart to provide this information. Also, questions about initial
education and professional development will be tailored to science teachers in one case, and otherwise
address more general issues.
37. As discussed at the PGB when the TQ option was submitted in April 2012, and at the QEG, it
will not be possible to measure teachers’ professional knowledge in PISA. However, related beliefs,
Page 12
EDU/PISA/GB(2012)28
12
especially self-efficacy beliefs, and non-cognitive factors, such as teacher enthusiasm and job satisfaction,
will be addressed. To check the validity of teachers’ self-reports of enthusiasm, (a) anchoring vignettes will
be provided, and (b) teacher engagement/enthusiasm will also be covered from the student perspective.
38. Finally, teachers will provide information on other policy issues, such as leadership, school
climate, parental involvement, and assessment practices.
Module 2: Science Teaching Practices
39. In line with the PISA approach of measuring scientific literacy as the capacity of students to
identify scientific questions, to explain phenomena scientifically, to use scientific evidence, and to value
scientific arguments, most science education experts nowadays recommend so-called inquiry-based
teaching as the via regia for fostering science literacy. Accordingly, PISA 2006 focused on various aspects
of inquiry-based teaching when asking students about their classroom learning experience. Four scales
were established: (1) interactive science teaching, (2) hands-on activities, (3) student investigations, and
(4) real-life applications. However, re-analyses showed that these scales were highly correlated. Also, the
QEG recommended extending the coverage of science teaching practices to capture learning environments
that are not aligned with the inquiry paradigm.
40. As a first step, PISA consortium (Core 6) experts revised the scales, shortening them and
aligning them with more recent research that identifies four aspects of inquiry-based teaching: procedural
aspects (e.g., students designing their own experiments), social aspects (e.g., class debating alternative
hypotheses), epistemic aspects (e.g., drawing conclusions from observations), and real life applications.
41. Secondly, two scales were added that ask about other kinds of science teaching practices. One
scale covers direct instruction (e.g., teacher providing lectures and designing well-structured practice
sessions) which is very popular, and has been shown to have positive effects especially on low-achieving
students. Another scale, named “opportunity to learn”, covers various aspects of media use, instructional
methods (such as teacher vs. student presentations) and interactive settings (e.g., small group vs. whole
class discussion).
42. All the measures discussed above are rather close to observable behaviour, mostly teacher
practices. Research on the quality of teaching, however, suggests that “latent” dimensions of teaching
quality are most predictive for student outcomes. Following experience with mathematics teaching quality
in PISA 2012, measures of teacher support, disciplinary climate, and cognitive activation (i.e. challenging
tasks) in science classes have been developed. Classroom discipline and teacher support have been
measured in most PISA cycles except for 2006, and have been shown to be highly correlated to student
achievement and student motivation, respectively. Both will be adapted to science education in the PISA
2015 Field Trial.
43. Student perceptions proved to be valid indicators of teaching practices and teaching quality in the
research literature. However, with an optional Teacher Questionnaire being introduced, there will also be a
scale reporting on classroom practices from teachers’ point of view. Teachers will be asked how they
perceive “inquiry-based science education”, and to what extent they implement its components. Teachers’
perceptions will be addressed by a set of vignettes, which can also serve as anchoring vignettes for
teachers’ self-reports, allowing to correct for response bias.
Module 3: School-level learning environments for science
44. Conceptually, this module has quite some overlap with other modules dealing with school-level
factors, such as Module 12, Learning Time and Curriculum, Module 15, School Leadership and
Management, or Module 19, Assessment, Evaluation, and Accountability (see below for brief reports on
Page 13
EDU/PISA/GB(2012)28
13
those). In addition to those, the QEG requested some questions in the School Questionnaire that would
directly focus on the status of science education in the school, and available resources. Accordingly, a
question has been developed to investigate the overall value of science within the school (Is science
education a shared priority for stakeholders?), along with questions on resources available for science
education: size of teaching staff per science area; physical resources such as laboratories and equipment for
student experiments; and cooperation with science museums, companies and other external partners.
Module 4: Science-related outcomes
45. Positive attitudes towards science and technology, but also motivation, value beliefs and
strategies that motivate students to engage in scientific issues are important educational and policy goals in
many countries. Moreover, such attitudes affect students’ learning and performance as well as their choice
of courses and future careers. However, the relationship between attitudes, motivation, value beliefs and
strategies on one hand and science performance on the other hand is reciprocal. They can cause higher
science performance as well as being a consequence of higher science performance. Therefore, they are
important outcomes of education themselves.
46. PISA 2015 will not use the “embedded measures” of content or task-specific interest that were
used in some PISA 2006 cognitive assessment units, because of limited validity and worries about negative
impact on achievement. However, classical questionnaire scales covering student interest (investigating
two aspects: science in general, and science as experienced in school) and instrumental motivation will be
developed. To ensure cross-cultural validity, and control for response bias, anchoring vignettes will be
provided.
47. While PISA 2006 implemented several highly correlated scales for student beliefs on
environmental issues, only one scale for “environmental awareness” will be kept in PISA 2015. Instead,
PISA 2015 will address more general student views on science, especially – as explicated in the PISA 2015
Science Framework – “Valuing scientific approaches to enquiry”. Another innovation in the PISA 2015
science framework is the treatment of technology as a substantial and integral aspect of science literacy,
rather than a matter of “application”. The Student Questionnaire reflects this change by asking about
frequency of, commitment to, beliefs about, and perceived competence in use of technology. Finally, the
2006 measure of science-related self-concept will be taken up, now complemented by self-efficacy – a
construct that in PISA 2012 was shown to be strongly related to student performance.
Module 7: Student SES, family and home background
48. In order to compare equity related to social and ethnic factors across PISA cycles, the PISA
consortium (Core 6) staff intend to keep measures of socio-economic status and other background
variables basically unchanged. However, the measures of home possessions will be updated, experience
with the new international coding system for occupations (ISCO) from PISA 2012 will be reviewed, and
the new international system of educational levels (ISCED) will be applied in PISA 2015. Furthermore,
alternative ways of asking about parental occupations in the Field Trial will be developed, because the
quality of these data may change when students need to type into a computer.
Module 9: Educational pathways in early childhood, Module 14: Parental involvement, and the Parent
Questionnaire option
49. When children enter primary school, they already differ in their language, pre-reading, and early
numeracy skills and these differences are often maintained later in life. Promoting school readiness and
better adjustment to school is hypothesised to be an efficient means of raising the achievement levels of all
children, but especially of those children who experience a lack of parental support or who grow up in
Page 14
EDU/PISA/GB(2012)28
14
disadvantaged circumstances. It has been argued that investing in early education programmes will have
large long-term monetary and non-monetary benefits. Early childhood education has been included in three
PISA cycles so far. The importance of pre-school quality has been acknowledged and analysed by OECD
reporting as well.
50. There are good reasons to gather more information on students' early educational pathways in
PISA. However, research on biographical memory suggests that regarding the most important aspects, e.g.
type, quality, or dosage of early child care, no reliable data will be gleaned from 15-year-olds answering
retrospective questions about educational experiences they made a decade or more ago. So, while keeping
a short question on ISCED 0 attendance in the Student Questionnaire, PISA consortium (Core 6) staff –
following advice from QEG and external experts – intends to implement a series of questions in the parent
questionnaire, expecting parents to be the more reliable source of information. Those countries applying
the optional parent questionnaire will gain information on basic characteristics of the early childhood
education and care arrangements of PISA participants and reasons for attending or not attending early
childhood education and care.
51. Over the past years, the involvement of parents in educational processes has gained importance in
the educational debate, and to some extent it has also become relevant for educational policy. Parents are
not only an important audience, but powerful stakeholders in education, thus information on parents’
opinions and engagement is highly valuable also for large scale assessments like PISA. PISA already
covers important aspects of parental involvement, mostly addressed in the international questionnaire
option of the “Parent Questionnaire”. In countries participating in this option, the parents of the students
sampled for PISA provide information on their engagement that focuses on the interaction with the
institutional learning setting, i.e. teachers and schools, and their engagement that focuses on the interaction
with the student, i.e. learning support. Aspects that are covered in this questionnaire include home to
school collaboration, the learning support parents provide to their child, the availability of home resources
for learning as well as parents’ attitudes and beliefs. In PISA 2015, the aspects of home to school
collaboration and parental learning support shall be covered in more depth than in previous cycles by
adding new constructs to the parent questionnaires.
52. Independently from the parent questionnaire option, aspects of parental involvement are
addressed in other questionnaires: Teachers will be asked about their collaboration with parents, for
example through scheduled parent-teacher meetings. More importantly – because this will allow the topic
to be covered in all participating countries -, a question from PISA 2012 has been adapted to address
parental influence on students’ views of science, and new items on parental support have been developed.
Following approaches that have been successfully implemented in research, several aspects of parental
support can be discriminated: stimulation/scaffolding, emotional warmth, guidance and boundaries, and
stability. Finally, school administrators will be asked about school policies on home to school collaboration
implemented in their schools. This includes the aspect of informing parents about their child’s progress,
but also different forms of enhancing parents’ engagement in school activities.
Module 10: Domain-general student behaviour and attitudes
53. Domain-general student behaviour and attitudes refer to the set of student factors or constructs
that cut across curricular topics, or that are independent of curricula, but that are nevertheless important for
and reflective of success in education. Alternative labels that have been used in the literature are “non-
cognitive skills”, “non-intellectual correlates of GPA”, or “21st Century Skills”. Recently economic
models have included this information to better understand labour market outcomes. Meta-analyses show
that these characteristics predict educational success to a degree that is comparable to cognitive ability
measures.
Page 15
EDU/PISA/GB(2012)28
15
54. So far, ICT-related skills and attitudes, and general attitudes towards the school (including “sense
of belonging”, a measure of well-being in school) have been the only types of domain-general outcomes
addressed in PISA. The former will be taken up in the optional ICT Questionnaire, while the latter will be
included in the Student Questionnaire, providing trend information relating to previous PISA assessments.
PISA 2012 added an assessment of dropout rates – a kind of negative outcome – by school administrators,
and an assessment of some basic student preferences related to problem solving, called “industriousness”
and “openness” in the psychological literature. Also, academic self-efficacy was assessed as a very general
indicator of self-related beliefs. PISA consortium (Core 6) intends to broaden the array of student
preferences and self-related beliefs, and provide anchoring vignettes to establish cross-cultural validity. For
example, grade-related expectations are added as a new aspect of self-related beliefs, and a link to Module
19 (student assessment), which is also a high priority area in PISA 2015.
55. All non-cognitive outcomes mentioned so far are conceptually related to student achievement,
learning and schooling. However, with support from PGB and QEG, the PISA consortium (Core 6)
intends to move into broader areas of non-cognitive outcomes, namely general well-being, health, and time
use. While student behaviour and attitudes have generally been rated very high in policy priority, there
may be some doubts as to the feasibility and relevance of these specific constructs. The QEG, at its recent
meeting, acknowledged these cautions, but advised to apply these questions in the Field Trial to see how
students react, and how the respective data relates to other student, family, and school related factors. Are
well-being and health related to student achievement? Is this relationship shaped by family background
and/or school policies and school climate? Addressing such questions for analysis and reporting should be
highly relevant for educational policy making. The time use data are important for Module 12 (learning
time) as well. This question has been simplified, shortened, and focused on learning activities; it is now a
very innovative computer-based approach to the assessment of learning activities.
Module 12: Learning Time and Curriculum, and the Educational Career optional questionnaire
56. Students’ educational achievement is influenced by a wide array of opportunities to learn. On a
very broad level of abstraction, students’ opportunities to learn can roughly be described by the quantity
and the quality of learning instances. As outlined above, Module 2 will predominantly introduce concepts
of quality of education and teaching in PISA 2015 and will thus cover the quality of learning opportunities.
Module 12 and the Educational Career optional questionnaire (ECQ) then complement Module 2 by
focussing on the current quantity of learning experiences. While these constructs have been part of most
previous PISA cycles, PISA 2015 will deliver critical innovations.
57. The learning time and curriculum experienced by students in their educational career are closely
related to student outcome. On a global level, learning time and curriculum are system characteristics, yet
the intended learning opportunities need to be realised on school and on classroom level, and finally
students must avail themselves of learning time and curriculum in order to learn. A number of factors
influence this implementation in schools and in the classroom, e.g. resources such as rooms, media
devices, and textbooks as well as staff. Also all levels involved in providing and using learning
opportunities hold the possibility to lose a share of the intended learning time and learning opportunities.
This happens when schools close unscheduled (e.g. due to bad weather or safety issues) or need to
reschedule due to poor resources or problematic teacher and student behaviour; on the individual student’s
part there remain problems of truancy and inattentiveness. Taking into account the different perspectives of
intended learning opportunities on the one hand (i.e. learning time provided and intended curriculum) and
loss of opportunities due to difficulties in realisation and implementation on system, school, classroom, and
individual levels on the other hand, PISA 2015 will provide more detailed and more valid information on
learning time than previous cycles. This approach to measuring learning time as it is offered, implemented,
used or not used was highly appreciated by the QEG; its feasibility will be checked in the Field Trial.
Page 16
EDU/PISA/GB(2012)28
16
58. Besides allowing for a detailed comparison of school systems on country level, the proposed
questionnaire components provide valuable indicators to compare educational opportunities across
different subgroups of students. Social or ethnic disparities in access to learning opportunities can thus be
analysed. The approach used in Module 12 will enable differentiation between disparities in provision and
use of educational opportunities.
59. Students’ learning does not only relate to education in schools alone. Achievement level at the
age of fifteen is determined by all synchronic and diachronic learning experiences. In recent years, more
and more non-school learning instances have gained importance worldwide. Besides looking only at
opportunities to learn in school, PISA needs to ask for time spent in other kinds of educational settings. In
previous cycles, a certain problem has always been to disentangle regular (mandatory) and additional
lessons, private supplementary lessons and private additional study, in- and out-of-school learning
experiences. Therefore, Module 12 will expand the assessment of learning time, including loss of intended
time, to some out-of-school environments. However, detailed coverage of such context goes beyond the
scope of the standard Student Questionnaire. Therefore, with strong support from the QEG, the PISA
consortium (Core 6) has added a focus on supplementary learning opportunities to the Educational Career
optional questionnaire (EC Option). Module 12 and the EC Option seamlessly connect to each other
providing a basic understanding of intended and used learning opportunities in school and supplementary
to school, in curricular and non-curricular domains. PISA 2015 will therefore provide insight in a large
array of learning instances tapping the most relevant sources of education in the students’ lives.
Module 15: School leadership and management
60. The PISA measure of educational leadership has seen a major revision in the 2012 study. Core 6
will carefully analyse the results and revise the measure if needed. An additional question focusing on
instructional leadership has been recommended by the QEG. The new optional Teacher Questionnaire will
be used to gather information on instructional and transformational leadership from teachers as well,
because research has shown that teachers’ perspective on leadership can differ from the positions held by
school administrators.
61. The PISA 2015 Field Trial will also address specific school policies in various areas of school
management and school improvement. Several new measures are spread across a number of Modules,
assessing policies on learning time and the school curriculum (Module 12), teacher recruitment and
professional development (Module 1), internal as well as external evaluation and teacher accountability
(Module 19), parental involvement (Module 14), and multicultural education (Module 8). Altogether, these
measures allow to draw a multi-faceted picture of key management activities.
Module 19: Assessment, evaluation, and accountability
62. Research has shown that assessment-related policies and practices are among the most influential
processes on the classroom, the school, and the system level. Educational reforms in many countries are
centred on assessment, evaluation and accountability. Instruments like performance standards, standard-
based assessment, centralised exams, internal and external evaluation, annual reports on student progress,
and school inspectorates have been promoted and implemented across continents. Analysing their
relationships with other characteristics of schools can add to our understanding of school policies and
student outcomes. The PISA questionnaires can help to describe assessment practices, to compare them
between countries, and to study their relationship with school level variables.
63. This module has received the highest rating in priority from the PGB members – probably
because the issue is very prominent in OECD reports on PISA, but still weakly represented in the
questionnaire design itself. In the most recent School Questionnaire (PISA 2012), student assessment
Page 17
EDU/PISA/GB(2012)28
17
practices and their purposes were covered, as well as school evaluation, teacher evaluation, and
accountability policies and practices. The very broad notion of “standardised tests” is used here – whether
these tests are high stakes or low stakes, local or national, etc.
64. Module 19 has been designed to draw a more sophisticated picture. Therefore, the module
systematically discriminates student assessment, which mostly takes place in classrooms, from evaluation,
which takes place at the school level. For both levels, information is gathered on (a) frequency of using
different assessment practices, (b) purposes and consequences of the assessment. While the questions on
student assessment focus on formative assessment and feedback (also called “Assessment for learning”),
the questions on evaluation mainly deal with internal rather than external evaluation, because research has
shown formative assessment and internal evaluation to have larger impact on student learning and school
development than summative assessment and external evaluation, respectively. Information from different
perspectives (student, teachers, administrators) will be combined. Results will also be important for
Modules 2 (teaching) and 15 (school leadership and management).
3. Questionnaire design and rotation in the Main Study StQ
3.1 Designing the PISA 2015 Field Trial
65. The full material which is proposed for the Field Trial is documented in the Appendix. Some
further changes to this material can be expected based on:
cognitive labs and focus group studies which are currently underway,
feedback from QEG (at its virtual meetings in December, 2012), from NPMs (through the web-
based forum which will open in January, 2013), from the Secretariat, and of course from the
PGB,
and further revisions made by the PISA consortium (Core 6) staff, questionnaire developers and
contracted experts.
66. A number of single items or questions may be changed or dropped as a result of these
deliberations. However, the overarching modular design of the questionnaires – as explained in section 1 of
this paper - and the framework of policy issues and constructs developed over the past year – as elaborated
in section 2 – will remain unchanged. Development of the constructs listed in the appendix has been based
on policy priorities defined by the participating countries, results from previous PISA studies, and an
extensive review of relevant research literature.
67. The material documented in the appendix altogether covers an estimated assessment time of 97
minutes for the FT Student Questionnaire, 55 minutes for the FT School Questionnaire, 27 minutes for the
FT Parent Questionnaire, 60 minutes for the FT Teacher Questionnaire, and about 18 minutes each for the
FT Educational Career Questionnaire and 14 minutes for the FT ICT Familiarity Questionnaire. In all
cases, the amount of material adheres to the time estimates for Field Trial instruments outlined in the
Integrated Design (see Document submitted by Core 3 and Core 6). This is possible because the Field Trial
design – in line with practices established in all previous PISA studies – will allow for different booklets
randomly being allocated to subgroups of students (for Student Questionnaire, Educational Career
Questionnaire option, and ICT Questionnaire option) and Teachers (for the Teacher Questionnaire option).
No person will have to sit on the questionnaire above time limits.
68. By field trialling this material, the following goals of questionnaire development in PISA can be
met:
Page 18
EDU/PISA/GB(2012)28
18
allowing for broad coverage of topics in all 19 modules;
assessing high priority modules with more material to provide deeper insights;
establishing close fit to the PISA 2015 major domain of assessment, namely science;
reviewing and - if feasible - reusing or revising any relevant material from earlier PISA cycles,
especially from PISA 2006, (a) to make efficient use of earlier translation and adaptation efforts,
and (b) to establish trend in reporting on context variables and non-cognitive outcomes;
trying out new, innovative questions and question formats – including interactive formats,
context-dependent questions, and formats intended to secure cross-cultural comparability, such as
anchoring vignettes;
ensuring feasibility of computer-based delivery for all previously established material.
69. The breadth of Field Trial material will allow for selecting the most appropriate content for the
Main Study. Introducing new, innovative content and format always bears the risk of not working on a
large scale, even if these innovations are based on sound research experience. This especially holds true
when a new mode of delivery (here: computer-based assessment) is being introduced at the same time.
From a policy point of view, using larger amounts of material in the Field Trial provides more degrees of
freedom for policy makers (PGB) when selecting Main Study content based on the Field Trial data.
3.2 Time restrictions and design implications for the Main Study
70. The Main Study will see much stronger time restrictions than the Field Trial (see Integrated
Design): approximately 30 minutes for the Student Questionnaire, 45 minutes for the School
Questionnaire, 30 minutes each for the Teacher and the Parent Questionnaire, and 10 minutes each for the
Educational Career and the ICT options. Thus, severe cuts will be made from the Field Trial to the Main
Study, especially regarding the two standard instruments, School and Student Questionnaire.
71. In preparation for this reduction exercise, the amount of material that might be taken up from
previous cycles has been estimated (cf. footnote in the Appendix). This hypothetical selection was based
on the PGB’s priorities, discussion at the QEG and a review of existing PISA reports (as in Table 1 above).
This selection of well established questions amounts to about 30 minutes for the Student Questionnaire,
and about 22 minutes in the School Questionnaire. Regarding the School Questionnaire, there is ample
space left for new, innovative content. Regarding the Student Questionnaire, however, these estimates
underpin the importance of finding ways of optimising testing time. Otherwise, the PISA 2015 Student
Questionnaire would be left without any innovation and improvement; neither in content nor in measures.
72. PISA 2012 introduced a rotated booklet design for the student questionnaire, with an eight
minutes “core” set of questions covering student and family background (in our terminology: Modules 7
and 8), while all other content was given to 2/3 of the students only. An estimated 33 minutes of non-core
content were formed into three clusters (A, B, C), and each student was randomly assigned to a
combination of two of these clusters (AB, BC, or CA). Random assignment of students to booklets ensured
that parameters reported for questionnaire constructs – like means, standard deviations, and percentiles –
would be unbiased. Altogether, the rotation scheme allowed for coverage of approximately 40 minutes of
estimated assessment time in the Student Questionnaire.
73. The test design has become more complex in PISA 2015. The computer-based tests will assess
four domains (science, reading, mathematics, and collaborative problem solving), while most individual
Page 19
EDU/PISA/GB(2012)28
19
students will be working on two of these domains only. These settings require more student scores (namely
50 per cent of all student scores) to be estimated (“imputed”) from achievement in other domains and
background information, thus putting high burden on the background information produced from
questionnaires. The suitability of a rotated design as used in PISA 2012 has been called into question
because of this increase in complexity for the assessment in PISA 2015.
74. Therefore the recommendation from the PISA 2015 Consortium and the QEG is that there should
be only one version of the student questionnaire in the PISA 2015 main study and no rotation, if enough
innovative material can be included. The results of the field trial would be used - taking advantage of the
more accurate information on timing which will be obtained from the computer-based administration - to
determine how much time will be needed to cover both established and innovative content in the main
study.. Furthermore, the time available for the administration of the student questionnaire in the main study
for 2015 might need to be revised based on the results of the field trial. The analysis of PISA 2012 data,
especially a careful evaluation of the rotation design applied, will also be considered when determining the
design of the background questionnaires in the PISA 2015 main study.
Page 20
EDU/PISA/GB(2012)28
20
REFERENCES
Almlund M., A.L. Duckworth, J.J. Heckman and T. Kautz (2011), "Personality Psychology and
Economics," IZA Discussion Papers 5500, Institute for the Study of Labour (IZA), Bonn.
Barnett, W.S. (2008), Preschool Education and its Lasting Effects: Research and Policy Implications,
Education and the Public Interest Center & Education Policy Research Unit, Boulder and Tempe,
http://epicpolicy.org/publication/preschooleducation, retrieved 27 September 2012.
Blau D. and J. Curie (2006), "Pre-School, Day Care, and After-School Care: Who's Minding the Kids?" in
Handbook of the Economics of Education, Elsevier, North Holland, pp. 1164-1278.
Bryk, A.S., P. Sebring, E. Allensworth, S. Luppescu and J. Easton (2010), Organizing School for
Improvement, Lessons from Chicago, The University of Chicago.
Creemers, B.P.M. and L. Kyriakides (2008), The Dynamics of Educational Effectiveness: A Contribution
to Policy, Practice and Theory in Contemporary Schools, Routledge, London.
Cunha, F., J.J. Heckman, L.J. Lochner and D.V. Masterov (2006), "Interpreting the Evidence on Life Cycle
Skill Formation,", in E. Hanushek and F. Welch (eds.), Handbook of the Economics of Education,
Elsevier, North Holland, pp. 697-812.
Hanushek, E.A. and L. Wößmann (2011), “The Economics of International Differences in Educational
Achievement”, in E.A. Hanushek, S. Machin and L. Wößmann (eds.), Handbook of the Economics
of Education, Vol. 3, North Holland, Amsterdam, pp. 89–200.
Heckman J.J., J. Stixrud and S.Urzua (2006), "The Effects of Cognitive and Noncognitive Abilities on
Labor Market Outcomes and Social Behavior", Journal of Labor Economics, University of Chicago
Press, Vol. 24, 3, pp. 411-482.
OECD (2007), Science Competencies for Tomorrow's World, Executive Summary, OECD Publishing,
Paris.
OECD (2009),” PISA 2009 Questionnaire Framework”, in PISA 2009 Assessment Framework: Key
Competencies in Reading, Mathematics and Science, OECD Publishing, Paris, pp. 149-169.
OECD (2010), PISA 2009 Results: What Students can do? Student Performance in Reading, Mathematics
and Science, Vol. 1, OECD Publication, Paris.
Sammons, P., K. Sylva, E. Melhuish, I. Siraj-Blatchford, B. Taggart, K. Toth, D. Draghici. and R. Smees
(2011), Effective Pre-School, Primary and Secondary Education Project (EPPSE 3-14): Influences
on Students’ Attainment and Progress in Key Stage 3: Academic Outcomes in English, Maths and
Science in Year 9, Institute of Education, University of London / Department for Education, London.
Page 21
EDU/PISA/GB(2012)28
21
Scheerens, J. and R.J. Bosker (1997). The Foundations of Educational Effectiveness, Pergamon Press,
Oxford.
Sykes G., B. Schneider and D.N. Plank (eds.) (2009), Handbook of Educational Policy Research,
Routledge, New York.
Wößmann, L., E. Lüdemann, G. Schütz and M.R. West (2007), “School Accountability, Autonomy, Choice,
and the Level of Student Achievement: International Evidence from PISA 2003”, OECD Education
Working Paper No. 13.
Page 22
EDU/PISA/GB(2012)28
22
APPENDIX: DRAFT OVERVIEW OF FIELD TRIAL CONTENT
Module Policy issue and construct area Specific construct to be
measured Status of the measure
Estimated
Time
(sec)
Instrument
1 Teacher qualification and professional knowledge
1 Teacher background Gender new to PISA, from TALIS 2013 10 TQ Core
1 Teacher background Age new to PISA, from TALIS 2013 10 TQ Core
1 Initial education Level of education new to PISA 15 TQ Core
1 Initial education
Teacher education or training
programme: completion, mode of
qualification, and duration
new to PISA 60 TQ Core
1 Initial education Subjects studied, subjects
teaching
new to PISA 100 TQ Core
1 Teacher background Workplace selection new to PISA 120 TQ Core
1 Teacher background Employment status new to PISA, from TALIS 2013 10 TQ Core
1 Teacher background Job experience – number of
schools new to PISA 10 TQ Core
1 Teacher background Job experience – number of years new to PISA, from TALIS 2013 20 TQ Core
1 Teacher background Teaching modal grade new to PISA 10 TQ Core
1 Professional development Professional development
activities new to PISA 110 TQ Core
1 Professional development
Obligation to participate in
professional development
activities
new to PISA 10 TQ Core
1 Professional development Support for professional
development activities new to PISA 20 TQ Core
1 Professional development
Content distribution for initial
Education and professional
development Activities
new to PISA 140 TQ General
Page 23
EDU/PISA/GB(2012)28
23
1 Professional development Teacher cooperation - general new to PISA 80 TQ General
1 Professional knowledge and teacher
beliefs
Self-efficacy in classroom
management new to PISA 40 TQ General
1 Professional knowledge and teacher
beliefs
Self-efficacy in maintaining
positive relations with students new to PISA 40 TQ General
1 Professional knowledge and teacher
beliefs Performance vs. needs orientation new to PISA 60 TQ General
1 Non-cognitive factors Job satisfaction new to PISA 100 TQ General
1 Initial education Content distribution for initial
education new to PISA 50 TQ Science
1 Professional development Science specific teacher
collaboration new to PISA 220 TQ Science
1 Professional development
Content distribution for
professional development
activities
new to PISA 50 TQ Science
1 Professional knowledge and teacher
beliefs
Self-efficacy related to science
content new to PISA 80 TQ Science
1 Professional knowledge and teacher
beliefs
Self-efficacy related to teaching
science content new to PISA 80 TQ Science
1 Non-cognitive factors Teacher enthusiasm (vignettes) new to PISA 80 TQ Science
1 Non-cognitive factors Teacher enthusiasm (scales) new to PISA 190 TQ Science
1 Initial education Initial education of all teaching
staff
previous (2000) (2003) (2006)
(2009) (2012) 120
* ScQ
1 Initial education Requirement for teacher
education or training programme new to PISA 20 ScQ
1 Initial education Teacher recruitment policies new to PISA 20 ScQ
1 Professional development Professional development policy new to PISA 150 ScQ
1 Professional development Participation in professional
development 2000, (2012) 20 ScQ
1 Non-cognitive factors Perceived teacher engagement new to PISA 70 StQ
2 Science teaching practices
2 Teaching practices
Science teaching: Inquiry-based –
(4 dimensions: procedural,
epistemic, social, and real life
applications)
previous (2006) and extended 130* StQ
2 Teaching practices Science teaching: direct
instruction changed (2012) 70 StQ
2 Teaching practices Science teaching: Opportunities new to PISA 120 StQ
Page 24
EDU/PISA/GB(2012)28
24
to learn
2 Teaching quality Teacher support previous (2000, 2003, 2012) 50* StQ
2 Teaching quality Perceived support for competence new to PISA 40 StQ
2 Teaching quality Perceived support of autonomy new to PISA 40 StQ
2 Teaching quality Disciplinary climate previous (2000, 2003, 2009, 2012) 50* StQ
2 Teaching quality Cognitive activation changed (2012) 70 StQ
2 Teacher perception Concepts of inquiry based science
teaching new to PISA 160 TQ Science
2 Teacher perception Science teaching practices new to PISA 120 TQ Science
3
School learning environment for science
(For other aspects of the school environment, see modules 8, 12, 13,
15, and-19)
3 Science teaching staff Number of teachers per science
area new to PISA 120 ScQ
3 Policy and actions taken to improve
teaching/
Value of science education in
school community new to PISA 80 ScQ
3 Policy and actions taken to improve
teaching/ Science-related collaborations new to PISA 40 ScQ
3 Policy and actions taken to improve
teaching/ Science-specific resources new to PISA 60 ScQ
4 Science-related outcomes: attitudes, motivation, value beliefs,
strategies
4 Motivation Interest in school science subjects changed (2006) 40* StQ
4 Motivation Interest in broad science topics changed (2006) 160* StQ
4 Motivation Anchoring Vignette Interest in
Science 60 StQ
4 Motivation Instrumental motivation changed (2006), previous (2012) 40* StQ
4 Beliefs about science General and personal value of
science previous (2006) 100 StQ
4 Beliefs about science Occupational prestige new to PISA 80 StQ
4 Beliefs about science Dealing with uncertainty and
ambiguity new to PISA 100 StQ
4 Beliefs about science Valuing scientific approaches to
enquiry new to PISA 80 StQ
4 Beliefs about science Students' view on science new to PISA 40 StQ
4 Self-related cognitions Self-Concept previous (2006) 50 StQ
4 Self-related cognitions Self-Efficacy new to PISA 80 StQ
4 Use of technology Use of technology: frequency new to PISA 90 StQ
Page 25
EDU/PISA/GB(2012)28
25
4 Use of technology Use of technology: perceived
competence new to PISA 50 StQ
4 Use of technology Use of technology: commitment new to PISA 100 StQ
4 Use of technology Use of technology: beliefs new to PISA 100 StQ
4 Environmental awareness Awareness of environmental
issues previous (2006) 70
* StQ
5 Out-of-school science experience
(see also module 12)
5 Out-of-school science experience Leisure time science activities
media related changed (2006) 60
* StQ
5 Out-of-school science experience Extra-curricular science activities
in school previous (2006) 110 ScQ
5 Out-of-school science experience Science activities at age 10 previous (2006) 50 PQ
6 Science career
6 Career aspiration Broad science aspiration previous (2006) 40* StQ
6 Career aspiration Specific career aspiration at age
30 previous (2006) 20 StQ
6 Students own preparation Student information on science
careers previous (2006) 40 StQ
7 Student SES, family and home background
7 Student background Grade PISA 2012 10* StQ
7 Student background Educational programme PISA 2012 10* StQ
7 Student background Age PISA 2012 10* StQ
7 Student background Sex PISA 2012 10* StQ
7 Family background Household members PISA 2012 40* StQ
7 Parental background Mother's profession PISA 2012 40* StQ
7 Parental background Mother's school education PISA 2012 20* StQ
7 Parental background Mother's tertiary education PISA 2012 20* StQ
7 Parental background Mother's employment status PISA 2012 20* StQ
7 Parental background Father's profession PISA 2012 40* StQ
7 Parental background Father's school education PISA 2012 20* StQ
7 Parental background Father's tertiary education PISA 2012 20* StQ
7 Parental background Father's employment status PISA 2012 20* StQ
7 Migration background Migration background PISA 2012 30* StQ
7 Migration background Age at immigration PISA 2012 10* StQ
7 Migration background Language spoken at home PISA 2012 30* StQ
7 Family background Home possessions - existence PISA 2012 120* StQ
7 Family background Home possessions - number PISA 2012 50* StQ
Page 26
EDU/PISA/GB(2012)28
26
7 Family background Number of books PISA 2012 20* StQ
7 Parental background Number of children in household previous (2009) 7 PQ
7 Parental background Age of parents previous (2006, 2012) 10 PQ
7 Parental background Parents' qualification (ISCED) previous (2006, 2009, 2012) 20 PQ
7 Parental background Parental occupation previous (2006, 2012) 60 PQ
7 Parents' background Spending on educational services previous (2006, 2009, 2012) 40 PQ
7 Parents' background Household income previous (2006, 2009, 2012) 15 PQ
7 Parents' background Home resources previous (2009) 20 PQ
7 ICT Availability of ICT at home previous (2009, 2012) 60 ICT
7 ICT General use of ICT previous (2012) 50 ICT
7 ICT Use of ICT outside of school (for
leisure activities) previous (2012) 60 ICT
7 ICT Use of ICT outside of school (for
school work activities) previous (2012) 60 ICT
8 Ethnicity and Migration
8 Ethnicity and Migration Cultural diversity of school new to PISA 30 TQ General
8
Multicultural school
culture/Concrete policies and
practices
Multicultural education new to PISA 50 TQ General
8
Multicultural school
culture/Concrete policies and
practices
Multicultural education new to PISA 50 ScQ
8 Multicultural school culture Basic assumptions on diversity new to PISA 80 ScQ
8 Multicultural school culture Basic assumptions on diversity new to PISA 80 TQ General
8 Multicultural school culture School intercultural climate new to PISA 60 StQ
8 Perceived discrimination in society Membership in a minority group new to PISA 20 StQ
8 Perceived discrimination in society Perceived reasons for
discrimination new to PISA 30 StQ
8 Migration background Language preferences for
communication previous (2012) 40 StQ
8 Parental involvement Barriers to parental involvement new to PISA 60 PQ
8 Parent’s migration background Country of birth previous (2012) 60 PQ
8 Parent’s migration background Citizenship previous (2012) 20 PQ
9 Educational pathways in early childhood (and thereafter)
9 Prior to school entry ISCED 0 attendance previous (2003), (2009), (2012) 10* StQ
9 Education biography School enrolment previous (2003), (2009), (2012) 10* StQ
9 After school entry Repeating grades previous (2003), (2009), (2012) 30* StQ
9 After school entry Skipping grades new to PISA 30 StQ
Page 27
EDU/PISA/GB(2012)28
27
9 School enrolment School enrolment relative to
mandatory school starting age new to PISA 30 PQ
9 School entry Age at school entry new to PISA 10 PQ
9 Prior to school entry Educational biography new to PISA 70 PQ
9 Education biography ECEC attendance prior to ISCED
1 new to PISA 10 PQ
9 Education biography Age at first ECEC attendance new to PISA 10 PQ
9 Education biography Type of predominant ECEC
arrangement new to PISA 30 PQ
9 Education biography ECEC dosage new to PISA 10 PQ
9 Education biography ECEC type of care arrangement new to PISA 20 PQ
9 Education biography ECEC reasons for attendance new to PISA 40 PQ
9 Education biography ECEC reasons for not attending new to PISA 80 PQ
9 Education biography Reasons for ECEC setting choice new to PISA 100 PQ
9 Education biography Change of school in ISCED 1 previous (2003, 2009) 15 EC
9 Education biography Change of school in ISCED 2 previous (2003, 2009) 20 EC
9 Education biography Change of school in ISCED 3 previous (2003, 2009) 20 EC
9 Education biography Change of study program in
ISCED 2 previous (2003) 20 EC
9 Education biography Change of study program in
ISCED 3 previous (2003) 20 EC
10 Domain-general student behaviour and attitudes
10 Student preferences and study skills Perseverance previous (2012) 30* StQ
10 Student preferences and study skills Industriousness new to PISA 30 StQ
10 Student preferences and study skills Organization new to PISA 60 StQ
10 Student preferences and study skills Procrastination new to PISA 60 StQ
10 Student preferences and study skills Openness (for Problem Solving) previous (2012) 50* StQ
10 Student preferences and study skills Anchoring Vignettes for student
preferences new to PISA 90 StQ
10 Self-related beliefs Test Anxiety new to PISA 60 StQ
10 Self-related beliefs Strategic Approach to Learning new to PISA 50 StQ
10 Self-related beliefs Academic Self Efficacy previous (2012) 30* StQ
10 Self-related beliefs Performance Self Efficacy new to PISA 30 StQ
10 Self-related beliefs Anchoring vignettes for self-
related scales new to PISA 120 StQ
10 Self-related beliefs Grade Goal and current grades new to PISA 15 StQ
10 Attitudes towards school Attitudes towards School -
Learning Activities previous (2012) 40
* StQ
Page 28
EDU/PISA/GB(2012)28
28
10 Attitudes towards school Attitudes towards School -
Learning Outcomes previous (2012) 30
* StQ
10 well-being General Life Evaluation new to PISA 60 StQ
10 well-being Life Satisfaction new to PISA 50 StQ
10 well-being Anchoring vignettes new to PISA 60 StQ
10 well-being Well-being at school/Sense of
belonging previous (2012) 90
* StQ
10 Health Over-/Underweight new to PISA 30 StQ
10 Health Physical Activities new to PISA 40 StQ
10 Health Nutrition new to PISA 60 StQ
10 Health Health-related school practices new to PISA 120 StQ
10 time use Sleeping Time new to PISA 20 StQ
10 time use Time spent at school new to PISA 20 StQ
10 time use
shortened day reconstruction time
protocol focused on
study/learning-related activities
new to PISA 200 StQ
10 Idea generation Consequences new to PISA 180 StQ
10 School dropout Number of dropouts previous (2012) 20* ScQ
10 ICT Engagement ICT Interest new to PISA 120 ICT
10 ICT Engagement Perceived ICT Competence new to PISA 120 ICT
10 ICT Engagement Perceived Autonomy related to
ICT Use new to PISA 120 ICT
10 ICT Engagement ICT as a topic in Social
Interaction new to PISA 120 ICT
11 Student dispositions related to collaborative problem solving
11 Collaboration and teamwork
dispositions
Teamwork dispositions:
Cooperate, guide and negotiate new to PISA 100 StQ
11 Experience related to collaborative
problem solving
In-school experience in
collaboration-related activities new to PISA 55 StQ
11 Experience related to collaborative
problem solving
Out-of-school experience in
collaboration-related activities new to PISA 25 StQ
12 Learning time and curriculum
12 Allocated learning time Instructional weeks per year new to PISA 20 ScQ
12 Allocated learning time Instructional days per year new to PISA 20 ScQ
12 Loss of learning time Loss on national/regional level new to PISA 40 ScQ
12 Loss of learning time Loss on school level new to PISA 60 ScQ
12 Provided additional learning time Additional learning time,
availability previous (2012) 10
* ScQ
Page 29
EDU/PISA/GB(2012)28
29
12 Provided additional learning time Additional learning time, purpose previous (2012) 30 ScQ
12 Provided additional learning time Additional other study support previous (2000) 20 ScQ
12 School policies School policies on additional
lessons new to PISA 10 ScQ
12 School policies School policies on additional
tutoring lessons content new to PISA 30 ScQ
12 School policies Handling of additional tutoring in
the school/school district new to PISA 80 ScQ
12 Students’ learning time Number of school lessons in
maths, science, and test language previous (2000-2012) 30
* StQ
12 Students’ learning time Learning time per science subject new to PISA 100 StQ
12 Students’ learning time All lessons per week previous (2012) 10* StQ
12 Students’ learning time Minutes per lesson previous (2012) 30* StQ
12 Loss of learning time Loss on classroom level new to PISA 10 StQ
12 Loss of learning time
Loss on individual level in
regular school lessons: within
lessons, whole lessons, full
school days
new to PISA 60 StQ
12 Loss of learning time Reasons for loss on individual
level new to PISA 10 StQ
12 Students’ additional learning time Amount of additional lessons and
study time previous (2000-2012) 180
* StQ
12 Students’ additional learning time
Loss on individual level in
regular school lessons: within
lessons, whole lessons, full
school days
new to PISA 50 StQ
12 Intended Science Curriculum Content: Components described
in the science curriculum new to PISA 30 TQ Science
12 Intended Science Curriculum Content: Approaches and
processes new to PISA 80 TQ Science
12 Curriculum coherence Transfer across classes and grades new to PISA 30 TQ Science
12 Textbook use Focus of curriculum content new to PISA 20 TQ Science
12 Realized curriculum Science courses taken new to PISA 50 StQ
12 Curriculum choice Choice in science courses new to PISA 20 StQ
12 Additional learning time Number of hours previous (2006) 180 EC
12 Additional learning time Content of additional lessons new to PISA 10 each EC
12 Additional learning time Type of additional lessons new to PISA 60 EC
12 Additional learning time Location of additional lessons new to PISA 50 EC
Page 30
EDU/PISA/GB(2012)28
30
12 Additional learning time Teacher in additional lessons new to PISA 40 EC
12 Comparing regular school and
additional lessons
Cognitive stimulation, adaptivity
of teaching new to PISA 150 EC
12 Comparing regular school and
additional lessons
Structuring content and classroom
management new to PISA 100 EC
12 Comparing regular school and
additional lessons Teacher-student relationship new to PISA 50 EC
12 Additional learning time Reasons for additional lessons new to PISA 80 EC
12 Additional learning time Reasons for no additional lessons new to PISA 100 EC
12 Education biography History of private supplementary
tutoring new to PISA 30 EC
13
School climate: Interpersonal relations, trust, expectations
(see also modules 2 for classroom climate and 14 for parental
involvement)
13 Teachers and students Student teacher relation at school previous (2000 ff.) with new items 115* StQ
13 Teachers and students Achievement pressure by teachers previous (2000) 40 StQ
13 Students Bullying new to PISA 70 StQ
13 Teacher and students Fairness new to PISA 60 StQ
13 Staff Teacher morale previous (2000, 2003, 2012) 40* ScQ
13 Staff/Students Teacher and student behaviour
affecting school climate previous (2012) 190
* ScQ
13 Teachers and students Student-teacher relation at school adapted from previous StQ 90 TQ
General
14 Parental involvement
14 Parental background Respondent previous (2006, 2009, 2012) 5 PQ
14 Parents' view on science Parents' view on science skills in
the job market previous (2006, 2012) 35 PQ
14 Parents' view on science Parent and peer influence in
student's view of science changed (2012) 60 StQ
14 School choice Perceived school quality previous (2006, 2009, 2012) 45 PQ
14 Home-to-school-collaboration
Parental engagement that focuses
on interaction with the
institutional learning setting
previous (2009, 2012) and new to
PISA 80 PQ
14 Home-to-school-collaboration Topics of parent-teacher
communication new to PISA 60 PQ
14 Home-to-school-collaboration Modes of parent-teacher
communication new to PISA 25 PQ
14 Home-to-school-collaboration Parents' satisfaction with school previous (2009) 120 PQ
Page 31
EDU/PISA/GB(2012)28
31
14 Home-to-school-collaboration Parents'motivation for
participation new to PISA 45 PQ
14 Home-to-school-collaboration Parents' social relationship and
involvement new to PISA 25 PQ
14 Home-to-school-collaboration
Parental engagement that focuses
on interaction with the
institutional learning setting
previous (2012) 80 ScQ
14 Home-to-school-collaboration Parental expectations towards
school previous (2006, 2009, 2012) 30 ScQ
14 Home-to-school-collaboration School policies for parental
involvement new to PISA 70 ScQ
14 Home-to-school-collaboration Topics of parent-teacher
communication new to PISA 60 TQ General
14 Home-to-school-collaboration Modes of parent-teacher
communication new to PISA 25 TQ General
14 Parental support
Emotional warmth, guidance and
boundaries,
stimulation/scaffolding and
stability
new to PISA 140 PQ
14 Parental support Parental support for learning at
home
previous (2009, 2012) focussing on
reading and mathematics 60 PQ
14 Parental support
Child's past science activities and
parental support for learning at
home
previous (2006) 35 PQ
14 Parental support
Emotional warmth, guidance and
boundaries,
stimulation/scaffolding and
stability
new to PISA 110 StQ
14 Parental support Help with homework or
additional study at home new to PISA 80 EC
14 Parental support Parents’ view on science careers previous (2006, 2012 for
mathematics) 35 EC
15
Leadership and school management
(For specific aspects of school management, see also modules 3, 8,
12, and 16-19)
15 Instructional leadership Leadership tasks new to PISA 50 ScQ
15 School management Educational leadership scale 2009(14), 2012(21) 135* ScQ
15 School policies and practices Curriculum 2012(3) 30 ScQ
Page 32
EDU/PISA/GB(2012)28
32
15 Use of ICT Resources ICT usage previous (2012) 25* ScQ
15 Transformational leadership Teachers' view on leadership new to PISA 70 TQ General
15 Instructional leadership Teachers' view on leadership new to PISA 110 TQ General
16 Resources
(see also module 3 for science-specific resources)
16 Structure and organisation of the
school Public or private school
2000(1), 2003(1), 2006(1), 2009(1),
2012(1) 10
* ScQ
16 Structure and organisation of the
school
Type of school location
community
2000(1), 2003(1), 2006(1), 2009(1),
2012(1) 15
* ScQ
16 Structure and organisation of the
school Number of students
2000(2), 2003(2), 2006 (2), 2009
(2), 2012(2 15
* ScQ
16 Structure and organisation of the
school Source of resources previous (2006), previous (2012) 40
* ScQ
16 Teaching staff Vacant positions previous (2006) 30 ScQ
16 Class size Average size of classes 2006(1), 2012(1) 11* ScQ
16 ICT ICT-Equipment previous (2012) 30* ScQ
16 Resources Problems due to little resources previous (2012) 130 ScQ
16 ICT Availability of ICT at school previous (2009, 2012) 60 ICT
16 ICT Use of ICT at school previous (2009, 2012) and new to
PISA 90 ICT
17 Locus of decision making within the school system
17 Decision making within the school
system Locus of decision making previous (2006, 2009, 2012) 120* ScQ
18 Admission, selection, and choice
18 Grade retention Policies on grade retention new to PISA 100 ScQ
18 Grouping Grouping policies previous (2006) 20 ScQ
18 Choice Competition between schools previous (2012) 30* ScQ
18 Choice Parental school choice previous (2009), previous (2012) 100 PQ
18 Choice Availability of schools previous (2009, 2012) 25 PQ
19 Admission Student admission policies previous (2000- 2012) 60* ScQ
19 Assessment, evaluation, and accountability
19 Student assessment/Instruments General assessment practice previous (2000, 2003, 2009, 2012) 60* ScQ
19 Student assessment/Instruments Classroom assessment
instruments new to PISA 80 TQ General
19 Student assessment/Instruments Teachers' grading practice new to PISA 170 TQ General
19 Student assessment/Use Adaption of instruction new to PISA 30 StQ
19 Student assessment/Use Adaption on instruction new to PISA 30 TQ Science
19 Student assessment/Use Perceived feedback new to PISA 50 StQ
Page 33
EDU/PISA/GB(2012)28
33
19 Student assessment/Use Source of feedback new to PISA 90 StQ
19 Student assessment/Use Use of feedback to guide learning new to PISA 60 StQ
19 Student assessment/Use Purpose of assessment results previous (2000, 2003, 2009, 2012) 100* ScQ
19 School evaluation/Norms Evaluation policies previous (2012) 100* ScQ
19 School evaluation/Norms Policies on teacher accountability new to PISA 30 ScQ
19 School evaluation/Instruments Foci of internal evaluation new to PISA 130 ScQ
19 School evaluation/Instruments Foci of internal evaluation new to PISA 130 TQ General
19 School evaluation/Instruments Teacher evaluation previous (2000, 2003, 2009, 2012) 40* ScQ
19 School evaluation/Instruments Teacher evaluation new to PISA 40 TQ General
19 School evaluation/Use Processes on internal evaluation new to PISA 70 ScQ
19 School evaluation/Use Processes on internal evaluation new to PISA 70 TQ General
19 School evaluation/Use Consequences of internal
evaluation new to PISA 130 ScQ
19 School evaluation/Use Consequences of internal
evaluation new to PISA 130 TQ General
19 Accountability Use of achievement data for
accountability previous (2006, 2009, 2012) 30
* ScQ
19 Accountability Teacher incentives previous (2012) 70* TQ General
19 School evaluation/Use Processes on external evaluation new to PISA 50 ScQ
*)
Hypothetical selection of material that might be taken up from previous cycles