BUILDING EDUCATIONAL INDICATORS FOR INTERNATIONAL COMPARISON Camila de Moraes - OCDE Brasília-DF | Junho 2018
BUILDING
EDUCATIONAL
INDICATORS FOR
INTERNATIONAL
COMPARISONCamila de Moraes - OCDE
Brasília-DF | Junho 2018
• Introduction
• Comparable raw data
• Comparable indicators
• Conclusion
OUTLINE
Introduction
• OECD mission: provide a forum in which governments can work together to
share experiences and seek solutions to common problems. An important part of
this work is building robust, comparable datasets that will help countries share
best practices.
• Education at a Glance: annual flagship publication on the state of education
around the world. Comparability is our main value-added.
• The report covers all 35 OECD countries and a number of partner countries
(Argentina, Brazil, China, Colombia, Costa Rica, India, Indonesia, Lithuania, the
Russian Federation, Saudi Arabia and South Africa).
Building comparable indicators
What do we want to measure?
Which indicator should be used to measure it and how should it be calculated?
What data do we need and how comparable is it?
Building comparable indicators
What do we want to measure?
What data do we need and how comparable is it?
Which indicator should be used to measure it and how should it be calculated?
Building comparable indicators
What do we want to measure?
What data do we need and how comparable is it?
Which indicator should be used to measure it and how should it be calculated?
1. Raw data
2. Indicator
development
Building comparable indicators: our general principles
• Which does not always mean comparability in the raw data collection
Main goal is indicator comparability
• Trade-off between stricter comparability and more data available
• Trade-off between the “most refined” and the “most comparable” data
Comparability implies trade-offs
• Must accept a degree of flexibility
100% comparability does not exist
• Introduction
• Comparable raw data
• Comparable indicators
• Conclusion
OUTLINE
Ensuring comparability in raw data: continuous process
Set harmonized standards/definitions
and map data
Collect metadata to assess comparability
Identify potential comparability
problems
Collect more information from
countries (e.g. ad-hoc surveys)
Provide a forum for countries and
experts to discuss
Adapt the data collection, indicator
calculation and reporting accordingly
Ensuring comparability in raw data: continuous process
Set harmonized standards/definitions
and map data
Collect metadata to assess comparability
Identify potential comparability
problems
Collect more information from
countries (e.g. ad-hoc surveys)
Provide a forum for countries and
experts to discuss
Adapt the data collection, indicator
calculation and reporting accordingly
ISCED classifications
• Collaboration between internationalorganizations, experts, and country representatives
• Classification must evolve over time toreflect important changes in educationsystems (e.g. ISCED 2011 classification reflects the Bologna Process in tertiary education)
Manuals for data submission
• Definition of concepts (even seemingly“obvious” ones like teachers, students)
• Scope (what should beincluded/excluded)
• Time-frame for data collection
• How to deal with borderline cases
Manuals: example of level of detail
Ensuring comparability in raw data: continuous process
Set harmonized standards/definitions
and map data
Collect metadata to assess comparability
Identify potential comparability
problems
Collect more information from
countries (e.g. ad-hoc surveys)
Provide a forum for countries and
experts to discuss
Adapt the data collection, indicator
calculation and reporting accordingly
Collect metadata
• Sources of data
• Reference year
• How does data reported differ from standard definition?
• How does data reported differ from specified scope?
• Any additional information
Collect metadata
• Sources of data
• Reference year
• How does data reported differ from standard definition?
• How does data reported differ from specified scope?
• Any additional information
+ Control codes and comments which also provide information on reported data
Ensuring comparability in raw data: continuous process
Set harmonized standards/definitions
and map data
Collect metadata to assess comparability
Identify potential comparability
problems
Collect more information from
countries (e.g. ad-hoc surveys)
Provide a forum for countries and
experts to discuss
Adapt the data collection, indicator
calculation and reporting accordingly
Identifying potential comparability problems
• ISCED Mappings
• Metadata
• Discussions with countries
• Priority rating exercise
• Countries are asked to rate indicators based on policy relevance and technical quality
• Helps prioritize the areas of work
Identifying potential comparability problems: Twoexamples
• Comparability in mapping:
• Early childhood education programmes
• Comparability in definitions/reporting:
• Household expenditure on education
Ensuring comparability in raw data: continuous process
Set harmonized standards/definitions
and map data
Collect metadata to assess comparability
Identify potential comparability
problems
Collect more information from
countries (e.g. ad-hoc surveys)
Provide a forum for countries and
experts to discuss
Adapt the data collection, indicator
calculation and reporting accordingly
Comparability of early childhood educationmapping
• OECD conducted an ad-hoc survey to collect more information on early childhood education programmes (ISCED 01 and ISCED 02).
• Survey collected data on which ISCED criteria were met for inclusion of programmes and which were not.
Overview of the different Early Childhood and Care systems across OECD countries
Comparability of reporting household expendituredata
Ensuring comparability in raw data: continuous process
Set harmonized standards/definitions
and map data
Collect metadata to assess comparability
Identify potential comparability
problems
Collect more information from
countries (e.g. ad-hoc surveys)
Provide a forum for countries and
experts to discuss
Adapt the data collection, indicator
calculation and reporting accordingly
Structure
Oversees strategic direction and coherence of OECD’s work on education
Set priorities and standards for data development and provide direction on the dissemination of the programme’s work
NESLI and LSO: technical discussions to develop and refine indicators
Focus on specific subjects and develop proposals that will be discussed at the INES Working Party and INES Networks
Partners:
Ensuring comparability in raw data: continuous process
Set harmonized standards/definitions
and map data
Collect metadata to assess comparability
Identify potential comparability
problems
Collect more information from
countries (e.g. ad-hoc surveys)
Provide a forum for countries and
experts to discuss
Adapt the data collection, indicator
calculation and reporting accordingly
How to deal with the comparability challenges in the raw data
Comparability issues in the raw data
Clarify the data collection or the manual
Document the comparability issues and set up working groups to continue to work on the topic
Build indicators that maximize comparability given the raw data
How to deal with the comparability challenges in the raw data
Comparability issues in the raw data
Clarify the data collection or the manual
Document the comparability issues and set up working groups to continue to work on the topic
Build indicators that maximize comparability given the raw data
Raw data is comparable
• Introduction
• Comparable raw data
• Comparable indicators
• Conclusion
OUTLINE
Ensuring comparability in indicators: continuousprocess
Restrict indicators to what is most comparable
Assess the impact of different data
sources/methodologies on indicator
Assess the comparability in
interpretation of the indicator
If possible, adapt or refine indicators
Agree on what is publishable and how to
document the comparability issues
Ensuring comparability in indicators: continuousprocess
Restrict indicators to what is most comparable
Assess the impact of different data
sources/methodologies on indicator
Assess the comparability in
interpretation of the indicator
If possible, adapt or refine indicators
Agree on what is publishable and how
to document the comparability issues
Restrict indicators to what is comparable: example of household expenditure
• Results from the ad-hoc survey on the comparability of household expenditure data
showed a considerable degree of variation in reporting across countries.
• Comparability is particularly challenging in reporting expenditure outside educational
institutions (e.g. transportation, rent). As a result, the indicator “expenditure per
student” only takes into account expenditure on educational institutions.
• The Secretariat will continue to work with countries to improve their reporting of
household expenditure. Degree of quality and comparability will be continuously
reassessed.
Restrict indicators to what is comparable: example of earlychildhood education
• Given the comparability challenges of reporting data on early childhood education
programmes, indicators for this level of education were removed from the totals in
Education at a Glance, and grouped into one chapter dedicated exclusively to this level
of education.
• For example, expenditure on education only shows the total from primary to tertiary
(ISCED1 to ISCED 8), and does not include ISCED 0. Expenditure on ISCED 0 is
reported separately in a specific chapter that more adequately explains the challenges and limits in reporting data for these programmes.
Ensuring comparability in indicators: continuousprocess
Restrict indicators to what is most comparable
Assess the impact of different data
sources/methodologies on indicator
Assess the comparability in
interpretation of the indicator
If possible, adapt or refine indicators
Agree on what is publishable and how to
document the comparability issues
Assess impact of different data sources/methodologies on indicator (1/2)
Compare methods:
• Simulations
• Data for countries where both methods are available
Conclusion:
• Given the magnitude of differences, agreed to present them separately and note the overestimation of cross cohort
Assess impact of different data sources/methodologies (2/2)
Ensuring comparability in indicators: continuousprocess
Restrict indicators to what is most comparable
Assess the impact of different data
sources/methodologies on indicator
Assess the comparability in
interpretation of the indicator
If possible, adapt or refine indicators
Agree on what is publishable and how
to document the comparability issues
Interpretability: importance of national context even with comparable raw data
• Luxembourg has the lowest first-time graduation rate for
tertiary education among OECD countries. However, that’s
because 75% of secondary school graduates pursue
tertiary studies abroad.
• First-time graduation rates in Australia and New Zealand
are the highest of all OECD countries, but only because of
the high share of international students.
Ensuring comparability in indicators: continuousprocess
Restrict indicators to what is most comparable
Assess the impact of different data
sources/methodologies on indicator
Assess the comparability in
interpretation of the indicator
If possible, adapt or refine indicators
Agree on what is publishable and how to
document the comparability issues
Evolution of indicators to increase comparability in interpretation
Gross graduation rate
Net graduation rate
First-time net graduation rate
First-time net graduation rate excluding international students
First-time net graduation rate excluding international students and below the typical age
Evolution of indicators to increase comparability in interpretation
Ensuring comparability in indicators: continuousprocess
Restrict indicators to what is most comparable
Assess the impact of different data
sources/methodologies on indicator
Assess the comparability in
interpretation of the indicator
If possible, adapt or refine indicators
Agree on what is publishable and how to
document the comparability issues
Metadata in published book
Footnotes
• Reference years
• Type of institution (e.g. only public institutions available)
• Inclusion/exclusion of a different category if not available in the table
Text analysis
• General limits to interpretability
• National context, policies, reforms, that can help interpret the data
• (e.g. Israel’s mandatory military conscription)
Annexes
• Country-specific details on definitions, coverage, methodology, context
• (e.g. non-inclusion of special-need institutions)
Metadata in online database
• The possibility of including metadata, methodology explanations and country-specific information in an online database is more limited.
• Avoid publishing indicators that require more metadata/explanations to be correctly interpreted (e.g. completion rates, financial returns to education)
• Introduction
• Comparable raw data
• Comparable indicators
• Conclusion
OUTLINE
Conclusion
• Ensuring comparability is a continuous process of improvement of both the raw data and the indicator.
• It is not possible to achieve 100% comparability, so there must be a degree of flexibility.
• Must balance between stricter comparability and less available data.
• Our main strategy to ensure comparability is to work closely with countries and provide opportunities for countries and experts to discuss.
Acesse nossas redes sociais Conheça nossas publicações
MUITO OBRIGADA!
TWITTER@OECDEduSkills
YOUTUBE@EduSkills OECD
PORTAL INEP portal.inep.gov.br
Education at a glance: www.oecd.org/education/education-at-a-glance-19991487.htm
Country notes:http://www.oecd.org/education/educationataglance2017-countrynotes.htm