The Implementation of National Educational Standards in Austria. Michael Bruneforth BIFIE - Austrian Federal Institute for Education Research, Innovation and Development of the Austrian School System EURASIAN EDUCATIONAL DIALOGUE 17 .-19. April, Yaroslavl. - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
BIFIE - Austrian Federal Institute for Education Research, Innovation and Development of the Austrian School SystemAreas of work International Studies (PISA, TIMSS, PIRLS, TALIS) Implementation of national education standards Formative and summative Evaluation of standards Central examination for upper secondary education
preparing for tertiary education National Education Reports
Very rich country (4th highest GDP per Capita in EU)High expenditure per student (140% of EU Average) in primary and secondary educationLow student teacher ratio and class sizeBut PISA shock in 2003 and 2009: Very low PISA results in reading (470, similar to Russia
459), no EU country scores less Strong relation between education outcomes and family
Since more than a decade there is a change in governance of schools and education in German speaking countries from input oriented control to evidence based output oriented governance and extension of school autonomy In Austria at the centre is shift towards competence oriented teaching based on the definition of nationwide compulsory education standards and a mechanism of formative and summative evaluation
From 2004: pilot phase with selected pilot schools2008: alteration of the law of teaching in schools to allow for introduction of educational standards2009: educational standards become compulsory for German, Math and English2009 (grade 8) and 2010 (grade 4): baseline-tests in samples of about 10.000 students eachFrom 2012 onwards yearly assessment of national standards (one subject per year)
Competency Models and descriptors are formulated as Can-Do-StatementsCompetency Models and descriptors are legally establishedEducational Standards can be interpreted as mandatory and agreed upon social objectives of instructionEducational Standards are derived from national curricula and clearly define learning outcome.
„fashionable term with a vague meaning“ (Weinert 2001)„cognitive prerequisites to coping with a specific range of situations“ (Klieme, Hartig & Rauch, 2008) „cognitive abilities and skills available to/acquired by individuals that enable them to solve specific problems, as well as the related motivational, volitional and social readiness and skills to utilize the solutions successfully and responsibly in variable situations“ (Weinert, 2002) BIFIE use: subject (domain) specific competency models, focusing on cognitive skills and proficiencies
Competency ModelsModels of competence structuresModels of competence levels Development models
Models describing the acquisition of skills and proficiencies May be linked to neurological/psychological theories of development Only a few exist Would be great to have a theoretical foundation for tests in different grades! (IKM – BIST)
The 4x4 grid leads to 16 nodes, using complexity as a 3rd dimension = 48 nodesNot very handy and the theoretical concept of complexity does not lend itself to assessment
P: „demonstrating, modelling “ C: „numbers and measures“
Formulated as can-do-statements: The students are able to transform given arithmetical data into (another)
mathematical representation. In doing so, direct use of basic skills is necessarytransform given arithmetical data into (another) mathematical representation. In doing so, connections to other mathematical contents (terms, definitions, representations) or activities have to be establishedmake and evaluate statements about the appropriateness as well as weaknesses and strengths of different mathematical representations (models) of arithmetical data
School-specific development of instruction Mandatory educational standards Census-survey of all students and schools Feedback to all students, teachers, schools Feedback moderation („Rückmeldemoderation“) Stimulation of quality development and support by MoE Strong linkage of monitoring and practical school and
instructional development Strengthening of schools, school management and
Main functions of standards according to the legal regulation
Output orientation: Sustainable output orientation in planning and implementation of instructionPromotion („Förderfunktion“): Diagnosis and specific individual remediation and challengeEvaluation: reliable information about the output of instruction, schools and education system
Supporting materials for instruction Praxishandbücher Themenhefte Aufgabenbeispiele, Interaktive Beispiele Best-Practice-Beispiele
Integration with teacher training institutions In-service teacher training (partially mandatory) Pre-Service teacher trainin Research and development work by members of teacher
Legal assessment cycle: three (school) yearsIn each cycle, the whole target population has to be assessed in all prescribed competency subdomains Grade 4: German, Mathematics Grade 8: German, Mathematics, English
Students, teachers, schools and regional education authorities must get feedback about the results (to be used for quality improvement)
Items are developed to measure sub-competencies or domains, eg. Content = numbers and units and process = demonstratingDimension „complexity“ is not used as such, since it can‘t be measured empiricallyWe have 16 nodes, upon which an item should fall. We strive for unidimensional items, i.e. items that can be classified as belonging to one specific node only
Items are developed to measure sub-competencies or domains, eg. Content = numbers and units and process = demonstratingDimension „complexity“ is not used as such, since it can‘t be measured empiricallyWe have 16 nodes, upon which an item should fall. We strive for unidimensional items, i.e. items that can be classified as belonging to one specific node only
Test blueprint specified the items to be developed: Mathematical node Item format: Multiple Choice, Constructed response, … Estimated difficulty of item
Further requirements: adequat for age group, no biased towards gender, race, religion, also taking into
account rural/urban differences. does not require other knowledge than maths contents. Items should be uni-dimensional
In order to secure validity of the test items, they were … written by school and university teachers reviewed by teachers (peer review and outsider review) trialled on a sample of grade 8 students – each item was
answered by at least 200 students item properties were reviewed by a team of
psychometricians and school teachers items with less than desirable or surprising properties
Using Rasch modelAllows to express difficulties of items and students ability on one common scaleAllows to match test cycles over timeAllows for complex test design
Standard SettingStandard setting goal: meaningfully described levels
criterion-referenced Phases:
Determination of number of levels
Labelling of levels (PLLs)
Description of levels (PLDs)
Establishing the cut scores between levels
4 levels in Austria in most domains0: standards not reached1: partially reached standards2: reached standards3: exceeded standards Easy to understand for „laymen“
3 – standards exceededStudents possess fundamental knowledge and skills in all parts of the mathematics curriculum and advanced knowledge structures, which exceed the requirements of level 2, specifically more pronounced abilities of abstraction and higher proficiency in combining parts of knowledge, methods or rules. They are able to apply these independently in novel situations in a flexible way.2 – standards reachedStudents posess fundamental knowledge and skills in all parts of the mathematics curriculum and can use these in a flexible way. They are able to find and apply problem solving strategies, to describe and reason an approach. They are able to handle verbal, graphical and formal representations of mathematical facts in a flexible way and can apply these appropriately. Theay are able to extract relevant information from a differently represented facts (e.g. texts, data material, graphics) and can interpret them in the respective context. They are able to relate their mathematical knowledge and can check, evaluate and/or reason mathematical statements.
1 – standards partially reachedStudents possess fundamental knowledge and skills in all parts of the mathematics curriculum and can master reproductive tasks and carry out routine procedures.
„ A standard setting method is a mechanism for helping panelists translate their intentions to the score scale.“ (Reckase, 2006) Goal: Match predetermined descriptors of competency levels to an empiric scale defined by test items
Determine cut scores between levels of competenceBifie invited a group of stakeholders to set the cut scores: teachers (lower and upper secondary), teachers of didactics of math; representatives of parents associations; representatives of the MoE; representatives of chamber of commerce; bifie staffMethod used: Item descriptor matching method
Information about school results:compared to AUTcompared to the defined goals (in the regulation) compared to other schools with similar basic conditions („fair comparison“)
Areas of conflict:- Educational Standards apply to all students –
and therefore to all schools- Conditions of schools vary widely
OutlookBringing IKM (grade(s) before BIST) and BIST assessments closer together. Using items in both assessments could lead towards a better understanding of learningBackwash from doing the assessments towards research in the areas of item development, standard setting, feedback … through evaluation of results Outcomes of the assessments should not only allow us to give feedback to schools, teachers and students but also to improve the competence models we use for the assessmentPrepare for computer based testing Open databases for external researchers and improve system level monitoring