Ceneval + TAO, improving the developers and test takers plataform César Chávez Álvarez Francisco M. Otero Flores TAO Days 2013 Bern, Switzerland October 1, 2013
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Ceneval + TAO, improving the developers and test takers plataform
César Chávez Álvarez
Francisco M. Otero Flores
TAO Days 2013 Bern, Switzerland
October 1, 2013
Contents
• Who are we?
• Ceneval’s platform
• Ceneval + TAO : Improving our content
2
Who are we?
3
• Ceneval is the National Center for the Evaluation of Higher Education.
• Is a Mexican non-profit institution founded in 1994 by the National Association of Universities and Higher Education Institutions (ANUIES).
• An institution committed to promote the quality of education by developing valid, reliable, and relevant assessments.
• Ceneval´s activities are focused on the evaluation of knowledge and skills acquired in the teaching-learning process at different levels in formal and informal education.
• Ceneval has a 20-member Board of Trustees integrated by Principals/Presidents of public and private universities, Presidents of the two Universities Associations and the Heads of Professional or Trade Associations.
4
Who we are in numbers
Tests in operation >250
Experts’ groups in charge of instruments >90
Members of experts’ groups >900
Teachers writing & reviewing items Thousands
Test takers per year >3 millions
Test administrations per year >2,100
High schools & universities served each year >1,500
Online test administrations events per year >170
Staff working full time 540
Government financial support $0.00
5
Ceneval’s tests
Ceneval’s tests can be grouped into four categories:
• Admissions or entrance tests (EXANI) for high school, undergraduate programs and postgraduate programs (basically masters degree programs)
• Higher Education Exit Assessments (EGEL) for undergraduate programs (40)
• Generic knowledge and skills assessments
• On demand assessments
6
Ceneval’s plataform
7
• Since 2002, Ceneval started a project which main purpose was to
create a set of tools to automatize most of the tasks required by the two productions lines of the Centre: items development and test administration.
• The first developments were the website, the secure items banks and the online registry. Over the years, new needs required new solutions.
• Nowadays, Ceneval’s platform consists of seven modules: portal, registry, testing, developers banks, security banks and BPM for test management.
8
• Ceneval’s website, which broadcast its services and
most important activities.
• Based on BPM technology, this module integrates the working flow to administer more than 2,100 test administrations events requested by our customers each year.
• This module allows test takers to register to a specific test session.
9
Ceneval’s platform modules (1)
• This module is open to all item developers and is
designed to capture and validate technical and grammatically new items that potentially will be used in a test form.
Annually this module adds more than 40,000 new items, which are validated by almost 1,000 different experts
• This module can only be accessed within Ceneval’s facilities, it stores items suitable to be pilot tested in a form, builds new forms, and stores psychometrical data.
Currently, this module stores more than 300,000 items in 256 test pool
• Integrated by a group of sub-modules, this module allows the candidates to take a test.
10
Ceneval’s platform modules (2)
- Test Information - Student guides - Test takers scores
11
Students
Administrations requests
Items contents & logs
Assessments’ forms
Psychometrical data & planning
Platform’s connectivity
12
Developers banks (e-BRAE)
• This module is open to all item developers and is designed to capture and validate, technical and grammatically new items that could potentially be used.
• In order to automate the item’s development process, the module defines: ü A working flow for item development attached to international standards ü Seven types of users profiles that may access one or more banks ü An item’s log which tracks its changes along the process, and also provides valuable
evidence at every stage ü A system’s log which tracks every user activity ü More than 30 different functions to administer the items and its content ü 17 different types of reports ü An automated connectivity with the security banks module when the item has
reached the end of the working flow ü The items contents and users passwords are encrypted at the database
• Annually this module adds more than 40,000 items developed and validated by almost 1,000 users in almost 256 different banks
13
Item’s life-cycle
Expert content create a new item
Experts content review
Grammar & Spelling review
Pilot test
Multiple choice + + +
14
15
Item’s life-cycle
Pilot test Psychometrical analysis
Ready for an Assessment
form
16
+ +
Multiple choice
Security banks (e-BASE)
• This module can only be accessed within Ceneval’s facilities, and its purposes are: ü Store the items cataloged at e-BRAE as suitable to be piloted in an
assessment form ü Build assessments forms according to the technical specifications, that may
be administered by paper and pencil or online ü Store the psychometric data obtained during one or more tests
administrations
• This module was the first developed, and shares many features with the e-BRAE, although most of the items administration functionality is restricted.
• Nowadays, the module stores more than 300,000 items for 256 banks
17
• In the last 10 years, the Mexican government encouraged public and private educational institutions to use widely the ICT as the mexican society transforms quickly into an information society. This government’s initiative has: • Enable the construction of more computer classrooms • The development or implementation of computer software as a supporting tool for
teachers • Increase the demand on bigger and low cost of broadband services
• The tendency at the urban institutions is to spread the use of computers for testing purposes, in order to have quick feedback and “low” the administration costs.
• Ceneval´s commitment to the educational institutions is to develop new test incorporating innovative computer-based items intended to measure knowledge, skills and abilities that are difficult to evaluate using only multiple-choice items.
• The new schema is intended to represent authentic, real word tasks in a competence
evaluation framework. 18
Increasing tendencies and new requirements
• During the last quarter of 2011, Ceneval had a group of systems aligned with the
organization requirements; allowing the development of new item, forms and administer test based only on single multiple choice items.
• At that moment, we were standing in a crossroad: • Continue with our system and develop new functionality • Acquire licensing for software that meet our requirements
• In our research, we meet a lot of providers and have a lot of remote meetings, with more or less the same results:
• Our software must be installed in your servers (so my items will no longer live with me?) • We don’t meet your requirements, but you can change your processes to meet my
system’s? (yes, indeed, when do we start) • We charge you for a user account (>2,000 item developers accounts + thousands of
reviewers) = $$$$$
19
Meeting our clients demands
Ceneval + TAO: Improving content
20
• Almost accepting that we might grant the demands of the new software providers, we found TAO; a socratic midpoint to our crossroads because:
• Can live in our data centers • We might have as many accounts as we need • As a GNU, we can have access to the source code
• In November 2012, TAO visited us in our headquarters at Mexico City with three
objectives: • Made a feature presentation of TAO to the organization • Showed TAO the items’ workflow: processes, tasks, reports • How TAO capabilities and Ceneval’s needs could be meet?
DON’T THROW AWAY ANYTHING,
USE WHAT YOU NEED! 21
Platform’s connectivity
22
Items contents & logs
Assessments’ forms
Psychometrical data & planning
May 2013
May 2013
+ TAO :: Items editor (QTI)
+ TAO :: Items editor (QTI)
December 2013
+ XHTML Content
- Test Information - Student guides - Test takers scores
Item’s life-cycle
Expert content create a new item
Experts content review
Grammar & Spelling review
Pilot test
TEXT 1) Choice 2) Inline choice 3) Associate 4) Order 5) Match 6) Gap match 7) Text entry 8) Extended text 9) Hottext
GRAPHIC 10) Hotspot 11) Graphic order 12) Graphic associate 13) Graphic gap 14) Select point 15) Slider 16) Upload documents
+ + +
23
Innovative Items (QTI)
Multiple choice
+
Pilot test Psychometrical analysis
Ready for an Assessment
form
24
+ +
TEXT 1) Choice 2) Inline choice 3) Associate 4) Order 5) Match 6) Gap match 7) Text entry 8) Extended text 9) Hottext
GRAPHIC 10) Hotspot 11) Graphic order 12) Graphic associate 13) Graphic gap 14) Select point 15) Slider 16) Upload documents
Item’s life-cycle
Innovative Items (QTI)
+
Multiple choice
25
User interface
Some other adjustments…
In order to implement this new assessment process, it is necessary:
• Ceneval’s system (easy part) • Update manuals and training courses • Adjust the item’s life-cycle • Spread the benefits of this type of evaluation among users
• Item development process (difficult part) • Update the policies and standards through the results of the pilot study to take
place next January • Include new roles and tasks • Retrain item developers and item reviewers
26
27
Related Documents
