Perception - Questionmark · PDF fileRoot Topic (Course) ... Practical Application in Resident Environment . 2012 Users Conference ... Password protected

Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

2012 Users Conference New Orleans March 20 - 23

Questionmark ™ Perception™

How e-Testing is Improving Assessment for the U.S. Coast Guard

Jim Parry, CPT, Test Development Manager

Art Stark, Instructor

USCG Training Center

Yorktown, VA

2012 Users Conference New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark

is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Background

Moving from the past to the 21st century



" The Coast Guard is using a slow and archaic paper testing process to measure the knowledge of its members. The Coast Guard has a duty to provide the best operational programs to ensure missions are completed in an exemplary manner. To allow the Coast Guard to continue its long tradition of exceptional service and superior foresight, we must move the current paper based testing system to an electronic medium."

Commanding Officer US Coast Guard Institute, 2005



Implementation Timeline

1999: USCG Institute (CGI) began transferring processes to electronic medium

2008: USCG awards contract to Questionmark ™

2012: CG wide implementation



Wide use expected

The Enterprise Solution



Over 300,000 test takers and over 800,000 test items available! http://www.cutteragent.com/atlas



Participants

US Coast Guard Personnel 41,000 Active Duty 7,000 Reserve 9,000 Civilian 33,000 Auxiliary NAF and Contracted personnel

National Maritime Center (NMC) Over 240,000 active merchant mariners



High Stakes Authors

Enlisted Advancement Testing (40) 19 enlisted specialties 6 testable pay grades in each (E-4 – E-9) Two test types Criterion-referenced Rating Advancement Test (RAT)

Norm-referenced Servicewide Examination (SWE)

Certification Exams (10)



High Stakes Testing

Rating Advancement Test (RAT) 2 or 3 for each specialty (E-4 – E-6) Minimum 50 questions each

Servicewide Exam (SWE) 2 parallel exams each exam cycle 150 questions each Active duty & Reserve Forces

Law Enforcement Certification Aviation Certifications



Low – Medium Stakes Authors

Resident Training (150 - 200) 15 locations Basic and advanced training courses Over 1300 courses One author may handle several courses



Approval Process

Validity and Reliability



Workflow

Test item requires minimum of 2 approvals before being published Author Writer/Editor Instructional Systems Specialist

Currently two CG workflows SWE/RAT Resident



SWE/RAT Workflow

Author generates/edits question

W/E edits, rejects or forwards

ISS edits, rejects or approves

No changes after final approval High stakes exams



Resident Workflow

Author generates/edits question

W/E edits, rejects or forwards

ISS edits, rejects or approves

Changes can be made Low/medium stakes



Standardization

Adds to defensibility of USCG testing



Standardized Test Item Topic Structure

REPOSITORY (OSC Server) Root Topic (Course) Sub Topic (Unit)

Sub-Sub Topic (TPO) Sub-Sub-Sub Topic (EO)

Test items are derived from the enabling/lesson objectives in the case of EOCT/RAT and resident courses and EPQ/RPQ for SWEs



Why Standard Structure?

Eliminates ownership confusion Better test item security

Root topic name includes course ID number Matches other CG databases

Allows drill down for coaching, statistical review & defensibility issues Many statistical and grading reports available



Example Topic Structure



Making it Real

Improving the Quality of the Testing Experience



The Enlisted Rating Advancement Training System



Purpose

To support the performance of assigned enlisted personnel in preparing for advancement or change in rating



Functions Define performance requirements Provide training and performance support Assess performance

Doctrine

Training

Assessment



Realistic Rating Advancement Testing

Able to embed content into test Allows more

realistic approach to solving problem

If reference is available in field it will be available in test item



Questionmark Secure™ will prevent “surfing”, printing, etc.



Selecting reference from list allows participant to view.



Practical Application in Resident Environment



Entering test items

As easy as pie!



How USCG Does It

Question Wizard Guides author through process

Question Editor Advanced users

Questionmark Live Contributers Allows work from home Cannot use for “For Official Use Only (FOUO)”

ASCII import Harvested from existing item banks



Defensibility and security

Getting it right



Use of Standards

ISO 10667-1 & 2 Assessment service delivery – Procedures and methods to

assess people in work and organizational settings AERA, APA, NCME Standards for educational and psychological testing

Interservice Enlisted Testing and Advancement Group (ISETAG) US Navy Advancement Center US Air Force Airman Advancement Division US Coast Guard Forces Command



Defensibility

Test Plans

Insert reference as feedback

Angoff score

Analytics to support cut score

Ability to track test and item statistics



Analytics

Test & Item Analysis

Coaching Reports

Gap Reports

Grade Book



Security

Password protected “strong” password required

Security, non-disclosure, etc. statement is first item on all high stakes exams

Authors, Admins, etc. Required to execute non-disclosure agreement



Potential Security Issues

Network printers Potential to send outside of secure area

Requires close monitoring when delivering assessments Test takers hidden behind screen



Standard Operating Procedures

CG established SOP/TTP for e-testing Defines various roles (profiles) Establishes workflow rules Establishes topic structure Provides job aids for certain functions



Community Sharing

Established collaboration site in the CG Portal Open access behind the USCG firewall Information and job aids available



Certification & Accreditation of Software

Not just plug and play…



Software C&A

USCG Telecommunications and Information

Systems Command (TISCOM) Checks for system image compatibility Reviews programming code for potential security holes Conducts penetration testing (Hacking)



Stakeholders

Who’s who…



The Key Players

CG Force Readiness Command (FC-Tadl) Operations and Systems Center (OSC) E-testing managers Coast Guard Institute (CGI) Personnel Services Center (PSC-adv) All Coast Guard Training Centers (TRACENs) Educational Services Officers (ESOs)



to the future and beyond

We are on the move!

A wheel is _____. A. flat B. round C. square D. cool



Shipboard and Remote Delivery

Sporadic or no connectivity Questionmark™ to Go Printing and scanning



What’s in Store?

Initial alpha/beta testing went relatively well Identified shortcomings Resolved most issues

Observational assessment Adaptive branching Growing pains are inevitable



Questions?

Perception - Questionmark · PDF fileRoot Topic (Course) ... Practical Application in Resident Environment . 2012 Users Conference ... Password protected

Documents