Page 1
Upcoming Caveon Events
• Caveon Webinar Series: Next session, May 17
“Using Decision Theory to Score Accurate Pass/Fail Decisions” Presented by Jamie Mulkey and Larry Rudner
• Caveon at NCSA – June 19-21, National Harbor, MD
Visit our booth and spend some time talking with us one-on-one!
Page 2
Latest Publications
• Handbook of Test Security – Now available for purchase! We’ll share a discount code before end of session.
• TILSA Guidebook for State Assessment Directors on Data Forensics – coming soon!
Page 3
Caveon Online
• Caveon Security Insights Blog– http://www.caveon.com/blog/
• twitter– Follow @Caveon
• LinkedIn– Caveon Company Page
– “Caveon Test Security” Group• Please contribute!
• Facebook– Will you be our “friend?”
– “Like” us!
www.caveon.com
Page 4
“What You Should Know about High Stakes Cheating in Your Schools”
Mike Stetter EdDSenior Security ConsultantFormer State Assessment Director
May 2, 2013
Caveon Webinar Series:
John Fremer PhDPresidentCaveon Consulting Services@TestSecurityGuy
Page 5
AGENDA
A. THE TEST SECURITY SITUATION
B. CHALLENGES FOR SCHOOL DISTRICT LEADERS
C. TOP TEN THINGS THAT YOU CAN DO TO
STRENGTHEN YOUR TEST SECURITY
D. GAINING A GREATER SECURITY ADVANTAGE
Page 6
A. THE TEST SECURITY SITUATION
Who cheats?
How many of them are there?
How often do they cheat?
Page 7
JOSEPHSON INSTITUTE SURVEY RESEARCH
• 95% admitted to cheating in high school
• 51% believed they must cheat to get ahead
• 90% were pleased with their own morality
40,000 College Students Surveyed in 2010
Page 8
Under 17 Over 50
Believe cheating is necessary 50% 10%
OK to deceive your boss 31% 8%
OK to keep incorrect change 49% 15%
OK to lie to your spouse 45% 22%
7,000 Survey Respondents
SURVEY OF ADULTS OVER 50 AND YOUTH UNDER 17
Page 9
29% felt pressure to cheat on standardized tests
34% felt pressure to change students’ scores for the better
21% knew educators that changed scores on students’ tests
8% admitted to changing students’ scores due to pressure
Detroit Free Press, July 26, 2011
MICHIGAN EDUCATOR SURVEY
Page 10
School District Cheating Scandals
In The News
Page 11
DC Public Schools
More than 100 schools
displayed suspicious
answer sheet erasures
So far, 2 teachers have been forced to resign
USA TODAY June 22, 2012
Page 12
Houston Independent
School District
Teachers at 2 elementary
schools caught cheating
Changed student answer sheets on State exams
The Houston Chronicle February 25, 2013
Page 13
Atlanta Public Schools
179 educators in
58 schools implicated
in dishonest test practices
41 have resigned or retired. 35 indicted by Grand Jury
Former Superintendent indicted on conspiracy
TIMENewsFeedApril 13, 2013
Page 14
Analyzed test results from
69,000 public schools
Suspicious test scores, similar to those in Atlanta,
were found in 196 school districts
Dishonest test practices on State level assessments
appears to be a national problem
April 13, 2013
Page 15
August 25, 2011
Duncan: No link between cheating and NCLB
Places the blame on school leaders
Regardless of who the perpetrators are . . .
School District Administrators will feel the heat!
Page 16
B. CHALLENGES FOR SCHOOL DISTRICT LEADERS
• Proctors’ potential conflict of interest
• Non-Standardized environment
• Examinee familiarity
• Transporting and storing test materials
Page 17
SCHOOL DISTRICT TEST SECURITY MATRIX
• District Superintendent
• District Assessment Coordinator
• School Test Coordinator
Page 18
DISTRICT TEST ROLES SECURITY MATRIX
The full Matrix is posted on the Caveon website:
http://bit.ly/testroles
Page 19
SCHOOL DISTRICT TEST SECURITY MATRIX
Essential Proactive
Steps
During Test Window
Before & After
Test Window
Superintendent
District Assessment
Director
Building Test Coordinator
Page 20
District Superintendent
• Assign clear responsibility & roles for test security
• Maintain communications with all local stakeholders on assessment activities, results
• Advocate for test program best practices
SCHOOL DISTRICT TEST SECURITY MATRIX
Page 21
District Assessment Director
• Supervise test training for all proctors
• Protect secure materials and test session integrity through detailed, accountable procedures
• Develop/Update district Test Security Plan
SCHOOL DISTRICT TEST SECURITY MATRIX
Page 22
Building Test Coordinator
• Confirm every test proctor completes training, and understands Do’s & Don’ts during testing
• Observe test sessions to confirm procedural compliance
• Directly supervise all handling of secure test materials
• Report suspected test irregularities
SCHOOL DISTRICT TEST SECURITY MATRIX
Page 23
C. TOP 10 THINGS YOU CAN DO
TO STRENGTHEN YOUR TEST SECURITY
Page 24
#1 – COMMUNICATION
• Raise test security awareness
• Include security references in communications
• Issue periodic security focused communications
Page 25
#2 – EXPECTATIONS
• Develop a system-wide Student Honor Code
• Emphasize the importance of honest test taking
• Issue clear expectations to test staff for professional behavior in test administration
Page 26
#3 – PROCEDURES
• Review all procedures for security related content
• Emphasize procedures for handling test materials
• Review / update security procedures annually
Page 27
#4 – TRAINING
• Review your test security training materials
• Track who completes the training
• Provide brush-up training annually
Page 28
#5 – OBSERVATIONS
• Perform announced pre-test session site visits
• Perform unannounced test session observations
• Follow-up on all observation outcomes
Page 29
#6 – FEEDBACK
• Seek feedback from test administrators / proctors
• Review procedures for reporting test irregularities
• Provide an anonymous tip line
Page 30
#7 – STUDY YOUR TEST DATA CAREFULLY
• Very Unusual Gains
• Other Indices
• Similarity
• Erasures
• Individual Patterns
• Timing (In Computer Testing)
Page 31
#8 – PREPARING FOR POSSIBLE SECURITY INCIDENTS
• Develop a “Security Incident Response Plan”
• Specify procedures, roles and responsibilities
• List potential outcomes for security infractions
Page 32
#9 – LEARNING FROM OTHERS
• CCSSO TILSA SCASS “Guidebook on Test Security”
• Wollack & Fremer “Handbook of Test Security”
• Caveon Webinars and Security Insights Newsletter
Learn about developments in prevention and detection
Page 33
#10 – IF YOU DEVELOP YOUR OWN TESTS
• Reduce exposure rate of test questions
• Randomize question locations
• Rotate answer options
Page 34
DATA FORENSICS
D. GAINING A GREATER SECURITY ADVANTAGE
Page 35
Data Forensics is the analysis of student test data
using sophisticated proprietary computer programs
to statistically identify instances likely representing
test fraud, collusion, and other security violations.
DATA FORENSICS
Page 36
A statistical measure of similarity within test results
or test response patterns that highlight instances
with a very high likelihood of a test security violation.
Similarity Index
DATA FORENSICS
Page 37
HOW CREDIBLE IS A FLAGGED SECURITY INDEX?
• Chance of being hit by lightning = 1 in 1 Million
• Chance of winning the lottery = 1 in 10 Million
• Chance of DNA false-positive = 1 in 30 Million
• Chance of students flagged for Similarity doing their own work = 1 in 1 Trillion
Page 38
Caveon’s data forensic diagnostic screening service for
school districts – a cost-effective means to monitor for
• Answer copying
• Test coaching
• Proxy test taking
• Collusion and other security violations
THE CAVEON SECURITY SCREEN
Page 39
• Custom forensic analyses of student test records
• Data output by subject, by school and by student
• Data interpretation with a Caveon test security specialist
• Advice on follow-up with highlighted security anomalies
• Consultation with test security planning and evaluation
• Security focused analysis of policies and procedures
THE CAVEON SECURITY SCREEN
Page 40
SAMPLE FORENSICS OUTPUT DATA REPORT
Page 41
Consultation
• Consultation with a Caveon Test Security Specialist
• Advice on Test Security planning and evaluation
• Interpretation of forensic data output reports
• Guidance on test irregularity follow-up
THE CAVEON SECURITY SCREEN SUMMARY
• Similarity Index analyses
• Perfect Test and Identical Test Analyses
• School level data reported by grade and by subject
• Flagged student output with cluster details for pairs and groups
Data Forensics
Page 42
Michael W. Stetter, D.Ed.
Senior Security Consultant
Caveon Test Security
(302) 415-0375
[email protected]
For information on the Caveon Test Security Screen
Page 43
HANDBOOK OF TEST SECURITY
Editors - James Wollack & John Fremer
Published March 2013
Preventing, Detecting, and Investigating Cheating
Testing in Many Domains
Certification/Licensure
Clinical
Educational
Industrial/Organizational
Don’t forget to order your copy at www.routledge.com
http://bit.ly/HandbookTS (Case Sensitive)
Save 20% - Enter discount code: HYJ82
Page 44
- Follow Caveon on twitter @caveon- Check out our blog…www.caveon.com/blog- LinkedIn Group – “Caveon Test Security”
Mike StetterSenior Security ConsultantCaveon Test Security
John FremerPresidentCaveon Consulting Services@TestSecurityGuy
Thank You!