June 10-15, 2012 Growing Community; Growing Possibilities High Stakes Testing with Samigo on a Locked-Down OS Image Ken Romeo Language Center Mike Queisser Academic Computing Services Stanford University
June 10-15, 2012
Growing Community; Growing Possibilities
High Stakes Testing with Samigo
on a Locked-Down OS Image
Ken RomeoLanguage Center
Mike QueisserAcademic Computing Services
Stanford University
2
Background: Why did we do this? Pieces: Where, what, and how?
◦ Hardware◦ Test settings◦ OS image◦ Preparation◦ Administration
Results: How well did it work? Lessons learned: What should/can we fix? The biggest victory: Collaborations
Roadmap
2012 Jasig Sakai Conference
2012 Jasig Sakai Conference 3
CourseWork = Sakai ACTFL = American Council on the Teaching
of Foreign Languages WPA = Writing Proficiency Assessment IME = Input Method Editor
Terminology
2012 Jasig Sakai Conference 4
Language Requirement
Accountability Speaking & Writing ACTFL Proficiency
guidelines
Background
2012 Jasig Sakai Conference 5
Paper test distribution
2012 Jasig Sakai Conference 6
Scans Project sites
2012 Jasig Sakai Conference 7
Why not just do it in samigo?
More secureBetter organizationTyping produces more language
2012 Jasig Sakai Conference 8
No space …
2012 Jasig Sakai Conference 9
“Faculty agree to avoid academic procedures that create temptations to violate the honor code.”
2012 Jasig Sakai Conference 10
Building 30
2012 Jasig Sakai Conference 11
2012 Jasig Sakai Conference 12
2012 Jasig Sakai Conference 13
2012 Jasig Sakai Conference 14
2012 Jasig Sakai Conference 15
2012 Jasig Sakai Conference 16
Lightweight, simple system Security Privacy
The OS image: Goals
2012 Jasig Sakai Conference 17
Minimize running services and applications Sophos A/V, Python 2.6, and CopSSH Automatic login XML file >> IME settings CMD.exe and Notepad.exe Windows updates disabled
Lightweight, Simple System
2012 Jasig Sakai Conference 18
Active directory Domain and local group policies Proxy server to allow access to only certain
sites Script read from imaging management
server
Security
2012 Jasig Sakai Conference 19
Group Policy Screenshot
2012 Jasig Sakai Conference 20
function FindProxyForURL(url, host) { //strings to return var allow = "DIRECT"; var block = "PROXY localhost:80"; if (shExpMatch(url,
"https://coursework.stanford.edu/*")) { return allow; } if (shExpMatch(url, "https://cw-tests.stanford.edu/*")) { return allow; } if (shExpMatch(url, "https://weblogin.stanford.edu/*")) { return allow; } if (shExpMatch(url, "*.microsoft.com/*")) { return allow; } if (shExpMatch(url, "*.windowsupdate.com/*")) { return allow; } if (shExpMatch(url, "*.microsoftupdates.com/*")) { return allow; } if (shExpMatch(url, "*.sophos.com/*")) { return allow; } return block; }
Proxy Configuration Script
2012 Jasig Sakai Conference 21
Clear temporary files at logout Reset user account completely at login
Privacy
2012 Jasig Sakai Conference 22
Typing: PC – IE8 (NOT a Mac, MSWord) Instructions
◦ Language bar◦ Don’t click the back button◦ Save after every sentence◦ Mistakes: ctrl-z / close browser◦ Submit when complete◦ Can re-open unsubmitted work◦ Can re-submit new work
User Preparation
2012 Jasig Sakai Conference 23
Coordinator group and 1-on-1 meetings Multiple emails to instructors Coordinator meetings with instructors Consulting on preparation activities
Instructor preparation
2012 Jasig Sakai Conference 24
Overview / schedule Preparation: steps and samigo export file Orientation
◦ Typing instructions: US International, Chinese, Japanese, Korean
◦ Test taking instructions Testing day instructions for administration
◦ Setup◦ Test start to finish◦ Error handling
http://wpa.stanford.edu
2012 Jasig Sakai Conference 25
Introduction: Honor code and instructions 2 items: text prompt, short answer
responses Settings:
◦ No release, due, retract dates◦ No time limit◦ Allow multiple submissions◦ Allow late submissions◦ High security: UserID and password
Concept: security through physical location
Tests and Quizzes (aka samigo)
2012 Jasig Sakai Conference 26
Previously observed Virtual server to avoid load balancer Certificate Vanilla Sakai In the end …
Unexplained errors
2012 Jasig Sakai Conference 27
2011: pilot 155 tests in Western languages 2012: 493 on samigo / 238 on paper,
◦ 31 Chinese, Japanese, and Korean pilot tests◦ No major data loss: 11 students lost 1-2
sentences◦ One test fell back to paper (10 students)
2013: expand CJK Remaining languages: Russian and Arabic
Results
2012 Jasig Sakai Conference 28
Students don’t read paper instructions Logistics are never, ever trivial Something was sketchy: load balancer? Overcautious instructions worked, sort of Non-soft controls (physical presence)
worked OS image totally worked (except notepad)
Lessons learned
2012 Jasig Sakai Conference 29
Block Notepad and CMD IE >> Firefox ESR Point out language bar
Improvement Opportunities
2012 Jasig Sakai Conference 30
Desktop (not web) app
Enrollment from Sakai Results to Sakai Cache results until
completely uploaded
Need: a better tool
SOPI application
2012 Jasig Sakai Conference 31
The end of the blue book Mid-terms, finals: like paper, but online Instructional technology for the 80% The beginning of learning analytics
High stakes testing
2012 Jasig Sakai Conference 32
The biggest victory
Imaging(Mike’s cube)
CourseWork
Team
Language Lab
(Ken’s office)
2012 Jasig Sakai Conference 33
Cannot do instructional technology *at scale* in isolation
Collaborations across units work! Face-to-face meetings Professional cooperation to improve
instruction = incentive
n.b. My explicit role is to facilitate these collaborations for the Language Center
Collaborations
2012 Jasig Sakai Conference 34
Proactively address needs Address problems systematically,
institutionally Cross institutional boundaries Generalize to contribute to the field
Instructional Technology should:
June 10-15, 2012
Growing Community; Growing Possibilities
More Info:http://wpa.stanford.edu
Thank you!Ken Romeo – [email protected]
Language Center
Mike Queisser – [email protected] Computing Services
Stanford University