High Stakes Testing on Sakai with Samigo and a Locked-down OS image

June 10-15, 2012

Growing Community; Growing Possibilities

High Stakes Testing with Samigo

on a Locked-Down OS Image

Ken RomeoLanguage Center

Mike QueisserAcademic Computing Services

Stanford University

2

Background: Why did we do this? Pieces: Where, what, and how?

◦ Hardware◦ Test settings◦ OS image◦ Preparation◦ Administration

Results: How well did it work? Lessons learned: What should/can we fix? The biggest victory: Collaborations

Roadmap

2012 Jasig Sakai Conference

2012 Jasig Sakai Conference 3

CourseWork = Sakai ACTFL = American Council on the Teaching

of Foreign Languages WPA = Writing Proficiency Assessment IME = Input Method Editor

Terminology

2012 Jasig Sakai Conference 4

Language Requirement

Accountability Speaking & Writing ACTFL Proficiency

guidelines

Background

2012 Jasig Sakai Conference 5

Paper test distribution

2012 Jasig Sakai Conference 6

Scans Project sites

2012 Jasig Sakai Conference 7

Why not just do it in samigo?

More secureBetter organizationTyping produces more language

2012 Jasig Sakai Conference 8

No space …

2012 Jasig Sakai Conference 9

“Faculty agree to avoid academic procedures that create temptations to violate the honor code.”

2012 Jasig Sakai Conference 10

Building 30

2012 Jasig Sakai Conference 11

2012 Jasig Sakai Conference 12

2012 Jasig Sakai Conference 13

2012 Jasig Sakai Conference 14

2012 Jasig Sakai Conference 15

2012 Jasig Sakai Conference 16

Lightweight, simple system Security Privacy

The OS image: Goals

2012 Jasig Sakai Conference 17

Minimize running services and applications Sophos A/V, Python 2.6, and CopSSH Automatic login XML file >> IME settings CMD.exe and Notepad.exe Windows updates disabled

Lightweight, Simple System

2012 Jasig Sakai Conference 18

Active directory Domain and local group policies Proxy server to allow access to only certain

sites Script read from imaging management

server

Security

2012 Jasig Sakai Conference 19

Group Policy Screenshot

2012 Jasig Sakai Conference 20

function FindProxyForURL(url, host) { //strings to return var allow = "DIRECT"; var block = "PROXY localhost:80"; if (shExpMatch(url,

"https://coursework.stanford.edu/*")) { return allow; }   if (shExpMatch(url, "https://cw-tests.stanford.edu/*")) { return allow; }   if (shExpMatch(url, "https://weblogin.stanford.edu/*")) { return allow; }   if (shExpMatch(url, "*.microsoft.com/*")) { return allow; }   if (shExpMatch(url, "*.windowsupdate.com/*")) { return allow; }   if (shExpMatch(url, "*.microsoftupdates.com/*")) { return allow; }   if (shExpMatch(url, "*.sophos.com/*")) { return allow; }   return block; }

Proxy Configuration Script

2012 Jasig Sakai Conference 21

Clear temporary files at logout Reset user account completely at login

Privacy

2012 Jasig Sakai Conference 22

Typing: PC – IE8 (NOT a Mac, MSWord) Instructions

◦ Language bar◦ Don’t click the back button◦ Save after every sentence◦ Mistakes: ctrl-z / close browser◦ Submit when complete◦ Can re-open unsubmitted work◦ Can re-submit new work

User Preparation

2012 Jasig Sakai Conference 23

Coordinator group and 1-on-1 meetings Multiple emails to instructors Coordinator meetings with instructors Consulting on preparation activities

Instructor preparation

2012 Jasig Sakai Conference 24

Overview / schedule Preparation: steps and samigo export file Orientation

◦ Typing instructions: US International, Chinese, Japanese, Korean

◦ Test taking instructions Testing day instructions for administration

◦ Setup◦ Test start to finish◦ Error handling

http://wpa.stanford.edu

2012 Jasig Sakai Conference 25

Introduction: Honor code and instructions 2 items: text prompt, short answer

responses Settings:

◦ No release, due, retract dates◦ No time limit◦ Allow multiple submissions◦ Allow late submissions◦ High security: UserID and password

Concept: security through physical location

Tests and Quizzes (aka samigo)

2012 Jasig Sakai Conference 26

Previously observed Virtual server to avoid load balancer Certificate Vanilla Sakai In the end …

Unexplained errors

2012 Jasig Sakai Conference 27

2011:  pilot 155 tests in Western languages 2012:  493 on samigo / 238 on paper,

◦ 31 Chinese, Japanese, and Korean pilot tests◦ No major data loss: 11 students lost 1-2

sentences◦ One test fell back to paper (10 students)

2013: expand CJK Remaining languages:  Russian and Arabic

Results

2012 Jasig Sakai Conference 28

Students don’t read paper instructions Logistics are never, ever trivial Something was sketchy:  load balancer? Overcautious instructions worked, sort of Non-soft controls (physical presence)

worked OS image totally worked (except notepad)

Lessons learned

2012 Jasig Sakai Conference 29

Block Notepad and CMD IE >> Firefox ESR Point out language bar

Improvement Opportunities

2012 Jasig Sakai Conference 30

Desktop (not web) app

Enrollment from Sakai Results to Sakai Cache results until

completely uploaded

Need: a better tool

SOPI application

2012 Jasig Sakai Conference 31

The end of the blue book Mid-terms, finals: like paper, but online Instructional technology for the 80% The beginning of learning analytics

High stakes testing

2012 Jasig Sakai Conference 32

The biggest victory

Imaging(Mike’s cube)

CourseWork

Team

Language Lab

(Ken’s office)

2012 Jasig Sakai Conference 33

Cannot do instructional technology *at scale* in isolation

Collaborations across units work! Face-to-face meetings Professional cooperation to improve

instruction = incentive

n.b. My explicit role is to facilitate these collaborations for the Language Center

Collaborations

2012 Jasig Sakai Conference 34

Proactively address needs Address problems systematically,

institutionally   Cross institutional boundaries Generalize to contribute to the field

Instructional Technology should:

June 10-15, 2012

Growing Community; Growing Possibilities

More Info:http://wpa.stanford.edu

Thank you!Ken Romeo – kenro@stanford.edu

Language Center

Mike Queisser – squeezer@stanford.eduAcademic Computing Services

Stanford University