10 September 2014, 07:00 GMT Bring-your-own-laptop e-Exam for a Large Class at NUS Seow Teck Keong and Jeffery Tay, National University of Singapore Your hosts Professor Geoff Crisp, Dean Learning and Teaching, RMIT University geoffrey.crisp[@]rmit.edu.au Dr Mathew Hillier, Institute for Teaching and Learning Innovation, University of Queensland mathew.hillier[@]uq.edu.au Transforming Assessment is supported by eAS Online Conference Just to let you know: By par'cipa'ng in the session you acknowledge and agree that: The session may be recorded, including voice and text chat communica'ons (a recording indicator is shown inside the webinar room when this is the case). We may release recordings freely to the public which become part of the public record. We may use session recordings for quality improvement, or as part of further research and publica'ons.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
10 September 2014, 07:00 GMT
Bring-your-own-laptop e-Exam for a Large Class at NUS Seow Teck Keong and Jeffery Tay, National University of Singapore
Your hosts
Professor Geoff Crisp, Dean Learning and Teaching, RMIT University geoffrey.crisp[@]rmit.edu.au
Dr Mathew Hillier, Institute for Teaching and Learning Innovation, University of Queensland mathew.hillier[@]uq.edu.au
Transforming Assessment is supported by
eAS Online Conference
Just%to%let%you%know:%%By#par'cipa'ng#in#the#session#you#acknowledge#and#agree#that: The#session#may#be#recorded,#including#voice#and#text#chat#communica'ons#(a#recording#indicator#is#shown#inside#the#webinar#room#when#this#is#the#case).#We#may#release#recordings#freely#to#the#public#which#become#part#of#the#public#record.#We#may#use#session#recordings#for#quality#improvement,#or#as#part#of#further#research#and#publica'ons.
BRING-YOUR-OWN-LAPTOP E-EXAM FOR A LARGE CLASS AT
NUS
SEOW#TECK#KEONG#Senior#Lecturer#
Department#Of#Biological#Sciences#
JEFFERY#TAY#Associate#Director#
Centre#For#Instruc'onal#Technology#
Na'onal#University#of#Singapore#
WHY DO WE DO E-EXAMS? • Authen'city#
• Showing#high#resolu'on#medical#images#rather#than#photocopied#versions#• Videos/Audio#clips#to#mimic#what#students#will#encounter#in#realUlife#situa'ons#in#their#
chosen#area#of#study#• Enable#ques'on#analy'cs#• Improved#efficiency#and#fairness#in#grading#and#repor'ng#• Audi'ng#• Ease#of#storage#(electronic#vs#hard#copy#scripts)#• Reduce#human#error#
• Administra've#processes#e.g#handling#of#scripts,#marks#transfer#• Illegible#student#handwri'ng#
• BYOD#(2012)#• Using#the#personal#learning#environment#(aka#laptop)#of#the#student#• Reduce#students’#examina'on#stress#• Reduce#system#delay#(transi'on#from#typing#to#pen#and#paper#during#exam)#
Lockdown Software
BYOD / PC Cluster
Inhouse solution PC Cluster
VPN PC Cluster
E-EXAMS SINCE 2006
0#
5#
10#
15#
20#
25#
2006/2007#
2007/2008#
2008/2009#
2009/2010#
2010/2011#
2011/2012#
2012/2013#
2013/2014#
Num
ber%o
f%assessm
ents%
Academic%Year%
CA#
Final#
EUAssessments#in#NUS#• Since#2006#(8#years)#• Average#200#students#per#run#• Both# con'nuous# assessment# as#well#as#high#stakes#
Facul'es#• Centre#for#English#Language#• Compu'ng#• Law#• Medicine#• School#of#Public#Policy#• Science#
Lockdown Software
BYOD / PC Cluster
Inhouse solution PC Cluster
VPN PC Cluster
E-EXAMS SINCE 2006
0#
5#
10#
15#
20#
25#
2006/2007#
2007/2008#
2008/2009#
2009/2010#
2010/2011#
2011/2012#
2012/2013#
2013/2014#
Num
ber%o
f%assessm
ents%
Academic%Year%
CA#
Final#
NUS E-EXAMS @ 2006
VPN#+#PC#Cluster#PC#clusters#• Laptop#are#costly#and#bulky#• Easier#to#manage#• Ownership#of#devices#reduces#risk#of#chea'ng#
VPN#• Provided#a#guest#environment#to#access#eUexam#resources#
• Restricted#access#based#on#creden'als#
NUS E-EXAMS @ 2006
Solu'on#• Limits#network#connec'on#only#to#exam#server#
• Loads#a#guest#environment#with#only#internet#explorer#available,#all#other#programs#are#hidden#
NUS E-EXAMS @ 2006
Pros#• Easy#for#student,#just#sit#down#and#take#exam#
• Easy#for#staff#to#create#ques'ons#• Auto#mark#solu'on#ensures#zero#errors#
Cons#• Lengthy#setup#and#teardown#• User#constantly#experience#system#lagging#
• Connec'vity#and#stability#issues#• Noisy#keyboards#caused#distrac'on#to#students#
• Not#scalable#• License#costs#• Does#not#completely#lock#down#the#system#
Lockdown Software
BYOD / PC Cluster
Inhouse solution PC Cluster
VPN PC Cluster
E-EXAMS SINCE 2006
0#
5#
10#
15#
20#
25#
2006/2007#
2007/2008#
2008/2009#
2009/2010#
2010/2011#
2011/2012#
2012/2013#
2013/2014#
Num
ber%o
f%assessm
ents%
Academic%Year%
CA#
Final#
NUS E-EXAMS @ 2009
Problems#• Prior#system#not#feasible#in#the#long#term#
• A#lot#of#possible#security#loopholes#make#it#unsuitable#for#high#stakes#exam#
Conduct#feasibility#study#on#commercially#available#solu'ons#• None#of#the#reviewed#solu'ons#are#able#to#meet#our#requirements#completely#
Conclusion#• Ahempt#to#do#an#inUhouse#solu'on#
NUS E-EXAMS @ 2009 Inhouse#Solu'on#+#PC#Cluster#PC#clusters#• Laptop#are#costly#and#bulky#• Easier#to#manage#• Ownership#of#devices#reduces#risk#of#chea'ng#
InUhouse#Solu'on#• Backend#dashboard#to#setup#and#manage#exam#
session#• Implement#the#following#lockdown#protocols#
• Limit#accessible#URLs,#Ingress#and#Egress#Firewall#
• Limit#keystrokes#and#key#combina'ons#• Restrict#access#to#local#storage#• Prevent#the#use#of#USB#devices#• Limited#'me#access#
NUS E-EXAMS @ 2009
Pros#• Easy#for#student,#just#come#and#take#exam#
• Easy#for#staff#to#create#ques'ons#• Auto#mark#solu'on#ensures#zero#errors#
• Ease#of#system#administra'on#
• Stricter#security#controls#compared#to#the#previous#implementa'on#
• Stricter#authen'ca'on#protocols#compared#to#previous#implementa'on#
Cons#• Lengthy#setup#and#teardown#• Connec'vity#issues#• User#constantly#experience#system#slowness#
• Noisy#keyboards#caused#distrac'on#to#students#
• Not#scalable#• Only#able#to#work#in#a#PC#Cluster#• License#costs#• Does#not#completely#lock#down#the#system#
Lockdown Software
BYOD / PC Cluster
Inhouse solution PC Cluster
VPN PC Cluster
E-EXAMS SINCE 2006
0#
5#
10#
15#
20#
25#
2006/2007#
2007/2008#
2008/2009#
2009/2010#
2010/2011#
2011/2012#
2012/2013#
2013/2014#
Num
ber%o
f%assessm
ents%
Academic%Year%
CA#
Final#
NUS E-EXAMS @ 2012 Problems#• Macbooks#and#BYOD#becoming#popular,#but#current#
system#is#incompa'ble#with#both#trends#
• Constant#upgrades#required#to#fix#new#security#vulnerabili'es#and#opera'ng#system#taking#a#toll#on#limited#resources#
• New#requirements#
• open#book#
• Increased#class#size#(>#200)#
Conduct#feasibility#study#on#available#solu'ons#• Respondus#Lockdown#Browser#found#to#be#suitable#
candidate#with#caveats#(LDB)#
NUS E-EXAMS @ 2012
LDB#+#PC#Cluster#or#BYOD#• LDB#supports#lockdown#of#both#Windows#and#Mac#plakorms#
• Implement#the#following#lockdown#protocols#
• Limit#accessible#URLs#
• Limit#keystrokes#and#key#combina'ons#
• Restrict#access#to#local#storage#
NUS E-EXAMS @ 2012
Pros#• Easy#for#staff#to#create#ques'ons#• Auto#mark#solu'on#ensures#zero#errors#
• Support#BYOD,#Windows#and#Mac#plakorms#
• Support#viewing#of#PDF#documents#
Cons#• Students#need#to#install#and#verify#that#LDB#runs#on#their#machine#before#the#exam#
• Limited#lockdown#capabili'es#compared#to#previous#implementa'on#
• BYOD#introduces#new#challenges#compared#to#using#a#PC#Cluster#e.g#dedicated#electrical#socket,#stable#network#connec'vity#
• S'll#no#support#for#Math#and#freehand#inputs#
• Possible#increase#in#manpower#requirements#due#to#BYOD#
CASE STUDY – LSM1301 E-EXAM @ NOV 2013
Background#• LSM#1301#–#General#Biology#• Enrollment#–#672#students#• Final#Examina'on#–#Open#book,#mul'ple#choice,#BYOD#
#First#LARGE#scale,#high#stakes#eUexam#in#NUS#
CASE STUDY – LSM1301 E-EXAM @ NOV 2013
Various#stakeholders#involved#to#ensure#objec'ves#are#met#• Computer#Centre:#Core#Infrastructure#
• Centre#for#Instruc'onal#Technology:#BYOD#eUexam,#technical#consultant,#single#point#of#contact,#training#
• Department,#faculty,#Registrar’s#office:#Logis'cal#support#
Objec've#• Ensure#students#are#acceptable#and#prepared#with#the#change#in#exam#format#
• Ensure#actual#eUexam#runs#as#smoothly#as#possible#for#the#672#students#
CASE STUDY – LSM1301 E-EXAM @ NOV 2013 Prepara'on#• Trial#run/mock#exam#
• Prepare#the#students#for#change#• First#point#of#contact#with#actual#devices#to#
be#used#in#exam#• Prior#iden'fica'on#of#poten'al#problem#
areas#to#prevent#these#to#occur#during#the#actual#exam#
• Standby#devices:#10%#of#cohort#• Equipment#test#at#actual#exam#loca'ons#• Standby#equipment#e.g#wireless#AP,#switches#
• Backup#exam#loca'on#
CASE STUDY – LSM1301 E-EXAM @ NOV 2013
Technical#issues#that#occurred#Mock#Exam#• Sonware#incompa'bility:#CIT#ahempted#to#fix#
the#incompa'bility,#most#of#them#were#caused#by#the#students#themselves#rather#than#the#sonware#
• Lockdown#circumven'on:#Under#certain#condi'ons#the#lock#down#is#circumvented,#these#are#then#reported#back#to#vendor#for#them#to#fix#
##
#
#Actual#Exam#• 5%#U#Power#surge:#Students#might#be#using#an#
incorrect#power#adapter,#causing#a#power#surge#and#blanking#out#power#to#other#students#
• 10%#U#Laptop#crashing/hanging/unable#to#connect#to#WIFI#
• 2%#U#Inconsistency/Limita'on#of#lockdown:#Due#to#the#different#opera'ng#systems#and#devices,#some#of#them#have#a#different#level#of#lockdown#compared#to#the#rest#(e.g#the#ability#to#use#the#built#in#dic'onary)#
STUDENTS’ PERCEPTION STUDY
Exam#–#22#Nov#2013#672#students#
PreUexam#survey#–#midUOct#2013#658#students#(98%#response)#
PostUexam#survey#–#midUDec#2013#399#students#(59%#response)#
Focus#group#discussions#–##mid#to#end#Mar#2014#15#students#(2%#response)#
PRE-EXAM AND POST-EXAM SURVEYS
0%#
10%#
20%#
30%#
40%#
50%#
60%#
70%#
80%#
PreUExam# PostUExam#Nega've# Posi've#
QUALITATIVE FEEDBACK – PRELIMINARY FINDINGS
Posi've#• More#efficient#
• Clicking#vs#shading#• Referencing#of#notes##(‘ControlUF’#or#‘CommandUF’#keys)#
• Personal#learning#space#• Familiarity#with#own#laptop#
• Well#organised#• Due#to#useful#trial#run#and#mock#exam#
Nega've#• Technical#issues#
• Dropped#connec'ons,#slow#downloading#of#notes#
• Unable#to#review#webcast#recordings#of#lectures#
• Limita'ons#of#lockdown#browser#
• Exam#taking#techniques#• Markings,#notes#on#ques'on#paper#
• Quaran'ne#
CLOSING NOTES
Conclusions#• Pedagogy#influences#examina'on#modes#which#in#
turn#affects#the#implementa'on#
• Closed#book#=#Full#or#limited#lockdown#
• Open#book#=#No#or#limited#lockdown#
• Closed#book/full#lockdown#is#close#to#impossible#with#a#BYOD#implementa'on#even#with#commercial#solu'ons#
• Open#book#with#no#or#limited#lockdown#is#possible#with#certain#commercial#solu'ons#
• Constant#maintenance#and#upgrade#required#to#ensure#solu'on#is#ahead#of#hackers#and#students#trying#to#beat#the#system.#
• Anything#short#of#no#lockdown#is#a#temp'ng#treat#for#Compu'ng#students,#showing#success#in#hacking#the#system#is#beher#than#scoring#100%#for#the#exam#
• High#stakes#nature#introduces#addi'onal#complexity#
• How#long#do#we#need#to#keep#the#data#and#ensure#backward#compa'bility?#
• Do#we#need#to#consider#a#more#stringent#electronic#supervision#to#ensure#security#and#fairness?#
• How#far#do#we#go#to#audit#student’s#ac'ons?#
• Is#there#a#need#to#cater#for#N+2#failure,#to#ensure#that#even#if#everything#fails,#there#is#s'll#a#backup?#
WHAT’S NEXT?
• Looking#for#an#easy#to#deploy#solu'on#that#is#as#effortless#as#the#flick#of#a#switch#• Currently#no#holy#grail#solu'on#that#is#able#to#address#all#modes#of#exams#
• Adapta'on#of#new#technologies#in#eLearning#is#picking#up#pace,#thus#a#comprehensive#solu'on#maybe#forthcoming.##
What#do#we#want?#
U A#secure#and#usable#solu'on#that#is#expensive##
U A#usable#and#cheap#solu'on#that#is#insecure,#or#
U A#secure#and#cheap#solu'on#that#is#not#usable?#
e-Assessment Scotland 2014 Online Presenter contact:
[email protected] [email protected]
With thanks from your hosts Professor Geoff Crisp, Dean Learning and Teaching, RMIT University geoffrey.crisp[@]rmit.edu.au Dr Mathew Hillier, Institute for Teaching and Learning Innovation, University of Queensland mathew.hillier[@]uq.edu.au
eAS Online Conference
Related Documents
