Online Learning Tools (workshop) John O’Byrne (Sydney) –Mastering physics software: 40 min Alex Merchant (RMIT) –Online assignments: 20 min Geoff Swan.

Online Learning Tools (workshop)

•John O’Byrne (Sydney)–Mastering physics software: 40 min

•Alex Merchant (RMIT)–Online assignments: 20 min

•Geoff Swan (ECU)–WebCT Quizzes: 20 min

Key Issues in Learning and Teaching in Undergraduate Physics – A National Workshop (16:00 to 17:30 Sept 28, 2005, University of Sydney).

WebCT Quizzes: Points of view

• Student viewpoint: ACTIVITY– Do the autc05 quiz at least twice checking grades and

individual question feedback

– Use student logins & passwords (autc1 to autc12)– http://thomson4.webct.com/public/swanserwaycowan/index.html

• Staff viewpoint: Show– Quizzes & individual questions (eg ECU example)

• Creating new questions and quizzes?

– Course management• Results (graph) & download

– Other?

THE END

Key to Participation

Geoff Swan

Physics Program

School of Engineering and MathematicsFaculty of Computing, Health and Science

Edith Cowan UniversityPerth, AUSTRALIA

AIP Congress, Canberra ACT (Feb 1, 2005)

Outline• PART 1: Setting the scene

– Background– Online quizzing environment– 2003 trial

• PART 2: Online quizzes in 2004– Major Adjustments from trial– Results

• Participation rates• Unit grades• Student evaluation• Open responses issues

• PART 3: Discussion

Background• Setting: Physics of Motion – first year first sem physics unit

– Engineering, Aviation, Physics major & Education students– Assessment: 60% Exam, 20% Labs, 20% test and/or assignments?– Homework: selected end of chapter problems (Serway &

Beichner/Jewitt). Not assessed. Students have worked solutions• Problem (anecdotal): Struggling students do insufficient

physics at home.– Time: Competing demands study vs work vs family etc– Motivation (affected by many factors)

• A Solution? Provide regular online quizzes– Engage students in regular problem solving– Help students keep up with the curriculum– Aim at struggling students – increase confidence– Encourage good study practices early (ie sem 1, year 1)– Provide early & continuous feedback (positive and negative)– Improve students’ overall performances

Motion Quizzes Trial. Sem 1, 2003Quiz No. & Topic(s) Set Week Set Assessment0 ECUGUEST: check out ID & Password + quiz site 1 NONE

1 Motion in one dimension, Vectors 2 Highest

2 Motion in two dimensions 3 Average

3 Newton's Laws 4 Average

4 Circular motion dynamics, 6 Average

Work & Kinetic Energy

5 Conservation of Energy, Momentum 7 Average-------------------------------------mid-semester break-----------------------------------------

6 Rotational Motion 9 Average

7 Static Equilibrium, Elasticity, 11 Average

Oscillatory Motion

8 Gravitation, Fluids 12 AverageNotes: 12 week term. Max 70 minutes for each quiz (4-6 questions). Available after

topic material covered. Allowed about 8 days to complete up to 2 attempts.

Student orientated 1

• Flexible delivery– Attempt: any place (home or ECU or ?) & any time

• Fit in with other commitments

– Guest quiz• Rewards

– Intrinsic: better understanding– Assessment: use to improve final grade (if better than

mid-semester test: 20%)– Revision: have extra revision questions and answers

for exam• No access at all if online quiz not attempted within

timeframe

Student orientated 2 (Feedback)• Feedback

– General (how am I going in this physics unit?) - Grades– Specific (how do I answer this question?) - Detailed

Feedback!• NO detailed feedback provided by publisher’s test bank

• Immediate Feedback (within seconds!)– Grades and detailed feedback

• Any time and any place

• Can act straight away on detailed feedback– Attempt a similar but different quiz (2003 wait of 30

minutes)• strike while the iron is hot!• Can improve grade• Formative assessment

• Significant advantages exist with an online environment

Online advantages for the Lecturer• Save TIME? (for large classes)

– existing publisher supplied WebCT test bank (1000 questions but no detailed feedback!) and server

– automatic marking– results easily transferred to excel mark sheet

• Reliable assessment– Individual tests (similar but different)

• Alternate multiple choice questions• Calculated questions (truth tables - usually 20 variations)• individual quizzes = harder to cheat plus 2nd attempt option

• Flexible settings– questions, assessment, access and timing options

Belief: Both pedagogical and practical advantages to using the online environment

WebCT Question types• Multiple choice

– 1000 question test bank (no detailed feedback)

• Calculated– Use formula to create truth table.

• ie same question but (say) 20 alternate sets of numbers

– Supported math operators:• ( ), +, -, *, /, **, sqrt, sin(x), cos(x), exp, log (base e ie ln)

• Also available (but not used):– Paragraph (tutor marked)– Matching– Short answer

Example: Masses on strings 1

Example: Masses on strings 2

FLAG – Your opinion??• RATE the detailed/general feedback

– Consider a borderline first year first semester physics student

– to SOLVE this question FOR THEMSELVES in a second attempt

1. Too much detail2. About right3. Not enough detail• Compare with the person next to you.• How does this compare with a worked

solution?

2003 online quizzes trial - RECAP

• Online quizzes: Physics of Motion trial in sem 1, 2003– Engage students with regular problem solving– 8 quizzes. Two attempts allowed (average mark normally used)– Optional assessment (if better than mid-sem test)– WebCT server (supplied by publisher)

• Two types of questions– Calculated

• Use formula to create truth table. – ie same question but (say) 20 alternate sets of numbers

• Supported math operators:– ( ), +, -, *, /, **, sqrt, sin(x), cos(x), exp, log (base e ie ln)

– Multiple choice• Based on 1000+ multiple choice question testbank (no detailed feedback)

• Detailed Feedback written for each question

2003 Results - SUMMARY• Students LIKED the Quizzes

– 85%+: Quizzes easy to access, relevant, improved my understanding of physics, helped me develop problem solving skills etc (N=29 with at least one quiz)

• 80% Detailed feedback about right (20% not detailed enough)

• Participation was POOR! (Of N=63 students who attempted the exam)– 17% attempted 6,7 or 8 quizzes

• No fails in this qroup! One student passed directly due to quizzes!!– 21% attempted 4 or 5 quizzes , 32% attempted 2 or 3 quizzes , 6% attempted 1 quiz

– 24% attempted ZERO quizzes.• Four times more likely to fail?

– the 19 students who completed zero or one quiz accounted for two thirds of the unit’s fail & borderline pass grades (ie 63% of these students compared with 14% of students who completed more than one quiz)

• Why zero quiz attempts? Lack of TIME!• Main Factors given were (N=7): bloody hard, laziness, problems with internet

connect - difficult to complete quizzes, time, don’t have time/hate physics anyway, hard to allocate solid 90 (sic) minutes in first few weeks and never got into it, work.

• MUST increase participation rate in 2004!

Changes in 2004• ASSESSMENT

– Compulsory assessment 20%• Replaced mid-semester test (now just for practice)

– Highest mark rather than average of two attempts• From 1 of 8 quizzes in 2003 to 6 of 8 quizzes in 2004

• Big incentive – high marks possible for struggling students

• OTHER– Orientation quiz in week 1 (not assessed)

– Quiz time decreased (from 70 to 60 minutes)

– New questions & topics rearranged

– Research: additional survey items on• The quiz environment for students: how they went about it

• The use of detailed feedback on individual questions

Quizzes in sem 1, 2004

Participation Rates

• Participation – how many attempted at least 6 of 8 quizzes?

• Consider only students who sat end of semester exam– 2003 N=63

– 2004 N=84

• 2003: Just 17%

• 2004: 85%– HUGE increase in participation rates

Quiz and Unit ResultsConsider only students (N=84. sem 1, 2004) who sat exam

• Pass rates– Quizzes: Pass rate 70% (Distinction rate 43%)

• Note: 15% students with 5 or less quizzes all failed here

– Unit: Pass rate 81% (Distinction rate 27%)

• 27% students received a distinction for the unit– 87% also received quiz distinction (ie 20 of 23 students)

• 19% students failed unit– 75% also failed quizzes (ie 12 of 16 students)

– Highest quiz mark for failing student was 58%• Below median average for quizzes

• Quizzes as a good indicator for success and failure– Use for early intervention!?

Student evaluation of quizzes• Students were extremely positive (% agreement)

Respondents (N=34) with at least 6 or 7 quizzes. Survey in penultimate teaching week.):

– Were easy to access (97%)– Were relevant to the unit content (97%)– Improved my understanding of physics (94%)– Provided me with necessary practice in solving physics

problems (91%)– Helped me develop problem solving skills (94%)– Detailed feedback was helpful (91%)– Overall, helped me learn physics (94%)

• Students normally accessed the quizzes from home (70%) and 91% of students thought that the number of quizzes (eight) was about right.

Assessment

• Assessment by online quizzes was fair and reasonable (71%)

• Future assessment– 71% online quizzes only– 29% combination of online quizzes and mid-sem test

• 62% prepared more thoroughly for the two average mark quizzes

Detailed Feedback• For my learning needs, the amount of feedback

provided to assist me in solving the problem for myself was– About right 67%– Not detailed enough 33%– Consider small increase for 2005?

• 81% found detailed feedback sometimes useful for correctly answered questions! Interesting– Not expected by author, but interesting! Subsequent

conversations reveal students like to compare the lecturer’s method (concepts used and application) with their own regardless of whether they answered the question correctly or not.

Open response questions

• Provided additional information on:– Preparation– Detailed feedback– Collaboration– Study habits

Preparation?

• Students listed a variety of ways in which they prepared for highest mark quizzes including– attending lectures– reading through lecture notes– reading chapters or chapter summaries in the textbook– doing set problems and even– reading the formula sheet.

• A significant minority were not well prepared, including one student whose preparation was:

“Nothing, I would waste the first go & work backwards from the solution”– unlikely to result in any meaningful learning– unlikely to improve their exam score.

Detailed feedback?

• How did students use detailed feedback? Three examples:

“To change the way I look at the questions”“To try and understand the concepts, but

sometimes it wasn’t enough”“I used it as a guide to lookup certain information

in the book and notes. I found it very good”• Student usage went beyond just getting the

correct answer. – Author was surprised and pleased!

Collaboration?• 44% provided help to other students• 36% received help from other students• some students seemed somewhat insulted with these

questions!?• Collaboration ranged form “nothing” or “verbal” to substantial.

Consider two substantial examples:1. “We all worked out an answer, & compared the way we did it &

the answers we got. It was through this that we were able to come to grips with theories”

2. “Group work. Do the quizzes together and work together to a solution”

FLAG: Should the collaboration described above (considering example 1 & example 2 separately) be:

– Encouraged (peer learning)– Neutral– Discouraged (it’s cheating!)

Further thoughts on collaboration• Students already collaborate in class time

– Labs: experiments done in small groups – Lectures: short small group discussions on (mainly

conceptual) questions. • Collaboration/group work/peer learning

– Necessary part of the curriculum– Supported by large body of literature & resources (eg

Mazur, McDermott etc)• Attitudes changed over time

– Individual with group support now OK• Original object was to replace mid-semester test (20% assessment)

– As long as individual learning occurs and each actual quiz attempt is done by the student logged on to the computer

• Delete second part of condition?? Students spending time interacting with and discussing physics is always a good thing!?

Study habits?

• Students fairly positive here:

• Helped me work more consistently over the semester (85%)

“The quizzes made sure that we were studying without forcing us to study”

• The quizzes encouraged and influenced both study at and away from the computer.– Consider preparation for quizzes reported by

students.

Conclusions• Provide opportunities to learn through

– Detailed feedback– Formative assessment– Collaboration

• Influence study habits– help students work consistently over the semester– Flexible time and place

• Overall help students learn physics (94%)• ECU evaluation (UTEI). BIG increase in overall

satisfaction for unit. Partly due to quizzes?– From 64% (N=28) in 2003 to 91% (N=45) in 2004

• Agree or strongly agree on 5 point scale (ie neutral counts as not satisfied)

• Online quizzes have made a difference– Adapted for second semester unit: Waves and Electricity 2004– Continue for 2005…

FURTHER INFORMATION?• Contact

– Dr Geoff Swan – Physics Program, SOEM– Edith Cowan University– 100 Joondalup Drive– Joondalup WA 6027– Tel: +61 08 6304 5447– Email: [email protected]

• Quiz site at

http://thomson4.webct.com/public/swanserwaycowan/index.html (login and password“ecuguest”)

Discussion?• How might these online quizzes be improved?

– Group work vs individual– Context: Questions in the quizzes are mostly de-

contextualised. Is there a pedagogical justification? (eg can reality be a distracter to learning physics?)

• How might resources like this be shared between institutions? (AUTC report recommendation)

• Students’ study habits and online quizzes– Is it our business to design curriculum to encourage good

study habits and/or teach study techniques?– Are we responsive enough to competing demands on students

who often combine uni & work & family? (AUTC report recommendation)

Paper references• References• Gordon, J.R., McGrew, R., & Serway, R.A. (2004). Student Solutions Manual & Study

Guide to accompany Volume 1 Physics for Scientists and Engineers Sixth Edition. Belmont: Brooks/Cole.

• Honey, M., & Marshall, D. (2003). The impact of on-line multi-choice questions on undergraduate student learning. In G.Crisp, D.Thiele, I.Scholten, S.Barker and J.Baron (Eds.), Interact, Integrate, Impact: Proceedings of the 20th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education. Adelaide, 7-10 December 2003.

• McInnis, C, James, R, & McNaught, C (1995). First year on campus: Diversity in the initial experiences of Australian undergraduates. Canberra: Australian Government Publishing Service.

• Serway, R.A., & Jewett, J.W. (2004). Physics for Scientists and Engineers with Modern Physics. (6th ed.). Belmont: Brooks/Cole.

• Swan, G.I. (2003). Regular motion. In G.Crisp, D.Thiele, I.Scholten, S.Barker and J.Baron (Eds.), Interact, Integrate, Impact: Proceedings of the 20th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education. Adelaide, 7-10 December 2003.

• Thelwall, M. (2000). Computer-based assessment: a versatile educational tool. Computers & Education, 34, 37-49.

• Thomson Learning (2004). WebCT server login page. Thomson Learning. Available: http://thomson4.webct.com/public/swanserwaycowan/index.html [July 2004]

• Additional: Peer instruction by Mazur. Tutorials in introductory physics by McDermott & Shafer.

SCP1112: Students usage and feedback• SCP1112 Waves and Electricity: Semester 2, 2001• Usage (5 full & 1 half quiz)

– Of 73 students who attempted the SCP1112 exam• 32 (44%) attempted 3 or more full quizzes

• 25 (34%) attempted at 1 or 2 full quizzes

• 16 (22%) did not attempt any quiz (for these students unit failure rate > 4 times that of other students)

• Feedback– Large majority extremely positive (mid-sem and end-sem)

• easy to use, relevant, improved understanding etc

• feedback detail about right (a few students wanted more detail!)

– More divided on• future compulsory assessment & No. of quizzes

• Helping to work more consistently over the semester