8.5 x 11 Raw CPOP form A New Classroom Practices Observation Protocol. Francis Jones <[email protected]> Science Education Initiative Instructional mode D Delivery (lecture, probs., Stu. q’ns, etc. … see “Happening”) E Experiment / Simulation / Demo M Media: Video, Anim’n, photo (other than usual ppt) S Socratic (continuous question posing) Q Question to students, not Socratic (open, simple clicker , etc.) A Active students (eg clicker sequence , worksheets, etc.) P Presentation by student(s) T Test or quiz (include groups if two-stage or TBL, etc.) Instructor is doing … s Static or low key - Talking m moving, interacting with screen, etc. - Talking w Wandering around class - Talking R Real time writing (board, doc. projector, etc.) H Helping or guiding student work (eg circulating) L Listening or marking (during presentations etc. ) O One-on-one: focus on individual (class may be listening) N Nothing – waiting for activity, etc. Happening or covering E Explain new content, knowledge or procedure S Summarizing or synthesizing R Review, revisit, refer to prior content OR knowledge framework. P Practice/Apply; problem, thinking, analysis, etc. C Case study or example(real world; may be with “P”) F Follow up, feedback on work or thinking. B Brainstorming or novel thinking M Motivational (“here’s why we’re doing this”, etc.) H Humor or just friendly A Administration (assign hmwrk, return tests, etc) O Other – explain in comments. Students Doing I Individual (listening or doing) C Contributing (e.g. explain in turns, etc.) Q Student asks question P Pairs (or peer instruction) G Groups (note group size in comments) W Whole class (“shout out”, discussion, etc) Blooms Taxonomy level (updated – more action oriented) R/U remembering / understanding A/A applying / analyzing E/C evaluating / creating Judge based on verbs that are in use (instructor or activity). Blooms Taxonomy level (original from 1956) K / C knowledge / comprehension A / A application / analysis S / E synthesis / evaluation Eng = Engagement observations: Here’s how to do this … 1) Select N students to observer. 2) Enter “X/N” = count of #students engaged (or not “disengaged”) in what’s going on. Key: “Key Flag” helps identify key messages for feedback to the instructor. Observation Codes Potential purposes • Characterize courses: - Student experiences, instructor actions, others • Course Transformations Impact Assessments - After transformation; pre-post if possible • Compare courses … eg. all 1xx; core vs. elective; etc. - Busy / idle students; balance of motivation, basics, context; theory / practical; etc. • Professional and peer development for instructors: - Based on code patterns and observer feedback. Precedent • EOS – “engagement observation protocol” (E. Lane) - Included in CPOP • UBC - Physics feedback forms (P. Newbury and C. Heiner) • RTOP – Reformed Teaching Observations Protocol • TDOP 1 – Teaching Dimensions Observation Protocol1 - Foundation of CPOP - Explored in Math (W. Code) Trials to date: EOAS courses: 1xx courses: 4 2xx courses: 5 3xx courses: 7 Eosc222: 12 classes, Jan-March Procedure and data: • Observation form stabile at version 10. • Forms printed on LiveScribe paper. • Audio records are keyed to coding on forms. • Coded forms transferred to spreadsheet template, including color and aggregation. eosc326, Nov 16th, 2011 DSQEA M P T sm wRHLON E S RP C F B M HAO I CQP G W RUAA EC 0 10% 2 20% 4 60% 6 75% 8 0% 10 0% DSQEA M P T s mwRH L ON E S RP C F B M HAO I CQP G W RUAA EC Eng 12 83% 14 69% 16 50% 18 50% 20 0% DSQEA M P T s mwRH L ON E S RP C F B M HAO I CQP G W RUAA EC Eng 22 65% 24 0% 26 0% DSQEA M P T s mwRH L ON E S RP C F B M HAO I CQP G W RUAA EC Eng 28 0% 30 94% 32 0% 34 0% 36 0% 38 0% 40 0% DSQEA M P T s mwRH L ON E S RP C F B M HAO I CQP G W RUAA EC Eng 42 0% 44 0% 46 0% 48 0% 50 0% Eng Inst Mode Inst. do. Happen/Cover Stu. doingBlooms eosc326, Nov 18th (active Friday), 2011 DSQEA M P T sm wRHLON E S RP C F B M HAO I CQP G W RUAA EC 0 2 4 6 8 10 DSQEA M P T s mwRH L ON E S RP C F B M HAO I CQP G W RUAA EC 12 14 16 18 20 DSQEA M P T s mwRH L ON E S RP C F B M HAO I CQP G W RUAA EC 22 24 26 DSQEA M P T s mwRH L ON E S RP C F B M HAO I CQP G W RUAA EC 28 30 32 34 36 38 40 DSQEA M P T s mwRH L ON E S RP C F B M HAO I CQP G W RUAA EC 42 44 46 48 50 DSQEA M P T s mwRH ON E S RP C F B M HAO I CQP G W RUAA EC 52 54 56 58 60 DSQEA M P T s mwRH L ON E S RP C F B M HAO I CQP G W RUAA EC Su act lect Inst Mode Inst. do. Happen/CoverStu. doingBlooms Mode Instructor Happening Covering Students Blooms Lecture “Active Friday” Mode Instructor Happening Covering Students Blooms Time Time Completed raw data form Transcribed raw data: 2 classes of eosc326 Data Characteristics and participation in SEI Some features to notice: • The “Not eosc” courses were active & application oriented, but did not use groups or peers much. • “No SEI”: These involved more delivery, more telling, and more solo time in class. • “SEI applied” tended to be more active, applying, in groups, and there is also a wider variety. Data recorded at 2-minute intervals in 1 st , 2 nd and 3 rd yr courses. 0% 50% 100% eosc114 eosc112 eosc221 eosc373 eosc326a eosc322 eosc222a eosc222b eosc332 eosc340 eosc110 eosc116 eosc270 eosc333 eosc326l atsc201 envr200a envr200b envr300 S,Q,A D,E,M 0% 50% 100% eosc114 eosc112 eosc221 eosc373 eosc326a eosc322 eosc222a eosc222b eosc332 eosc340 eosc110 eosc116 eosc270 eosc333 eosc326l atsc201 envr200a envr200b envr300 P,C,B E,S,R,M,H,A 0% 50% 100% eosc114 eosc112 eosc221 eosc373 eosc326a eosc322 eosc222a eosc222b eosc332 eosc340 eosc110 eosc116 eosc270 eosc333 eosc326l atsc201 envr200a envr200b envr300 P,G,W I,C,Q EOS; SEI applied Not EOSC EOS; No SEI Instructional mode: Active vs Delivery What happens in class: Applying vs Telling etc. What students are doing: Pairs/groups vs Solo Six weeks of EOSC222 classes: What analysis possibilities? Address questions by combining codes: Variations versus time: – More group work in Feb. – “Active” tends to follows basics. Sort by Bloom’s score: Score = 1*R+2*U+3*A+4*A+5*E+6*U - Higher Blooms … more active. 0% 50% 100% active deliver 0% 50% 100% active deliver Lower Bloom’s Higher Bloom’s 0% 50% 100% pairs/grps solo Students: solo vs. pairs or groups Instructional mode Some Patterns – 19 different classes Sorted by Blooms score (see codes, below left) low Bloom: – mostly 1st, 2nd yr. (BUT see exceptions) – high "telling" instr. Mode – high "explain or review" cover high Bloom: – mostly 2nd, 3rd yr. (BUT see exceptions) – more students working together – exception is eosc222 poster session eosc112 – high blooms, but… – low engagement eosc114 – low blooms, but… – strong engagement envr300 – high blooms, and.. – most diverse instructor doing – All guided group work eosc340 – moderate blooms – very low "explaining“ – diverse instr. & stu. doing – engage starts low stays high throughout 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% eosc222a envr200b eosc114 eosc332 eosc270 eosc333 eosc116 eosc110 eosc322 atsc201 eosc221 eosc373 eosc340 eosc326l eosc326a eosc222b eosc112 envr300 envr200a Blooms C E A A U R 0 2 4 6 8 10 12 14 16 18 20 eosc222a envr200b eosc114 eosc332 eosc270 eosc333 eosc116 eosc110 eosc322 atsc201 eosc221 eosc373 eosc340 eosc326l eosc326a eosc222b eosc112 envr300 envr200a Happen / cover R S E 0 2 4 6 8 10 12 14 eosc222a envr200b eosc114 eosc332 eosc270 eosc333 eosc116 eosc110 eosc322 atsc201 eosc221 eosc373 eosc340 eosc326l eosc326a eosc222b eosc112 envr300 envr200a Stu. doing W G P 0 2 4 6 8 10 12 14 16 18 20 eosc222a envr200b eosc114 eosc332 eosc270 eosc333 eosc116 eosc110 eosc322 atsc201 eosc221 eosc373 eosc340 eosc326l eosc326a eosc222b eosc112 envr300 envr200a Instr. mode M E D 0% 20% 40% 60% 80% 100% 0 10 20 30 40 50 60 Percentage of sample who were "engaged" Minutes into the class eosc340 eosc114 eosc110 eosc112 Engagement: 4 courses, 1 class each Comparing two science electives: D 67% S 0% Q 11% E 0% A 22% M 0% P 0% T 0% Instr. mode, 373 D 30% S 7% Q 13% E 0% A 50% M 0% P 0% T 0% Instr. mode, 340 I 46% C 6% Q 8% P 8% G 24% W 8% Stu. doing 340 I 78% C 7% Q 0% P 15% G 0% W 0% Stu. doing 373 Instructional mode Students doing EOSC 340 Climate change EOSC 373 Oceanography II NOTES and caveats: - Each course was observed only once, and no information is used relating to what students do outside of class. - Also, the observation protocol has not yet been validated by having several people conduct simultaneous observations. - There may be other shortcomings; the project is definitely in only the “development” stages. ABSTRACT: A new observation procedure is being developed aimed at helping characterize instructional practices, student actions, "Bloom's levels" and other classroom characteristics, on a timeline of the class. The intention is to develop a procedure that is useful for characterizing any type of class, rather than to make judgments about "quality" in any sense. Preliminary examples of trials are shown with some possibilities for analysis; including comparisons of several classes from one course, & single classes from 16 different courses.