With a classmate: IDENTIFY at least TWO ways to assess your students’ learning in new/modified ways CLARIFY how these ways of assessing learning.

TESTING IN THE WORLD OF HEALTH EDUCATION With a classmate:

IDENTIFY at least TWO ways to assess your students’ learning in new/modified ways

CLARIFY how these ways of assessing learning could help (a) your students, (b) you as the instructor, or (c) both!

CREATE one or two assessment items of a type that could contribute to your project (or to your own teaching)

Evidence-Based Teaching, Spring, 2012Drs. Anne Belcher & Linda Adamson, Instructors

Session 9

OBJECTIVES

At the end of this session, the learners will be able to demonstrate the ability to use the following strategies for the evaluation of teaching and learning: Formative feedback Summative/course evaluation Peer review

FORMATIVE FEEDBACK

Purposes of formative feedback Guides future learning and teaching (evidence

suggests that appropriate, well-delivered formative evaluation can significantly improve learning processes and outcomes)*

Promotes reflection in the learner and in the teacher

Shute (2007); Epstein (2007)

FORMATIVE FEEDBACK

More effective when it provides details about how to improve (elaboration) rather than just indicate the student’s/teacher’s work is correct (verification)

FORMATIVE FEEDBACK

Is intended to help the learner to modify their thinking or behavior

Should permit comparison of actual performance to an established standard

Think: non-evaluative; supportive; timely; credible; specific; learner-driven

FORMATIVE FEEDBACK AND THE MURDER MYSTERY*

Motive—the learner needs the motivation

Opportunity—the learner receives the feedback in time to use it

Means—the learner is ready and willing to use the feedback

*Shute; Elliott

REFERENCES ON FORMATIVE FEEDBACK Archer, J.C. (2010). State of the science in health

professional education: Effective feedback. Medical Education, 44, 101-108.

Black, P. & William, D. (2007). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80 (2). http://www.pdkintl.org/kappan/kblas9810.htm

Epstein, R.M. (2007). Assessment in medical education. New England Journal of Medicine, 365 (4), 387-396.

Shute, V.J. (2007). Focus on formative feedback. Princeton: Educational Testing Service.

COURSE EVALUATION

Review examples from Schools of Nursing, Medicine, and Public Health that have been made available in class tonight

ISSUES TO CONSIDER WITH REGARD TO STUDENT RATINGS

“just a popularity contest” “just plain unreliable and invalid” Students “too immature, inexperienced, and

capricious” Concern that teacher can “buy” good ratings Easier to get good ratings in higher level

courses Rating of required course harsher than of

elective course

COURSE EVALUATION

Why are the response rates low and what can be done about that?

What should be done with the data/reports?

COURSE EVALUATION REFERENCE

Arreola, R.A. (2007). Developing a comprehensive faculty evaluation system. A guide to designing, building, and operating large-scale faculty evaluation systems. Bolton, MA: Anker Publishing Company, Inc.

JHU SCHOOL OF NURSING PEER REVIEW OF TEACHING

The two components of the JHUSON process are peer observation in the classroom setting and peer review of course materials (including Blackboard course sites)

According to Arreola (2007), peers can provide the following categories of information:

Content-dependent performance, including currency of the content, appropriateness of the sequencing of the content to help learners achieve the objectives of the course;

Content-independent performance, for example, service to the institution

BACKGROUND OF PEER REVIEW

Is formative evaluation of classroom interactions clinical instruction, and/or teaching materials

Is intended to be developmental rather than judgmental

Provides opportunity for teacher to receive collegial feedback that complements that received from students

GENERAL GUIDELINES FOR PEER REVIEW Non-participant observation At least once during the course Date, time and length of observation

negotiated with teacher Ideally a content reviewer and a

pedagogy reviewer

TEACHER INFORMATION TO REVIEWER Number of students in the class Description of the classroom

environment Copy of the course syllabus and lesson

plan Audiovisual materials, including power

point List of issues/strategies/concerns

PEER REVIEW FEEDBACK

Creation of a written report Participant debriefing

ISSUE TO CONSIDER

What is the relative advantage of a review: “live recorded

REFERENCES

Arreola, R.A. (2007). Developing a comprehensive faculty evaluation system. Bolton, MA: Anker Publishing Company, Inc.

Chism, N. V. N. (2007). Peer review of teaching. A sourcebook. Bolton, MA: Anker Publishing Company, Inc.

 For the purposes of this JHUSON peer review process, the focus will be on content-dependent performance and pedagogy.The general guidelines proposed for the peer observation are:Each observation will, whenever possible, be conducted by a content expert and an expert in pedagogy; these experts can be internal (within the SON) or external (within JHU)

The length of the observation will also be negotiated (a sample in time versus the entire class)The revised Faculty Evaluation Forms (based on form created by Berk et.al, as well as a seminar form to be created) will be used by the reviewer for recording observations in the classroom setting.

The expert/reviewers will have been trained in the use of the Faculty Evaluation Forms and the procedures to be followed.

The non-participant observations should occur at least once in the course, the date and time to be negotiated by the faculty reviewee in collaboration with the reviewer(s). If possible the reviewee should arrange to have the class media-sited for followup anaylsis with the reviewer.

 The faculty member to be reviewed will share a list of any concerns/questions about their teaching in preparation for the observations as well as the course syllabus, “lesson plan” and/or power point presentation one week prior to the class. If the review is of teaching materials, access to the Blackboard course site will also be provided.

The reviewer(s) will conduct a debriefing session with the reviewee within a reasonable period of time after the observation(s). The reviewer(s) will provide the reviewee with a written report which the reviewee may choose to submit, in combination with the reviewee’s response to the report and plans for faculty development, with his/her portfolio to the A&P Committee.Evaluative data to be submitted by the reviewer(s) and the reviewee to the OTE is the completed Peer Review Survey regarding satisfaction with the peer review process and usefulness of the process.

ReferencesArreola, R.A. (2007). Developing a comprehensive faculty evaluation system. Bolton, MA: Anker Publishing Company, Inc.Chism, N. V. N. (2007). Peer review of teaching. A sourcebook. Bolton, MA: Anker Publishing Company, Inc.

Next Steps in Teaching Project:

Evaluation Items/Rubric: How will you gather evidence of your students’

learning?

How will you use it to identify different levels of learner success?

How will results help inform your subsequent teaching?

Chism (1999) cites evidence that peers are most suited to judge the following teaching-related behaviors of faculty:Up-to-date knowledge of subject matterKnowledge of what must be taughtKnowledge and application of the most appropriate or most adequate methodology for teaching specific contentKnowledge and application of adequate evaluative techniques for the objectives of his/her coursesProfessional attributes and behaviorsAttitude toward and commitment to students When peer observation of classroom performance is used for summative evaluation, the following guidelines should be followed (Arreola, 2007):Develop or adopt a valid, reliable observational checklistAssemble and train a peer observer teamSchedule multiple visitsPrepare the studentsPrepare the instructorSchedule a post-observation conference For the purposes of this JHUSON peer review process, the focus will be on content-dependent performance and pedagogy.The general guidelines proposed for the peer observation are:Each observation will, whenever possible, be conducted by a content expert and an expert in pedagogy; these experts can be internal (within the SON) or external (within JHU)The expert/reviewers will have been trained in the use of the Faculty Evaluation Forms and the procedures to be followed.

The non-participant observations should occur at least once in the course, the date and time to be negotiated by the faculty reviewee in collaboration with the reviewer(s). If possible the reviewee should arrange to have the class media-sited for followup anaylsis with the reviewer.

The length of the observation will also be negotiated (a sample in time versus the entire class)The revised Faculty Evaluation Forms (based on form created by Berk et.al, as well as a seminar form to be created) will be used by the reviewer for recording observations in the classroom setting.Rubrics are to be developed for review of course materials using Quality Matters as a guide.Factors to be addressed/data to be collected by the expert/observers include:

The number of students in the class

The classroom environmentThe time of dayThe focus of the course—

didactic, didactic/clinical, or seminar The faculty member to be reviewed will share a list of any concerns/questions about their teaching in preparation for the observations as well as the course syllabus, “lesson plan” and/or power point presentation one week prior to the class. If the review is of teaching materials, access to the Blackboard course site will also be provided.The reviewer(s) will conduct a debriefing session with the reviewee within a reasonable period of time after the observation(s). The reviewer(s) will provide the reviewee with a written report which the reviewee may choose to submit, in combination with the reviewee’s response to the report and plans for faculty development, with his/her portfolio to the A&P Committee.Evaluative data to be submitted by the reviewer(s) and the reviewee to the OTE is the completed Peer Review Survey regarding satisfaction with the peer review process and usefulness of the process.ReferencesArreola, R.A. (2007). Developing a comprehensive faculty evaluation system. Bolton, MA: Anker Publishing Company, Inc.Chism, N. V. N. (2007). Peer review of teaching. A sourcebook. Bolton, MA: Anker Publishing Company, Inc.

Revisit tonight’s learning objectives.

Evaluate the extent to which each objective was successfully addressed.

Describe any new understandings, greater clarity, or additional insights that you developed during this session.

Provide a suggestion for how this session could have been more effective.

Reflect on Learning

Elzubieir, M. & Rizk, D. (2002). Evaluating the quality of teaching in medical education: Are we using the evidence for both formative and summative purposes? Medical Teacher, 24(3), 313-319. Retrieved from http://informahealthcare.com/doi/abs/10.1080/01421590220134169

Hatzipanagos, S. & Lygo-Baker, S. (2006). Teaching observations: A meeting of minds? International Journal of Teaching and Learning, 17(2), 97-105. Retrieved from http://www.isetl.org/ijtlhe/pdf/IJTLHE52.pdf

Nilson, L. B. (2010). Teaching at its best: A research-based resource for college instructors (3rd ed.). San Francisco: Jossey-Bass, 315-328.

Additional References for Further Reading