1 HOW TO BE A GOOD COURSE DIRECTOR PART 2 - HOW TO IMPROVE A COURSE Master Teacher Guild Mike Fennewald, PhD (SGPDS) Daniel Bareither, PhD (SCPM) Lecia.

1

HOW TO BE A GOOD COURSE DIRECTOR PART 2 -

HOW TO IMPROVE A COURSEMaster Teacher Guild

Mike Fennewald, PhD (SGPDS) Daniel Bareither, PhD (SCPM)

Lecia Apantaku, MD (CMS) Roberta Henderson, PT, PhD (CHP)

Fred Sierles, MD (CMS)John Becker (SCPM)

December 15, 2011

2

WHO IS RESPONSIBLE FOR IMPROVEMENT?

• Course Director:– This individual has been selected by the

department and the college to coordinate the learning activities for the course (or clerkship). They must understand that the course is not THEIRS but rather they have the responsibility for providing an excellent educational experience for the students. They are responsible for the selection and evaluation of the faculty that participate in the course.

• Department Chairs: – Responsible for selecting qualified faculty to

serve as course directors and for reviewing their performance.

3

WHO IS RESPONSIBLE FOR IMPROVEMENT?

• Deans: – Responsible for making sure the departments are

providing excellent educational experiences, that there is assessment of the curriculum, and that the students are achieving the learning outcomes.

• Assessment Committee:– Responsible for coordinating the methods of

assessment used to collect data on the curriculum, the analysis of the data, and identifying any problems and making recommendations to the Curriculum Committee and Dean.

• Curriculum Committee:– Responsible for management of the curriculum and

taking action regarding recommendations from the Dean and Assessment Committee.

4

WHAT INFORMATION CAN BE USED FOR IMPROVEMENT?

• Course/Clerkship Evaluations - student input regarding: – Organization of the

course/clerkship– Learning objectives– The learning outcomes that the

course/clerkship helps to develop– Method of evaluation– Use of learning management

system (D2L)– Integration of basic science and

clinical science information– Faculty availability– Faculty contribution to their

understanding of the course material

– Constructive comments• Note: Process for review and

feedback from evaluations (assessment committee).

5

WHAT INFORMATION CAN BE USED FOR IMPROVEMENT?

• Process:– Development of the course evaluation (assessment and

curriculum committees)– Determine method for students to complete course

evaluation– Evaluation results compiled by someone other than the

course director– Evaluation results provided to the department chair– Department chair reviews evaluation results with the

course director and recommends any actions to be taken– Evaluation results and recommended actions forwarded to

the Assessment Committee and Dean– Assessment Committee makes recommendations to the

Curriculum Committee and/or Dean– If recommendation is to the Curriculum Committee then

they make a recommendation to the Dean.• Note: Correspondence from Dr. Marinelli regarding review of the

graduate curriculum and the Oversight Committee

6

WHAT INFORMATION CAN BE USED FOR IMPROVEMENT?

• Peer Evaluations – faculty input regarding:– Course materials available to

students– Established learning objectives– Quality of visual materials– Instructor – knowledge,

interaction with students, utilization of AV, professional conduct

– Presentation – organized, understandable, emphasized major points, integrated basic science and clinical information, delivered at a good pace, timely

• Note: Need a process for review and feedback from evaluations (department chair).

7

WHAT INFORMATION CAN BE USED FOR IMPROVEMENT?

• Assessment Committee – identifies a variety of internal and external methods to assess the achievement of the student learning outcomes/competencies, and the achievement of the programmatic outcomes, and curriculum/teaching: – Internal assessment -

• ongoing performance in a course (individual exams) • overall course performance (grade distribution) compared to

other courses occurring at the same time• mean of the overall course performance compared to previous

years• performance on comprehensive exams• course evaluations• curriculum evaluations• peer evaluations

– External assessment -• performance on board exams• performance in clerkships• performance in residency • performance on licensure exams

8

WHAT INFORMATION CAN BE USED FOR IMPROVEMENT?

• Curriculum Committee:– major mechanism by which the faculty develop and review

the curriculum– coordinates the process for course evaluations– coordinates the review of the syllabi, the methods of

instruction, the scheduling of courses, the size and sequencing of courses, and receives and evaluates course proposals

– annual evaluation of the curriculum to look at the sequencing of courses, the integration of basic science and clinical science information, the development of critical thinking, and the utilization of the learning management system

– develops and reviews academic policies• Student Focus Groups:

– focus groups of students can be used to obtain additional information on a particular issue identified in the assessment results

9

WHAT INFORMATION CAN BE USED FOR IMPROVEMENT?

• Curriculum Committee:– major mechanism by which the faculty develop and

review the curriculum– coordinates the process for course evaluations– coordinates the review of the syllabi, the methods

of instruction, the scheduling of courses, the size and sequencing of courses, and receives and evaluates course proposals

– annual evaluation of the curriculum to look at the sequencing of courses, the integration of basic science and clinical science information, the development of critical thinking, and the utilization of the learning management system

– develops and reviews academic policies

10

FACULTY PERFORMANCE AND PROBLEMS

• Prevent problems.• Monitor teaching closely.• Continue to orient, mentor and

monitor.• Expect, praise and reward

improvement.• If a few cycles fail to yield

improvement, reassign the faculty member.

11

Prevent Problems (Primary Prevention)

• Primary prevention. Prevent problems before they occur.– Consider teaching ability when hiring.– Assign classes to best teachers• Not necessarily most expert on the

topic– Orient and mentor new teachers.• Prepare new teachers for first class.

12

Primary Prevention (cont.): Reward Good Teaching

• Reward Good Teaching – Offer praise,

encouragement.– Suggest—to your

chair—raises, travel support, promotion, tenure.

– Write recommendation letters.

– Nominate for teaching awards.

– In considering tenure, your chair (with your input) should ask, “What will he or she do if he or she loses this grant?”

13

Monitor Teaching Closely

• For new or “struggling” teachers – Attend, help set up, give “tech

support” for • First few lectures, some (or all) subsequent lectures• Early small group, occasional subsequent small groups

14

Monitor Teaching Closely: Prompt Oral Feedback

• Give prompt oral feedback.– During a break– Soon after class (same day, next couple of

days)• How to give feedback.

– Begin with praise.– Empathize with faculty member’s feelings,

perceptions and experiences in the class.– Follow with constructive criticism.– Summarize hopefully, end on a positive

note.

15

Monitor Teaching Closely: Review Written Feedback

• Obtain, read, distribute and review with faculty member– Student evaluations of teaching, test item

preparation– Student exam performance on faculty

member’s topic • Department exams• National exams

16

Monitor Teaching Closely: Keep Chair Posted; Maintain Records

• Keep chair apprised of, and give chair access to, above indicators.

• Obtain and maintain records of feedback received by faculty member.– Documentation crucial if faculty

member challenges decisions based on his or her teaching.

17

Follow-up Beyond Early Interactions and Feedback

• Continue to orient, mentor, monitor.• Expect and praise each effort and success at

improvement.• If sub-par teaching and failure to improve

continue after two or three cycles of monitoring and feedback,– Suggest, to chair, reassignment of faculty

member to other duties.

18

Course Director Vignettes

What Would You Do in Each Case?

19

Vignette 1• The Course Director is attending a two hour

lecture presented by an early-career faculty member who is bright, confident and flexible. During the first 25 minutes of the first 50 min., the lecturer gives multiple colorful, vivid, relevant illustrations of patients she has treated, regularly poses questions to the class, and the class is attentive and energized. Inexplicably, during the second 25 min. of the first 50 min., the lecturer provides no illustrations, poses no questions, and many class members—as well as the Course Director, nod off periodically to sleep. The Course Director should…

20

Vignette 2

• A part-time associate professor of psychiatry with nine years’ teaching experience gives a two hour, first-time, one-presentation per year lecture on a topic on which she is expert. She presents the topic at the intellectual level of an introductory college psychology class, a topic that is already “known by heart” by most of the students. Her voice is monotonous. She poses no questions to the class, takes no questions, and gives only two illustrative examples of patients she has seen. During the class, several students—and the Course Director—“nod off.” Post-course, the students give her low ratings, and their narrative comments point out the weak points noted above. The Course Director should…

21

Vignette 3

• In the anonymous student feedback following a third year required psychiatry clerkship, two students give an assistant psychiatry professor an overall rating of “F,” and each of them mentions—in the narrative—something like, ”Just awful. The only thing he had us do was draw blood for his lab experiments. He gave us no patient care responsibilities and spent virtually no time teaching us.” This fits with this faculty member’s previous behavior. The Clerkship Director should…

22

Vignette 4

• In the written anonymous student feedback following a third year required psychiatry clerkship, five of the 23 clerks spontaneously mention, in their narrative comments, something like: “Resident Dr. A gave us awesome off-the-cuff mini-lectures, and is a terrific supervisor and clinician.” Prior to his psychiatry residency, Dr. A published 17 psychiatry papers in refereed journals. The Clerkship Director should…

23

STUDENT PERFORMANCE

Using internal measures to adjust curriculum Exams Clinical Performance End of course evaluations

Using external measures to adjust curriculum National Board of Medical Examiners (NBME) subject exams Residency director questionnaires Medical School Graduation Questionnaires

24

NBME SUBJECT EXAMINATIONCONTENT AREA ITEM ANALYSIS

REPORT

25

NBME STEP I

26

Medical School Graduation Questionnaire

2011 All Schools Summary Report

27

MEDICAL SCHOOL GRADUATION QUESTIONNAIRE

28

MEDICAL SCHOOL GRADUATION QUESTIONNAIRE

29

M1 AND M2 QUESTIONNAIRES

 Please rate the professionalism with which you were treated in this course by the course faculty. Excellent, Good, Adequate, Unacceptable, Cannot evaluate

 Please rate the professionalism with which other students were treated in this course by the course faculty.

Please rate the professionalism with which you were treated by your fellow students in this course.

 Additional comments on professionalism in this course:

Lecturers and Notes/Labs: Please rate Dr. _______’s lectures: Please rate Dr. _______’s notes and related materials (e.g. audio, video, PowerPoint):

 Constructive comments.

30

M1 AND M2 QUESTIONNAIRESI attended about _______ of the lectures:

The D2L site for this course was:ExcellentGoodAdequateUnacceptableCannot evaluate

 The clinical relevance of this course was: The effectiveness of the exams to assess course objectives was:

 Integration of course content with other courses was:

 This course overall was:

31

M3 QUESTIONNAIRE

Question 1

Please rate how well the specific clerkship objectives and clerkship policies (e.g. grading criteria, examination format) were identified early in the clerkship.

Superior Very Good Acceptable Deficient

Question 2

Please rate how well significant facts, principles, and basic science concepts in this clinical area were explained.

Question 3

Please rate how well lectures and/or seminars related to the objectives.

Question 4

Constructive written comments on the lectures and/or seminars in this clerkship: (text box)

Question 5

Please rate how well chart, bedside or team rounds were conducted. Please rate how patient care opportunities directly applied the facts, concepts and principles presented in the didactic sessions.

32

RESIDENCY PROGRAM DIRECTOR QUESTIONNAIRE

• Can collect data from the following sources: patient history or history from other relevant information sources (e.g. the family) and physical exams.

• Has basic science knowledge sufficient to interpret data gathered from the patient interview and physical examination.

•  Knows the sources available for securing additional data through laboratory procedures, and can interpret such data.

•  Can construct a differential diagnosis and a plan for its confirmation or rejection.

•  Treats each patient as an integrated whole, rather than as a disease entity.

• Uses current literature and other educational resources to understand and resolve diagnostic problems.

• Can develop multidimensional treatment plans which include diet, place of care, use of medications and therapeutic procedures, and which take into account the patient’s socioeconomic and psychological status.

33

RESIDENCY PROGRAM DIRECTOR QUESTIONNAIRE

• Learns new concepts in medical literature.• Critically evaluates new concepts and interprets new data • Respects other health professionals and facilitates the

development of their skills. • Knows concepts of primary prevention, epidemiology and

healthcare delivery and accounts for socioeconomic status. • Participates effectively in peer review, especially in

providing constructive criticism of professional behavior. • Able to choose a career track appropriate to his/her abilities

and performance. • Has the basics of comprehensive health care delivery,

regardless of long-term specialty choice. • Is self-directed in solving problems.

34

LEARNING ENVIRONMENT

Using internal and external data to improve the learning environment Space Technology

35

CONTINUOUS QUALITY IMPROVEMENT

How do you measure success?

Set goals

Re-measure