Running head: EVALUATION PLAN 1 Brett Stachler & Kathryn Ward-Mytinger Loyola University Chicago
Dec 21, 2015
EVALUATION PLAN 2
Contents Theoretical Framework ................................................................................................................... 4!History of Global Health Resident Program ................................................................................... 5!Global Health Certificate Details .................................................................................................... 7!
Goals and Objectives .................................................................................................................. 7!Goals, Objectives, and Components ........................................................................................... 8!Potential Limitations ................................................................................................................... 9!
Logic Model .................................................................................................................................. 10!Inputs ........................................................................................................................................ 10!Outputs ...................................................................................................................................... 11!Outcomes .................................................................................................................................. 11!Assumptions .............................................................................................................................. 13!External Factors ........................................................................................................................ 14!
Evaluation Approach .................................................................................................................... 14!Evaluation Questions and Focus ................................................................................................... 15!Evaluation Rationale ..................................................................................................................... 15!Quantitative Evaluation Approach ................................................................................................ 16!
Participants ................................................................................................................................ 17!Research Design ....................................................................................................................... 17!Survey Instrument ..................................................................................................................... 18!Implementation Plan ................................................................................................................. 20!Statistical Analyses ................................................................................................................... 22!Quantitative Results Presentation ............................................................................................. 23!
Quantitative Evaluation Approach ................................................................................................ 23!Participant Demographics ......................................................................................................... 24!Instrumentation and Validity .................................................................................................... 25!Implementation ......................................................................................................................... 26!Analysis .................................................................................................................................... 27!
Display Results ............................................................................................................................. 29!Next Steps ..................................................................................................................................... 29!References ..................................................................................................................................... 30!Appendix A: Northwestern Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Heath Logic Model ..................................................................................... 32!Appendix B: McGaw Global Health Curriculum ......................................................................... 34!Appendix C: McGaw Graduate Medical Education Certificate of Global Health Survey Construct Map and Item Matrix for PRE-TEST ........................................................................... 35!Appendix D: McGaw Graduate Medical Education Certificate of Global Health Survey Construct Map and Item Matrix for POST-TEST ........................................................................ 42!Appendix E: E-mail Draft of Pre-Test Survey .............................................................................. 46!Appendix F: Pre-Test Survey ........................................................................................................ 47!
EVALUATION PLAN 3
Appendix G: E-mail Draft of Post-Test Survey ............................................................................ 53!Appendix H: Post-Test Survey ..................................................................................................... 54!Appendix I: Pre-Test Resident Survey Email Reminder .............................................................. 60!Appendix J: Post-Test Resident Survey Email Reminder ............................................................ 61!Appendix K: General Survey Reminder Email ............................................................................ 62!Appendix L: Resident Semi-Structured Interview Protocol ......................................................... 63!Appendix M: Semi-Structured Interview Resident Demographic Questionnaire ........................ 66!Appendix N: Semi-Structured Resident Interview Protocol Summary Notes .............................. 67!Appendix O: Resident Semi-Structured Student Interview E-Mail .............................................. 68!Appendix P: Mentor Semi-Structured Interview Protocol ............................................................ 69!Appendix Q: Semi-Structured Interview Mentor Demographic Questionnaire ........................... 72!Appendix R: Mentor Semi-Structured Interview Protocol Notes ................................................. 73!Appendix S: Mentor Semi-Structured Student Interview E-Mail ................................................. 74!Appendix T: Lecturer Semi-Structured Interview Protocol .......................................................... 75!Appendix V: Lecturer Semi-Structured Interview Protocol Notes ............................................... 79!Appendix W: Lecturer Semi-Structured Lecturer Interview E-Mail ............................................ 80!
EVALUATION PLAN 4
Evaluation Plan for McGaw Global Health Clinical Scholars Certificate Program
The world has become increasingly interconnected and globalization now affects
virtually every person’s life. Increases in the flow of people, products, services, and information
between and among countries and continents are having a dramatic influence on the world’s
health and health care delivery. This evaluation proposal will give an overview of the McGaw
Global Health Clinical Scholars Certificate Program, the reasoning for the inquiry; how the
assessors will approach evaluate the program from both a quantitative and qualitative evaluative
approach, and recommended next steps for key stakeholders. It is our goal to provide a solid
evaluative foundation to this innovative and groundbreaking program.
Theoretical Framework
Delivering care from a global health perspective is necessary for physicians and health
care providers. Researchers in the field contend residents who complete a focused global health
curriculum learn to practice medicine with limited access to laboratory tests and expensive
diagnostic procedures, relying on strengthened physical examination skills and depending less on
laboratory values, radiologic imaging, and other diagnostic testing, and they develop a deeper
appreciation for global public health issues and become more culturally sensitive (Drain,
Holmes, Skeff, Hall, & Gardner, 2009). Within the medical community, the term “global health”
refers to a broad spectrum of health issues encountered by physicians worldwide.
Acknowledgment of its importance in helping health care workers learn to provide quality care
for patients is often overlooked, likely because the definition of global health is sometimes
confused or used interchange- ably with both “international health” and “public health”. This
was addressed by Koplan, Merson, Reddy, Rodriguez, & Sewankambo (2009):
[Global health is] an area for study, research, and practice that places a priority on
EVALUATION PLAN 5
improving health and achieving equity in health for all people world- wide. Global
health emphasizes transnational health issues, determinants, and solutions; involves many
disciplines within and beyond the health sciences and promotes interdisciplinary
collaboration; and is a synthesis of population-based prevention with individual-level
clinical care (p. 1993).
Global health training programs are necessary in graduate medical education given resident
interest across all specialties as well as the globalization of health (Chase & Everret, 2011; Kern,
Thomas, & Hughs, 2009). There is a paucity of such training programs, particularly
competency-driven programs, in the United States (A. Doobay-Persaud, personal
communication, September 2, 2014). In addition, many residents across universities participate
in international rotations that have not been evaluated for safety, educational value, social
responsibility, or accountability (Cherniak, Drain, & Brewer, 2013). Of note, recent surveys
have shown that between 50-60% of residents have chosen their programs based on global health
training opportunities (Drain, et al., 2009).
History of Global Health Resident Program
In early 2014 the Center for Global Health, housed within the world renowned Fienberg
School of Medicine at Northwestern University, in order to answer the call by medical
residences and global need for global health practitioners, took steps to join an emerging group
of elite residency programs that offer comprehensive competency-based Global Health
Certification (A. Doobay-Persaud, personal communication, September 2, 2014). Developed
and managed by Dr. Ashti Doobay-Persaud, the Faculty Development Chair at the Center for
Global Health and Director of the Section of Global Health within the Division of Hospital
Medicine at Northwestern University, program development began in 2013 when Dr. Doobay-
EVALUATION PLAN 6
Persaud invited a small group of medical practitioners dedicated to global health to a round table
to engage in a substantive dialogue on the development of a Global Health curriculum for
medical residents. The dialogue led Dr. Doobay-Persaud to collaborate with the McGaw
Medical Center of Northwestern University, an Illinois not-for-profit corporation that sponsors,
administers, and supports graduate medical education, to assist financially in the development
and implementation of a three-year global health certificate for medical residents of the Fienberg
School of Medicine. While many global health programs are offered at various medical
institutions very few are grounded in the Accreditation Council for Graduate Medical Education
(ACGME) recommended competencies for the training of residents (A. Doobay-Persaud,
personal communication, September 2, 2014). An important tenant of the certificate program is
to focus on building competency essential for practice in under-resourced or disaster-effected
areas at home and abroad.
The process to develop the preliminary certificate curriculum involved two steps: a
review of existing programs and competencies as well as interviewing and assessing needs from
key stakeholders; and an internet search of existing programs generated with the following
terms: health; medicine; global; international; residency; curriculum; competencies; and
education (A. Doobay-Persaud, personal communication, September 2, 2014). Twenty residency
programs were reviewed, and Dr. Doobay-Persaud contacted the directors of the five programs
that had competencies for guidance; four of the five responded (A. Doobay-Persaud, personal
communication, September 2, 2014). In addition to corresponding with peer medical
institutions, Dr. Doobay-Persaud conducted a PubMed literature search limited to the last ten
years, using the internet search terms world health, international educational exchange, program
development, graduate medical education and internship; 448 articles were retrieved, and 33
EVALUATION PLAN 7
were included as they described specific competencies and skills (A. Doobay-Persaud, personal
communication, September 2, 2014). Finally, current expert opinion from the Consortium of
Universities for Global Health on inter-professional global health competencies also guided
program development (A. Doobay-Persaud, personal communication, September 2, 2014).
Utilizing information gathered from peer expertise, research, and existing ACGME competencies
Dr. Doobay-Persaud drafted the preliminary certificate curriculum. On September 2, 2014, the
Dean of Graduate Medical Education, program directors, senior faculty across specialties, the
Center for Global Health and the Division of Hospital Medicine leadership, as well as
development and administrative professionals, met to review the preliminary curriculum for
approval.
Global Health Certificate Details
The meeting on September 2, 2014 produced a robust and comprehensive curriculum
plan for the certificate program officially entitled, the McGaw Graduate Medical Education
Certificate of Global Health. The certificate grounded in ACGME competencies will require
interested residents to complete eight certificate components over a two-year period.
Goals and Objectives
The components combine to meet the goal initially established by Dr. Doobay-Persaud,
to create a global health certificate program with an educational curriculum for residents and
fellows that will train them in clinical care delivery in resource-limited settings worldwide. The
concept of the global health education began within the Feinberg School of Medicine as a
combination of core concepts such as high-quality global health practicums, clinical experiences
with mentorships and evaluations, and a scholarly project. However, the September 2, 2014
meeting led to several additions, requests, and changes that have been approved to better support
EVALUATION PLAN 8
the goals and objectives of the program. Program directors requested, in acknowledgement of
limited time and schedule diversity of residents, the didactic elements be made available online
(A. Doobay-Persaud, personal communication, September 22, 2014). In addition, senior faculty
specialists introduced several new lecture topics they believed critical to a well-rounded global
health education. Furthermore, each program director and faculty member present committed to
leveraging global health clinic partnerships to provide structured ACGME-based global
experiences. Directors and faculty members also committed to creating relationships with onsite
mentors for residents when completing the global clinic visit component of the program (A.
Doobay-Persaud, personal communication, September 22, 2014). At the conclusion of the
September 2, 2014, after comprehensive debate and cross-specialty dialogue, the goals,
objectives, and components of the McGaw Graduate Medical Education Certificate of Global
Health were established.
Components
The McGaw Graduate Medical Education Certificate of Global Health requires Fienberg
School of Medicine residents to complete eight components within two years to attain a
certificate of global health. The eight components are: nine core lectures, five specialty-specific
lectures, participate in a quarterly journal club, seek faculty mentorship, utilize web-based
training modules, contribute to cross-residency skills exchange, travel to global health clinic for
skills training and exchange, and produce and present scholarly project upon return from their
global experience (See Appendix B). These components were created to meet the following
global health competencies: describe global burden of disease and demonstrate understanding of
epidemiologic tools and methods; understand health implications of travel, migration, and trade;
recognize the major determinants of health; explain the role of community-engagement strategies
EVALUATION PLAN 9
in capacity strengthening; demonstrate high standards of ethical conduct and quality in global
healthcare; practice and professionalism in a resource-limited setting: provide culturally sensitive
care and support to patients; demonstrate a basic understanding of the relationship between
health and human rights; develop global health focused scholarly work online, currently, there is
not a universal assessment tool to ensure standards are met across programs, it is dependent upon
the program to individually assess each resident prior to certification. (A. Doobay-Persaud,
personal communication, September 22, 2014). The components and corresponding
competencies are ambitious, however Dr. Doobay-Persaud believes the breadth and depth of the
certificate will attract residents eager to expand their medical knowledge, improve patient care,
and increase their understanding of global need (See Appendix B).
Key Stakeholders
Along with Dr. Doobay-Persaud many within the Fienberg School of Medicine and the
McGaw Medical Graduate program are dedicated to the creation and implementation of this
certificate program. Deans of both McGaw and Fienberg have committed support through
funding, promotion, and authoring Program Directors to allow residents time to participate in a
global site rotation. In addition to both schools, the Center for Global Health at Fienberg School
of Medicine has committed funding, physical office and meeting space, web-hosting and
recording of lectures, and faculty. The McGaw Global Health Resident Certificate program is a
new program with significant support across the diverse medical community of Northwestern
University.
Potential Limitations
As a new program, Dr. Doobay-Persaud and key stakeholders shared concerns in regards
to the next steps of resident recruitment and program implementation. The primary concern is
EVALUATION PLAN 10
certificate funding, which will affect both the recruitment of residents as well as implementation
of the certificate program. As of September 25, 2014 multiple departments, including McGaw
Graduate Medical Education Organization, the Center for Global Health, and the Fienberg
School of Medicine have verbally committed to financial support of both the administrative,
faculty, and resident costs of the program online (A. Doobay-Persaud, personal communication,
September 28, 2014). However, no funds have been received. Additionally, a program funded
by multiple entities will be subject to a complex set of financial and administrative oversight that
will add a layer of complexity to the evaluative goals and process.
Logic Model
The McGaw Global Health Resident Certificate program logic model describes the
resources and planning before the classes begin the role of residents, faculty, and staff while the
program is in session, and the outcomes from a short-to long-term timeline based on residents’
participation in the program (Wholey, Hatry, & Newcomer, 2010). For a visual graph of the
logic model, please refer to Appendix A.
Inputs
The inputs describe the research, planning, and supplies needed before the program is
able to begin and while it is in session. As described earlier, the individuals involved in the
program begin with the staff members who have previously evaluated the needs for the McGaw
Global Health Resident Certificate program. They have also constructed the curriculum for the
faculty members who will serve as lecturers and mentors for the residents enrolled in the McGaw
Global Health Resident Certificate program. Finances have been verbally committed from
various Feinberg School of Medicine entities to cover the staff creating and planning the
program, as well as the faculty who will be providing the lectures and mentorship to residents.
EVALUATION PLAN 11
With the lectures of the program in the evening after residency hours during the day, staff will be
providing dinners for the residents as an incentive to enroll in the program. Finances are
provided for various items related to residents visiting their global health sites. Time and
materials relate to the planning and implementation of the McGaw Global Health Resident
Certificate program. Research, a needs evaluation, and development of the program came from
various stakeholders within the Feinberg School of Medicine, with stakeholders now marketing
and enrolling residents to begin the program in 2015.
Outputs
The courses will be taught in-class by faculty members unless residents are physically
located in a space they cannot leave due to conflicts with their residency status (i.e., Family
Medicine). The residents enrolled in the program will complete 9 of the 12 core curriculum
lectures, and 5 of the 6 specialty lectures, participate in a quarterly journal club, and engage in
various scholarly and professional development projects. The global health sites will be in a
variety of different locations. Medical insurance will be provided to residents, and global health
sites will have medical practitioners and faculty providing mentorship. Prior to departing to
global health sites, residents will participate in a mandatory pre-departure training.
Outcomes
Outcomes are defined as the results of residents engaging and participating in the McGaw
Global Health Resident Certificate program during their participation, upon completion, and in
years after completing (Wholey, et al., 2010). The short-term goals focus on the experiences of
residents during the three years of engagement in the program. Residents will begin to
understand the global context and inequities of health and medicine, in both their lectures and in
global health visits to various countries. In discovery of other health and medical contexts,
EVALUATION PLAN 12
residents will discover different sociopolitical contexts in the countries they study in, which will
highlight different socio-economic restraints and inequalities and political contexts that may
reject or find western health and medicine hostile. Residents will begin to examine health and
medicine outside the paradigm of western medicine, further identified as traditional medicine
(not recognized by the Accreditation Council for Graduate Medical Education). Residents are
also exposed to different ethical dilemmas related to global health and medicine, and are offered
a lecture to explore these concepts further. During all of these various experiences and
engagement, residents will also develop professional skills such as creating scholarly works, and
presenting on various topics. Residents will also be exposed to different careers in global health,
different mentors, and connections to supplement their experiences during and after the McGaw
Global Health Resident Certificate program.
Medium-term outcomes are defined as the results of residents completing the two-year
program upon time of completion into their professional positions as fellows or licensed medical
practitioners. Residents will be able to define various global health contexts and inequities, and
use their knowledge in practice with a high level of ethical medical care. Residents will in
addition complete the program with knowledge of systemic structures and policies that shape
healthcare and medicine to where graduates can work within and shift those structures when
necessary. Residents will also gain knowledge and know how to practice medicine within the
community and cultural context, and know how to operate healthcare under financial restrictions.
As the capstone to the certificate program, residents will engage in scholarly work related to
global health. In relation to long-term outcomes, graduates will contribute to the medical
community with the knowledge and experience of global heath differences and inequity.
EVALUATION PLAN 13
Graduates are expected to continue to strive for global health equity in their various capacities
and scholarly work.
Assumptions
The evaluators must carefully consider the assumptions made of the residents, faculty,
and staff involved in the McGaw Global Health Resident Certificate program, which may
influence the accuracy of the survey, and the results outlined from the evaluation (Wholey, et al.,
2010). Considering the McGaw Global Health Resident Certificate has input from various
stakeholders throughout the Feinberg School of Medicine, and is in its infancy of being
implemented, the inputs and outputs of the program must be accounted for as assumptions. Prior
knowledge of a programs’ development is one of the key factors in understanding a program’s
inputs, outputs, assumptions, and external factors (Weiss, 1998). The understanding of the
program may change as it developed, and must be continuously monitored for the evaluation to
be effective (Weiss, 1998). Residents are currently being enrolled, but we are under the
assumption they are interested in the program, can attend the on-campus lectures, participate in
the global sites, and complete the program in three years. Assumptions are also made regarding
the program curriculum and process. We do not know if the lectures, mentorship, global site
visits, and other components of the program will meet the outcomes outlined by McGaw Global
Health Resident Certificate faculty and staff, and the Accreditation Council for Graduate
Medical Education (ACGME). Some assumptions under this umbrella to consider are that: the
ACGME can be taught by lectures and understood by residents, whether the global site mentors
have training in ACGME outcomes, and whether ACGME outcomes are desirable outcomes
outside of the United States medical context.
EVALUATION PLAN 14
External Factors
The evaluators must consider external factors that can alter the intended outcomes of the
McGaw Global Health Resident Certificate program. Among the most fundamental of external
factors is the Feinberg School of Medicine continuing to enroll residents, which are needed for
the McGaw Global Health Resident Certificate to exist. Program stakeholders have prepared for
this reality, but have already seen progressive enrollment as off December 2014. Program
funding must also continue and key stakeholders for the program must remain in their positions
for the program to meet the outcomes for students. Within the context of the global sites,
communication between faculty and staff of Feinberg and the global health sites must be strong
to ensure the sites are meeting ACGME outcomes. It is difficult to control the geopolitical and
sociopolitical contexts at the global sites, which will factor resources at the sites, the type of
health and medical experiences residents will have, and the overall safety of the site and resident.
It will be difficult to control external factors once the residents graduate, which may challenge
long-term goals specifically.
Evaluation Approach
Our approach will be evaluating the outcomes of the McGaw Global Health Resident
Certificate by focusing on the program in its infancy, constituting a formative approach, which
seeks to provide evaluative information for the purpose of program improvement (Fitzpatrick,
Sanders, & Worthen, 2011). Enrollment for winter 2015 is underway and has multiple residents
enrolled. Staff and faculty facilitated a needs assessment for the creation of the McGaw Global
Health Resident Certificate program supplementing the creation of learning strategies used in the
programs’ curriculum. Unique to this global health program is the desire to meet ACGME
EVALUATION PLAN 15
standards, which are often not met in graduate medical global health programs (A. Doobay-
Persaud, personal communication, September 22, 2014).
Evaluation Questions and Focus
These same outcomes created by program faculty and staff, under the standards of
ACGME will be under evaluation by the authors. The overarching themes guiding our
evaluation stem from the educational strategies and outcomes the McGaw Global Health
Resident Certificate program anticipates for students who finish their certificate. Will students
describe global burden of disease and demonstrate understanding of epidemiologic tools and
methods? Will they understand health implications of travel, migration, and trade? Will they
recognize the major determinants of health? Will they understand the role of community-
engagement strategies in the context of healthcare? Will they demonstrate high standards of
ethical conduct and quality in global healthcare? Will they practice professionalism in a resource
limited setting by providing culturally sensitive care, support to patients, and understanding the
role of traditional medicines? Will they demonstrate a basic understanding of the relationship
between health and human rights? Will they develop global health focused scholarly work?
These outcomes will guide our approach in evaluating the McGaw Global Health Resident
Certificate program.
Evaluation Rationale
The program outcomes-orientated approach will allow the authors to evaluate the
specified outcomes and purposes of the McGaw Global Health Resident Certificate program to
focus on the extent to which the outcomes and purposes are achieved (Fitzpatrick, et al., 2011).
Using quantitative and qualitative measurements, we will attempt to measure the developmental
changes residents undergo related to the outcomes created by the McGaw Global Health
EVALUATION PLAN 16
Resident Certificate program. We also hope to measure students retaining the short-term
outcomes and purposes, thus leading to medium- and long-term stated outcomes measured in
future evaluations. Because this program has been established and is ready to be measured for
its outcomes for the first time, our assessment will be formative by measuring the program in its
progress, rather than assessing whether or not the program should be continued (Fitzpatrick, et
al., 2011).
One of the two evaluators has extensive knowledge of the program, although a
developing context of global health programs outside of Feinberg. Although familiarity with the
program is a strength for the evaluation plan, it has the potential to cause bias in contributing to
the final product. Additionally, global heath contexts will be needed from stakeholders.
Although the formative evaluation seeks to define the context for programs, the McGaw Global
Health Resident Certificate program will continue to develop and change due to a variety of
external factors (Fitzpatrick, et al., 2011). The evaluators will continue to evaluate these external
factors, and recognize changes with the McGaw Global Health Resident Certificate program.
Quantitative Evaluation Approach
This section will detail the implementation of quantitative measurements for our
evaluation of the McGaw Graduate Medical Education Certificate of Global Health, including
our research design, statistical analysis, survey instrument, and implementation plan. The logic
model contextualizes the quantitative approach and methodology by seeking to understand short,
medium, and long-term goals of the McGaw Graduate Medical Education Certificate of Global
Health.
EVALUATION PLAN 17
Participants
Fienberg School of Medicine residents in the OBGYN, Dermatology, Family Medicine,
General Medicine, and Surgery programs who enroll in the McGaw Graduate Medical Education
Certificate of Global Health will be the focus of this evaluation. The McGaw Graduate Medical
Education Certificate of Global Health is a new program; therefore, enrollment projections for
the inaugural winter 2015 quarter are unknown. The sampling will be census, meaning the entire
portion of the evaluative population, inviting all students enrolled in the McGaw Graduate
Medical Education Certificate of Global Health to complete the pre- and post- test surveys as a
proposed requirement for receiving their certificate, pre-Institutional Review Board approval
(Schuh, 2009).
Research Design
The most effective quantitative measure to assess the outcomes of the McGaw Graduate
Medical Education Certificate of Global Health is a pre-experiential longitudinal study using a
pre-test and post-test survey (Creswell, 2009). The McGaw Graduate Medical Education
Certificate of Global Health is interpreted as an intervention, as it is a unique program with a set
of outcomes not currently present in the Feinberg School of Medicine. The quantitative
measurements will be used to measure the direction of change, the amount of change that
occurred, and the amount of consistency in which the resident changed by using a set of Likert
scale questions (Creswell, 2009). The quantitative portion of the longitudinal study will begin
with residents taking the pre-test before their lectures begin in their first quarter, and end with
residents completing the post-test upon completion of the McGaw Graduate Medical Education
Certificate of Global Health curriculum. As mentioned above, the type of sampling used for the
longitudinal study will be census, inviting all residents enrolled to participate (Wholey, et al.,
EVALUATION PLAN 18
2010). Participation in the quantitative portion of the evaluation is being proposed as required to
the IRB for students to receive their certificate, ensuring near a 100% response rate, limiting the
internal threat of mortality (Creswell, 2009). Though census sampling will provide a holistic
context of the residents in the McGaw Graduate Medical Education Certificate of Global Health,
the small sample size of approximately 5-7 students per quarter will also adversely affect the
validity of the findings (Wholey, et al., 2010).
The pre-experiential nature of the quantitative design will measure the single group of
residents participating in the McGaw Graduate Medical Education Certificate of Global Health,
without a control group to measure, classified as a one-group pre-test post-test design (Creswell,
2009). A control group will not be utilized for this evaluation, limiting validity that is offered in
quasi- and true experimental design (Creswell, 2009). However, a control group and random
selection are not feasible in this context, since the material and time, resources do not exist to
administer a control group, and the population to survey is not large enough for random selection.
The survey questions were constructed using a survey construct matrix to link survey questions
to the short-, medium-, and long-term outcomes detailed in the logic model of the McGaw
Graduate Medical Education Certificate of Global Health (Appendix A).
Survey Instrument
The McGaw Graduate Medical Education Certificate of Global Health program, designed
to be completed in a semi-linear manner, will result in the sample of residents completing the
program at various times throughout the calendar year. Therefore, the ideal evaluation
instrument is an online survey to be emailed upon the submission of their request for certificate
completion (Appendix E; Creswell, 2009). To account for non-sequential completion, an online
survey will also attend to variation of schedules and limited time medical residents cite as
EVALUATION PLAN 19
roadblocks to completing similar surveys. Primary stakeholder, Dr. Doobay-Persaud has agreed
the completion of the survey be a condition of receipt of the official certificate of completion
(written communication, 2014). The required completion of the online survey therefore is a
feasible option for assessing the McGaw Graduate Medical Education Certificate of Global
Health outcomes.
The McGaw Graduate Medical Education Certificate of Global Health program will
enroll the first class of residents in November of 2014, to begin in January of 2015; therefore, it
is essential the survey be designed specifically for this program to assess desired outcomes.
However, while the program may be in infancy, program assessment surveys have become
standard throughout higher education. As such, many questions have been borrowed directly or
modified slightly from existing survey questions. The sources from which these questions were
drawn include: Center for Global Education at the University of California, Los Angeles SOAR
Survey (2012), Center for Experiential Learning at Loyola University Chicago Survey (2013),
and Central Connecticut State University Center for International Education Study Abroad
Survey (2013).
The development of the survey began with a request from key stakeholders to assess if
the certificate program is a successful intervention in expanding residents’ global health
awareness and knowledge in tandem with increasing medical competency and patient care. To
meet this request a survey construct map (Appendix A) allow for the ordering of questions and
ease of relation to the program outcomes stated in the logic model. The survey construct map
(Appendix C) includes both pre and post – test survey questions; the questions are intended to
measure if residents who completed the certificate program met the intended outcomes. The pre-
test survey begins by requiring the student identification number so the pre and post-test surveys
EVALUATION PLAN 20
may be compared. In addition, several general questions ascertain the residents’ previous
experience abroad to control for prior knowledge or experience. The assessment tool is intended
to assess resident knowledge, aptitude, and perception of global healthcare prior to the
completion of the program. The pre-test consists of 16 questions divided into seven distinct
groups.
Of the seven groups, three groups: Skill Development, Global Health Awareness, and
Professionalism addressed short-term outcomes with a four-option Likert scale. For continuity, a
four-option Likert scale is standard throughout the pre and post-test surveys. While Wholey, et
al. (2010), recommends five-option Likert scale, this survey deviates from that recommendation
to eliminate the middle choice, to encourage deep assessment of global health experience and
awareness. In addition, the exclusion of a neutral option in this initial assessment may give a
clearer idea of effectiveness of program components. The pre-test survey continues with
questions designed to assess the residents’ familiarity with the core concepts of the certificate
program. The medium outcomes are where the greatest change between pre and post-test are
anticipated. Finally, the pre-test survey concludes with a series of questions that seek to ascertain
general perception of the necessity and impact of global health. Similar to the pre-test, the post-
test survey will include all questions except demographic information, as we may link the pre
and post-test via the student number. In addition to the pre-test questions, additional process goal
questions have been included to assess additional aspects of the program.
Implementation Plan
In order to attend to the inaugural implementation of the program and survey, a pilot test
will be conducted to improve clarity of survey questions as well as create a baseline for validity
(Creswell, 2009). The pilot test, of the pre-test survey, will be administered to a convenience
EVALUATION PLAN 21
sample of current third year residents (not eligible for two year program) associated with the
Center for Global Health within the Fienberg School of Medicine. Once the pilot test and
feedback from those surveyed have been collected, reviewed, and assessment instrument refined
the pre-test survey will be emailed to the participating resident upon acceptance to the Global
Healthcare Residence Certificate Program. The email will communicate the estimated time
needed to complete the survey and require the resident to complete the survey within one week;
with email reminders occurring each day the survey remains incomplete. If incomplete after two
business days, the resident will be contacted via phone and email by the certificate program
director, Dr. Doobay-Persaud (written communication, 2014).
To complete the survey loop, the post-test mirrors the pre-test in distribution mode. The
post-test assessment will be conducted via online survey. The survey link will be emailed to the
resident upon the approval of certificate completion. However, to ensure survey completion the
email will communicate the survey must be completed within one week. If incomplete after one
week, the resident will be contacted via phone and email by the program assistant to make sure
email was received and encourage completion of survey (written communication, 2014). It is the
stakeholder’s hope that through verbal and written reminders to complete the survey, along with
the dedication of residents to Global Health the survey will have a high completion rate, at this
time the stakeholders are unwilling to provide additional incentives (written communication,
Doobay-Persaud, 2014). Finally, upon receipt of both the pre and post-test the residents will
receive an automated email thanking them for their time and invite them to provide additional
feedback via email or in-person meeting (written communication, Doobay-Persaud, 2014).
EVALUATION PLAN 22
Statistical Analyses
The statistical analyses will help evaluate and understand the direction, amount, and
consistency residents changed during the McGaw Graduate Medical Education Certificate of
Global Health. SPSS will be used to conduct measures of central tendency, a paired sample t-
test, and an ANOVA analysis, to evaluate the previous aspects mentioned. The null hypothesis
we seek to reject in the ANOVA is that there are no mean differences between the different
groups we seek to test, such as time 1 and time 2 differences in global health awareness, beyond
a statistical significance.
The descriptive statistics will describe the pre- and post-test changes between the mean,
median, and mode from the Likert scale questions asked in the survey, providing general patterns
(Wholey, et al., 2010). Central tendency measures will also provide a context of data in
relationship with demographics of the residents such as race/ethnicity, gender, sexual orientation,
program year, intended professional practice, and international experience by comparing means,
medians, and modes between the pre-test and post-test. Cross tabulations and chi-square test will
determine if these categorical variables differed across demographic categories. These analyses
will provide a basic understanding of residents’ development throughout the program.
A paired sample t-test will help us evaluate the pre- and post-test results by comparing
data between the pre- and post-test. This will allow us to understand if significant changes
happened across different groups, and across different dependent variables (our different
outcomes measured in the Likert scale questions). An ANOVA test will expand our argument by
seeking to invalidate the null hypothesis depending on whether the results establish a threshold
for statistical significance (Wholey, et al., 2010). The ANOVA will also allow us to compare
scores across three or four groups, provide deeper analysis by determining if there is statistical
EVALUATION PLAN 23
significance between three or more different identities in the goals and outcomes measured in the
Likert scale. Regression tests will be used to determine the variance in any statistically
significant results.
Quantitative Results Presentation
The results of the assessment are essential to the development, improvement, funding,
and continuation of the certificate program, as many stakeholders appear to be data driven. Key
stakeholders anticipate presenting the results through professional presentations that include
poster presentation, global health panels, and the annual Consortium of Universities for Global
Health in 2017. In addition, several stakeholders have expressed interest in authoring a peer-
reviewed paper on the success of Global Health programs for residents. Finally, the results will
be used to improve resident experience, streamline program operations, and increase funding for
additional global site experiences.
Quantitative Evaluation Approach
This section will outline the quantitative approach to the evaluation of the McGaw Global
Health Resident Certificate Program to add depth and meaning to the evaluation results by using
methods that utilize in-depth interviews (Schuh, 2009). For the quantitative approach to the
evaluation, semi-structured interviews will be utilized with residents, lecturers, and resident
mentors. Semi-structured interviews will give further meaning and depth to the sample provided
in the qualitative results, and increases the sample size by gathering a larger amount of
information from a small group of individuals (Schuh, 2009). Semi-structured will be also be
utilized, as opposed to focus groups, due to the vast differences in experiences, schedules, and
development we foresee interviewing between three different populations comprising the
McGaw Global Health Resident Certificate Program. The Semi-structured interviews will
EVALUATION PLAN 24
expand on the questions asked in the quantitative survey, deepen their reflection on their
experiences in the program, and provide the interviewers a chance to provide deeper
explanations related to the qualitative findings. The questions will be orientated around the logic
model outcomes, particularly the short- and medium-term outcomes.
Participant Demographics
All of the medical resident participants for the interview will be gathered from the
surveyed participants. The last question of our survey asks participants to disclose their e-mail
address if they are interested in participating in the interview process. For the lecturers and
mentors, we will email (See Appendices S & W), inviting them to participate. The respondents
will then participate in a semi-structured interview process to capture both the residents story as
well as the mentors and lecturers who the program placed in a position of strong influence on
both the program and participants for more complete assessment (Wholey et al., 2010). It is
important to the key stakeholders that there is at least one resident and mentor from each global
site, to had better evaluate the site as well as the overall program. However, the beneficiary
group, medical residents, is historically difficult to schedule due to rigorous professional and
academic schedules. Therefore, while bias will increase, it is more realistic to first see that
volunteers for the qualitative portion of the program evaluation.
The key stakeholders have indicated they would ideally have at minimum two Resident
and Mentor participants from each global site, as well as, all lecturers and mentors. As this
group will be the first to complete the certificate program, it is essential to capture many
experiences and perspectives. It is especially important to focus on the global site experience
within the qualitative portion, as there will be no standard experience to measure qualitatively, as
each resident will have an experience unique to each global site. In addition to the residents, the
EVALUATION PLAN 25
lecturers and mentors will provide a critical eye as experienced professionals within the field of
global health, which will enable the key stakeholders to improve and grow the program.
Instrumentation and Validity
The protocol instruments (Appendices L, P, & T) that we have created focuses on several
distinct subjects: reasons for enrolling in the McGaw Global Health Resident Certificate Program,
experiences while enrolled such as mentorship and global health site experiences, and areas of
improvement for the program. The protocol seeks to guide the questions for semi-structured
interviews and includes probes in case the participants do not answer the questions with the
depth sought in the questions (Wholey, et al, 2010). The protocol’s focus on reasons for
enrolling seeks to understand how students heard about the program, since it is a new and
developing program. Probes include questions based on different facets of the program weighing
on their decision to enroll such as: medical colleagues, different global health sites, or their
different program directors. One probe also asks if their experiences as a resident have changed
since their enrollment in the McGaw Global Health Resident Certificate Program. The
protocol’s focus on experiences while enrolled asks residents about their experiences while
enrolled generally, experiences with global health sites, and mentorship relationships. Specific
probes ask for more depth information on the three subjects and include: most rewarding and
challenging experiences, and the impacts the latter two pieces of the McGaw Global Health
Resident Certificate Program had on the residents. The last specific protocol focus is
effectiveness and needs and seeks to understand the strengths and weaknesses of the program so
that it can further be developed. Specific probes include asking if any new initiatives would help
develop the program, or how any of the existing pieces of the program could further be
developed.
EVALUATION PLAN 26
Due to strict time constraints of those participating in the semi-structured interviews, the
time of the interviews is to be no longer than 60 minutes. Although the quantitative portion of
the evaluation is a program requirement for residents to complete the certificate, the qualitative
section will not be required for residents. Completion time of the semi-structured interviews
may be impacted if interpretation assistance is needed for residents who do not use English as a
first language.
In order to improve the validity of the interview protocol, members of the McGaw Global
Health Resident Certificate Program, including Dr. Ashti Doobay-Persaud and Kathryn Ward-
Mytinger, will review the materials. By having the protocol reviewed by stakeholders, we hope
to improve the face validity of the qualitative data. Face validity ensures the experts evaluate the
content of the research instrument, and indicate the degree to which the measurement criteria or
objectives are valid (Schuh, 2009). To further develop the accuracy of the qualitative instrument,
we will administer a pilot test with a group of no more than seven students, with the hope of one
student per demographic group.
Implementation
As program director, Ashti Doobay-Persaud has requested that Katie Ward-Mytinger, the
program graduate intern and co-constructor of the survey, conduct the semi-structured interviews.
In addition, it is again important to attend to the complex and saturated schedules of our
interview participants. Therefore, participants who indicated willingness to participate will be
contacted via email (See Appendices O, S, & W) to alert them of their selection, time
commitment of 30 minutes, a request to complete an online form indicating dates and times
available for interview, and the demographic questionnaire. The form will also contain a choice
stating “unavailable for stated interview times,” if the participant checks that box they will be
EVALUATION PLAN 27
prompted to contact the interviewer to set up a time and date outside the offering, possibly via
Skype.
Interviews will take place at the Center for Global Health offices that are conveniently
located within a block of Northwestern Memorial Hospital, where a majority of the participants
work. This space provides both relaxed and formal spaces that have been committed to the
program for interview purposes. Interviewees will have the opportunity to choose the interview
space in an effort to create a comfortable interview space. Those interviewees who are not able
to interview in person will be asked to interview over Skype to best mirror a face-to –face
environment. However, there is a possible bias created in using two different modes of
interviewing. It is our hope that the mode of interview, in-person versus virtual, will not
significantly impact the data collected.
Analysis
The focus group and semi-structured interviews facilitated will be analyzed via
transcription and notes. Participants are aware their responses are being recorded, and particular
themes will require facilitators to take notes. The data will be analyzed by administrators in the
McGaw Global Health Resident Certificate Program, using software for qualitative data analysis
(i.e., NVivo, Atlas.ti).
The methodical approach evaluators will take, once data are collected and organized, is
open coding. By using open coding, organize and prepare the data, divide the data into
categories, determine how categories will be represented, interpret and make meaning of the data
based off the categories, and connect the categories to the qualitative data and assessment
objectives (Schuh, 2009). This will allow us to describe the data using themes, which are then
EVALUATION PLAN 28
discussed and interpreted, backed up by quotes, and put into conversation with similar and
different perspectives (Schuh, 2009).
To assess the validity of the results, we will use several strategies including member
checking, peer debriefing, and triangulation (Schuh, 2009). Member checking is described by
taking the initial findings of the qualitative instruments back to the participants to determine their
accuracy (Schuh, 2009). In order for member checking to feasibly happen, transcriptions and
coding will need to happen quickly after the focus group sessions, so that interviewees can have
a polished view of the data analysis as opposed to the raw transcripts (Creswell, 2009). We will
also utilize peer debriefing by having individuals not familiar with the study, but familiar to the
field of global health, to review our findings and ask questions (Schuh, 2009). To further assess
the validity of our qualitative results, we will triangulate it with our quantitative results, and
continue to use existing results to enrich the data found in future evaluations (Schuh, 2009).
We will discuss limitations in an effort to control for them as much as possible. Personal
bias is possible from the interview process, to coding the data, and describing the data in its
written and final form. Individuals involved with the evaluation will be asked to participate in
trainings, construct expectations, participate in personal reflections, and create strategic plans so
that personal bias can be limited as much as possible. With interviews, personal bias can
interfere as interviewers can ask leading questions and/or close-ended questions, hindering the
responses of the interviewees. Although the evaluators will seek to create rich qualitative data
by using coding and direct quotes, the evaluators will be presented with a large amount of data.
The evaluations will need to reflect and be mindful of inter-rater reliability, especially as the
codes are developed, since data may be interpreted in multiple ways, based on each evaluator’s
bias. A balance must be created between richness and feasibility to ensure the assessment, which
EVALUATION PLAN 29
is orientated towards action as opposed to research, strives for its intention to create meaningful
change (Banta & Blaich, 2010).
Display Results
The final report will be a combination of statistical analyses as well as direct quotations
from the focus group interviews. Direct quotes will serve as a powerful and detailed way to
understand the aspects of the McGaw Global Health Resident Certificate Program that achieved
the intended outcomes as well as aspects that need to be omitted or altered in future years.
Similarly, direct quotes will allow evaluators to make meaning of aspects the residents
experienced that were unanticipated and unaccounted for in the research design. The coding
process and data presentation will allow stakeholders to quickly see the extent to which students
shared experiences that were suggestive of similar themes.
Next Steps
The evaluators will meet with key stakeholders in the coming days for final approval; this
meeting is set for December 4, 2014. Any sort of revisions will be completed before residents
are enrolled in the second week of January. The evaluators will submit the final mock up and
pilot test to begin building the quantitative sections, and to determine if any revisions need to be
made to the qualitative and quantitative questions. The final proposal will include a budget
request, requesting the budget is approved and secured before the evaluation begins. Once
residents are enrolled, they will be asked to complete the pre-test.
EVALUATION PLAN 30
References
Banta, W. T. & Blaich, C. (2010). Closing the assessment loop. Change: The Magazine of
Higher Learning, 43, 22-27.
Central Connecticut State University. (2013). Center for International Education Study Abroad
Survey. Retrieved from:
web.ccsu.edu/uploaded/departments/AdministrativeDepartments/Center_for_Internationa
l_Education/Courses_Abroad/Study_Abroad_Student_Assessment_Form.pdf
Chase, J.A. & Evert, J. (Eds.) (2011). Global health training in graduate medical education: A
guidebook (2nd Ed.). San Francisco, CA: iUniverse, Inc.
Cherniak, W.A., Drain, P.K., & Brewer, T.F. (2013). Educational objectives for international
medical electives: A literature review. Academic Medicine. 88, 1778-1781.
Creswell, J. (2009). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches
(3rd Ed.). SAGE Publication: Thousand Oaks, CA.
Drain, P.K., Holmes, K.K., Skeff, K.M., Hall, T.L., & Gardner, P. (2009). Global health training
and international clinical rotations during residency: Current status, needs, and
opportunities. Acad Med. 84, 320-325.
Fitzpatrick, J., Sanders, J., & Worthen, B. (2011). Program evaluation: Alternative approaches
and practical guidelines (4th Ed.). New York: Longman.
Gupta, A., Talavlikar, R., Ng, V., Chorny, Y., Chawla, A., Farrugia, M., Lorette, J., Raza, D., &
Vyvey, M. (2012). Global health curriculum in family medicine: Resident perspective.
Canadian Family Physician, 58, 143-6.
Koplan, J.P., Bond T.C., Merson, M.H., Reddy, K.S., Rodriguez, M.H, & Sewankambo, N.K.
(2009). Toward a common definition of global health. Lancet. 373 (9679), 1993-5.
EVALUATION PLAN 31
Kern, D.E., Thomas, P.A., & Hughs, M.T. (2009). Curriculum development for medical
education: A six-step approach. Baltimore, MD: The John Hopkins University Press.
Loyola University, Chicago. (2013) Center for Experiential Learning LUROP Student Survey.
Retrieved from: http://www.luc.edu/lurop/luropstudentsurvey/
Schuh, J.H. & Associates (2009). Assessment methods for student affairs. San Francisco, CA:
Jossey-Bass.
University of California, Los Angeles. (2012) Center for Global Education SOAR Survey.
Retrieved from: http://www.globaled.us/cccsoar/about-ccc-soar-surveys.asp#top
Weiss, C. H. (1998). Evaluation (2nd Ed.) Upper Saddle River, NJ: Prentice Hall.
Wholey, J.S., Hatry, H.P., & Newcomer, K.E. (Eds.) (2010). Handbook of practical program
evaluation). (3rd Ed.) Jossey-Bass: San Francisco
Central Connecticut State University. (2013) Center for International Education Study Abroad
Survey. Retrieved from:
http://web.ccsu.edu/uploaded/departments/AdministrativeDepartments/Center_for_International_
Education/Courses_Abroad/Study_Abroad_Student_Assessment_Form.pdf
EVALUATION PLAN 32
Appendix A: Northwestern Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Heath Logic Model
Inputs
Outputs Outcomes -- Impact
Activities Participation Short Medium Long
People • 5 Program Directors • Program Manager • 2 Deans: Dean of
Feinberg; Dean of Graduate Medical Education
• 8 members in the Center for Global Health
• (?#) of Feinberg Residents
• Mentors for enrolled residents
Finances • Hiring staff for program
lectures • Dinner for Residents • Discretionary funds for
global sites • Printed materials • Transportation to and
from Global Medical Sites
• Medical Insurance Time
• Research and development
• Program design • Benchmark and planning
meetings • Pre-departure
orientation • Curriculum design • Teaching courses
Materials • Simulation lab • Online lecture software • Website • Poster explaining
program • Classroom and office
space Physical Global Medical
Sites • Comprehensive list in
Appendix …
Faculty
• Building university partnerships
• Mentoring • Lectures
Residents
• Residents complete 9 out of 12 core lectures
• Residents complete 5 out of 6 specialty lectures
• Quarterly journal club • Cross residence skills
practicum • Pre-departure training • Global site specific
training • Mentorship program at
Northwestern and global specific site
• Global field experience • Scholarly project • Professional
presentation
• Faculty • Residents • Deans • Residency Program
Directors and Chief Residents
• Local and global healthcare workers
• Local and global healthcare facilities
• Center for Global Health administrators and staff
During students 1- 1 ½
year tract in program
• Residents understand global contexts in health.
• Residents understand global health inequities.
• Residents create cross residency skills to diversify medical knowledge and improve patient care.
• Development of professional skills (i.e. scholarly presentations, writing)
• Increased medical knowledge of global medical issues
• Develop knowledge of traditional medicine (otherwise defined as non-ACGME board certified medicine)
• Expanded understanding of ethics in global healthcare
• Understanding of sociopolitical contexts in relation to global and domestic health
• Exposure to career opportunities in global medicine
Upon completion of
certificate program
• Residents can describe global burden of disease and demonstrate understanding of epidemiologic tools and methods
• Residents will understand health implications of travel, migration and trade
• Residents will recognize the major determinants of health
• Residents can explain the role of community-engagement strategies in heath contexts
• Residents will demonstrate high standards of ethical conduct and quality in global healthcare
• Residents will practice and professionalism in a resource limited setting by providing culturally sensitive care and support to patients, and by understanding the role of traditional medicine
• Residents will demonstrate a basic understanding of the relationship between health and human rights
• Residents will develop global health focused scholarly work
Years following
certificate program
combined with practical
experience.
• Graduates will build action capacity for addressing health resource inequities domestically and globally.
• Graduates will work toward global health equity
• Graduates will be able to provide context outside of western medicine
• Graduates can create global health initiatives within local communities
• Global healthcare equity
EVALUATION PLAN 33
Assumptions
External Factors (thinking broadly) • Residents will be interested in Global Health Certificate program. • Residents will be available to attend on-campus lectures or online lectures if necessary. • Faculty is available to teach lectures. • Faculty is willing to mentor residents. • Lectures will engage students and meet learning outcomes. • Global Health curriculum meets Accreditation Council of Graduate Medical Education. • Initial funding is enough to sustain program. • Residents will be able to complete the certificate program in the proposed timeline of
three years. • Global sites will available and maintain accreditation. • Residents will be able to complete their required global site visit. • Pre-departure component will adequately prepare residents for site visit.
• Program external factors o Residents are developmentally prepared for global health curriculum. o Funding from internal and external sources. o Communication is in place between domestic and global sites. o Key stakeholders will remain in decision-making positions. o Feinberg School of Medicine continues to offer residency programs.
• Global external factors o Political/geographical stability of location o Resource distribution at global health sites o Availability of medical factuality to train residents and meet mentorship
requirements.
EVALUATION PLAN 35
Appendix C: McGaw Graduate Medical Education Certificate of Global Health Survey Construct Map and Item Matrix for PRE-TEST
Pre$test:(McGaw(Graduate(Medical(Education(Certificate(of(Global(Health(Logic(Model(Component( SQ(#( Question( Answer(Type(( Answer(Options((
General(Information(
Blank&& Student&ID&#&Text&Box&
N/A&&
1& Please&indicate&which&residency&program&you&are¤tly&participating&in& N/A&&
2& Which&statement&best&describes&your&primary&reason&for&participating&in&the&program&&
CheckFbox/Likert&Scale&&
4&point,&essential&to¬&important&
3& Please&indicate&the&site&you&are&interested&in&completing&your&global&practicum&& CheckFlist&
N/A&&
4& Have&you&had&any&experience&in&global&healthcare&settings?& Answer&option&yes/no&
5& If&yes,&please&describe&indicate&location(s)&and&briefly&describe&responsibility(s)&& Text&Box& N/A&&
12& Please&indicate&your&program&year& CheckFlist& N/A&13& Please&describe&how&you&identify&your&gender&
Text&Box&
N/A&14& Please&describe&how&you&identify&your&race/ethnicity& N/A&15& Please&describe&how&you&identify&your&sexual&orientation& N/A&16& Please&describe&your&intended&professional&practice& N/A&&
17& Prior&to&this&certificate&program,&what&kinds&of&international&experience&have&you&had?&
CheckFlist&&
Answer&options:&no&past&international&experience,&vacation,&lived&abroad&for&more&than&3&months,&volunteered&abroad,&worked&abroad,&served&in&the&military&abroad&for&more&than&3&months,&participated&in&a&study&abroad&program&&&
EVALUATION PLAN 36
&&
&& && && && &&Logic(Model(Component( SQ(#& Question& Answer(Type(& Answer(Options(&
Short(Term(Criteria(1($(Skill(Development((
6& Prompt:&How&would&you¤tly&rate&yourself&in&the&following&areas:&&
CheckFbox/Likert&
Scale&&
Very&experienced,&some&experience,&little&
experience,&no&experience&
6A& Scholarly&Writing&6B& Scholarly&Presentations&6C& Ability&to&work&with&diverse&groups&of&people&6D& Ability&to&work&in&teams&6E& Patient&care&&6F& Ability&to&take&initiative&
&& && && && &&
Short(Term(Criteria(2($(Global(Health(Awareness((
7& Prompt:&How&would&you¤tly&rate&your&understanding&in&the&following&areas:&
CheckFbox/Likert&
Scale&
Very&knowledgeable,&some&knowledge,&little&
knowledge,&none&
7B& The&relationship&between&health&and&human&rights&7C& Major&global&&health&determinants&7D& Familiarity&with&nonFwestern&health&practices&7E& The&role&of&community&engagement&stratifies&in&healthcare&7F& SocioFpolitical&contexts&in&relationship&to&global&health&&
&& && && && &&
Short(Term(Criteria(3($(Professionalism(
&
&
8& Prompt:&To&what&extent&do&you&agree&or&disagree&with&the&following:&&
CheckFbox/Likert&
Scale&
Strongly&agree,&agree,&disagree,&strongly&
disagree&8A&
Global&health&practitioners&have&the&ability&to&communicate&effectively&with&patients&and&health&practitioners&with&a&different&culture/ethnicity&different&than&their&own.&
8B& Global&health&practitioners&have&a&diverse&skill&set,&outside&their&chosen&specialty&
EVALUATION PLAN 37
& 8C& Global&health&practitioners&develop&relationships&with&other&experienced&healthcare&professionals&
8D& Global&health&practitioners&share&global&health&knowledge&with&novices&in&the&field&
8E& Global&health&practitioners&expand&the&understanding&ofðics&in&global&healthcare&
&& && && && &&
Medium(Term(Criteria(1($(Understand(Core(Concepts(
9& Prompt:&How&would&you¤tly&rate&your&understanding&in&the&following&areas:& CheckF
box/Likert&Scale&
Very&knowledgeable,&some&knowledge,&little&
knowledge,&none&9A& Global&burden&of&disease&9B& Global&burden&of&maternal&and&child&health&
Logic(Model(Component( SQ( Question& Answer(Type(& Answer(
EVALUATION PLAN 38
#& Options(&
Medium(Term(Criteria(1($(Understand(Core(Concepts(
9C& Global&burden&of&mental&health&
& &
9D& Global&burden&of&reproductive&health&9E& Global&burden&of&surgical&disease&&9F& Global&health&systems&&9G& Health&and&Humanitarian&Disaster&Response&9H& Impact&of&climate&change&on&global&heath&9I& Effective&use&of&local&medicine&
&& && && && &&
Medium(Term(Criteria(2($(Scholarly(Work(
10& Prompt:&How&would&you¤tly&rate&yourself&in&the&following&areas:&&
CheckFbox/Likert&
Scale&
Very&experienced,&
some&experience,&
little&experience,&no&experience&
10A& Ability&to&synthesize&&global&health&concepts&into&written&work&
10B&
Familiarity&with¤t&research&in&global&healthcare&10C& Ability&to&inform&others&about&the&global&health&context& && &&
&& && && && &&
Long(Term(Criteria($(Transference(
11& To&what&extent&do&you&agree&or&disagree&with&the&following:&&
CheckFbox/Likert&
Scale&
Strongly&agree,&agree,&disagree,&strongly&disagree&
11A& Equitable&healthcare&is&a&human&right&11B&
Healthcare&capacity&of&underFresourced&community&directly&effects&the&health&of&all&communities&
11C& I&have&anðical&obligation&to&have&competency&in&global&healthcare&11D&
I&believe&medical&practice&outside&the&western&tradition&is&vital&to&global&healthcare&
11E& I&presently&incorporate&global&health&practices&into&my¤t&practice&11F& I¤tly&participate&in&global&health&initiatives&
EVALUATION PLAN 39
Logic(Model(Component( SQ(#(
Question( Answer(Type((
Answer(Options((
General(Information( Blank&&
Student&ID&#& Text&Box& N/A&&
1& Please&indicate&which&residency&program&you&are¤tly&participating&in&
CheckFlist& N/A&&
7& At&which&site&did&you&complete&your&global&health&experience& Text&Box& N/A&&11& While&participating&in&the&certificate&program&did&you&complete&any&
additional&global&health&rotations?&&&Check&Box& Yes/No&
12& If&yes,&please&indicate&the&program,&location,&and&duration:& Text&Box& N/A&&13& Who&is&&your&global&health&mentor& Text&Box& N/A&&14& Please&describe&your&intended&professional&practice& Text&Box& N/A&&
&& && && && &&Short(Term(Criteria(1($(Skill(
Development((2& Prompt:&How&would&you¤tly&rate&yourself&in&the&following&areas:&& CheckF
box/Likert&Scale&&
Very&experienced,&
some&experience,&
little&experience,&no&experience&
A& Scholarly&Writing&B& Scholarly&Presentations&C& Ability&to&work&with&diverse&groups&of&people&D& Ability&to&work&in&teams&E& Patient&care&&F& Ability&to&take&initiative&
&& && && && &&Short(Term(Criteria(2($(Global(Health(
Awareness((3& Prompt:&How&would&you¤tly&rate&your&understanding&in&the&
following&areas:&CheckF
box/Likert&Scale&
Very&knowledgeable,&
some&knowledge,&
little&knowledge,&
none&
A& The&relationship&between&health&and&human&rights&B& Major&global&&health&determinants&C& Familiarity&with&nonFwestern&health&practices&D& The&role&of&community&engagement&stratifies&in&healthcare&E& SocioFpolitical&contexts&in&relationship&to&global&health&&
&& && && && &&Short(Term(Criteria(3($(
Professionalism(4& Prompt:&To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckF
box/Likert&Scale&
Strongly&agree,&agree,&
disagree,&strongly&disagree&
A& Global&Health&Practitioners&have&the&ability&to&communicate&effectively&with&patients&and&health&practitioners&with&a&different&culture/ethnicity&different&than&their&own.&
B& Global&Health&Practitioners&have&a&diverse&skill&set,&outside&their&chosen&specialty&
C& Global&Health&Practitioners&develop&relationships&with&other&experienced&healthcare&professionals&
EVALUATION PLAN 40
D& Global&Health&Practitioners&share&global&health&knowledge&with&novices&in&the&field&
E& Global&Health&Practitioners&expand&the&understanding&ofðics&in&global&healthcare&
&& && && && &&Medium(Term(Criteria(1($(Understand(
Core(Concepts(5& Prompt:&How&would&you¤tly&rate&your&understanding&in&the&
following&areas:&CheckF
box/Likert&Scale&
Very&knowledgeable,&
some&knowledge,&
little&knowledge,&
none&
A& Global&burden&of&disease&B& Global&burden&of&maternal&and&child&health&C& Global&burden&of&mental&health&D& Global&burden&of&reproductive&health&E& Global&burden&of&surgical&disease&&F& Global&health&systems&&G& Health&and&Humanitarian&Disaster&Response&H& Impact&of&climate&change&on&global&heath&I& Effective&use&of&local&medicine&
&& && && && &&Medium(Term(Criteria(2($(Scholarly(
Work(6& Prompt:&How&would&you¤tly&rate&yourself&in&the&following&areas:&& CheckF
box/Likert&Scale&
Very&experienced,&
some&experience,&
little&experience,&no&experience&
A& Ability&to&synthase&global&health&concepts&into&written&work&B& Familiarity&with¤t&research&in&global&healthcare&
C& Ability&to&inform&others&about&the&global&health&context& && &&&& && && && &&Long(Term(Criteria($(Transference( 7& To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckF
box/Likert&Scale&
Strongly&agree,&agree,&
disagree,&strongly&disagree&
A& Equitable&healthcare&is&a&human&right&B& Healthcare&capacity&of&underFresourced&community&directly&effects&the&
health&of&all&communities&C& I&have&anðical&obligation&to&have&competency&in&global&healthcare&D& I&believe&medical&practice&outside&the&western&tradition&is&vital&to&global&
healthcare&E& I&presently&incorporate&global&health&practices&into&my¤t&practice&
EVALUATION PLAN 41
F& I¤tly&participate&in&global&health&initiatives&&& && && && &&Process(Goal(1(Criteria($(Mentorship(
Experience(8& To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckF
box/Likert&Scale&
Strongly&agree,&agree,&
disagree,&strongly&disagree&
A& My&mentorship&experience&benefited&my&practice&of&medicine&B& My&mentor&was&committed&to&my&success&C& My&mentor&expanded&my&knowledge&of&global&healthcare&&D& My&mentor&and&I&effectively&communicated&while&I&was&abroad&E& My&mentor&assisted&in&the&development&of&my&professional&presentation&&
&& && && && &&Process(Goal(2(Criteria($(Global(Site(
Experience(9& To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckF
box/Likert&Scale&
Strongly&agree,&agree,&
disagree,&strongly&disagree&
A& I&felt&the&preFsite&orientation&prepared&me&for&my&global&site&experience&B& The&lecture&series&enhanced&my&site&experience&C& My&onsite&mentor&worked&to&expand&my&medical&and&professional&skills&D& My&global&experience&was&positive&E& My&site&experience&gave&me&the&chance&to&add&to&improve&my&medical&
skills&F& My&site&supervisor&was&committed&to&making&it&a&good&learning&
experience&&& && && && &&
Process(Goal(3(Criteria($(Didactic(Instruction(
10& To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckFbox/Likert&
Scale&
Strongly&agree,&agree,&
disagree,&strongly&disagree&
A& The&Global&Health&Lecture&series&expanded&my&medical&knowledge&outside&my&specialty&
B& The&Global&Health&Lecture&series&schedule&worked&well&with&my&residency&program&schedule&
C& The&lecture&series&met&my&expectation&D& The&lecturers&were&informative&E& The&cross&residency&skill&exchange&improved&my&patient&care&at&my&global&
site&
EVALUATION PLAN 42
Appendix D: McGaw Graduate Medical Education Certificate of Global Health Survey Construct Map and Item Matrix for POST-TEST
Logic(Model(Component( SQ(
#(Question( Answer(
Type((Answer(Options((
General(Information( Blank&&
Student&ID&#& Text&Box& N/A&&
1& Please&indicate&which&residency&program&you&are¤tly&participating&in&
CheckFlist& N/A&&
7& At&which&site&did&you&complete&your&global&health&experience& Text&Box& N/A&&11& While&participating&in&the&certificate&program&did&you&complete&any&
additional&global&health&rotations?&&&Check&Box& Yes/No&
12& If&yes,&please&indicate&the&program,&location,&and&duration:& Text&Box& N/A&&13& Who&is&&your&global&health&mentor& Text&Box& N/A&&14& Please&describe&your&intended&professional&practice& Text&Box& N/A&&
&& && && && &&Short(Term(Criteria(1($(Skill(
Development((2& Prompt:&How&would&you¤tly&rate&yourself&in&the&following&areas:&& CheckF
box/Likert&Scale&&
Very&experienced,&
some&experience,&
little&experience,&no&experience&
A& Scholarly&Writing&B& Scholarly&Presentations&C& Ability&to&work&with&diverse&groups&of&people&D& Ability&to&work&in&teams&E& Patient&care&&F& Ability&to&take&initiative&
&& && && && &&Short(Term(Criteria(2($(Global(Health(
Awareness((3& Prompt:&How&would&you¤tly&rate&your&understanding&in&the&
following&areas:&CheckF
box/Likert&Scale&
Very&knowledgeable,&
some&knowledge,&
little&knowledge,&
none&
A& The&relationship&between&health&and&human&rights&B& Major&global&health&determinants&C& Familiarity&with&nonFwestern&health&practices&D& The&role&of&community&engagement&stratifies&in&healthcare&E& SocioFpolitical&contexts&in&relationship&to&global&health&&
&& && && && &&Short(Term(Criteria(3($( 4& Prompt:&To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckF Strongly&agree,&
EVALUATION PLAN 43
Professionalism( A& Global&Health&Practitioners&have&the&ability&to&communicate&effectively&with&patients&and&health&practitioners&with&a&different&culture/ethnicity&different&than&their&own.&
box/Likert&Scale&
agree,&disagree,&strongly&disagree&B& Global&Health&Practitioners&have&a&diverse&skill&set,&outside&their&chosen&
specialty&C& Global&Health&Practitioners&develop&relationships&with&other&experienced&
healthcare&professionals&D& Global&Health&Practitioners&share&global&health&knowledge&with&novices&
in&the&field&E& Global&Health&Practitioners&expand&the&understanding&ofðics&in&global&
healthcare&&& && && && &&Medium(Term(Criteria(1($(Understand(
Core(Concepts(5& Prompt:&How&would&you¤tly&rate&your&understanding&in&the&
following&areas:&CheckF
box/Likert&Scale&
Very&knowledgeable,&
some&knowledge,&
little&knowledge,&
none&
A& Global&burden&of&disease&B& Global&burden&of&maternal&and&child&health&C& Global&burden&of&mental&health&D& Global&burden&of&reproductive&health&E& Global&burden&of&surgical&disease&&F& Global&health&systems&&G& Health&and&Humanitarian&Disaster&Response&H& Impact&of&climate&change&on&global&heath&I& Effective&use&of&local&medicine&
&& && && && &&Medium(Term(Criteria(2($(Scholarly(
Work(6& Prompt:&How&would&you¤tly&rate&yourself&in&the&following&areas:&& CheckF
box/Likert&Scale&
Very&experienced,&
some&experience,&
little&experience,&no&experience&
A& Ability&to&synthase&global&health&concepts&into&written&work&B& Familiarity&with¤t&research&in&global&healthcare&
C& Ability&to&inform&others&about&the&global&health&context& && &&&& && && && &&Long(Term(Criteria($(Transference( 7& To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckF Strongly&agree,&
EVALUATION PLAN 44
A& Equitable&healthcare&is&a&human&right& box/Likert&Scale&
agree,&disagree,&strongly&disagree&
B& Healthcare&capacity&of&underFresourced&community&directly&effects&the&health&of&all&communities&
C& I&have&anðical&obligation&to&have&competency&in&global&healthcare&D& I&believe&medical&practice&outside&the&western&tradition&is&vital&to&global&
healthcare&E& I&presently&incorporate&global&health&practices&into&my¤t&practice&F& I¤tly&participate&in&global&health&initiatives&
&& && && && &&Process(Goal(1(Criteria($(Mentorship(
Experience(8& To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckF
box/Likert&Scale&
Strongly&agree,&agree,&
disagree,&strongly&disagree&
A& My&mentorship&experience&benefited&my&practice&of&medicine&B& My&mentor&was&committed&to&my&success&C& My&mentor&expanded&my&knowledge&of&global&healthcare&&D& My&mentor&and&I&effectively&communicated&while&I&was&abroad&E& My&mentor&assisted&in&the&development&of&my&professional&presentation&&
&& && && && &&Process(Goal(2(Criteria($(Global(Site(
Experience(9& To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckF
box/Likert&Scale&
Strongly&agree,&agree,&
disagree,&strongly&disagree&
A& I&felt&the&preFsite&orientation&prepared&me&for&my&global&site&experience&B& The&lecture&series&enhanced&my&site&experience&C& My&onsite&mentor&worked&to&expand&my&&medical&and&professional&skills&D& My&global&experience&was&positive&E& My&site&experience&gave&me&the&chance&to&add&to&improve&my&medical&
skills&F& My&site&supervisor&was&committed&to&making&it&a&good&learning&
experience&&& && && && &&
Process(Goal(3(Criteria($(Didactic(Instruction(
10& To&what&extent&do&you&agree&or&disagree&with&the&following:&& CheckFbox/Likert&
Scale&
Strongly&agree,&agree,&
disagree,&strongly&disagree&
A& The&Global&Health&Lecture&series&expanded&my&medical&knowledge&outside&my&specialty&
B& The&Global&Health&Lecture&series&schedule&worked&well&with&my&residency&program&schedule&
C& The&lecture&series&met&my&expectation&D& The&lecturers&were&informative&
EVALUATION PLAN 45
E& The&cross&residency&skill&exchange&improved&my&patient&care&at&my&global&site&
Running head: EVALUATION PLAN 46
Appendix E: E-mail Draft of Pre-Test Survey
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: PRE-TEST SURVEY
You are being invited to complete this survey as an enrolled student in the McGaw Graduate
Medical Education Certificate of Global Health. Although your participation is a prerequisite for
completion of the McGaw Graduate Medical Education Certificate of Global Health, your
experience will aid our evaluation of the certificate as a means to continuously improve your
experience.
The pre-test is the first component of the evaluation procedure. At the end of the course, you will
be invited to take a post-test survey. All information you provide will remain confidential, but
please indicate your student ID number so we can effectively analyze results.
Student ID Number:
Should you have any questions please feel free to contact: Ashti Doobay-Persaud, MD, Faculty Development Chair [email protected]
EVALUATION PLAN 47
Appendix F: Pre-Test Survey
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: PRE-TEST SURVEY
Questions 1 – 5 will ask you general information: 1. Please indicate which residency program you are currently participating in:
Drop Down Menu
2. How important is this Global Health training to your medical education:
Essential Very Important
Somewhat Important
Not Important
3. Please indicate the site you are interested in completing your global practicum:
Check list will be compiled once sites are finalized, TBA.
4. Have you had any experience in global healthcare settings?
Yes No
5. If yes, please describe indicate location(s) and briefly describe responsibility(s):
Question 6 will ask you to rate your various skills:
6. How would you currently rate yourself in the following areas:
A Major Strength
Somewhat Strong
Average Somewhat weak
A Major weakness
Scholarly Writing
Scholarly Presentations
Ability to work with diverse groups of people
Ability to work in teams
Patient care
Ability to take initiative
EVALUATION PLAN 48
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: PRE-TEST SURVEY
Question 7 will ask about your awareness in global health:
7. How would you currently rate your understanding in the following areas:
Very Knowledge-
able
Some Knowledge
Little Knowledge
None
The relationship between health and human rights
Major determinants of global
health
Familiarity with non-western health practices
The role of community engagement strategy in healthcare
Socio-political contexts in relationship to global health
EVALUATION PLAN 49
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: PRE-TEST SURVEY
Question 8 will ask about your opinions and attitudes:
8. To what extent do you agree or disagree with the following:
Strongly Agree
Agree Disagree Strongly Disagree
Global health practitioners have the ability to communicate effectively with patients and health practitioners with a culture/ethnicity different from their own.
Global health practitioners have a diverse skill set, outside their chosen specialty
Global health programs develop relationships with other experienced healthcare professionals
Global health programs share global health knowledge with novices in the field
Global health programs expand the understanding of ethics in global healthcare
EVALUATION PLAN 50
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: PRE-TEST SURVEY
Question 9 and 10 will ask about your awareness of core concepts in global health
9. How would you currently rate your understanding in the following areas:
Very Knowledge-
able
Some Knowledge
Little Knowledge
None
Global burden of disease
Global burden of maternal and child health
Global burden of mental health
Global burden of reproductive health
Global burden of surgical
disease
Global health systems
Health and humanitarian disaster response
Impact of climate change on global heath
Effective use of local medicine
10. How would you currently rate yourself in the following areas:
Very Knowledge-
able
Some Knowledge
Little Knowledge
None
Ability to synthesize global health concepts into written work
Familiarity with current research in the importance of equitable global healthcare
Ability to inform others about the importance and impact of global health
EVALUATION PLAN 51
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: PRE-TEST SURVEY
Question 11 will ask about your opinions and attitudes:
11. To what extent do you agree or disagree with the following:
Strongly Agree
Agree Disagree Strongly Disagree
Equitable (access, care, affordability) healthcare is a human right
Healthcare capacity of an under-resourced community directly effects the health of all communities
I have an ethical obligation to have competency in global healthcare
I believe medical practice outside the western tradition is vital to global healthcare
I presently incorporate global health health care practice into my current practice
I currently participate in global health initiatives (volunteer, research, education initiatives, community outreach, etc.)
Questions 12 – 17 will ask you general information:
12. Please indicate your program year: Check list will be compiled once residency ranking is finalized, TBA
13. Please give your current gender identity: Fill in Blank
14. Please give your race/ethnicity: Fill in Blank
15. Please give your sexual orientation (voluntary):
EVALUATION PLAN 52
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: PRE-TEST SURVEY
17. Prior to this certificate program, what kinds of international experience have you had?
Check Box If, Yes please give number of occurrences
No past international experience
Vacation
Lived abroad for more than 3 months
Volunteered abroad
Worked abroad
Served in the military abroad for more than 3
months
Participated in a study abroad program
Thank you for taking the time to complete the McGaw Graduate Medical Education Certificate
of Global Health pre-test survey!
EVALUATION PLAN 53
Appendix G: E-mail Draft of Post-Test Survey
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: EMAIL POST-TEST SURVEY
You are being invited to complete this follow-up survey as an alumnus of the McGaw Graduate
Medical Education Certificate of Global Health. Although your participation is a prerequisite for
completion of the McGaw Graduate Medical Education Certificate of Global Health, your
experience will aid our evaluation of the certificate as a means to continuously improve residents’
experiences.
All information you provide will remain confidential, but please indicate your student ID number
so we can effectively analyze results.
Student ID Number:
Should you have any questions please feel free to contact: Ashti Doobay-Persaud, MD, Faculty Development Chair [email protected]
EVALUATION PLAN 54
Appendix H: Post-Test Survey
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: POST-TEST SURVEY
Questions 1 will ask you general information: 1. Please indicate which residency program you are currently participating in:
2. At which site did you complete your global health experience:
Question 3 will ask you about your various skills:
3. How would you currently rate yourself in the following areas:
Very Experienced
Some Experience
Little Experience
No Experience
Scholarly Writing
Scholarly Presentations
Ability to work with diverse groups of people
Ability to work in teams
Patient care
Ability to take initiative
Question 4 will ask about your awareness in global health:
4. How would you currently rate your understanding in the following areas:
Very Knowledge-
able
Some Knowledge
Little Knowledge
None
The relationship between health and human rights
Major global health
determinants
Familiarity with non-western health practices
The role of community engagement stratifies in healthcare
Socio-political contexts in relationship to global health
EVALUATION PLAN 55
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical
Education Certificate of Global Health: POST-TEST SURVEY Question 5 will ask about your opinions and attitudes:
5. To what extent do you agree or disagree with the following:
Strongly Agree
Agree Disagree Strongly Disagree
Global health practitioners have the ability to communicate effectively with patients and health practitioners with a culture/ethnicity different from their own.
Global health practitioners have a diverse skill set, outside their chosen specialty
Global health programs develop relationships with other experienced healthcare professionals
Global health programs share global health knowledge with novices in the field
Global health programs expand the understanding of ethics in global healthcare
EVALUATION PLAN 56
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical
Education Certificate of Global Health: POST-TEST SURVEY Question 6 and 7 will ask about your awareness of core concepts in global health
6. How would you currently rate your understanding in the following areas:
Very Knowledge-
able
Some Knowledge
Little Knowledge
None
Global burden of disease
Global burden of maternal and child health
Global burden of mental health
Global burden of reproductive health
Global burden of surgical
disease
Global health systems
Health and humanitarian disaster response
Impact of climate change on global heath
Effective use of local medicine
7. How would you currently rate yourself in the following areas:
Very Knowledge-
able
Some Knowledge
Little Knowledge
None
Ability to synthesize global health concepts into written work
Familiarity with current research in the importance of equitable global healthcare
Ability to inform others about the importance and impact of global health
EVALUATION PLAN 57
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: POST-TEST SURVEY
Question 8-11 will ask about your opinions and attitudes:
8. To what extent do you agree or disagree with the following:
Strongly Agree
Agree Disagree Strongly Disagree
Equitable (access, care, affordability) healthcare is a human right
Healthcare capacity of an under-resourced community directly effects the health of all communities
I have an ethical obligation to have competency in global healthcare
I believe medical practice outside the western tradition is vital to global healthcare
I presently incorporate global health care practice into my current practice
I currently participate in global health initiatives (volunteer, research, education initiatives, community outreach, etc.)
9. To what extent do you agree or disagree with the following:
Strongly Agree
Agree Disagree Strongly Disagree
My mentorship experience improved my practice of medicine
My mentor was committed to my professional development
My mentor was committed to academic development
My mentor expanded my medical knowledge of global healthcare
My mentor expanded my cultural knowledge of global healthcare
My mentor and I effectively communicated while I was abroad
My mentor assisted in the development of my professional presentation
EVALUATION PLAN 58
10. To what extent do you agree or disagree with the following:
Strongly Agree
Agree Disagree Strongly Disagree
I felt the pre-site orientation prepared me for my global site experience
The lecture series enhanced my site experience
My onsite mentor worked to expand my medical and professional skills
My global experience was positive
My site experience gave me the chance to add to improve my medical skills
My site supervisor was committed to making it a good learning experience
11. To what extent do you agree or disagree with the following:
Strongly Agree
Agree Disagree Strongly Disagree
The Global Health Lecture series expanded my medical knowledge outside my specialty
The Global Health Lecture series schedule worked well with my residency program schedule
The global health content of the lecture series met my expectation
The lecturers were informative
The cross residency skill exchange improved my patient care at my global site
Questions 12 – 17 will ask you general information:
12. While participating in the certificate program did you complete any additional global health rotations?!!
Yes No
13. If yes, please indicate the program, location, and duration:
14. Who is your global health mentor:
EVALUATION PLAN 59
16. Please identify your intended professional practice:
Thank you for taking the time to complete the McGaw Graduate Medical Education Certificate
of Global Health follow-up survey!
EVALUATION PLAN 60
Appendix I: Pre-Test Resident Survey Email Reminder
Email Subject: McGaw: Congratulations & Next Step
Dear Resident First_Name (merge field),
Once again congratulations on your acceptance to the McGaw Graduate Medical Education
Certificate of Global Health!
The next step in the completion of your certificate is to take a few moments to complete a brief online survey (link below) as a means of continuing to improve the student experience. The pre-test is the first component of an evaluation procedure. Upon completion of the certificate, you will be required to take a post-test survey. All information you provide will remain confidential and will have no impact on approval for certificate, but please indicate your Feinberg School of Medicine ID number so we can effectively analyze results. Please complete the survey within two business days.
Survey Link: linkplaceholder.com
Due: within one week
We look forward to working and learning with you as you complete your certificate of global
health.
Should you have any questions, concerns, or requests please feel free to contact:
Dr. Ashti Doobay-Persaud, Director of the McGaw Graduate Medical Education Certificate of
Global Health program
EVALUATION PLAN 61
Appendix J: Post-Test Resident Survey Email Reminder
Email Subject: McGaw: Congratulations & Final Step
Dear Resident_First_Name (merge field),
Once again congratulations on your completion of the McGaw Graduate Medical Education
Certificate of Global Health!
The final step in the completion of your certificate is to take a few moments to complete a brief online survey (link below) as a means of continuing to improve the student experience. All information you provide will remain confidential and will have no impact on approval for certificate, but please indicate your Feinberg School of Medicine ID number so we can effectively analyze results. Please complete the survey within two business days.
Survey Link: linkplaceholder.com
Due: within two business days
We look forward to your feedback and to working with you in the future!
Should you have any questions, concerns, or requests please feel free to contact:
Dr. Ashti Doobay-Persaud, Director of the McGaw Graduate Medical Education Certificate of
Global Health program
EVALUATION PLAN 62
Appendix K: General Survey Reminder Email
Email Subject: McGaw: Survey Reminder
Dear Resident First_Name (merge field),
The next step in the completion of your certificate is to take a few moments to complete a brief online survey (link below) as a means of continuing to improve the student experience. Please complete the survey immediately.
Survey Link: linkplaceholder.com Due: Immediately All information you provide will remain confidential and will have no impact on approval for certificate, but please indicate your Feinberg School of Medicine ID number so we can effectively analyze results. Should you have any questions, concerns, or requests please feel free to contact:
Dr. Ashti Doobay-Persaud, Director of the McGaw Graduate Medical Education Certificate of Global Health program
EVALUATION PLAN 63
Appendix L: Resident Semi-Structured Interview Protocol
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Resident Semi-Structured Interview
Protocol Housekeeping & Consent Obtainment
Thank you for agreeing to participate in this interview today. My name is ___________ and I
will be your interviewer. In advance of our starting our discussion, there are a few housekeeping
items to attend to.
• We have planned this interview to last no longer than thirty minutes. • To facilitate our note taking and use of time, we would like to audio tape our
conversation. • Only the program director and evaluators on the project will be privy to the tapes, which
will be destroyed after transcription. Further, no identifying information will be shared in our final evaluation report.
• Before we start, we ask that you complete two forms. First, a questionnaire that asks for some demographic information; second, consent form, which is required to be included in our research.
• The consent form states: o All information will be held in confidence o Participation is voluntary and you may opt out at any time o We do not intend to inflict any harm
Introduction
You have been selected to speak with us today because of your participation in the McGaw
Global Health Resident Certificate (MGHRC) Program through the Fienberg School of Medicine.
Our evaluation focuses on both the outcomes achieved by the MGHRC and processes used to
accomplish these outcomes. We expect through this interview that we will also capture
improvement opportunities for MGHRC program.
EVALUATION PLAN 64
As I mentioned earlier, I have planned this interview to last no longer than one hour. I would like
to discuss several aspects of the program with you. At times, it may be necessary for me to
interrupt you in order to ensure we are able to discuss all key points. Please understand if this
occurs. Also, you are free to skip any questions you do not wish to answer or stop this interview
at any time. Do you have any questions before we begin?
A. I would like to begin by asking you some questions about your background and global health connection. A1. Tell me about your decision to enroll in the certificate program?
Probes:
• How did you learn about the opportunity? • Who contacted you or did you reach out to you Program Director? • What impact, if at all, did the global site locations have in terms of
your decision? • What effect, if at all, did your fellow residents have in terms of your
decision? B. Main Program Components
I would like to get a better understanding of your experience in completing the certificate
program.
B1. Has your decision to complete this certificate added to your practice?
Probes:
• What has been your favorite or most rewarding experience? • What part of the program most challenged you?
B2. Tell me about your Global Site experience.
Probes:
• For what reasons did you choose that specific site? • In what ways, if at all, has your global site experience affected your
practice?
B3. Tell me a bit more about your mentor/mentee experience
Probes:
• What, if any, impact did this relationship have on your professional development?
EVALUATION PLAN 65
• What is it like being able to spend time with an instructor outside of the hospital environment?
• In what ways, if at all, has your mentor influenced your practice?
C. Effectiveness and Needs
C1. Earlier, you reflected on the certificate program as it relates to your current practice. I
want to better understand this so we can improve the program for future residents. Can
you give me ideas of what, if anything should change and continue in the future?
Probes:
• Do you have specific ideas about what should change or remain in the program going forward? Areas of focus may be, Journal Club, Mentorship, Lectures, Global Experience ect.
• Do you have specific ideas about what should change or remain in how the program is organized? Areas of focus may be, Journal Club, Mentorship, Lectures, Global Experience ect.
• Do you have any specific areas that should be added to the certificate?
C2. How, if at all, has your experience differed from your expectations prior to the completion of this certificate? C3.Is there anything about the program that I did not ask you that you would like to share?
Thank you for your participation. Your feedback will be enable us to continue to
improve and grow our program. Please feel free to reach out should you like to share any
additional feedback.
EVALUATION PLAN 66
Appendix M: Semi-Structured Interview Resident Demographic Questionnaire
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Resident Demographic Questionnaire
Please complete the following questions in advance of our interview.
Interviewee Background & Resident Program
1. What is your residence program _______________________________
2. Did you complete a Global Rotation prior to this
program_______________________
a. If so where, ____________________________________________
3. Do you currently volunteer in the community ___________________________
a. If so where, ________________________________________________
Thank you for your participation.
EVALUATION PLAN 67
Appendix N: Semi-Structured Resident Interview Protocol Summary Notes
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Resident Interview Protocol
Resident Interview Protocol Summary Notes
Date:_________________________
Interviewee:___________________________________________________
Interviewer:___________________________________________________
Other Topics
Discussed:_____________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
___________________________
Documents
Obtained:______________________________________________________________________
______________________________________________________________________________
Post Interview Comments:
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
EVALUATION PLAN 68
Appendix O: Resident Semi-Structured Student Interview E-Mail
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Resident Interview Email
Thank you for volunteering your time to participate in an interview regarding your experience as
an enrolled student in the McGaw Graduate Medical Education Certificate of Global Health.
This interview will last no longer than 30 minutes, and be held at The Center for Global Health
offices (address below). In order to attend to your schedule we have blocked out several
interview dates and times, hopefully one will work with your schedule. If not, please contact Dr.
Doobay-Persaud to set up a more convenient time.
Please follow the link below to schedule your interview date and time.
(Doodle Link)
Please follow the link below to complete a brief pre-interview demographic form.
(Link Here)
All feedback and information you provide will remain private.
Should you have any questions please feel free to contact:��
Ashti Doobay-Persaud, MD, Faculty Development Chair
EVALUATION PLAN 69
Appendix P: Mentor Semi-Structured Interview Protocol
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Mentor Semi-Structured Interview
Protocol Housekeeping & Consent Obtainment
Thank you for agreeing to participate in this interview today. My name is ___________ and I
will be your interviewer. In advance of our starting our discussion, there are a few housekeeping
items to attend to.
• We have planned this interview to last no longer than thirty minutes. • To facilitate our note taking and use of time, we would like to audio tape our
conversation. • Only the program director and evaluators on the project will be privy to the tapes, which
will be destroyed after transcription. Further, no identifying information will be shared in our final evaluation report.
• Before we start, we ask that you complete two forms. First, a questionnaire that asks for some demographic information; second, consent form, which is required to be included in our research.
• The consent form states: o All information will be held in confidence o Participation is voluntary and you may opt out at any time o We do not intend to inflict any harm
Introduction
You have been selected to speak with us today because of your participation as a Mentor in the
McGaw Global Health Resident Certificate (MGHRC) Program through the Fienberg School of
Medicine. Our evaluation focuses on both the outcomes achieved by the MGHRC and processes
used to accomplish these outcomes. We expect through this interview that we will also capture
improvement opportunities for MGHRC program.
As I mentioned earlier, I have planned this interview to last no longer than one hour. I would like
to discuss several aspects of the program with you. At times, it may be necessary for me to
interrupt you in order to ensure we are able to discuss all key points. Please understand if this
EVALUATION PLAN 70
occurs. Also, you are free to skip any questions you do not wish to answer or stop this interview
at any time. Do you have any questions before we begin?
A. Interviewee Background and Global Health Connection
A1. Tell me a little about your decision to become a Mentor for the certificate program?
Probes:
• How did you learn about the opportunity? • Who contacted you or did you reach out to you Program Director? • What impact, if at all, did your mentee have on your Global Health
practice? • What effect, if at all, did your colleagues have in terms of your
decision? B. Main Program Components
I would like to get a better understanding of your experience as a Mentor with the certificate
program.
B1. In what ways, if any, has your decision to be a Mentor added to your practice?
Probes:
• What has been your favorite or most rewarding experience? • What part of the program most challenged you?
B2. Tell me about your communication with your mentee while on their Global Site rotation.
Probes:
• How often did you communicate? • How did you utilize the Mentor/Mentee contract? • Would you recommend a particular communication protocol in the future?
If so what?
B3. In what ways, if any, has your mentee influenced your practice?
Probes:
• What, if any, impact did this relationship have on your professional development?
• What is it like being able to spend time with a resident outside of the hospital environment?
C. Effectiveness and Needs
EVALUATION PLAN 71
C1. Earlier, you reflected on the certificate program as this relates to your Global Health
practice. I want to better understand this so we can improve the program for future
Mentors. Can you give me ideas of what should change and continue in the future?
Probes:
• Do you have specific ideas about what should change or remain in the program going forward? Areas of focus may be, Journal Club, Mentorship, Lectures, Global Experience ect.
• Do you have specific ideas about what should change or remain in how the program is organized? Areas of focus may be, Journal Club, Mentorship, Lectures, Global Experience ect.
• Do you have any specific areas that should be added to the certificate?
C2. How, if at all, has your experience differed from your expectations prior to your mentee’s completion of this certificate?
C3. Is there anything about the program that I did not ask you that you would like to
share?
Well, thank you very much for taking the time to do this interview with us and sharing
your story. Your participation and understanding of the program is critical in our
evaluation. Please feel free to reach out should you like to share any additional feedback.
EVALUATION PLAN 72
Appendix Q: Semi-Structured Interview Mentor Demographic Questionnaire
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Mentor Demographic Questionnaire
Please complete the following questions in advance of our interview.
Interviewee Background & Resident Program
1. What is your Title _____________________________
2. What is your Department:_______________________________
3. Have you formally Mentored a Resident prior to this
program_______________________
a. If so where, ____________________________________________
4. Do you currently volunteer in the community ___________________________
a. If so where, ________________________________________________
Thank you for your participation.
EVALUATION PLAN 73
Appendix R: Mentor Semi-Structured Interview Protocol Notes
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Mentor Interview Protocol
Student Interview Protocol Summary Notes
Date:_________________________
Interviewee:___________________________________________________
Interviewer:___________________________________________________
Other Topics
Discussed:_____________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
___________________________
Documents
Obtained:______________________________________________________________________
______________________________________________________________________________
Post Interview Comments:
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
EVALUATION PLAN 74
Appendix S: Mentor Semi-Structured Student Interview E-Mail
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Mentor Interview Email
Thank you for volunteering your time to participate in an interview regarding your experience as
a Mentor in the McGaw Graduate Medical Education Certificate of Global Health. This
interview will last no longer than 30 minutes, and be held at The Center for Global Health offices
(address below). In order to attend to your schedule we have blocked out several interview dates
and times, hopefully one will work with your schedule. If not, please contact Dr. Doobay-
Persaud to set up a more convenient time.
Please follow the link below to schedule your interview date and time.
(Doodle Link)
Please follow the link below to complete a brief pre-interview demographic form.
(Link Here)
All feedback and information you provide will remain private.
Thank you again for your time and participation, have a wonderful day!
Should you have any questions please feel free to contact:�
Ashti Doobay-Persaud, MD, Faculty Development Chair
EVALUATION PLAN 75
Appendix T: Lecturer Semi-Structured Interview Protocol
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Lecturer Semi-Structured Interview
Protocol Housekeeping & Consent Obtainment
Thank you for agreeing to participate in this interview today. My name is ___________ and I
will be your interviewer. In advance of our starting our discussion, there are a few housekeeping
items to attend to.
• We have planned this interview to last no longer than thirty minutes. • To facilitate our note taking and use of time, we would like to audio tape our
conversation. • Only the program director and evaluators on the project will be privy to the tapes, which
will be destroyed after transcription. Further, no identifying information will be shared in our final evaluation report.
• Before we start, we ask that you complete two forms. First, a questionnaire that asks for some demographic information; second, consent form, which is required to be included in our research.
• The consent form states: o All information will be held in confidence o Participation is voluntary and you may opt out at any time o We do not intend to inflict any harm
Introduction
You have been selected to speak with us today because of your participation in the McGaw
Global Health Resident Certificate (MGHRC) Program through the Fienberg School of Medicine.
Our evaluation focuses on both the outcomes achieved by the MGHRC and processes used to
accomplish these outcomes. We expect through this interview that we will also capture
improvement opportunities for MGHRC program.
As I mentioned earlier, I have planned this interview to last no longer than one hour. I would like
to discuss several aspects of the program with you. At times, it may be necessary for me to
EVALUATION PLAN 76
interrupt you in order to ensure we are able to discuss all key points. Please understand if this
occurs. Also, you are free to skip any questions you do not wish to answer or stop this interview
at any time. Do you have any questions before we begin?
A. Interviewee Background and Global Health Connection
A1. Tell me a little about your decision to lecture for the certificate program?
Probes:
• How did you learn about the opportunity? • Who contacted you or did you reach out to Dr. Doobay-Persaud? • What impact, if at all, did a previous global health experience have in
terms of your decision? • What affect, if at all, did your colleagues have in terms of your
decision? • How, if at all, has your experience differed from your expectations
prior to your lecture? B. Main Program Components
I would like to get a better understanding of your lecture experience with the certificate
program.
B1. Tell me about your overall experience in lecturing for the program.
Probes:
• What has been your favorite or most rewarding experience? • What part of the program most challenged you? • How would you describe the students’ reception of the lecture? • In what ways did you structure your lectures to meet learning outcomes?
C. Effectiveness and Needs
C1. Earlier, you discussed your expectations of the lecture experience. I want to better
understand this so we can be sure to improve the program for future residents. Can you
give me ideas of what should change and continue in the future?
Probes:
EVALUATION PLAN 77
• Do you have specific ideas about what should change or remain in the program going forward? Areas of focus may be, Journal Club, Mentorship, Lectures, Global Experience ect.
• Do you have specific ideas about what should change or remain in how the program is organized? Areas of focus may be, Journal Club, Mentorship, Lectures, Global Experience ect.
• Do you have any specific areas that should be added to the certificate?
C2. Is there anything about the program that I did not ask you that you would like to
share?
Thank you for your participation. Your feedback will enable us to continue to improve
and grow our program. Please feel free to reach out should you like to share any
additional feedback.
EVALUATION PLAN 78
Appendix U: Semi-Structured Interview Lecturer Demographic Questionnaire
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Lecturer Demographic Form
Please complete the following questions in advance of our interview.
Interviewee Background & Resident Program
4. Please give your title __________________________________
5. Please give your program _______________________________
6. Did you give this lecture prior to this program_______________________
a. If so where, ____________________________________________
7. Do you currently volunteer in the community ___________________________
a. If so where, ________________________________________________
Thank you for your participation.
EVALUATION PLAN 79
Appendix V: Lecturer Semi-Structured Interview Protocol Notes
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Lecturer Interview Protocol Summary Notes
Date:_________________________
Interviewee:___________________________________________________
Interviewer:___________________________________________________
Other Topics
Discussed:_____________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
___________________________
Documents
Obtained:______________________________________________________________________
______________________________________________________________________________
Post Interview Comments:
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
EVALUATION PLAN 80
Appendix W: Lecturer Semi-Structured Lecturer Interview E-Mail
Northwestern University Feinberg School of Medicine, McGaw Graduate Medical Education Certificate of Global Health: Lecturer Interview Email
Thank you for volunteering your time to participate in an interview regarding your experience as
a Lecturer for the McGaw Graduate Medical Education Certificate of Global Health. This
interview will last no longer than 30 minutes, and be held at The Center for Global Health offices
(address below). In order to attend to your schedule we have blocked out several interview dates
and times, hopefully one will work with your schedule. If not, please contact Dr. Doobay-
Persaud to set up a more convenient time.
Please follow the link below to schedule your interview date and time.
(Doodle Link)
Please follow the link below to complete a brief pre-interview demographic form.
(Link Here)
All feedback and information you provide will remain private.
Should you have any questions please feel free to contact:��
Ashti Doobay-Persaud, MD, Faculty Development Chair