Standardizing Learner Surveys Across the Enterprise Francis Kwakwa, MA, Radiological Society of North America Valerie Smothers, MA, MedBiquitous.

Standardizing Learner Standardizing Learner Surveys Across the Surveys Across the

EnterpriseEnterpriseFrancis Kwakwa, MA, Francis Kwakwa, MA,

Radiological Society of North Radiological Society of North AmericaAmerica

Valerie Smothers, MA, Valerie Smothers, MA, MedBiquitousMedBiquitous

DisclosureDisclosure

We have no financial We have no financial relationships to disclose.relationships to disclose.

ObjectivesObjectives

At the completion of this session, you At the completion of this session, you will be able to:will be able to:

Adopt strategies to improve the Adopt strategies to improve the collection of consistent evaluation collection of consistent evaluation data from learners data from learners

Adopt strategies to improve the Adopt strategies to improve the analysis of evaluation data across analysis of evaluation data across the CME enterprise the CME enterprise

OverviewOverview

1.1. Challenges in analyzing learner Challenges in analyzing learner surveyssurveys

2.2. MedBiquitous and MEMSMedBiquitous and MEMS

3.3. RSNA’s Implementation of a RSNA’s Implementation of a Standardized SurveyStandardized Survey

4.4. Results of RSNA course evaluationResults of RSNA course evaluation

5.5. Challenges faced by RSNAChallenges faced by RSNA

6.6. Key strategies for improving data Key strategies for improving data collection and analysiscollection and analysis

Challenges in Analyzing Challenges in Analyzing Learner SurveysLearner Surveys

Most of us use Most of us use surveyssurveys

Surveys often differ Surveys often differ based on activitybased on activity

Survey data may be Survey data may be in different systems in different systems or formatsor formats

The result: it’s hard The result: it’s hard to analyze results to analyze results across activitiesacross activities

RSNARSNA

Radiological Society of North AmericaRadiological Society of North America ““to promote and develop the highest to promote and develop the highest

standards of radiology and related standards of radiology and related sciences through education and sciences through education and research”research”

Over 40,000 membersOver 40,000 members Online and in-person CME activitiesOnline and in-person CME activities Member of MedBiquitousMember of MedBiquitous Francis Kwakwa, Chair of the Francis Kwakwa, Chair of the

MedBiquitous Metrics Working Group MedBiquitous Metrics Working Group

MedBiquitousMedBiquitous

Technology standards developer for Technology standards developer for healthcare educationhealthcare education ANSI AccreditedANSI Accredited Develops open XML standardsDevelops open XML standards 60 members (societies, universities, 60 members (societies, universities,

government, industry)government, industry) 7 working groups7 working groups

The Focus on MetricsThe Focus on Metrics ““Without the creation Without the creation

of a standard data of a standard data set for reporting set for reporting CME program CME program outcomes … it is outcomes … it is difficult to obtain difficult to obtain consistent metrics of consistent metrics of those outcomes. And those outcomes. And if you can’t measure if you can’t measure it, you can’t improve it, you can’t improve itit.” .”

Medical Education Medical Education Metrics – MEMSMetrics – MEMS

Ross Martin, MD, Director Healthcare Informatics Group, Pfizer

Another PerspectiveAnother Perspective

““I need this to I need this to better better understand how understand how my program as a my program as a whole is doing.”whole is doing.”

Nancy Davis, American Academy of Family Physicians

The MedBiquitous Metrics The MedBiquitous Metrics Working GroupWorking Group

Mission:Mission: to develop XML standards … for the exchange of aggregate evaluation data and other key metrics for health professions education.

Originally a subcommittee of the Originally a subcommittee of the Education Working GroupEducation Working Group

Became a working group in April Became a working group in April 20052005

We’re all using the same measuring stick…

--Francis

Who is Involved?Who is Involved? Francis Kwakwa, Francis Kwakwa,

RSNA, ChairRSNA, Chair Linda Casebeer, Linda Casebeer,

Outcomes Inc. Outcomes Inc. Nancy Davis, AAFPNancy Davis, AAFP Michael Fordis, Baylor Michael Fordis, Baylor

College of MedicineCollege of Medicine Stuart Gilman, Stuart Gilman,

Department of Department of Veterans AffairsVeterans Affairs

Edward Kennedy, Edward Kennedy, ACCME *ACCME *

Jack Kues, University Jack Kues, University of Cincinnatiof Cincinnati

Tao Le, Johns Hopkins Tao Le, Johns Hopkins UniversityUniversity

Ross Martin, PfizerRoss Martin, Pfizer Jackie Mayhew, PfizerJackie Mayhew, Pfizer Mellie Pouwels, RSNAMellie Pouwels, RSNA Andy Rabin, CE CityAndy Rabin, CE City Donna Schoonover, Donna Schoonover,

Department of Department of Veterans AffairsVeterans Affairs

* Invited experts

What’s in MEMSWhat’s in MEMS Participation Metrics Participation Metrics

how many participantshow many participants Learner Demographics Learner Demographics

profession, specialtyprofession, specialty Activity Description Activity Description

name, typename, type Participant Activity Evaluation Participant Activity Evaluation

survey resultssurvey results Other types of evaluation metrics planned Other types of evaluation metrics planned

for future versionsfor future versions

For more information:For more information:

Metrics Working Group PageMetrics Working Group Pagehttp://www.medbiq.org/working_grohttp://www.medbiq.org/working_groups/metrics/index.htmlups/metrics/index.html

MedBiquitous WebsiteMedBiquitous Websitehttp://www.medbiq.orghttp://www.medbiq.org

DiscussionDiscussion

Describe the learner surveys that Describe the learner surveys that you are using and how they differ you are using and how they differ from or are similar to the survey from or are similar to the survey described. What are the benefits or described. What are the benefits or drawbacks of using a standardized drawbacks of using a standardized survey? survey?

RSNA’s Project…RSNA’s Project…

Adoption of MEMS survey instrument Adoption of MEMS survey instrument coincided with implementation of a coincided with implementation of a new Learning Management Systemnew Learning Management System

Currently MEMS is used to evaluate Currently MEMS is used to evaluate over 300 online coursesover 300 online courses

RSNA’s Project…RSNA’s Project…

Types of online courses using MEMSTypes of online courses using MEMS

Cases of the Day (COD)Cases of the Day (COD)

RadioGraphics CME Tests/Education RadioGraphics CME Tests/Education Exhibits (EE)Exhibits (EE)

Refresher Courses (RSP)Refresher Courses (RSP)

Results…Results…

COD-45 (N = 24)COD-45 (N = 24)The course achieved its learning objectivesThe course achieved its learning objectives

0

10

20

30

40

50

60

Strongly Agree Agree Neutral Disagree StronglyDisagree

%

EE-355 (N = 32)EE-355 (N = 32)The course achieved its learning objectivesThe course achieved its learning objectives

0

10

20

30

40

50

60

70


%

RSP-2904 (N = 43)RSP-2904 (N = 43)The course achieved its learning objectivesThe course achieved its learning objectives

0

10

20

30

40

50

60

70

80

90


%

COD-45 (N = 24) COD-45 (N = 24) The course was relevant to my clinical The course was relevant to my clinical

learning needslearning needs

0

10

20

30

40

50

60


%

EE-355 (N = 32)EE-355 (N = 32)The course was relevant to my clinical The course was relevant to my clinical


0

10

20

30

40

50

60


%

RSP-2904 (N = 43)RSP-2904 (N = 43)The course was relevant to my clinical The course was relevant to my clinical


0

10

20

30

40

50

60

70

80

90


%

COD-45 (N = 24) COD-45 (N = 24) The course was relevant to my personal The course was relevant to my personal


0

10

20

30

40

50

60


%

EE-355 (N = 32)EE-355 (N = 32)The course was relevant to my personal The course was relevant to my personal


0

10

20

30

40

50

60


%

RSP-2904 (N = 43)RSP-2904 (N = 43)The course was relevant to my personal The course was relevant to my personal


0

10

20

30

40

50

60

70

80

90


%

COD-45 (N = 24) COD-45 (N = 24) The online method of instruction was The online method of instruction was

conducive to learningconducive to learning

0

10

20

30

40

50

60

70


%

EE-355 (N = 32)EE-355 (N = 32)The online method of instruction was The online method of instruction was


0

10

20

30

40

50

60


%

RSP-2904 (N = 43)RSP-2904 (N = 43)The online method of instruction was The online method of instruction was


0

10

20

30

40

50

60

70

80

90


%

COD-45 (N = 24) COD-45 (N = 24) The course validated my current practiceThe course validated my current practice

0

10

20

30

40

50

60

70


%

EE-355 (N = 32)EE-355 (N = 32)The course validated my current practiceThe course validated my current practice

0

10

20

30

40

50

60

70


%

RSP-2904 (N = 43)RSP-2904 (N = 43)The course validated my current practiceThe course validated my current practice

0

10

20

30

40

50

60

70


%

COD-45 (N = 24) COD-45 (N = 24) I plan to change my practice based on what I I plan to change my practice based on what I

learned in the courselearned in the course

0

10

20

30

40

50

60


%

EE-355 (N = 32)EE-355 (N = 32)I plan to change my practice based on what I I plan to change my practice based on what I


0

5

10

15

20

25

30

35

40

45

50


%

RSP-2904 (N = 43)RSP-2904 (N = 43)I plan to change my practice based on what I I plan to change my practice based on what I


0

5

10

15

20

25

30

35

40


%

COD-45 (N = 24) COD-45 (N = 24) The faculty provided sufficient evidence to The faculty provided sufficient evidence to

support the content presentedsupport the content presented

0

10

20

30

40

50

60


%

EE-355 (N = 32)EE-355 (N = 32)The faculty provided sufficient evidence to The faculty provided sufficient evidence to


0

10

20

30

40

50

60


%

RSP-2904 (N = 43)RSP-2904 (N = 43)The faculty provided sufficient evidence to The faculty provided sufficient evidence to


0

10

20

30

40

50

60


%

COD-45 (N = 24) COD-45 (N = 24) Was the course free of commercial bias Was the course free of commercial bias

towards a particular product or company?towards a particular product or company?

Yes100%

EE-355 (N = 32)EE-355 (N = 32) Was the course free of commercial bias Was the course free of commercial bias

towards a particular product or towards a particular product or company?company?

Yes100%

RSP-2904 (N = 43)RSP-2904 (N = 43) Was the course free of commercial bias Was the course free of commercial bias

towards a particular product or towards a particular product or company?company?

Yes98%

No2%

COD-45 (N = 24) COD-45 (N = 24) Did the course present a balanced view of Did the course present a balanced view of

clinical options?clinical options?

Yes96%

No4%

EE-355 (N = 32)EE-355 (N = 32) Did the course present a balanced view Did the course present a balanced view

of clinical options?of clinical options?

Yes100%

RSP-2904 (N = 43)RSP-2904 (N = 43) Did the course present a balanced view Did the course present a balanced view

of clinical options?of clinical options?

Yes98%

No2%

Group DiscussionGroup Discussion

What challenges to survey data What challenges to survey data collection and analysis have you collection and analysis have you faced?faced?

Challenges Faced by Challenges Faced by RSNARSNA

Survey is optional; little data Survey is optional; little data available for some coursesavailable for some courses

Little variation in the dataLittle variation in the data Some disconnect with educators on Some disconnect with educators on

how the data is usedhow the data is used Difficult to get data out of the LMSDifficult to get data out of the LMS Surveys for live events are not Surveys for live events are not

includedincluded

Key StrategiesKey Strategies

Data CollectionData Collection Common core set of survey questionsCommon core set of survey questions Common format for evaluation dataCommon format for evaluation data

Data AnalysisData Analysis Compare within type and modalityCompare within type and modality Compare across type and modalityCompare across type and modality Look for trends and variationLook for trends and variation Look for red flagsLook for red flags

An Added BenefitAn Added Benefit

Assists with program analysis and improvement required by the ACCME “The provider gathers data or information

and conducts a program-based analysis on the degree to which the CME mission of the provider has been met through the conduct of CME activities/educational interventions.”

--ACCME Updated Accreditation Criteria,

September 2006

Standardizing Learner Surveys Across the Enterprise Francis Kwakwa, MA, Radiological Society of North America Valerie Smothers, MA, MedBiquitous.

Documents

medbiquitous slide

medbiquitous metrics

group slide

surveys surveys

consistent metrics

data collection

analysis slide

analysis of evaluation