Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM.

Assessing clinical judgment using the script concordance test: The

importance of using specialty-specific experts to develop the scoring key

Petrucci AM. MD1, Nouh T. MD1, Boutros M. MD1, Gagnon R. PhD3, Meterissian S. MD1, 2

Department of Surgery, McGill University 1

Center for Medical Education, McGill University 2

Faculty of Medicine, University of Montreal 3

Association for Surgical Education

Paper Session 1

March 22, 2012

Disclosures

The authors have no disclosures to declare

Current Assessment Tools

• MCQ

• Rich context MCQ

• Orals

• Short answer questions

• Long answer questions

The perfect assessment tool?

Graduating general surgery residents

Multiple choice examination Oral examination+

Institute of Applied Research in Health Sciences Educationhttp://www.script.md/portal/en/script_en.html

What is Reasoning?

SCT: from memorizing to reasoning

• New tool intended to assess mental processes in medical practice

• Can be used to evaluate a trainee’s approach to ill-defined problems including those encountered by expert surgeons

Example

-2 Contra-indicated or almost totally-1 Not useful or even detrimental 0 Nor less nor more useful+1 Useful+2 Necessary or absolutely necessary

The Scoring Grid (Modal Experts’ Choice)

-2 -1 0 +1 + 2

Number of experts choosing answer 0 0 5 4 1

Score 0 0 5/10 4/10 1/10

Transformed score 0 0 5/5 4/5 1/5

Credit per item 0 0 1 0.8 0.2

Nouh, T., et al. (2012). The script concordance test as a measure of clinicalreasoning: A national validation study. The American Journal of Surgery, In press

What have we done in the past?

Preliminary Study:McGill University

n= 36

Meterissian, S. et al. (2007). Is the script concordance test a valid instrument for assessment ofintraoperative decision-making skills? The American Journal of Surgery, 193, 248-251.

Pan Canadian Studyn= 202

Nouh, T., et al. (2012). The script concordance test as a measure of clinicalreasoning: A national validation study. The American Journal of Surgery, In press

Purpose

To determine whether using a specialty-specific scoring key would result in a progression of scores from R1 to R5, with the R5 residents having the highest scores and therefore improving the validity of the scoring key

Methods

Participants

n= 25McGill

n= 202(Nouh, T., et al, 2012)

n tot= 227

Specialty Specific Scoring Key

Colorectal

Hepatobiliary

Endocrine

Surgical Oncology

Thoracic

Trauma/ ACS

EXPERTS

Colorectal 1 1 1 1 1

Hepatobiliary 1 1 1 1 1

Endocrine1 1 1 1 1

Surgical Oncology 1 1 1 1 1

Thoracic 1 1 1 1 1

Trauma/ ACS1 1 1 1 1

EX

PE

RT

2

EX

PE

RT

3

EX

PE

RT

4

EX

PE

RT

5

EX

PE

RT

1

Creation of amalgamated expert exams

• Reliability: Cronbach alpha coefficient

• Analysis of item-to-total item correlation used to select the best items for the final analysis

• Construct validity tested with a one-way ANOVA with post-hoc comparisons test and planned contrasts

• All p values at alpha<5% were considered significant

Statistical Analysis

RESULTS

Final exam100-question testCronbach α = 0.81

153-question test Cronbach α = 0.67

Elimination of items with a negative item-to-total item correlation

Results

?

Scores rising with increasing level of residency: p< 0.001

Number of expert panel members matters!

Gagnon et al. Medical Education, 2005:

• The mean of residents’ scores increased with increasing panel size.

“…the number of points earned on an SCT is influenced by the number of members used in the panel”.

Mean Score Graph

Scores rising with increasing level of residency: p< 0.001

Juniors vs Seniors

Total N

R1/ R2 45.1 (7.6) 109

R3/ R4/ R5 50.4 (8.0) 118

The average score of junior residents proved to be significantly lower than the average score of senior residents: p< 0.001

Discussion

• The results of this study suggest that the scoring key should be developed through the use of specialty- specific experts

• This may be due to the increasing subspecialization of General Surgery

• This has important implications in the application of the SCT on a wider level

Limitations• We used only 5 amalgamated general

surgery experts for each surgical discipline instead of the recommended >10

Gagnon et al. Medical Education 2005

• The resident pool consisted mostly of the same residents from our last study with the addition of 25 new residents from our institution

Future implications…

National wide study using the specialty-specific expert scoring key.

The SCT might be ready for a national in-training formative examination.

Acknowledgements

• Dr. Sarkis Meterissian• Dr. Robert Gagnon• Dr. Thamer Nouh• Dr. Marylise Boutros

Thank you