School Performance Variation on COMLEX Level 1 Linjun Shen, Ph.D., MPH National Board of Osteopathic Medical Examiners.

School Performance School Performance Variation on COMLEX Level Variation on COMLEX Level

11

Linjun Shen, Ph.D., MPHLinjun Shen, Ph.D., MPH

National Board of Osteopathic Medical National Board of Osteopathic Medical

ExaminersExaminers

PurposesPurposes

Provide more information to school Provide more information to school administrators and educatorsadministrators and educators

Help school administrators and Help school administrators and educators to interpret and analyze educators to interpret and analyze COMLEX dataCOMLEX data

Demonstrate analytical approaches Demonstrate analytical approaches individual schools can adopt in individual schools can adopt in digesting their own COMLEX datadigesting their own COMLEX data

LimitationsLimitations

Analysis is limited to Level 1 June Analysis is limited to Level 1 June 2002 administration2002 administration

Variables available for this analysis Variables available for this analysis is limited, so as the resultsis limited, so as the results

A model of the medical A model of the medical school-licensing examination school-licensing examination

relationshiprelationship

Practice

LicensingExams

Pre-MedMed Sch

Scope of This AnalysisScope of This Analysis

Interest of the analysis: school mean Interest of the analysis: school mean scores and school passing rates on scores and school passing rates on Level 1.Level 1.

Potentially influential factors: gender Potentially influential factors: gender of students, sector of school, school of students, sector of school, school mean MCAT.mean MCAT.

Analytical approach: mainly descriptive Analytical approach: mainly descriptive and graphical with a brief statistical and graphical with a brief statistical modeling.modeling.

Summary of June 2002 Summary of June 2002 Level 1Level 1

N Mean/SD Passing

2696 500/79 90%

June 2002 Level 1June 2002 Level 1

StandardScore

Min. Passing 400(-1.26sd)

Mean 500

Lowest 225(-3.48sd)

Highest 862(4.58sd)

Level 1 School MeanLevel 1 School Mean

440

450

460

470

480

490

500

510

520

530

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Highest and Lowest on Highest and Lowest on Level 1Level 1

RANK

19.001.00

TO

TA

L800

750

700

650

600

550

500

450

400

350

300

250

200

School Mean Score School Mean Score DistributionDistribution

LEVEL1

530.0

525.0

520.0

515.0

510.0

505.0

500.0

495.0

490.0

485.0

480.0

475.0

6

5

4

3

2

1

0

Std. Dev = 16.54

Mean = 500.3

N = 19.00

School Failing Rate School Failing Rate DistributionDistribution

FAILING PERCENTAGE

16.014.012.010.08.06.04.0

5

4

3

2

1

0

Std. Dev = 4.65

Mean = 9.5

N = 19.00

Failing Rate by Ranking of Failing Rate by Ranking of MeansMeans

Rank by Mean

19

18

17

16

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

FA

ILIN

G P

ER

CE

NT

AG

E18

16

14

12

10

8

6

4

2

0

Passing Rate by Ranking of Passing Rate by Ranking of MeansMeans

Rank by Mean Score

19

18

17

16

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

PA

SS

ING

RA

TE

98

96

94

92

90

88

86

84

82

80

School Mean PerformanceSchool Mean Performance

Two ways to rank schools: by mean Two ways to rank schools: by mean score and passing ratescore and passing rate

Numerical ranking is not as Numerical ranking is not as important as qualitative groupingimportant as qualitative grouping

Variation among school mean scores Variation among school mean scores is substantial (53 point between the is substantial (53 point between the highest and lowest, 40 point between highest and lowest, 40 point between above and below average groups)above and below average groups)

Significant Gender Significant Gender Differences on June 2002 Differences on June 2002

Level 1Level 1

NMean/SD Passing

Male 1534 509/82 91.3%

Female 1162 486/73 88.6%

Gender Difference in Gender Difference in Mean Scores by SchoolMean Scores by School

400

420

440

460

480

500

520

540

560

1 3 5 7 9 11 13 15 17 19

MaleFemale

Gender Difference in Gender Difference in Failing Percentage by Failing Percentage by

SchoolSchool

0

5

10

15

20

25

1 3 5 7 9 11 13 15 17 19

MaleFemale

Gender Difference in Gender Difference in Osteopathic Medical Osteopathic Medical

SchoolsSchools

There are substantial gender There are substantial gender differences in terms of individual differences in terms of individual student Level 1 performancestudent Level 1 performance

For a few schools, the differences in For a few schools, the differences in both mean scores and passing rates are both mean scores and passing rates are alarmingalarming

Gender differences can also be an Gender differences can also be an opportunity for schools to improve if opportunity for schools to improve if efforts are made to narrow the gap efforts are made to narrow the gap

Sector Differences on June Sector Differences on June 2002 Level 12002 Level 1

Mean/SD Passing

Sector A 498/79 89.9%

Sector B 505/79 91.2%

Gender and Sector Gender and Sector InteractionInteraction

on Mean Scoreon Mean Score

516

492

508

484

460

470

480

490

500

510

520

Male Female

Sector ASector B

Gender and Sector Gender and Sector InteractionInteraction

on Failing Rateon Failing Rate

9.38.3

12

8.8

0

2

4

6

8

10

12

14

Male Female

Sector ASector B

Effects of Sector of Effects of Sector of Osteopathic Medical Osteopathic Medical

SchoolsSchools

There is no evidence that sector There is no evidence that sector significantly affects Level 1 performancesignificantly affects Level 1 performance

According to a study 10 years ago, sector According to a study 10 years ago, sector was a powerful factor in determining was a powerful factor in determining osteopathic Board exam performanceosteopathic Board exam performance

It is a very positive improvement of It is a very positive improvement of osteopathic medical educationosteopathic medical education

More formal study is neededMore formal study is needed

School Mean MCAT School Mean MCAT DistributionDistribution

MCAT

9.008.758.508.258.007.757.50

6

5

4

3

2

1

0

Std. Dev = .42

Mean = 8.32

N = 19.00

School Mean MCAT of Class School Mean MCAT of Class 20002000

Mean 8.32

Lowest 7.40

Highest 9.00

Sector A 8.36

Sector B 8.29

Mean MCAT by RankMean MCAT by Rank

Rank

19

18

17

16

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

Me

an

MC

AT

9.5

9.0

8.5

8.0

7.5

7.0

Mean MCAT and Mean Mean MCAT and Mean Level 1Level 1

(r = .30, not significant)(r = .30, not significant)

Mean MCAT

9.59.08.58.07.57.0

Me

an

LE

VE

L1

530

520

510

500

490

480

470

Mean MCAT and Mean Mean MCAT and Mean Passing Rate (r = .27)Passing Rate (r = .27)

Mean MCAT

9.59.08.58.07.57.0

Pa

ssin

g R

ate

98

96

94

92

90

88

86

84

82

MCAT and Level 1 MCAT and Level 1 PerformancePerformance

The implication of correlation of The implication of correlation of school means of MCAT and a single school means of MCAT and a single Level 1 administration is highly Level 1 administration is highly limitedlimited

Individual student MCAT and Level Individual student MCAT and Level 1 score relationship can be very 1 score relationship can be very different from school mean MCAT different from school mean MCAT and school mean Level 1and school mean Level 1

Conventional RegressionConventional Regression

GenderSectorscoreLevel *23*85081

Results of Conventional Results of Conventional RegressionRegression

The model is significant. Both gender The model is significant. Both gender and sector are significant predictor.and sector are significant predictor.

This regression model with two This regression model with two variables only explained 3% of the variables only explained 3% of the student. performance variation. Most student. performance variation. Most of the variation remains unexplained.of the variation remains unexplained.

More information is needed to explain More information is needed to explain student performance.student performance.

Hierarchical Linear Hierarchical Linear RegressionRegression

11211101

00201000

10score1Level

MCATSector

MCATSector

Gender

Models Level School

Model LevelStudent

Results of HLM AnalysisResults of HLM Analysis Difference of students Level 1 Difference of students Level 1

performance across school is significantperformance across school is significant Gender is a predictor for Level 1 scores Gender is a predictor for Level 1 scores

but the difference of gender is not but the difference of gender is not significant across schoolssignificant across schools

Neither Sector and school mean MCAT Neither Sector and school mean MCAT predict student Level 1 performancepredict student Level 1 performance

More data will help: student MCAT, race, More data will help: student MCAT, race, curriculum type, etc.curriculum type, etc.

Thank You!Thank You!