Finding the ‘Right’ Students: The Search for Predictive Validity in Applicant Screening for the Health Sciences Rosemary Lysaght, Ph.D. Catherine Donnelly, M.Sc. Michelle Villeneuve, M.Sc. School of Rehabilitation Therapy
Dec 18, 2015
Finding the ‘Right’ Students:
The Search for Predictive Validity in Applicant Screening
for the Health Sciences
Rosemary Lysaght, Ph.D.Catherine Donnelly, M.Sc.Michelle Villeneuve, M.Sc.
School of Rehabilitation Therapy
Background• Impetus for examination of our admissions
process was the creation of the new masters entry-level program in 2004
Goals for admission included: • An evidence-based admissions process• Increase the applicant pool• Efficiency in admission process
Overall Aim of SelectionTo select students who will:
• Succeed in the academic program• Perform credibly in professional practice • Possess the traits of character and ethical
values desired of a professional person
(Nayer, 1992)
Commonly Used Selection Criteria in Health Sciences• Pre-admission academic grades• Discipline-specific aptitude tests (MCAT, HOAE)• Interviews• Written submissions• Letters of reference• Prerequisites
(Auriemma, 2002; Salvatori, 2001)
Predictors of academic success
• Evidence across disciplines supports the predictive validity of – Pre-admission academic grades
• No clear support found for – Discipline-specific aptitude tests – Interviews– Written submissions– Letters of reference– Prerequisites
(Bridle, 1987; Caplan et al, 1996; Howard & Jerosh-Herold, 2000; Lewis & Smith, 2002; Kirchner et al., 2000; Kirchner & Holm, 1997; Salvatori, 2001)
Predictors of clinical performance
• Unclear relationship between pre-admission performance and clinical performance
• Valid outcome measures difficult to identify• Lack of consistent raters• Variability across settings
(Howard & Jerosch-Herold, 2000; Kirchner & Holm, 1997; Katz & Mosey, 1980; Tan et al., 2004)
Research Questions
1. What admissions screening tools best predict academic performance in a masters level OT program?
Selected academic courses and Clinical performance
2. Does undergraduate coursework predict success in topically-related coursework?
Method
• Analysis of existing data for 128 students admitted to the OT masters program– Multiple regression – Models created for each research question
• Project received approval by the Queen’s University’s Research Ethics Board
• Sample included 3 cohorts of students (1st three years of the new professional masters program)
Factors Considered in OT Admissions
– GPA– Academic transcript– Letter of intent– 2 referee rating forms/letters– Supporting data (foreign/non-traditional
applicants)
Predictor Variables
• Undergraduate Grade Point Average (GPA) – from ORPAS
• Letter of Intent (LOI) • Average rating, 2 raters
• Referee Rating• 5 point scale, 12 items• Average total score of 2 external referees
Predictor Variables (con’t)• Additional ratings were created for each student
based on file review (1 = weak to 5 = exceptional) Physical Sciences Experience/Preparation
• e.g. Anatomy, physiology, kinesiology (r for 2 raters = .98)
Social Sciences Experience/Preparation• e.g. Psychology, Sociology, Family Studies (r=.91)
Experience with People with Disabilities/Vulnerable Populations
• Rated for number, duration, and relevance (r=.79)• Information drawn from Letter of Intent, referee letters and
experience questionnaire
Criterion Variables
• Program GPA (1st year)• Communication Skills Practicum• Targeted course grades in:
Physical Determinants of Occupation Cognitive-Neurological Determinants Psycho-Emotional Determinants
Descriptive Statistics - Predictor Variables
Variable N Mean SDUndergraduate GPA 129 3.24 .278
Letter of Intent Rating 129 71.70 12.86
Referee Rating 129 4.72 .25
Bkg. Physical Sciences 127 3.92 1.35
Bkg. Social Sciences 127 2.90 1.66
Bkg. Experience with PWD 127 3.70 .88
Descriptive Statistics - Criterion Variables
Variable N Mean SDFirst Year GPA 128 80.29 3.0
0
Communication Skills Rating 112 8.44 1.19
Physical Determinants Grade 129 77.72 4.05
Cogntive-Neuro Det. Grade 126 78.97 5.13
Psycho-Emotional Det. Grade 126 78.89 5.03
Results – First Year GPA
• Predictor Variables: GPA, Letter of Intent, Referee Rating• Model is significant (R2 = .101; F= 4.65, p <
.001)• GPA is only variable with significant beta
score (p = .005)• Significant correlations between all three
variables and program GPA
Results – Communication Skills • Predictor Variables: GPA, Letter of
Intent, Referee Rating, all experience ratings,
• Model is not significant (R2 = .021; F= .62)• Referee rating is significantly correlated
with performance rating• No background experience ratings were
correlated with performance rating
Results – Physical Determinants
• Predictor Variables: GPA, all experience ratings
Result:• Model not significant (R2 = .009; F=.275)• No significant correlations between any
background experience rating and outcome
Results – Cognitive-Neuro
• Predictor Variables: GPA, all experience ratings
Result:• Model is significant (R2 = .14; F=5.0, p = .004)• GPA only variable with significant beta score (p
= .00)• Background in physical sciences was
significantly negatively correlated with outcome
Results – PsychoEmotional Det.
• Predictor Variables: GPA, all experience ratings
Result:• Model is significant (R2 = .082; F = 2.6, p< .05)• Social science bkgd only variable with
significant beta score (-.2 )• GPA is significantly positively correlated with
outcome, while SS bkgd is significantly negatively correlated.
Additional Observations• The contribution of the undergraduate GPA to all
models was reduced by the addition of the 3rd cohort, which had significantly higher GPA ratings than the first 2 cohorts and less spread in scores
• Referee ratings also had small but significant correlations with – physical determinants grade (r = .18, p = .02)– communication skills grade (r = .25, p = .003)
Conclusions
• Findings relative to GPA as a significant predictor of academic performance in a health science program supports previous research
• Other positive correlates with 1st year GPA:– Referee ratings– LOI ratings
suggest that these measures have some value in the admissions process.
• Referee ratings have even broader potential value, given positive correlations with performance in Physical Determinants & Communications Skills courses.
Conclusions (con’t)
• No support for the requirement of specific academic pre-requisites
• Previous experience with PWD/vulnerable populations did not affect academic grades or communication skills performance
• No support for the requirement of specific academic pre-requisites
• Previous experience with PWD/vulnerable populations did not affect academic grades or communication skills performance
Discussion
• Fairness of admissions process– Pre-requisite requirements not justified if predictive validity of courses
not substantiated– No control over authorship of LOI
• Inherent biases in process– LOI requirement may bias selection towards persons from the same
culture who highlight issues salient to that culture, females, and strong writers
– Applicants with strong social science background may be over-represented if GPA is primary selection factor
• Impact on program– Elimination of pre-requisites broadens applicant pool, may result in
higher calibre class– More diversity of students in program/field
Discussion
• Role of Pre-Requisites and other Screening Tools– LOI, Referee ratings useful as screen out, rather
than selection criteria?– Certain pre-requisites may make academic
progress easier for student– Admissions requirements may have value beyond
identification of best applicants• Credibility of process and applicant• Interviews may help form relationships, promote program
Limitations
• Findings limited to one health science program format within a Canadian context, and may not generalize
• Other potential screening tools not available for consideration in this study
• Course grades are subject to unsystematic scoring errors
• Ratings of background experience and LOI subject to rater error
References• Auriemma, D. (2002). Admission methods of professional occupational therapy
programs in the united states. Education Special Interest Section Quarterly, 12(3), 1-4.
• Bridle, M. J. (1987). Student selection: A comparison of three methods... queen's university occupational therapy program. Canadian Journal of Occupational Therapy, 54(3), 113-117.
• Caplan, R.M, Kreiter, C., & Albanese, M. (1996). Preclinical science course ‘preludes’ taken by premedical students: do they provide a competitive advantage? AMJ, 71 920-922.
• Howard, L., & JeroschHerold, C. (2000). Can entry qualifications be used to predict fieldwork and academic outcomes in occupational therapy and physiotherapy students? British Journal of Occupational Therapy, 63(7), 329-334.
• Katz, G.M., & Mosey, A.C. (1980). Fieldwork performance, academic grades, and pre-selection criteria of occupational therapy students. American Journal of Occupational Therapy, 34(12), 794 – 800.
• Kirchner, G. L., & Holm, M. B. (1997). Prediction of academic and clinical performance of occupational therapy students in an entry-level master's program. American Journal of Occupational Therapy, 51(9), 775-779.
• Kirchner, G. L., Stone, R. G., & Holm, M. B. (2000). Use of admission criteria to predict performance of students in an entry-level master's program on fieldwork placements and in academic courses. Occupational Therapy in Health Care, 13(1), 1-10.
• Lewis, M., Smith, S. (2002). Selection of pre-registration physiotherapy students: Changing to a more objective process. Physiotherapy, 88(11), 688 – 698.
• Nayer, M. (1992). Admission criteria for entrance to physiotherapy schools: How to choose among many applicants. Physiotherapy Canada, 44, 41 – 46.
• Salvatori, P. (2001). Reliability and validity of admissions tools used to select students for the health professions.see comment. Advances in Health Sciences Education, 6(2), 159-175.
• Tan, K. P., Meredith, P., & McKenna, K. (2004). Predictors of occupational therapy students clinical performance: An exploratory study. Australian Occupational Therapy Journal, 51(1), 25-33.