Rethinking Letters of Recommendation: Why? What? … · Rethinking Letters of Recommendation: Why? What? Who? How Julie Masterson Kenn Apel Karen Jacobs ... Special appearance by

Rethinking Letters of Recommendation: Why? What?

Who? How

Julie Masterson Kenn Apel Karen Jacobs

Associate Provost/Dean of the Graduate College Professor and Chair, CSD Vice President, Client Success

Missouri State University University of South Carolina Liaison International

Special appearance by Robert Augustine, Senior Vice PresidentCouncil of Graduate Schools

It must be December…….

Audience Participation

• What value do you put on Letters of Recommendation?

• Critical for making admission decisions

• Somewhat important for making admission decisions

• Minimally important for making admission decisions

• Not worth using

• Maybe they matter, but I hate writing them so much that I don’t care

“If letters were a new psychological test, they would not come close to meeting minimum profession criteria (i.e., Standards) for use in

decision-making”.

Brihl & Wasieleseski, 2004

Strong FACE Validity

• LOR shown to be most heavily weighted non-objective (second order) criteria. Of 248 masters programs, 62.5% report that LOR are very important criteria in admission decisions, 28.5% report “somewhat important”, 6.3% report minimal importance and 1.2% do not use LOR (Briihl & Wasieleski, 2004)

• Individuals who have had direct, relevant contact with the applicant should be in best position to provide a holistic evaluation.

Concurrent Validity

• Correlations with standardized test scores: verbal ability (.14) & quantitative ability (.08) (Kuncel et al., 2014)

• Correlations with prior grades (.26), personal statements (.41) and interviews (.18) (Kuncel et al., 2014)

• Addition of LOR increased R2 .01 for faculty ratings of student performance in medical school (R2=.128 to R2=.139) (Kuncel et al., 2014)

Predictive Validity• Kuncel et al. (2014): Addition of LOR increased R2

• .003 for Grad GPA (R2=.112 to R2=.115)

• .24 degree attainment, increase was much larger (R2=.031 to R2=.055)

• Letters of recommendation did not add incremental validity to the combo of GRE scores and undergrad GPA (Aamodt, 2012)

• Halberstam & Redstone (2005): Correlated with graduate GPA (r=0.51, p<.05).

• Most letters rated as being excellent, very good or good. Therefore, with little variance across ratings, range restriction may be problematic.

• Hall, O’Conner and Cook (2017): Numerically rated letters predicted graduate school productivity.

• Students with higher mean LOR ratings had 3+ first author publications in Graduate School.

• Again, range restriction is a potential issue with results.

Master’s Admission Attributes: Current Status and Missing Evidence: CGS/ETS

• The Data: 1.4 of 1.8 Million Enrolled- 75.2% of Graduate Candidates

• The Impact: Responsive & Innovative: Demand + Competencies + Metrics

• The Problem: Lack of Evidence: Admission – Matriculation – Completion – Careers

• The Starting Point: Admissions Survey: Program Directors & University Leaders

• The Method: Identify Current & Needed Admission Attributes + Evidence + Value

• The Outcome: Evidence Based Admissions Practices

• The Future: Admissions Evidence to Support Matriculation – Completion – Careers

Why (doesn’t the LOR have higher predictive validity)?• Degree attainment is difficult to predict

• Predictive power of LOR was better/equal to all other traditional predictors but standardized tests of field specific knowledge (i.e., GRE subject test).

• Few references are negative. For example, in two of the studies less than 7% of letters rated applicants as being average or below average

• Reference providers may also avoid giving low ratings or include negative ratings out of fear that the applicant will be angry or file a defamation lawsuit.

• Superficially positive in tone; applicants choose letter writers and are motivated to avoid selecting an individual who might give them a negative reference.

• Low inter-rater reliability between evaluators for the same applicant

• .22 (Baxter et al., 1981) and .40 (Kuncel et al., 2014)

• More agreement between recommendations written by the same person for two different applicants.

• Two reference providers may see applicants in very different situations

Why (continued)?

• Letters may say more about the person writing the letter than the subject of the recommendation

• Letter readers tend to favor letters written by people they know. CGS/ETS study indicates that these are not subtle biases, but rather somewhat blatant!

• Bias extends to the institution with LORs from faculty at Big Name University preferred over Directional School because of differences in perceived grade inflation, which is not supported by evidence.

• LORs are interpreted by admissions committees and combined judgments may decrease effectiveness.

Structured LORs from Other Disciplines• SLOEs – Standard Letter of Evaluation

• Provides a global perspective on an applicant’s candidacy for training by providing meaningful comparisons to peers applying for training in emergency medicine

• Provides comparative data to peers in addition to important information regarding the distinguishing non-cognitive characteristics (e.g. maturity, professionalism, leadership, compassion, initiative, enthusiasm) that an applicant possesses

• Standardized

• Concise

• Discerning/Discriminating

• Standardized Letter of Recommendation

• http://onlinelibrary.wiley.com/doi/10.1002/j.2333-8504.2007.tb02080.x/epdf

• Committee Letter

• Summary of letters written by a group of evaluators

• May also include an interview of the student

• PI – Personality Inventory

• Any of several tests that attempt to characterize the personality of an individual by objective scoring of replies to a large number of questions concerning his or her own behavior

• Minnesota Multiphasic Personality Inventory

What (might we do)?

• Content-valid, structured approaches seem to work better than generic reference checklist

• Rather than asking about broad traits, ask specific questions related to desired behaviors

Audience Participation: Part 2

• If CSD adapted a specific format that all recommenders must use/address, what types of information would you want to see included?

Suggestion: Structured LOR for CSD

References

• Aamodt, M. G. (2012, February). Are letters of recommendation worth the effort?

Recommendation letters do not predict performance. Psychology Today.

• Baxter, J. C., Brock, B., Hill, P. C., & Rozelle, R. M. (1981). Letters of recommendation:

A question of value. Journal of Applied Psychology, 66(3), 296

• Briihl, D. S., & Wasieleski, D. T. (2004). A survey of master's-level psychology

programs: Admissions criteria and program policies. Teaching of Psychology, 31(4), 252-

256.

• Halberstam, B., & Redstone, F. (2005). The predictive value of admissions materials on

objective and subjective measures of graduate school performance in speech‐language

pathology. Journal of Higher Education Policy and Management, 27(2), 261-272.

• Hall, J. D., O’Connell, A. B., & Cook, J. G. (2017). Predictors of Student Productivity in

Biomedical Graduate School Applications. Plos ONE, 12(1), 1-14.

• Kuncel, N. R., Kochevar, R. J., & Ones, D. S. (2014). A meta‐analysis of letters of

recommendation in college and graduate admissions: Reasons for hope. International

Journal of Selection and Assessment, 22(1), 101-107.