CORE COMPETENCIES (CCs) FOR LEADERS IN BIOMEDICAL RESEARCH Sample Rubric: Full rubric copies available, on paper. 2016 RATINGS 2015 RATINGS ASSESSMENT MUST BE BASED ON OUTCOMES: • As students are integrated into science, iterative and outcome- oriented feedback is imperative. ² Formative and summative assessments are needed ² Skill-based performance assessment differentiate student progression to degree ² Such assessments validate habits that are rooted in the behaviors of successful students and should be transparent to all students. FACULTY DEVELOPMENT NEEDS: • Faculty must be trained initially, and then developed to assess appropriately and consistently. • Involvement of faculty in developing institution-specific Core Competencies is critical for understanding and commitment. STUDY DESIGN: • This type of outcomes-oriented surveys in repeated manner with analysis benefits design (and revision) of effective programs. • Longitudinal data are required, with repeated measures due to the complexity of admissions, student variability, and inter- rater reliability. Students becoming professionals: transparent assessment using Core Competencies Julie Davis Turner, Patty Farrell - Cole, and Steven J. Triezenberg Van Andel Institute Graduate School, 333 Bostwick Ave., NE, Grand Rapids, Michigan 49503 Background & Rationale REFERENCES Appreciation to these resources as we developed the framework for Core Competencies: 1. Walker, G.E., C.M. Golde, L.Jones, A.C.Bueschel, P.Hutchings. (2008) The Formation of Scholars (San Francisco: Jossey-Bass). 2. Lovitts, B. (2007) Making the implicit explicit: creating performance expectations for the dissertation. (Sterling, VA: Stylus Press). 3. NRC. (2001) Knowing what students know. (cited 1/20/2017) http://www.nap.edu/openbook.php?isbn=0309072727 4. Council of Graduate Schools, Assessment & Review of Graduate Prog, 1990, 2005, 2011. http://cgsnet.org/assessment-and-review-graduate-programs Evidence of progression across cohorts • Domains of Research & Ethic/Professional Practice show the largest increase in development across cohorts. • Domains of Knowledge and Communication demonstrate less even levels of mastery, possibly due to individual cohort attributes. EDUCATIONAL PHILOSOPHY • Interactive, problem-based curriculum gives students responsibility for learning. • Proposal-driven problems measure content mastery, creativity, and scientific communication. • Like science, optimal learning requires collaboration among students and faculty. WHAT WE LEARNED MISSION The mission of VAIGS is to train PhD scientists to be leading scholars in cellular, molecular, and genetic biology of human disease. Claim 1: CC assessment identifies a developmental progression. Claim 2: CCs define mastery between thesis adviser and student. Claim 3: Evidence from annual CC assessing drives program changes. CONCLUSIONS • Core Competencies articulate a strong set of explicit expectations for student learning outcomes at VAIGS. • The rubric, as a developmental framework, aids thesis advisers in monitoring and understanding their students’ progress and facilitates student accountability with common expectations. • The rubric provides faculty with language to convey clear expectations and graduated ratings for objective rationale. VAIGS cohorts increase in mastery of four Core Competency Domains (Knowledge, Research, Communications, Ethics & Professional Practice), assessed by Thesis Adviser (2016 averages; cohort = 2-4 students, n=18 total) Exceptional Heightened Advancing Intermed. Beginning Exceptional Heightened Advancing Intermed. Beginning cohort Based on “expected outcomes” (set per cohort): • By 6 th Year, all students have achieved Heightened to Exceptional (as anticipated – shown in WHITE). • Isolated areas of Clinical Context (Knowledge), Scientific Ethics, or Authorship show repeated lags. Comprehensive Program Review recommendations (2016) employed these preliminary data to address lags: • Curricular modifications in written communication and grantsmanship • Revitalized course in “translational research” • Refresher course for scientific ethics Future faculty committee mandates will be informed by 2017-18 study data. Taken together, Core Competency outcomes provides a real-time “dashboard” for prioritizing program changes. KNOWLEDGE Beginning Intermediate Advancing Heightened Exceptional Describe key concepts in biomedical science Describe general concepts with gaps and errors Describe general concepts correctly and clearly Explain concepts with expanded historical and contextual detail Recognize and apply key concepts as they appear in new settings Evaluate new challenges or controversies relevant to concepts Place core concepts in the relevant clinical context Describe thesis research and disease association in general and limited terms Describe clinical context of molecular mechanisms as presented by others Connect clinical issues and molecular mechanisms accurately and broadly Recognize and apply key concepts as they appear in new clinical settings Use clinical information to elicit gaps in collective knowledge Know scientific literature relevant to the research area Understand biomedical textbook descriptions of key concepts Understand review article descriptions of emerging concepts Understand and evaluate a primary research article Compare contradictory publications in the dissertation research area Evaluate literature beyond the dissertation research area RESEARCH Cohort ratings determine curricular “gaps”. • Thesis Adviser ratings were averaged and compared to “expected outcomes” for each cohort • Those competencies in which ratings lag are shown: Value of comparing Student to Thesis Adviser ratings • Clarifies mentor expectations in iterative manner with consistent language • Identifies miscommunications in the mentor-student relationship • Provides tangible, consistent behaviors that all students can work toward coordinately Concordance equals: average [(faculty score x ) – (student score x )] across domain History of Core Competencies at VAIGS Domain / Core Competencies 2 nd yr 3 rd yr 4 th yr 5 th yr 6 th yr Knowledge (3 total competencies) Describe key concepts in biomedical science Know scientific literature relevant to research area Place core concepts in the relevant clinical context Research (7 total competencies) Apply creative / appropriate experimental controls Use controls appropriately Integrate results into relevant models Communications (3 total competencies) Speaks effectively Writes effectively Ethics & Professional Practice (6 competencies) Manage data with scientific integrity Engage in best authorship practices Address ethical problems in scientific research Comply with safety & regulatory standards Display appropriate lab citizenship Work collegially & effectively as a team/collaborator LEGEND: LAGS in 2015 LAGS in 2016 Lags in both 2015 and 2016 Method: deidentified data collection (grouped by cohort) • Using Survey Monkey © and unique identifer codes for students and mentors, VAIGS gathers and deidentifies ratings, maintaining mentor-student association. Higher mentor Higher student -1.2 -0.9 -0.6 -0.3 0.0 0.3 0.6 0.9 1.2 6th Year (n=2) 5th Year (n=4) 4th Year (n=3) 3rd Year (n=4) 2nd Year (n=4) Communications Research Ethics & Professional Practice Knowledge Concordance POSTER # 11 While early in the study, preliminary data show: • Lack of concordance (in 2015) necessitated reliance on mentor ratings. • Cohort-specific attributes may skew concordance. • Improved concordance occurred in 2016 relative to 2015. – This improvement may be due to enhanced rubric familiarity. Growth over time (limited display: only 2 longitudinal data sets) • For most of domains, mentor ratings increase from 2015 to 2016. • Ratings are complicated by issues with timing and consistency 5 th – 6 th year 2 nd – 3 rd year -1.2 -0.9 -0.6 -0.3 0.0 0.3 0.6 0.9 1.2 6th Year (n=3) 5th Year (n=3) 4th Year (n=4) 3rd Year (n=4) 2nd Year (n=4) Research (7 Competencies) Communica/ons (3 Competencies) Knowledge (3 Competencies) Ethics & Professional Prac/ce (6 Competencies) Concordance Higher mentor Higher student Communication ü Speak effectively ü Write effectively ü Communicate to diverse audiences Ethical & Professional Practice ü Manage data with scientific integrity ü Engage in best authorship practices ü Address ethical problems in scientific research ü Comply with safety and regulatory standards in lab ü Display appropriate lab citizenship ü Work collegially and effectively as a team or collaborator Summer Intermittent use by faculty/students February 2016 December 2016 Revision of CC rubric by faculty subcommittee 1 st Core Competency Assessment (100% compliance) 2 nd Core Competency Assessment (100% compliance) Knowledge ü Describe key concepts in biomedical science ü Place core concepts in the relevant clinical context ü Know scientific literature relevant to the research area Research skills ü Define sound rationale / identify gap in knowledge ü Frame appropriate hypotheses ü Apply creative and appropriate experimental design ü Use controls appropriately ü Execute experiments with technical skill ü Demonstrate critical analysis and thinking ü Integrate results into relevant models