Leveraging Partnerships to Enhance Assessment Practices Beth M. Lingren Clark, Ph.D. Director: Orientation & First-Year Programs University of Minnesota-Twin Cities
Dec 28, 2015
Leveraging Partnerships to Enhance Assessment
Practices
Beth M. Lingren Clark, Ph.D.
Director: Orientation & First-Year Programs
University of Minnesota-Twin Cities
Outcomes
• Evolution into assessment framework• Developing assessment plans • Involving key campus partners/stakeholders• Examples of using the data to increase the
effectiveness of measures, themes and impact institutional projects
Old practices in assessment• Orientation & First-Year Programs (OFYP) discovered
assessment methods only measured the level of satisfaction with our program/specific components.
• OFYP asked questions unveiled that participants were satisfied to very satisfied.
• Satisfaction not outcome orientated or as helpful when trying to enhance or improve your program.
• OFYP aligned orientation program questionnaires. We worked with the Office of Measurement Services to help us identify common questions across all questionnaires.
Old practices in assessment
• Shared data with presenters to enhance content
• Shared with collegiate representatives to enhance college components
• Influenced orientation program content and communications
WITH THIS DATA…
Needed more information to identify content, make enhancements/improvements, inform partners to all populations served
TAKING A BROADER APPROACH
Think about what motivates a study(topic) and what shapes a study (question or problem).
“Research is a structured process of inquiry that draws upon existing knowledge and theory to advance general understanding or impact practices in the field at large.”
-Henscheid, J.M.& Keup, J.R. (2011, p.ix)
Research: Topics/Questions
• How does what we implement impact retention?
• Is parent involvement important?• Why do students leave?• How are we impacting personal and
academic success?• What key factors predict transfer student
success?
Process: Topics/Questions
• Who completes online Orientation?• Who attends on campus Orientation?• Who lives in Living Learning Communities• Who has completed the Strengths
assessment?• What attended Transfer Welcome Day?• Which International students are coming?
Content: Topics/Questions
• What are the gaps between expectations and reality of experiences?
• How can we reduce anxieties/concerns?• How can we increase parent involvement?• How do we create a sense of belonging?
EVOLUTION TO ASSESSMENT FRAMEWORK
Where we started…
Purpose of Committee: (Continued) – Identify existing instruments to collaborate data collection– Discuss implementation of the program evaluation– Discuss attendance tracking at events
Representatives from the following areas: Provost’s office, Orientation & First-Year Programs, Office of Measurement Services, Institutional Research, College of Liberal Arts, Student Activities, Center for Teaching and Learning, Department of Educational Policy & Administration, Career Center, Student Leadership & Engagement
Welcome Week Committee Structure
The process• Followed cohorts from old program – New Student Weekend (2001 and
2002 cohorts)
• Welcome Week planning occurred simultaneously with the development of the First Year Assessment Plan (baseline cohort 2007)
• Did not want to implement WW in isolation• Support the efficacy of WW
• Created a mapping processing and aligned with what mattered at the institution
• Identified what else existed system-wide to support first year content and themes
Assessment Cycle • Identify learning objectives/goals for program• Map learning objectives/program goals to Student Learning
and Development Outcomes• Conduct gap analysis• Change/enhance content based upon gap analysis and
program survey results from previous year (Common questions)
• Implement program introducing objectives and outcomes to population
• Evaluate program • Analyze and share results• Repeat
How Do We Create a Common Language of Assessment?
• Goal: Initiating undergraduates into Transformative Education– Student learning & development outcomes
• Two primary transformations…– Transforming the campus into the curriculum– Transforming the undergraduate experience
• Assessing first-year programs– Have goals we care about– Know what you know– Map to goals
Understanding the assessment framework
How Do We Create a Common Language of Assessment?
• Mapping process is a conversation– Meant to be iterative; refined over time– Each step informs the next
• What we needed to begin mapping– Schedule and description of
goals/sessions/events– List of student learning and
development outcomes– Mapping spreadsheet (essentially a
matrix of sessions/outcomes)
Mapping Outcomes
How Do We Create a Common Language of Assessment?
• Was done on a large scale (for all of Welcome Week’s content)• Mapping also done on a smaller scale
– College Day - College of Liberal Arts
Mapping Outcomes
The Power of the Common Question
• We wanted to:– Create a common language beyond
orientation programming– Create an intentional plan for content and
evaluative measures– Implement transformational practices on
campus– “We wanted to speak a common language”
The Power of the Common Question (continued)
• We wanted to:– Create a profile of people completing our
program questionnaires– Create longitudinal data from one year to
the next as well as within the cohort/year– Ask more specific questions, AND ask
participants to elaborate on items where they may not have been satisfied
– Understand the utility of common mapping
DEVELOPMENT OF A FORMAL FIRST-YEAR ASSESSMENT PLAN
First-Year Assessment Plan
Intent of FYA Plan• The intent of this process provides data on
student behavior related to:• Matriculation process• Student participation• Student satisfaction• Student success both personally and academically• Longitudinal data utilized• Measure the impact on student retention and other
programs
• The plan helps to inform institution on the things we were learning about from our students and their experiences
Involving Campus Partners
• Office of Measurement Services• Undergraduate Education• Collegiate Units• Academic Support Resources• Admissions, Housing, Athletics, Honors,
International • Office of Institutional Research• Strengths• Office for Student Affairs
EXAMPLES OF USING THE DATA
Matriculation
• Tracking…– Online Orientation completion– Orientation attendance– Document check for international– Parent attendance– Welcome Week/Transfer Welcome Day
attendance– Strengths completion
ParticipationClass of 2016 5514 (10th day enrolled)
Commuter 754 13.7%
Residential 4760 86.3%
MN 3493 63.3%
Out of State 1714 31.1%
International 285 5.2%
UHP 572 10.4%
Athletes 139 18.4%
SOC 1077 19.5%
Band 74 1.3%
First-generation 1382 25.7%
NHS Attended OR 5572 103.8% 5501 (10th Day Enrolled)
NHS Parents Attended OR 4791
June OR 3626 67.5%
July OR 1276 23.8%
Attended OR 8/22 129 2.34% (3 non 10th day, 53 international)
Attended OR 8/23 189 3.43% (4 non 10th day, 76 international)
Attended OR 8/27 307 5.57% (139 international)
0 Attended Parent OR 2267 41.1%
1 Attended Parent OR 1810 32.8%
2 Attended Parent OR 1370 24.8%
3 Attended Parent OR 66 1.2%
4 Attended Parent OR 1 0.1%
Survey Data• Pre-Orientation Survey
– Expectations/Welcome messages– Orientation content (student and parent)
• Common Questions– Cross college– Theme/Program content
Survey Data• Program Evaluations
– Improvement– Outcomes based
• Check-In Surveys– Reality of expectations– Additional support and resources– Newsletter content
Institutional Data• National Survey on Student Engagement
– Comparing freshmen and transfer student experiences
• Student Experience in a Research University– Welcome Week and Sense of Belonging– Transfer Research Grant results
Institutional Data• Leaver’s Analysis
– Longitudinal data/Collegiate specific– Importance of Parent/Family Engagement– @ Home in MN
• Student Retention• Transfer Grant
– Influence overall experience and engagement
Additional Themes…
• Interactions with faculty are an important part of the college experience.
• The majority of the students spend between 11-20 hours attending class per week.
• On average, students spend up to 10 hours per week on social networking (facebook).
• Transfer students have been successful adjusting to the academic demands and developing effective study skills but have had more of a difficult time getting to know their peers and their faculty
Additional Themes…• Student’s top major anxiety/concerns:
– academic coursework– making friends – financial concerns– deciding on a major – developing skills related to note-taking/studying – time management – stress management– transfer of credits**
Additional Themes…
• The University can feel like a big place so it is important for students to create community.– Students can do this in their residence hall– Through the Commuter Connection program– In class– At work– Joining a student organization. – Especially important for students who are not from the
Twin Cities area.
Additional Themes…• New students leave the U for various reasons:
– Financial concerns– Academic– Medical– Family– Other personal issues
Additional Themes…
• At the six to eight week mark, students are:– seeking more contact with advisers– want help with study skills– more opportunities to socialize**– information on campus libraries**– more interactions with faculty**
Sharing the Data
– College/unit level data is reported to committees and administrators
– Mapping Process revisited to ensure meeting goals and desired programmatic outcomes and University Student Development and Learning Outcomes
Sharing the Data
– Program evaluation data will be distributed to content creators and future planning committees
– Data presented to the students (e.g., Advice from the Class of 2016/Transfer?)
– Annual reports disseminated to leadership and decision-makers within Undergraduate Education
Where Do We Go From Here?
• Utilize the Data– Use the findings for continuous improvement efforts
• Did we do what we set out to do?• What can we do better?• What was successful and should be continued?• How do we alter content to address new issues?
– Compare findings year to year• Are students attitudes/perceptions changing?• Are there new transformational issues?
– Longitudinal Analysis• Has Welcome Week impacted the student experience?• Are there correlations between student attendance, satisfaction
retention, graduation rates, etc.? (Utilize orientation database)
Inspiring Our Work
• Influence program content, delivery, and messaging
• Integrate data results directly into what we share with students and families
• Share with colleges and units to influence the way we do our work
• Strengthens partnerships• Provide evidence to work with units to make
strategic changes
“Educators who work to improve the experience of college students, especially students facing transitional periods, are typically driven by passion for the work. At their best, these educators combine this passion with a solid understanding of how students experience college, what helps or hinders the students’ progress, and what outcomes result from their experience.
-Henscheid, J.M.& Keup, J.R. (2011, p.17)
Resourceshttp://www.ofyp.umn.edu/en/welcome-week.html
http://www.ofyp.umn.edu/en/survey-results.html
http://www.ofyp.umn.edu/ofypmedia/pdfs/highered/reports/2011_OFYPAnnual.pdf
http://www.oir.umn.edu/
http://www.oir.umn.edu/surveys/seru
Reference
Henscheid, J.M. & Keup, J.R. (2011). Crafting and conducting research on student transitions. Columbia, SC: University of South Carolina, National Resource Center for the First-Year
Experience and Students in Transition.
“When done well, quantitative research has the potential to enhance knowledge and inform programs, policies and pedagogies that educators use to serve students and contribute to their success”
(Henscheid & Keup, 2011, p.84)
Questions/Comments
Beth Lingren Clark, Ph.D.Director: Orientation & First-Year Programs