Developing and Assessing Outcomes in the Co-Curricular Experience Prepared for Auburn University’s Division of Student Affairs A-Team April 2, 2013 Sandi.

Developing and Assessing Outcomes in the Co-Curricular Experience

Prepared for Auburn University’s Division of Student Affairs A-Team

April 2, 2013

Sandi Osters, Ph.D. Retired Director, Student Life Studies

Texas A&M University3001 Aztec Street

College Station, TX [email protected]

Workshop Overview

Student integrative learning Outcomes-based assessment Learning outcomes – the heart of

the matter Working sessions Assessment methods Working sessions Reflective dialogue

Today’s Program Outcomes

The presenter will: Work on issues important to participants Provide a general understanding about the

importance of outcomes based assessment in student affairs

Emphasize the importance of alignment with division and university mission, goals and planning

Emphasize the use of data to inform practice

Today’s Learning Outcomes

Participants will: Describe learning and program

outcomes for their own program, office or activity

Employ assessment methods for their outcomes

Develop initial action steps Articulate what they have learned and

what they still need to know

Handouts

PowerPoint Writing Outcomes Assessment Plan Template Rubrics Everything You Ever Wanted to

Know about Assessment – on One Page

It’s All About Students’ Integrative Learning A strong emphasis on student learning is

the primary key to retention through graduation

Learning is the primary activity and goal of the college environment – both inside and outside of the classroom

The integration of academic programs and co-curricular activities is important

The ability of students to integrate their learning both inside and outside of the classroom – be responsible for their own learning – is of the greatest importance

What Is Student Learning in the Co-curricular?

“Learning Reconsidered defines learning as a comprehensive, holistic, transformative activity that integrates academic learning and student development, processes that have often been considered separate, and even independent of each other.”“Student learning produces both educational and developmental outcomes; distinguishing them is pointless and potentially harmful, and the goal of higher education should be the integration of all domains of learning and of the work of all educators.”

Keeling, R. P. (Ed.) (2004). Learning Reconsidered. Washington, D.C.: NASPA and ACPA.

What is Assessment and Evaluation?

“Assessment is any effort to gather, analyze, and interpret evidence which describes institutional, departmental, divisional or agency effectiveness.”

“Evaluation is any effort to use assessment evidence to improve institutional, departmental, divisional, or agency effectiveness.”

Upcraft, M. & Schuh, J. (1996). Assessment in student affairs: A guide for practitioners. San Francisco: Jossey-Bass.

The Purpose of Outcomes Based Assessment To answer some simple questions:

What is it that we do? Why do we do it? How well do we do it? How do we know that we do it well? What improvements and changes do we make from

what we learn? Do the improvements work?

To genuinely engage faculty, staff and students in the day-to-day reflection of answering these simple questions

To demonstrate a commitment to systematic examination of the quality of all we do to improve our programs and departments

The Purpose of Outcomes Based Assessment, continued To celebrate our successes and learn from our

failures To inform our policy, programs, processes and

resource allocations Increase our confidence that we are putting our time

and energy into activities that results in the outcomes we and our students value

To have the data to support why our programs generate the learning Auburn wants for its students in times of financial downsizing

To be able to articulate exactly how student affairs professionals contribute to student learning and student development

To gather and display data that will allow us to satisfy accrediting agencies and program reviews (CAS or your own)

Advantages of Using Outcomes Language in Student Affairs

Make it clear to students who participate in our programs, activities and services what they can expect to gain from doing so

Make it clear to others what the program will accomplish and, where appropriate, what students will learn

Help staff select appropriate strategies and assessment measures/methods to reach the outcomes

Advantages of Using Outcomes Language in Student Affairs

Apply the results of our assessments for meaningful improvement and/or change

Move beyond student satisfaction and tracking the use of services as the sole means of describing student affairs effectiveness

Connect Student Affairs to the academic enterprise – we all are about student learning – we all are educators

What Any Assessment is NOT Assessment does not exist for

assessment’s sake It informs but does not drive where you

want to go and what you want to be as an office, activity, program

It is not used for personnel evaluation It is not a one time thing – but

systematic and iterative It does not assess all things, all the time

Outcomes Based Assessment and Decision Making

Most importantly, Outcomes based decision making and assessment should be

Understood = by faculty/professionals and students Meaningful = faculty/professional (i.e., expert) driven Inclusive = involve as many faculty, student affairs

professionals and students as possible Manageable = takes into account varying resources Flexible = takes into account assessment learning curves Truth-seeking/objective/ethical Iterative and systematic

Inform decisions for continuous improvement or provides evidence of proof

Promote a culture of accountability, of learning, and of improvement

Bresciani, J. J., Zelna, C. L., & Anderson, J. A. (2004). Techniques for assessing student learning and development: A handbook for practitioners. Washington, D.C.: NASPA, Inc.

Mission/Purpose

Goals

Objectives/Outcomes

Implement Methods to Deliver

Outcomes and Methods to Gather

Data

Gather Data

Interpret Evidence

Make decisions to improve programs; enhance student learning and development;

inform institutional decision-making, planning,

budgeting, policy, public accountability

Politics

Scho

lars

hi

p Accountability

Environment

Assessment Cycle

Budget

Writing Outcomes Outcomes – Detailed, specific, measurable or

identifiable, and personally meaningful statements that are derived from goals and articulate what the end result of a unit, program, course, activity, or process is. Learning Outcomes – An easily identified

action that a student is expected to demonstrate in terms of knowledge, skills, and attitudes upon completion of a program/activity/event.

Program/process outcomes – The end result of what a program or process is to do, achieve, or accomplish.

Write simply and with one outcome per statement

Bloom’s Taxonomy (1956)

A structure for developing learning outcomes

Learning Outcomes? Students will be satisfied with the

information received. Students will rate the service received

positively. Students will create their own personal

leadership philosophy based on three models taught in Lead 101 in the fall semester

After the service project, students will be able to articulate the root cause of homelessness in Auburn, Alabama.

Disability Services Example Learning outcome

Students will be able to describe their functional limitations in an academic setting

Assessment method Pre-test/post-test (intake form) Rubric

Criteria for success/achievement target Students post-scores will improve over pre-

test

Disability Services Example Learning outcome

Students will be able to describe how their disability may affect them in the workplace and what accommodations they may need to ask for from their employers

Assessment method Exit interviews with graduating students

Criteria for success/achievement target 80% of students in their graduating

semester will be able to identify at least one way their disability might effect them in the workplace.

Program/Process Outcomes? Students will like the conference banquet The professional development sub-

committee will offer two dining etiquette programs in the fall semester for graduating students

Develop and expand the Student Health Services web page in order to increase student access to health information, patient services and educational programming

Examples of Program Outcomes

The Department of Residence Life will be able to respond to maintenance requests within 24 hours of their walk-in time

The Student Health Service will be able to admit students for check-ups within 24 hours of their walk-in time.

The Financial Aid Office will have full award letters out to the on-time complete applicants by April 15th.

Program Outcome Example Program outcome

Develop and expand the Student Health Services webpage in order to increase student access to health information, patient services, educational programming

Assessment methods Data retrieval from webpage interactions Annual patient survey to determine student

utilization of the web page Survey instruments by Health Education program to

determine student utilization of web page Criteria for success/achievement target

Increase in the utilization of the SHS webpage regarding health information, patient services and educational programming.

Common Mistakes in Writing Learning Outcomes The learning outcomes don’t align with department,

division or university goals Outcomes include words that are hard or impossible to

measure (understand, appreciate, know about, become familiar with, learn about, become aware of)

Outcomes include too many skills in one statement Outcomes measure satisfaction or performance

evaluation rather than learning of the student There are too many learning outcomes Only one person wrote, reviewed, edited and

implemented the outcome

Working Session Work in small groups of two or three Develop at least two learning outcomes and

two program outcomes for the Division of Student Affairs Global Service Initiative (mission provided)

Don’t worry about being an expert on this initiative. Pretend you are. What’s important is working on the outcomes

Share time

Assessment Methods

Qualitative and quantitative One time and longitudinal Direct - requires students to display

their knowledge, behavior or thought processes

Indirect – asks students to reflect upon their knowledge, behaviors or though processes

Local (Auburn developed) and national assessments (like the NSSE or CIRP)

Direct Measures Benefits of direct measures to Student Affairs

Provides faculty and administrators more confidence in the measures and their results because it is the academic standard

Helps with accreditors who are looking for student learning in the co-curricular and measures of same

Challenges Time and expertise needed to develop Assessing the “fuzzies” – attitudes and values Small “n’s” and the validity of the studies

Bottom line Use multiple indirect measures to offset the lack of

direct measures

Common (and still viable) Assessment Measures Surveys to inform of needs and satisfaction

Surveys to assess outcomes Focus groups/interviews to inform future assessments or to dig more deeply into existing survey results

Tracking use and participation (numbers)

New Wave of Assessment Measures

Exit Interviews Learning contracts Photography Pre and post tests of knowledge Observations with documentation One minute papers Rubrics Reflective journals Reflective conversations/writing Combination of the above

Working Session Working individually, develop at least two

learning outcomes for a program or activity of interest to you. If you do not work directly with student learning, consider learning outcomes for your student workers

Develop at least one assessment measure for each outcome and address how you will know if you have been successful (criteria for success/achievement target)

Work in small groups and share your outcomes and measures.

As a group, develop an action plan (see Assessment Plan Template, Implementation of Assessment Process) for one of the learning outcomes in your group

Share

Sharing Your Assessment Data Have clear objectives/purpose Know who cares about your results Effectively analyze the data Effectively report/present results

Determine the best format – written reports, websites, newsletters, publications, oral presentations

Use the data yourself – role model to others Share with each other – assessment road

shows/poster sessions at division meetings/monthly during the A-Team meetings

Share with students!! What actions have you taken from what you have learned from them?

In Summation

Be clear about who you are, why you do what you do, how well you do what you do, and how you improve and change

Check in annually to be sure your program/activity and event outcomes align with your departmental mission/goals and that your departmental mission aligns with the division’s mission/goals and strategic plan and that your division mission/goals align with institutional mission/goals/strategic plan/QEP

Be patient; you will get better at writing outcomes; you always will have more to learn!

Practice, practice, practice and learn by teaching others Work with your students and student groups to

articulate outcomes for their programs Work with colleagues, faculty and others with expertise Celebrate your accomplishments

Contact Information

Sandi Osters, Ph.D. Retired Director, Student Life Studies

Texas A&M University3001 Aztec StreetCollege Station, TX 77845

[email protected]