Conducting Assessment of Academic Advising 2013 NACADA Summer Institute Jacksonville, Florida Dr. Karen Boston Dr. Rich Robbins NACADA Executive Office.

Conducting Assessment of

Academic Advising

2013 NACADA Summer InstituteJacksonville, Florida

Dr. Karen BostonDr. Rich Robbins

NACADA Executive OfficeKansas State University

2323 Anderson Ave, Suite 225Manhattan, KS 66502-2912

Phone: (785) 532-5717 Fax: (785) 532-7732

e-mail: [email protected]

© 2013 National Academic Advising Association

The contents of all material in this presentation are copyrighted by the National Academic Advising Association, unless otherwise indicated. Copyright is not claimed as to any part of an original work prepared by a U.S. or state government officer or employee as part of that person's official duties. All rights are reserved by NACADA, and content may not be reproduced, downloaded, disseminated, published, or transferred in any form or by any means, except with the prior written permission of NACADA, or as indicated below. Members of NACADA may download pages or other content for their own use, consistent with the mission and purpose of NACADA. However, no part of such content may be otherwise or subsequently be reproduced, downloaded, disseminated, published, or transferred, in any form or by any means, except with the prior written permission of, and with express attribution to NACADA. Copyright infringement is a violation of federal law and is subject to criminal and civil penalties. NACADA and National Academic Advising Association are service marks of the National Academic Advising Association.

The Global Community for Academic Advising

mailto:[email protected]

• Review of reasons for performing assessment• Review of differences between evaluation and

assessment• The assessment cycle• Identifying stakeholders• Advising is teaching• Identifying student learning outcomes from various

sources• Outcome measurement• Acting upon the results• Assessment as research

Overview


• Program effectiveness

• Program improvement

• Program accountability• Enhancing student

• success• persistence• retention

• Activities aimed at student success, program improvement and accountability are all important

Assessment Has Multiple Purposes


Assessment is

• An on-going cycle of activity

• A gathering of a variety of information and data

• A team effort with faculty, staff, students, and administrators all actively engaged

• A complex process of comparison


Remember…

“…a lack of assessment data can sometimes lead to policies and practices based on intuition, prejudice, preconceived notions, or personal proclivities – none of them desirable bases for making decisions”

Upcraft and Schuh (2002. p. 20)


A Distinction with a Difference

Assessment Evaluation

Focus Programmatic Individual

Process Continuous & Embedded

Episodic

Measures Outcomes Effectiveness

Part of assessment, part of measuring outcomes


The Assessment Cycle (Maki, 2002, 2004)

Gather

Evidence

Interpret

Evidence

Identify

Outcomes

Implement Change

Mission/PurposesEducationalObjectives


The Assessment Flowchart

Student Learning OutcomesCognitive, Psychomotor, AffectiveProcess/Delivery Outcomes

Mapping the ExperienceWhat experiences?When or by when?

Gathering EvidenceWhen gathered?Where gathered?

How often gathered?From whom gathered?

How gathered?Minimum performance criteria for success?

ValuesVision

MissionGoals

Programmatic Outcomes

Sharing and Acting Upon the ResultsInterpret how results inform practice

How and with whom to share interpretationFollow up on implemented changes

Start the process all over again!

(adapted from Darling, 2005, 2010)


The Assessment Matrix/TableInstitutional Mission Statement

Local Mission Statement

Specific Goal or Objective

Specific Process/Delivery Outcome or Student Learning Outcome

Where Outcome Occurs

By When You Want Outcome To Occur

Outcome Measure

Data Instruments

Minimum Performance Criteria for Success (Threshold)

Action(s) Based on Outcome Data

Mapping of Outcomes

Adapted from Robbins, R. L. (2009). Evaluation and assessment of career advising. In K. Hughey, D. N. Burton Nelson, J. Damminger, and B. McCalla-Wriggins (Eds.) The Handbook of Career Advising (chapter 12). San Francisco: Jossey-Bass.

Robbins, R. (2011). Assessment and accountability of academic advising. In J. Joslin & N. Markee (eds.), NACADA Monograph #22: Academic Advising Administration: Essential Knowledge and Skills for the 21st Century (chapter 4). Manhattan, Kansas: NACADA.

Robbins, R. & Zarges, K.M. (2011). Assessment of Academic Advising: A Summary of the Process. NACADA Clearinghouse of Academic Advising Resources: http://www.nacada.ksu.edu/Clearinghouse/AdvisingIssues/assessment-Robbins-Zarges.htm


Exercise 1:Identify Your Stakeholders


Key Terms• Student Learning Outcomes – Articulate what students are

expected to know, do, and appreciate as a result of involvement in the academic advising experience

• Process/Delivery Outcomes – Articulate the expectations for how academic advising is delivered and what information should be delivered through the academic advising experience

• Mapping – The process of determining when, where and how the outcomes for advising will be accomplished over the students’ academic career and beyond


Process/Delivery OutcomesProcess/Delivery Outcomes

• Do not assess student learning, but evaluate what processes occurred during the advising interaction

• Typically what is evaluated via student satisfaction surveys• e.g., “the advisor provided accurate information”• e.g., “the setting for the advising session was

appropriate”• etc.


Focus on Student Learning:Advising is Teaching


Advisor as TeacherAdvisor as Teacher

• facilitator of communication

• coordinator of learning experiences

• referral agent who connects students with all of the institution’s resources and co-curricular opportunities that can help them be successful


Exercise 2:What skills used in teaching are also used in advising?


Teaching

Knowledge of subject matter

Preparation - planning and organization of course material

Advising

Knowledge of institutional policies, procedures, programs, resources

Preparing for advising meetings


Teaching

Engaging students in participation of their learning

Giving students feedback on their progress

Helping students learn to analyze and problem solve

Advising

Guiding students to be self-directed and autonomous

Working with student to regularly evaluate the student’s goals and progress

Assisting students in decision-making skills


Teaching

Clarity of presentation of subject matter

Establishment of dialogue with students in the classroom

Demonstration of excellent listening skills

Advising

Sharing information in a clear manner

Leading students to question and interact with the advisor

Listening to what advisees are saying verbally and non-verbally


Teaching

Assigning out-of class work

Working from a developmental perspective

Providing a learner-centered environment

Advising

Assigning out-of-session tasks

Working from a developmental perspective

Providing a learner-centered environment


Teaching

Identification and communication of student learning outcomes for the curriculum or course

Use of a course syllabus

Advising

Identification and communication of student learning outcomes for the advising process/interaction

Use of an advising syllabus


Focus on the Learner/AdviseeFocus on the Learner/Advisee

As a result of academic advising, what do we want students to demonstrate that they…

• Know (cognitive student learning outcome)

• Are able to do (behavioral student learning outcome)

• Value and appreciate (affective student learning outcome)


It’s All AboutStudent Learning


Student Learning Outcomes

What do you expect students to be able to

• Know (cognitive learning)• Do (behavioral learning)• Value and Appreciate (affective learning)

as a result of participating in academic advising?


Cognitive Elements

What do we want students to KNOW as a result of participating in academic advising?

Know general education requirements

Know about academic support services

Know how to use the student information system to register

Know how to use the catalog

Etc…….


Exercise 3:Identify Three Things You Want Students to Know as a Result of

Academic Advising


Tips on Developing SLOsTips on Developing SLOs

• avoid compound SLOs, e.g., “students understand and value the purpose of a liberal arts education,” “students understand and utilize their degree progress reports”

• as you are developing each SLO, think about where and when the opportunities for student learning may occur for that desired SLO

• as you are developing each SLO, think about how you might measure whether or not the SLO has been achieved


Skills/Behavioral/Psychomotor Elements

What do we want students toDo as a result of participating in academic advising?

Generate their degree audit

Make advising appointments

Keep advising appointments

Ask for help

Access course descriptions and degree requirements using the online catalog

Etc….


Exercise 4:Identify Three Things You Want Students to Do as a

Result of Academic Advising


Affective Elements

What do we want students to Value or Appreciate

as a result of participating in academic advising?

Value/Appreciate general education

Value/Appreciate the advising relationship

Value/Appreciate the process of learning

Etc….


Exercise 5:Identify Three Things You

Want Students to Value or Appreciate as a Result of

Academic Advising


Developing SLOs from Programmatic Goals


What do students need to demonstrate they know, can do, or appreciate in order to say that your programmatic goals have been achieved?


Exercise 6:

• Identify the services your advising program offers

• Identify 1 student learning outcome for each service

• Prioritize• If you could assess only two to start, which

ones would they be?


Deriving SLOs from other sources


Clear and measurable student learning outcomes for academic advising may also be derived from

• the NACADA Concept of Academic Advising

• the NACADA Core Values

• the CAS Standards for Advising


NACADA Concept of AdvisingNACADA Concept of Advising

http://www.nacada.ksu.edu/Clearinghouse/AdvisingIssues/Concept-advising-introduction.htm

• multi-dimensional and intentional

• grounded in teaching and learning

• includes specific purpose and content

• includes specific student learning outcomes for academic advising




NACADA Statement of Core Values

• Six values that NACADA adopted in 2004 as being important regarding what academic advisors are expected to do as part of the advising interaction

• May be easily translated into Student Learning Outcomes…

http://www.nacada.ksu.edu/Clearinghouse/AdvisingIssues/Core-Values.htm


http://www.nacada.ksu.edu/Clearinghouse/AdvisingIssues/Core-Values.htm

NACADA Core Values: Advisors NACADA Core Values: Advisors Teach Students…Teach Students…

• to value the learning process

• to apply decision-making strategies

• to put the college experience into perspective

• to set priorities and evaluate events

• to develop thinking and learning skills

• to make choices


NACADA Core Values as Student NACADA Core Values as Student Learning OutcomesLearning Outcomes

As a result of academic advising, students will

• value the learning process• apply decision-making strategies• put the college experience into perspective• set priorities and evaluate events• develop thinking and learning skills• make choices


Using the CAS Standards in Program Assessment

Council for the Advancement of Standards in Higher Education (CAS) Standards and Guidelines for Academic Advising

http:// http://www.nacada.ksu.edu/Resources/Clearinghouse/View-Articles/CAS-Advising-Standards.aspx


http://www.nacada.ksu.edu/Clearinghouse/AdvisingIssues/CAS.htm

http://www.nacada.ksu.edu/Resources/Clearinghouse/View-Articles/CAS-Advising-Standards.aspx

http://www.nacada.ksu.edu/Resources/Clearinghouse/View-Articles/CAS-Advising-Standards.aspx

CAS Standards for Academic AdvisingCAS Standards for Academic Advising

• 16 “relevant and desirable” student learning outcomes for academic advising (revised in 2006)

• Developed based on every type of higher education academic setting

• Each outcome will be achieved based on exposure to student and student maturity

• May require a systematic approach re: introducing these concepts and expectations in order for students to achieve these student learning outcomes


Advising Programs…Advising Programs…

…promote learning and development in students by encouraging experiences which lead to:

• intellectual growth• the ability to communicate effectively• appropriate career choices• leadership development• the ability to work independently and collaboratively• and so on for all 16 student learning outcomes…


It is not necessary to attempt to use all of the CAS Standards as your outcome statements…

…for those you may choose to use, be sure to develop related student learning outcomes as well as process/delivery outcomes


Developmental SLOs

• The offering of opportunities for students to learn the desired SLOs will vary by SLO

• The time by which your students need to demonstrate achievement of a specific SLO will vary by SLO

• Not all SLOs are achieved at same time


Thus, you may also take a developmental approach, and utilize different SLOs as desired outcomes for students at different times during their college careers


Mapping of Outcomes

The process of determining when, where and how the SLOs for academic advising will be accomplished over the students’ academic career and beyond


Mapping the Learning Experience

• What should be learned: e.g., student knows the components of the institution’s General Education requirements

• Where it should be learned: e.g., orientation workshops, advising sessions, personal reading of catalog or curriculum guide

• When it should be learned: e.g., prior to first year (orientation); by end of first year (via advising sessions); by end of first year (via personal reading)


Mapping of Outcomes

The Assessment Matrix/Table

Institutional Mission Statement



Specific Process/Delivery Outcome or Student Learning Outcome


When or By When Outcome Occurs

Outcome Measure

Data Instruments




Exercise 7:Mapping Your Desired

Outcomes:Identifying Where/When

Outcomes Occur


Measuring Outcomes

Once the desired Process/Delivery outcomes and Student Learning outcomes have been identified, as well as when and where they will occur, the next step is to determine who or what will be measured and how the data will be gathered…

…using multiple measures of varying types


Initial ConsiderationsTrue assessment of academic advising should involve multiple measures

– beyond a simple satisfaction survey or advisor evaluation

– individual measures evaluate (rather than assess) a single dimension of a single phenomenon


Types of Measures• Qualitative

– exploratory– small samples– open-ended– emerging information– subjective, inductive interpretation of data

• examples– focus groups

– case studies

– naturalistic observation

Information/data in form of rich, in-depth responses (words)


Types of Measures continued…

• Quantitative– descriptive– large samples– structured– objective, deductive interpretation of data

• examples– questionnaires

– surveys

– experiments

Information/data in form of numbers, measures (statistics)



• Direct– may be qualitative or quantitative– examples

• direct observation of advising interaction• pre-test/post-test of variable leading to desired outcome• standardized test or inventory measuring student learning• tracking of student data (enrollment rates, retention rates, GPAs,

transcript analysis, etc.)• counts of use of services• advisor:student ratios



• Indirect– may be qualitative or quantitative– examples

• focus groups• surveys, questionnaires• interviews• reports• tracking of student perceptions (satisfaction, ratings of advisors,

ratings of service, etc.)• tracking of advisor perceptions (student preparedness, estimation

of student learning, etc.)


Examples of Existing Instruments

• To be used as just one measure among multiple measures– ACT Survey of Academic Advising– Noel-Levitz Student Satisfaction Inventory (SSI)– Winston and Sandor’s Academic Advising

Inventory (AAI)– NACADA Assessment of Advising Commission

www.nacada.ksu.edu/Commissions/C32/index.htm


http://www.nacada.ksu.edu/Commissions/C32/index.htm

Other Sources

• To be used as just one measure among multiple measures– Data from National Survey of Student

Engagement (NSSE)– Data from other nationally normed, standardized

instruments- Existing institutional data


Institutional Data• Why reinvent the wheel, so to speak, if there are existing

institutional data that relate to a question you would like answer regarding your advising program?

• Become knowledgeable regarding what type of data your institution is already collecting

• Get to know your institutional data people, and include them as a stakeholder in the assessment of your advising program

• Remember that any single type of data should be utilized as just one measure among multiple measures for each phenomena you are evaluating within your overall assessment effort


For both process/delivery and student learning outcomes, you need to identify the minimum criteria for success of the outcome measure, e. g.,•number of students exhibiting a specific learning performance•percentage of students exhibiting a specific learning performance• advisor rating of student performance• student rating of specific aspect of advising process• advisor rating of specific aspect of advising process• etc.


In the absence of any existing relevant measures of the identified student learning outcomes for academic advising, the initial cycle of assessment for any given desired student learning outcome should be considered a baseline data gathering effort

Suggestion


Mapping of Outcomes

Adapted from Robbins, R. L. (2009). Evaluation and assessment of career advising. In K. Hughey, D. N. Burton Nelson, J. Damminger, and B. McCalla-Wriggins (Eds.) The Handbook of Career Advising (chapter 12). San Francisco: Jossey-Bass

The Assessment Matrix/Table

Institutional Mission Statement



Specific

PDO or SLO


When or By When Outcome Occurs

Outcome Measures

Data Instruments




Exercise 8:Identifying Multiple Measures

Select one student learning outcome, and identify three ways to measure whether or not that specific outcome has been met


An Additional And Important Consideration

- even if your outcome data supports the achievement of the desired SLO – how do you know that the student learning was a result of academic advising???

- you literally have to ask students “where did you learn the information,” “how did you know to do the behavior,” and/or “how did you develop the appreciation” in regard to the SLO


Advising Syllabus as a Tool for Communicating Key Elements

• If academic advising is teaching…• Advisors are teachers• Teachers have a discipline• The advisor’s discipline is academic advising• Academic advising is a discipline• Individual academic advisors offer the “course” academic

advising• Individuals in disciplines author unique courses• Each course has a syllabus


What Does an Advising Syllabus Include?

• Purpose of academic advising• Scope of academic advising• Tools, texts, resources• SLOs for academic advising• Advisor responsibilities• Advisee responsibilities• Criteria for successful academic advising experiences

(outcome measurements)• Other elements as individually appropriate


Acting Uponthe Outcome Data


Determine How and With Whom Results Are Shared

• Administration: President, Provost, various committees

– via annual report, strategic plan, white paper, Web sites, etc.

• Faculty: all faculty, curricular committees, faculty advisors

– via performance reviews, annual reports, strategic plans, Web sites, etc.

• Students: all students, student advisees, student senate, student groups

– via newsletters, annual reports, Web sites, etc.

• Budgeting entities

– via annual reports, budget requests, Web sites, etc.

• Accreditors

– via self-studies, accreditation reports, Web sites, etc.


• Revise pedagogy or curriculum

• Develop/revise advisor training programs

• Design more effective programming– advising, orientation, mentoring, etc.

• Increase out-of-class learning opportunities

• Shape institutional decision making– planning, resource allocation

• Other…

Interpret How Results Will Inform Decision Making


Decide How You WillFollow-up on Implemented

Changes• Timetable to implement changes

– implement all or specific components on a schedule

• Assessment of implemented changes– repeat assessment cycle again

• Continuous assessment– assessment is on-going


Professional Development• Using assessment to inform and support

professional development

• Revise advisor training and development programming accordingly

• Demonstrate the need for additional training and development

• Demonstrate the need for additional resources to meet goals


Building a Culture and Capacity for Assessment

• The Culture• Commitment• Communication• Collaboration

• The Capacity• Support


Exercise 9:Acting Upon the Outcome Data

Identify at least one use for the outcome data of your assessment of advising


At the end of the day, assessment of academic advising

is all about…• developing consensus around collective expectations about student learning that should occur in advising

• gathering evidence in order to understand student learning resulting from academic advising

• using this evidence to support improvements in academic advising that will contribute to improvements in student learning


assessment is much morethan just a single evaluation

Remember:


Assessment as Research


Why Research Academic Advising?Why Research Academic Advising?

• Academic advising is a collective experience among most college students

• Academic advising is an integral part of student development

• Academic advising is teaching, with a discipline and pedagogy

• Professional disciplines utilize inquiry and resulting data to inform decision making

• There is much anecdotal information re: the power of effective advising, but little empirical research


Wide Range of Audiences

• Peers in academic advising

• Advising administrators

• Higher Education

• Field of Advising

• Deans, Provosts, and Presidents

• Individual advisors

• Students

• Parents


Scholarly Inquiry May Include…Scholarly Inquiry May Include…

• Hypothesis testing• Replication of existing knowledge in new

setting• Discovery of a novel phenomenon• Development of a new theory• Creation of new knowledge• Evaluation of effectiveness of new

implementation or approach


Methods of InquiryMethods of Inquiry

• Experimental• Quasi-experimental• Ex-post facto (after-the-fact)• Correlational• Historical• Ethnographic (cultural interpretation)• Phenomenological (description of experience)• Case study• Longitudinal• Program Assessment


Continuing Assessment as Continuing Assessment as ResearchResearch

Once you have obtained assessment data and act upon it via implementing a new strategy or model, the subsequent assessment of the new implementation serves as research:

• experimental assessment• current versus historical data• ethnographic inquiry• qualitative inquiry• quantitative inquiry• other research/inquiry


Cognitive Learning Outcomes:• Understand the differences between assessment and

evaluation• Understand the differences between Process/Delivery

Outcomes (PDO) and Student Learning Outcomes (SLO)

• Understand the concept of “mapping” of desired outcomes

• Understand the importance of using multiple measures for any single desired outcome

Learning Outcomes for Presentation


Behavioral Learning Outcomes:• Identify your stakeholders in the assessment of advising• Identify at least one SLO for your assessment of

advising process• Map at least one SLO• Identify multiple measures for at least one SLO• Identify at least one use for the outcome data of your

assessment of advising



Affective Learning Outcomes:• Value the cyclical process of assessment• Appreciate the importance of assessment (besides

accountability)• Appreciate the value of student learning in advising• Appreciate the importance of assessing student learning

in advising• Value the process of identifying opportunities for

learning (mapping) in the advising process• Appreciate the need for multiple measures for any single

desired PDO or SLO• Value the role of outcome data in informing decisions



Questions/Discussion


THANK YOU!


Conducting Assessment of Academic Advising 2013 NACADA Summer Institute Jacksonville, Florida Dr. Karen Boston Dr. Rich Robbins NACADA Executive Office.

Documents