Assessment of Academic Advising: An Overview NACADA Executive Office Kansas State University 2323 Anderson Ave, Suite 225 Manhattan, KS 66502-2912 Phone:
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Assessment ofAcademic Advising:
An Overview
NACADA Executive OfficeKansas State University
2323 Anderson Ave, Suite 225Manhattan, KS 66502-2912
The contents of all material in this presentation are copyrighted by the National Academic Advising Association, unless otherwise indicated. Copyright is not claimed as to any part of an original work prepared by a U.S. or state government officer or employee as part of that person's official duties. All rights are reserved by NACADA, and content may not be reproduced, downloaded, disseminated, published, or transferred in any form or by any means, except with the prior written permission of NACADA, or as indicated below. Members of NACADA may download pages or other content for their own use, consistent with the mission and purpose of NACADA. However, no part of such content may be otherwise or subsequently be reproduced, downloaded, disseminated, published, or transferred, in any form or by any means, except with the prior written permission of, and with express attribution to NACADA. Copyright infringement is a violation of federal law and is subject to criminal and civil penalties. NACADA and National Academic Advising Association are service marks of the National Academic Advising Association.
Rich Robbins, Bucknell University
The presenters acknowledge and appreciate the contributions of NACADA colleagues Susan Campbell, Charlie Nutt, Mike Kirk-Kuwaye, Lynn Higa, Sharon Aiken-Wisniewski, Tom Grites, and Eric White in preparation of materials for this presentation
This General Session serves as an overview of the assessment process for academic advising…
…a more in-depth, applied opportunity will be offered as a Workshop on Assessment of Academic Advising on Thursday from 9:00 – 11:45
The Global Community for Academic Advising
• Understanding assessment• Definitions of assessment• Purposes for conducting assessment• Evaluation versus assessment• Key terms and concepts
• Engaging in assessment of academic advising
• Use of outcome data
Agenda
The Global Community for Academic Advising
Assessment
“Assessment is the systematic collection, review, and use of information about educational programs* undertaken for the purpose of improving student learning* and development*”
(Marchese, 1993)
* Advising is part of the educational process, not simply a “service”
The Global Community for Academic Advising
Assessment
“Assessment is an ongoing process of collecting information* that is aimed at understanding and improving student learning and personal development”
(Angelo, 1995)
* what we like to call “evidence”
The Global Community for Academic Advising
Assessment
“Assessment is a process that focuses on student learning, a process that involves reviewing and reflecting on practice as academics have always done, but in a more planned and careful way”
(Ewell, 2000)
The Global Community for Academic Advising
Assessment
“Assessment is the means used to measure the outcomes of education and the achievement of students with regard to important competencies”
(Pellegrino, Chudowsky, and Glaser, 2001)
The Global Community for Academic Advising
For Academic Advising…
“Assessment is the process through which we gather evidence about the claims we are making with regard to student learning and the process/delivery of academic advising in order to inform and support improvement”
(Campbell, 2008)
Assessment
The Global Community for Academic Advising
What Is Assessment –The Intentions
• Assessment is intended to be a positive process, yet its connotations are often negative
• The focus has often been on activities that demonstrate accountability to the exclusion of those that are aimed at improvement
The Global Community for Academic Advising
• Program effectiveness
• Program improvement
• Program accountability
• Activities aimed at both improvement and accountability are important
• Most compelling purpose is “institutional curiosity” (Maki, 2002; 2004)– i.e, student learning and student
achievement
Assessment Has Multiple Purposes
The Global Community for Academic Advising
Assessment or Evaluation?What Distinguishes Assessment from Evaluation?• evaluationevaluation usually measures usually measures effectivenesseffectiveness• assessmentassessment usually measures usually measures outcomesoutcomes• assessment focuses on programmatic issues while assessment focuses on programmatic issues while
evaluation focuses on individual performances of advisorsevaluation focuses on individual performances of advisors• assessment should be continuous and imbedded in the assessment should be continuous and imbedded in the
culture while evaluation is episodicculture while evaluation is episodic• evaluation of individual performance and evaluation of evaluation of individual performance and evaluation of
effectiveness of effectiveness of processes may be used as part of an processes may be used as part of an overall assessment designed to measure program overall assessment designed to measure program outcomesoutcomes
The Global Community for Academic Advising
What Assessment is NOT
• Assessment is NOT episodic
• Assessment is NOT just about measurement
• Assessment is NOT about evaluating the performance of an individual staff / faculty / student
• Assessment is NOT solely an administrative process
• Assessment is NOT easy or quick
The Global Community for Academic Advising
Assessment is
• An on-going cycle of activity• A gathering of a variety of information and
data• Using this feedback for improvement of
individual or program performance• A team effort with faculty, staff, students, and
administrators actively engaged• A complex process of comparison
“…a lack of assessment data can sometimes lead to policies and practices based on intuition, prejudice, preconceived notions, or personal proclivities – none of them desirable bases for making decisions”
Mapping the ExperienceWhat experiences?When or by when?
Gathering EvidenceWhen gathered?Where gathered?
How often gathered?From whom gathered?
How gathered?Minimum performance criteria for success?
ValuesVision
MissionGoals
Programmatic Outcomes
Sharing and Acting Upon the ResultsInterpret how results inform practice
How and with whom to share interpretationFollow up on implemented changes
Start the process all over again!
(adapted from Darling, 2005)
The Global Community for Academic Advising
The Assessment Matrix/TableInstitutional Mission Statement
Local Mission Statement
Specific Goal or Objective
Specific Process/Delivery Outcome or Student Learning Outcome
Where Outcome Occurs
When or By When Outcome Occurs
Outcome Measure
Minimum Performance Criteria for Success
Data Instruments
Action(s) Based on Outcome Data
Mapping of Outcomes
Adapted from Robbins, R. L. (2009). Evaluation and assessment of career advising. In K. Hughey, D. N. Burton Nelson, J. Damminger, and B. McCalla-Wriggins (Eds.) The Handbook of Career Advising (chapter 12). San Francisco: Jossey-Bass.
and
Robbins, R. (2011). Assessment and accountability of academic advising. In J. Joslin & N. Markee (eds.), NACADA Monograph #22: Academic Advising Administration: Essential Knowledge and Skills for the 21st Century (chapter 4). Manhattan, Kansas: NACADA.
The Global Community for Academic Advising
Identifying Key Stakeholders:Who Should Be Involved?
• Colleagues, faculty, administrators, institutional researchers, staff, students, and institutional community
• Decide how the assessment team will interact, overlap, and/or support other institutional efforts
• Encourage stakeholders on and off campus
• Continuous communication and feedback is a must!
The Global Community for Academic Advising
• Building of a shared trust
• Building of a shared motivation
• Building of a shared language
• Building of support for academic advising institutionally-wide
• The result is a shared ownership and belief in the process
Benefits of a Collective and Collaborative Process
The Global Community for Academic Advising
Need to Involve Stakeholdersat Each Step
• Pre-assessment• Establishment of vision, mission, goals, and outcomes• Planning for assessment• Development of a shared definition and philosophy of
academic advising and assessment• Identification of assessment criteria and methodology• Implementation• Reporting of results• Facilitating change
The Global Community for Academic Advising
• Values – What is considered important in regard to academic advising on the campus
• Vision – The aspirations of what academic advising can be on the campus
• Mission – The statement which reflects the purpose of academic advising on the campus that serves as the institution’s roadmap to reach its vision and affirm its values for academic advising
Key Terms
The Global Community for Academic Advising
Key Terms• Process/Delivery Outcomes – Articulate the expectations
for how academic advising is delivered and what information should be delivered through the academic advising experience
• Student Learning Outcomes – Articulate what students are expected to know, do, and value/appreciate as a result of involvement in the academic advising experience
• Mapping – The process of determining when, where and how the outcomes for advising will be accomplished over the students’ academic career and beyond
The Global Community for Academic Advising
Process/Delivery OutcomesProcess/Delivery Outcomes• Typically what is evaluated via student
satisfaction surveys• e.g., “the advisor provided accurate information”• e.g., “the setting for the advising session was
appropriate”• etc.
• Do not assess student learning, but evaluate what processes occurred during the advising interaction
The Global Community for Academic Advising
Cognitive SLOs
What do we want students to KNOW as a result of participating in academic advising?
Know general education requirements
Know about academic support services
Know how to use the student information system to register
Know how to use the catalog
Etc…….
The Global Community for Academic Advising
Behavioral/Psychomotor SLOs
What do we want students toDo as a result of participating in academic advising?
Generate their degree audit
Make advising appointments
Keep advising appointments
Ask for help
Access course descriptions and degree requirements using the online catalog
Etc….
The Global Community for Academic Advising
Affective SLOs
What do we want students to Value or Appreciate
as a result of participating in academic advising?
Value/Appreciate general education
Value/Appreciate the advising relationship
Value/Appreciate the process of learning
Etc….
The Global Community for Academic Advising
Mapping of Outcomes
The process of determining when, where and how the Student Learning outcomes for academic advising will be accomplished over the students’ academic career and beyond
The Global Community for Academic Advising
Mapping the Learning Experience
• What should be learned: e.g., student knows the components of the institution’s General Education requirements
• Where it should be learned: e.g., orientation workshops, advising sessions, personal reading of catalog or curriculum guide
• When it should be learned: e.g., prior to first year (orientation); by end of first year (via advising sessions); by end of first year (via personal reading)
The Global Community for Academic Advising
Mapping of Outcomes
The Assessment Matrix/Table
Institutional Mission Statement
Local Mission Statement
Specific Goal or Objective
Specific Process/Delivery Outcome or Student Learning Outcome
Where Outcome Occurs
When or By When Outcome Occurs
Outcome Measure
Minimum Performance Criteria for Success
Data Instruments
Action(s) Based on Outcome Data
The Global Community for Academic Advising
Measuring Outcomes
Once the desired Process/Delivery outcomes and Student Learning outcomes have been identified, as well as when and where they will occur, the next step is to determine who or what will be measured and how the data will be gathered…
…using multiple measures of varying types
The Global Community for Academic Advising
Multiple Measures• The multiple measures utilized should vary in
type– Qualitative– Quantitative– Direct– Indirect
• This will help to insure the validity of the results of any single measure
The Global Community for Academic Advising
Examples of Existing Instruments
• To be used as just one measure among multiple measures– ACT Survey of Academic Advising– Noel-Levitz Student Satisfaction Inventory (SSI)– Winston and Sandor’s Academic Advising
Inventory (AAI)– NACADA Assessment of Advising Commission
• To be used as just one measure among multiple measures– Data from National Survey of Student
Engagement (NSSE)– Data from other nationally normed, standardized
instruments- Existing institutional data
The Global Community for Academic Advising
Dangers of Satisfaction Surveys
there is often a difference between an advisee receiving good, effective academic advising and being satisfied with the advising process:
– if any negative information is exchanged during the advising interaction, the student may respond negatively to the survey items even though the information provided was correct and the process of the interaction was appropriate
– the student will likely rate the advising provided based on the type of interaction desired (e.g., informational, relational)
The Global Community for Academic Advising
Measures can (and should) include existing institutional data
• Information from Institutional Research, Admissions, Registrar, etc. can provide tracking data, GPAs, retention rates, and other information you can utilize as assessment data
– this can be a source of some of the multiple measures utilized (in addition to formal instruments, satisfaction inventories, and others)
The Global Community for Academic Advising
For both process/delivery and student learning outcomes, you need to identify the minimum criteria for success of the outcome measure, e. g.,•number of students exhibiting a specific learning performance•percentage of students exhibiting a specific learning performance• advisor rating of student performance• student rating of specific aspect of advising process• advisor rating of specific aspect of advising process• etc.
The Global Community for Academic Advising
In the absence of any existing relevant measures of the identified student learning outcomes for academic advising, the initial cycle of assessment for any given desired student learning outcome should be considered a baseline data gathering effort
Suggestion
The Global Community for Academic Advising
Mapping of Outcomes
The Assessment Matrix/Table
Institutional Mission Statement
Local Mission Statement
Specific Goal or Objective
Specific Process/Delivery Outcome or Student Learning Outcome
Where Outcome Occurs
When or By When Outcome Occurs
Outcome Measure
Minimum Perfor-mance Criteria for Success
Data Instru-ments
Action(s) Based on Outcome Data
The Global Community for Academic Advising
So I Have The Data – Now What?
• Sharing and Acting Upon Results– Interpret results regarding how they inform the
advising process/delivery, student learning, and decision-making
– Determine with whom and how the results are reported
– Decide how you will implement changes based on the results
– Start assessment cycle again…
The Global Community for Academic Advising
At the end of the day, assessment of academic advising
is all about…• developing consensus around collective expectations about student learning that should occur in advising
• gathering evidence in order to understand student learning resulting from academic advising
• using this evidence to support improvements in academic advising that will contribute to improvements in student learning
The Global Community for Academic Advising
assessment is much morethan just a single evaluation
Remember:
The Global Community for Academic Advising
Assessment as Research
The Global Community for Academic Advising
NACADA View of ResearchNACADA View of Research
NACADA views research as scholarly inquiry into all aspects of the advising interaction, the role of advising in higher education, and the effects that advising can have on students. It regards consuming and producing research as the collective responsibility of all members of the higher education advising community, including advisors, faculty, administrators, and students.
NACADA Task Force on Infusion of Research, 2008
The Global Community for Academic Advising
Scholarly Inquiry May Include…Scholarly Inquiry May Include…
• Hypothesis testing• Replication of existing knowledge in new
setting• Discovery of a novel phenomenon• Development of a new theory• Creation of new knowledge• Evaluation of effectiveness of new
implementation or approach
The Global Community for Academic Advising
Methods of InquiryMethods of Inquiry
• Experimental• Quasi-experimental• Ex-post facto (after-the-fact)• Correlational• Historical• Ethnographic (cultural interpretation)• Phenomenological (description of experience)• Case study• Longitudinal• Program Assessment
The Global Community for Academic Advising
Remember…
…a more in-depth, applied opportunity will be offered as a Workshop on Assessment of Academic Advising on Thursday from 9:00 – 11:45