Academic Program Assessment & Evaluation Fred Rovai Jim Downey
Feb 22, 2016
Academic Program Assessment & Evaluation
Fred RovaiJim Downey
Background
• “College standards are becoming diluted and there is a fuzziness about what faculty teach and what is expected from students” (Commission on the Future of Higher Education).
• “Outcome-based education means starting with a clear picture of what is important for students to be able to do, then organizing the curriculum, instruction, and assessment to make sure that this learning ultimately happens” (Spady, 1994).
SACS Comprehensive Standards
• 3.3.1 – The Institution identifies expected outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results.
• 3.4.1 – …establishes and evaluates program and learning outcomes.
• 3.5.1 – …identifies college-level competencies within the general education core and provides evidence that graduates have attained those competencies.
Student Learning Outcomes (SLOs)
• Skills, knowledge, and attitudes that students are expected to acquire in a program and be able to demonstrate upon course and program completion.
• Similar to instructional objectives .• Characteristics
– Specific – SLOs should specify what students are to achieve.– Measurable – You should be able to measure the degree to which
SLOs are met.– Achievable - Are the SLOs achievable and attainable?– Realistic – Can students realistically achieve the outcomes with
available resources?– Time bounded – When are the students to achieve the outcomes?
SLOs
• SLOs are expressed and measured at the course, program, or institutional level.
• SLOs reflect a shift of focus from “What am I teaching” to “What are my students learning”?
Curriculum Alignment
• SLOs• Essential questions
Intended Curriculum
Achieved Curriculum
• Assessment for Learning– Formative– “In-progress”– Provides corrective feedback
• Assessment of Learning– Summative– “After the fact”– High stakes
Taxonomy of the Cognitive Domain
Level Sample Action Verbs
Creating Generating, planningEvaluating Critiquing, recommendingAnalyzing Differentiating, organizingApplying Executing, implementingUnderstanding Summarizing, explainingRemembering Retrieving, recalling
Taxonomy of the Affective DomainLevel Sample Action Verbs
Characterization by value set
Avoiding, embracing
Organization Theorizing, formulating
Valuing Supporting, debating
Responding Following, volunteering
Receiving Accepting, listening
Taxonomy of the Psychomotor DomainLevel Sample Action Verbs
Non-Discursive Communication
Choreographing, gesturing
Skilled movements Dancing, acting
Physical activities Moving precisely
Perceptual Punting, catching
Basic Fundamental Movement
Running, manipulating
Reflex movements Stretching, extending
Significant Learning Categories• Learning how to learn: self-regulation, organizing
time, data-driven decision making, etc.• Motivation/interest/values• Human dimension: acquiring professional behaviors,
leadership, teamwork, etc.• Cognition• Integration/connection• Application/problem solving/critical thinking• Skills: using technology/communication, etc.
Course Syllabus
• Course goals, instructional objectives, and/or SLOs must be consistent with: – University mission and goals– School mission and goals– Program goals and SLOs– National disciplinary norms
• Assessments must be identified and linked to each instructional objective or SLO
Assessment
• Dressel, P. (1957) writes:"...a grade (is)...an inadequate report of an
inaccurate judgment by a biased and variable judge of the extent to which a student has attained an undefined level of mastery of an unknown proportion of an indefinite amount of material." (p. 6.)
Assessment
• Anderson et al. (1975) provide the classical description of assessment:
“Assessment, as opposed to simple one-dimensional measurement, is frequently described as multitrait-multimethod; that is, it focuses upon a number of variables judged to be important and utilizes a number of techniques to assay them – Its techniques may also be multisource … and/or multijudge” (p. 27).
Assessing SLOs• Focus is on assessing student:
– Skills (psychomotor domain)– Knowledge (cognitive domain)– Attitudes (affective domain)as a result of the educational program.
• Purpose: To obtain information that can be used by program faculty to answer the following questions:– Are our students learning what we think is important?– Are they learning what they need to succeed in this field or profession?– Are we improving in our ability to help students learn?– Should our curriculum or teaching strategies be modified?– Are there other techniques or additional resources that would help our
students learn more effectively?
Sample SLO Assessments
Direct Measures• Standardized test results• Performance on licensure or
professional exams• Embedded assignments in
specific courses• Internships• Performances• Portfolios of work samples• Job placement rates
Indirect Measures• Student surveys• Graduate follow-ups• Focus groups• Exit interviews
Student Learning Assessment Cycle
Define SLOs
Identify evidence
Identify tasks & methods
Administer assessments
Review results & identify changes
Evidence Requirements
• Identification of SLOs and how they are addressed for each program, e.g., curriculum maps
• Assessment of each SLO• Efforts to:– improve student learning – make the program more effective
based on these assessments
Simple Program Curriculum MapProgram
SLOs Courses
Course 1 Course 2 Course 3 Course 5 Capstone Course
SLO 1 Introduced Reinforced & assessed
Reinforced & assessed
Reviewed
SLO 2 Introduced & assessed
Reviewed & assessed
SLO 3 Introduced & assessed
Reinforced& assessed
Reviewed
SLO 4 Introduced & assessed
Reviewed & assessed
SLO 5 Introduced & assessed
Reinforced Reviewed & assessed
Assessment Opportunities
• Embedded assignments in specific courses– Traditional vs. authentic, application driven
• Student performances– Student thesis/dissertation defenses
• Internships• Program comprehensive examinations• Course evaluations (self, peer, student, & supervisor)• Standardized surveys• Focus groups
Putting Assessment into Practice at Regent• Assessment schedule will be advanced to address SACS concerns
and start building the culture• New assessment matrices will be used• Executive summaries of results will be generated for reporting to
school deans and Office of IE• Identify one high-stakes outcome for you program that would be a
major issue if not achieved• Begin to think about program outcomes beyond SLOs
– Job placement rates– Retention rates– Employer satisfaction– Etc.