Closing the Loop Standards 7 and 14 R. Dean Foster, Ph.D. Director of Assessment [email protected] Joseph M. Thomas, Ph.D. Dean, Undergraduate Studies [email protected].

Closing the Loop Standards 7 and 14

R. Dean Foster, Ph.D.Director of Assessment

[email protected]

Joseph M. Thomas, Ph.D.Dean, Undergraduate Studies

[email protected]

Brian Buerke, Ph.D. Associate Professor of Physics

[email protected]

Albright College

• Small Private (1,650 traditional students)• Liberal Arts Mission• Interdisciplinary Emphasis• Primarily Residential• Reading PA (medium-sized urban setting)• Strong Faculty Governance

Goals of Presentation

Participants will • Better understand concept of Closing

the Loop• Be aware of several examples of Closing

the Loop onStandard 7Standard 14

Closing the Loop DefinedUsing assessment results to • Plan improvements• Advocate for allocation or reallocation of

budget and other resources (when those resources are needed to implement improvements)

• Implement improvements• Re-assess results (data and information)

Closing the Loop on Standard 7 Institutional Effectiveness

•Health Services•IT Support•Academic Support Services•Facilities - Classrooms and Labs

Closing the Loop on Standard 7 Institutional Effectiveness

Data Sources

• Higher Education Data Sharing (HEDS) Senior Survey • Association of Independent Colleges and

Universities of Pennsylvania End of First Year (Student) Survey (AICUP)

• National Survey of Student Engagement (NSSE)

Closing the Loop on Standard 7 Institutional Effectiveness

Initial Data

Selected areas of low student satisfaction–Health Services < 70% . . 2.7/5– IT Support < 80% . . 2.7/5–Academic Support . . < 3.0/5– Facilities – Classrooms and Labs . . 2.7/5

Qualitative feedback from students and faculty reinforced these concerns.

Closing the Loop on Standard 7

Health ServicesRatings < 70% . . 2.7/5 - Closing the Loop byDeveloped/administered the Gable Health Center

Satisfaction Survey. Student-users identified overall high satisfaction. Lower rating on location, hours.

Based on results, the College relocated and improved the Health Center, changed service hours, changed intake process. 100% satisfaction on location.

National survey results still not at benchmark, assuming non-user ratings, will survey broader student body Spring 2011.

Closing the Loop on Standard 7

IT Services

Ratings < 80% . . 2.7/5 - Closed the Loop byFollow-up surveys with students and discussions with

faculty identified areas of dissatisfaction which included

• Lack of wireless capability in residence halls.• Faculty concerns with technology support in

classrooms.

Closing the Loop on Standard 7

IT ServicesClosed the Loop by• $3/4 million for wireless residence halls and Campus

Center.• Move IT under provost. Classroom support was

restructured and improved. • $1 million over 5 years to renovate classrooms.Student satisfaction increased to 2.9-3.2 in 2009.Faculty acknowledged improvements.

Closing the Loop on Standard 7

Academic SupportRating < 3.0/5 - Closing the Loop by• Increased from 1 FT to 4.5 staff• Achieved CRLA certification of tutoring• Expanded skills workshops, tutoring times, and study

groups• Added services for students on academic probation• Improved disability servicesChanges too recent to be reflected in national surveys BUT . . .

Closing the Loop on Standard 7Impact also on Standard 14

Academic Support

Results of Increased Services for Students on Academic Probation (59 students)

• 27 fulfilled all requirements - 98% increased term GPA; 50% increased cumulative GPA

• 21 fulfilled some requirements - 60% increased term GPA; 21% increased cumulative GPA

• 13 fulfilled no requirements - 6% increased term GPA; no increase in cumulative GPA.

AND . . .

Closing the Loop on Standard 7Impact also on Standard 14

Academic Support

Retention of “at risk” students increased from • 32% in AY 2007-08 to • 56% in AY 2009 - 10

Closing the Loop on Standard 7

Facilities – Classrooms and Labs

Ratings < 70% . . 2.7/5 - Closing the Loop by• Completed a Campus Master Plan (facilities).• Making improvements in classroom technology as

already reported above.• Allocating $27 million for an addition to and

renovation of the Science Building.• Continuing to receive gifts (recent $5 million) to

support Strategic and Master Plans.• Continuing to implement the Master Plan

Closing the Loop on Standards 7 and 14:

Academic Program Review(APR) as an Assessment Hub

Closing the Loop on Standards 7 and 14

APR’s Cross-College Connections

APR and resulting academic improvements connect:

• Most institutional operations and goals• MSCHE Standards – 7, 11, 12, 14• Departmental planning to institutional

planning

Closing the Loop on Standards 7 and 14

APR Defined• Joint administrative-faculty process• Periodic (5-7 years), spans 2 years• Formalizes self-reflection • Department’s bully pulpit to describe

activities and needs• Vehicle for administration and other faculty

to evaluate support of mission

Closing the Loop on Standards 7 and 14

APR Process Overview

1. Self-study: analyzes data about faculty, curriculum, students, and resources

2. Site visit: provides analysis and consultation from experienced external peers given access

3. Action plan: specifies improvements in context of institutional goals and resources

All final documents available to Faculty and Board.

Closing the Loop on Standards 7 and 14

Sample APR Results

• BIO retention: + 8% from smaller sections, revised labs, formative assessments

• HIS curriculum gap: add World History• SOC rigor: add pre-reqs, theory requirement,

and quantitative capstone• Theater staffing, equipment, and facilities:

upgrades and planning cycle underway

Closing the Loop on Standards 7 and 14

APR Detail: First-Year Seminar

• Data: Low satisfaction (rated 2.3/5)• Analysis: Lack academic focus; non-credit• Responses: o2006-8 topical focus (rated 3.4/5)o2009 full credit seminar (rated 3.9/5)

• Faculty approval + Provost budget

Closing the Loop on Standards 7 and 14

APR Detail: Math Retention

• Data: Low grades in pre-Calc and Calc I• Analysis: Initial student preparation;

academic support; variable sections• Responses: placement; remedial course/SI;

tutor training; cross-section standards• Faculty approval + resources through

Provost’s budget and staffing cycle

Closing the Loop on Standards 7 and 14

APR Detail: Math Content and Mission

• Data: Curriculum gap; senior info?; low # of co-concentrators

• Analysis: Need “Real Analysis” course; more info on senior satisfaction and competency; more collaboration with other departments

• Responses: Create and require course; implement ETS test and senior survey; create new track with Business appeal

• Faculty approval + Provost’s budget cycle

Closing the Loop for Student Learning Outcomes (Standard 14)

• Course levelModern Astronomy (PHY 102)

• Conceptual learning

• Objective assessmentImpact of changes

Closing the Loop (Standard 14): Course

PHY 102 is a survey lab course with a standard syllabus

- Introduction/ancient astronomy - Planetary motion and gravity- Solar system - Characteristics and evolution of stars- Stellar explosions and remnants- Structure and evolution of the

Universe

Closing the Loop (Standard 14): Course

Lab includes project-based and observational activities

- Building a quadrant - Galileo’s gravity experiment - Building a telescope - Observing the sun - Nighttime viewings

A team of students tests the quadrant they designed and built

Closing the Loop (Standard 14): Conceptual Learning

One student learning outcome is to improve conceptual understanding

• Provide framework for deep learning

• Misconceptions- are common- hard to overcome- prevent effective learning

Closing the Loop (Standard 14): Objective Assessment

Conceptual knowledge is measured by a concept test (LST)1

• Distractor based• Nationally validated• Given pre- and post-semester• Direct measure of knowledge gains (g)

post preg =

100% pre

1. Alan Lightman and Philip Sadler, The Physics Teacher 31, 162 – 167 (1993).

Closing the Loop (Standard 14)

Dissatisfaction with conceptual learning inspired changes

• Fall 2000: Average gain (g) = 0.19

• Change to interactive lecture notes- improve student focus- reinforce mental engagement- increase opportunities for discussion

Closing the Loop (Standard 14): Assessment of Changes

Gains doubled after changes

0 1 2 3 4 5 6 7 8 90

0.1

0.2

0.3

0.4

0.5

0.6

LST concept test

Semesters since spring 2000

Aver

age

norm

aliz

ed g

ain

Interactive lecture notes introduced

Avg. Gain(7 semesters):

g = 0.44 ± 0.05

Closing the Loop Standards 7 and 14

Summary

Questions and Discussion

Closing the Loop Standards 7 and 14

R. Dean Foster, Ph.D.Director of Assessment

[email protected]

Joseph M. Thomas, Ph.D.Dean, Undergraduate Studies

[email protected]

Brian Buerke, Ph.D. Associate Professor of Physics

[email protected]