Using Accountability to Support Quality

Network Conference

Using Accountability to Support Quality

Aligning Program, Campus, System, State, and Federal Models of Higher Education Effectiveness

Victor M. H. Borden, Ph.D.Associate Professor of Psychology

Associate Vice ChancellorInformation Management and Institutional Research

Indiana University-Purdue University IndianapolisPresident, Association for Institutional Research

Network Conference

Victor M. H. Borden – April 1, 2004

In ConclusionSuccessful accountability systems are sensitive to diverse perspectives on qualityThe best accountability measures derive from systematic assessment and evaluationAlignment across levels is optimized by:

Recognizing and rewarding multiple kinds of qualityActive listening to multiple constituent groups

Responsible accountability educates all constituents

Network Conference


Part I

What is a quality higher education?

Network Conference


A Pause for Reflection

What qualities of a college experience are most valued by…

Students?Parents?Faculty?Trustees?Legislators?Accreditors?Business and Industry?

Network Conference


Qualities of the College ExperienceStudents

Excellent teachers; access to rewarding careers/professions; academic support; personal enrichment; quality of student life; exposure to people

ParentsPersonal and professional development; safety; access to positive role models; stay out of trouble

FacultySmall classes; good students; space and resources (library, equipment, etc.) for research and scholarship; good colleagues; support for venture development; lack of bureaucracy

TrusteesEffective use of resources; faculty accomplishments (recognition and funding); student progress and achievement; contributions to economic development

LegislatorsAccess by constituents; employability of graduates; contributions to state welfare and development; effective use of resources; ties with business and industry; controlled costs

AccreditorsAdequate organizational structures and processes (administration and governance); appropriate credentials of faculty; effective use of resources; demonstrated student learning outcomes; quality assurance processes

Business and IndustryPreparation of graduates for workforce; support for research and development

Network Conference


Quality

+

Cost AccessTo Student

+To State

Undifferentiated

+Differentiated

Program

Higher Education System Quality

Network Conference


Quality Cost Access Students Teachers, programs, support, career

dev., student life, friends Low cost or

high aid To a point

Parents Career & personal dev., safety, role models, safety

Low cost or high aid

To a point

Faculty Small classes, good students, resources, colleagues, work environ.

Lots of funds Diverse and well prepared

Trustees Effective use of resources, faculty accomplishments; student progress, economic development

Managed costs; diverse

resources

Diverse, well prepared, fit to

mission Legislators Employability, contributions to state

welfare and development, effective use of resources, ties with business and industry

Low costs Broad access

Accreditors Org infrastructure and capacities, faculty credentials, effective use of resources, student learning, quality assurance

Fiscal viability, controlled

costs

Appropriate to mission

Business Employability, support R&D Managed Enough to fill workforce needs

Network Conference


Models of Quality Reputation and Resources

Ratings and rankings

Institutional EffectivenessQuality assurance

Value-Added ApproachesStudent learning outcomes

Student EngagementQuality of academic experience

Higher Education Balanced Score CardMeasuring Up: The State Report Card

Network Conference


Reputation and ResourcesPrevailing view?Revolves around institutional archetypes

Research university; Liberal arts college; Experience of residential 18-22 year olds

Peer judgments included academic reputation

Measures emphasize inputs and resourcesSelectivity, faculty/student ratio, library volumes, research and salary dollars

Graduation rates as outcomesAnother reflection of selectivity and residential nature

Epitomized by U.S. News & World Report

Network Conference


Value-Added ApproachesMore talked about than implementedFocus on change from input to output

Student developmentRegional improvement

Favors longitudinal measurementControversy about amount of change vs. reaching standards of excellence (or even adequacy)Highlights key question: To what degree is institutional quality related to selectivity?Can be incorporated as contextual element of other models

Network Conference


Institutional Effectiveness

Mechanistic approachRatio of resources to inputs, processes, and outcomes

Qualitative approachPeer review of structures, credentials, and processes“Audit” of quality assurance processes, including student outcomes assessment

Traditional accreditation approach

Network Conference


Student EngagementFocuses on quality of college experienceDerived from “best practices” in undergraduate education

Wingspread; Chickering & Gamson

Basis of National Survey of Student Engagement (NSSE)

Fastest growing and most widely used survey in U.S. Higher Education HistoryEngagement Indices analyzed within context of student population characteristics

Network Conference


Principles of Good PracticeEncourages contact between students and faculty Develops reciprocity and cooperation among students Encourages active learning Gives prompt feedback Emphasizes time on task Communicates high expectations Respects diverse talents and ways of learning

Network Conference


HE Balanced Score Card

Kaplan & Norton (1976) propose business model

Financial performanceCustomer service and satisfactionProcess effectiveness and efficiencyOrganizational learning

Ruben (1999) adapts to higher education

http://www.odl.rutgers.edu/pdf/score.pdf

Network Conference


Ruben’s HE BSCTeaching/Learning

Programs/Courses, Student Outcomes

Service/OutreachUniversity, profession, alumns, state, prospective students, families employers

Scholarship/ResearchProductivity/Impact

Workplace satisfactionFaculty/staff

FinancialRevenues/expenditures

Network Conference


Measuring UpState-Level FocusPreparation

H.S. grad rate; math/science course taking; math, science, reading, writing proficiency, college entrance exams; AP scores

ParticipationH.S.college rate; young adult enrollment; working adult enrollment

AffordabilityFamily abil. to pay at CC; family ability to pay at 4-Yr; need-based financial aid; low-priced colleges; low student debt

CompletionRetention and grad rates at 2- and 4-yr. colleges

BenefitsAdults w/bachelor’s + degree; Increased income from BA degree; increased income from some col/2-yr degree; population voting; charitable contributions; quant literacy; prose literacy; document literacy

Network Conference


Measuring Up Student LearningAll states received an incomplete in the 2000 and 2002 roundsSeveral position papers regarding approaches to assessing college-level learning (Peg Miller and Peter Ewell)Several ongoing studies, e.g., RAND/CAEKentucky pilot, using licensing exams; GRE, MCAT, LSAT, etc; National Adult Literacy Survey; National Survey of Student EngagementMove toward a college-level NAEP?

Network Conference


Part I SummaryDifferent people have different views of what defines the quality of higher educationAny attempt to create a comprehensive view will require

Multiple dimensions, multiple perspectives, and multiple measuresBalance between quantitative and qualitative components

But is a comprehensive view necessary, or can we take multiple, less complex views and then consider how well they align?

Network Conference


Part II

How do we measure quality in higher

education?

Network Conference


Common Methods

Peer Review of Self StudyBenchmarkingPerformance Indicators

Network Conference


Peer Review of Self StudyHow would you assess the quality of the history major at Harvard University as compared to at Miami-Dade College?Self-study requires unit to take stock and reflect upon effectiveness in attaining goals, which requires

Having goals that relate to overall missionHaving capacity (infrastructure, resources, and processes) to effect goalsMonitoring effectiveness and adjusting

Network Conference


Peer Review of Self Study

Peer review provides expert and credible judgment sensitive to contextManifestations

Faculty-led assessmentAcademic program reviewSpecialized (professional and disciplinary) accreditationRegional accreditationU.K. External Examiners

Network Conference


Benchmarking Definition

Evaluation against an established standard

A process for evaluating business operations by detailed comparison with those of another business, in order to establish best practice and improve performance

Network Conference


Best PracticeA process, technique, or innovative use of technology, equipment or resources that has a proven record of success in providing significant improvement in cost, schedule, quality, performance, safety, environment, or other measurable factors which impact an organizationContext specific example – substance abuse

"best practices" are those strategies and programs which are deemed research-based by scientists and researchers at:

National Institute for Drug Abuse (NIDA), Center for Substance Abuse Prevention (CSAP), National Center for the Advancement of Prevention (NCAP), Office of Juvenile Justice and Delinquency Prevention (OJJDP), and Centers for Disease Control and Prevention (CDC).

Network Conference


Best PracticeStanley Fish in his Chronicle column…practices that had worked for some people

in some context where some problem had been identified and was addressed successfully by some solution"Best Practices" is itself a practice, an industry focused on itself and equipped with its own internal machinery including a version of the Academy Awards that allows practitioners to recognize and honor one another publicly

Network Conference


Data and MeasuresComprise the metrics of benchmarking, but are not benchmarking itselfOften mistaken for outcome goals

U.S. News & World ReportCarnegie Classification

More correctly viewed as process indicatorsExamples

Graduate program admissions selectivity and yieldStudent credit hours per faculty FTE

Network Conference


Realities of BenchmarkingOften used outside process context

Measures seen as having meaning in and of themselvesHow well do we do compared to others?This is why ratings and ranking systems are inherently invalid

But measures can be starting point for process inquiry

Network Conference


Performance Indicators

History of external mandatesRecent efforts more closely related to internal evaluation and improvement efforts

BSCPerformance reports

Network Conference


Criteria for Effective PIs*

Start with purposeAlign throughout organization (or system)Align across input, process, outputCoordinate a variety of methodsUse PIs in decision making

*Banta, T. W. and Borden V. M. H. (1994). Performance indicators for accountability and improvement. In V. M. H. Borden & T. W. Banta (eds.) Using performance indicators to guide strategic decision making. New Directions for Institutional Research, 82. San Francisco: Jossey-Bass.

Network Conference


Principles for Effective PI Reporting*

FrameworkTransparentInclusiveAuditable

ContentCompleteRelevantSustainable

Quality & Reliability

AccurateNeutralComparable

AccessClearTimely

*Source: Global Reporting Initiative – Sustainability Reporting Guidelines

Network Conference


PIs as Measures

Inductive – Deductive Cycle

Network Conference


Measurement - Theory

Validity and reliabilityUnless very careful attention is paid to one’s theoretical assumptions and conceptual apparatus, no array of statistical techniques will suffice – Blalock, 1982, p.9

i.e., garbage in, garbage oute.g., graduation rate, funding per FTE, etc.

Network Conference


APQC MIPO

An American Productivity and Quality Center (APQC) benchmarking study

Measuring Institutional Performance Outcomes (Higher Education)

Network Conference


Effective Performance Measures…

…communicate the institution’s core values…are carefully chosen, reviewed frequently, and point to action to be taken on results…may be stimulated by external requirements…are best used as “problem detectors” to identify areas for management attention and further exploration…are linked to resource allocation indirectly (non-punitively)

Network Conference


Effective Performance Measures…

…are publicly available, visible, and consistent across the organization…are best considered in the context of a wider transformation of organizational culture…take time to develop, require considerable “socialization” of the organization’s members, and are enhanced by stable leadership…change the role of managers and the ways in which they manage

Network Conference


E.G.: [email protected]

Network Conference


Network Conference


Network Conference


Network Conference


Assessable Outcomes

Instrumentation

Data Collection & Analysis

Application of Findings

Culture ofEvidence

Improvement

1. Reporting to internal constituents 2. Demonstrating accountability to

external stakeholders

3. Applying findings in campus improvement initiatives

4. Proposing improvement initiatives

5. Improving assessment methods

Web-based data Electronic portfolios

Evaluation

1. Academic and administrative program reviews

2. Evaluation of process effectiveness

3. Assessment of learning outcomes

• in major

• in general education

5. Student assessment

6. Constituent surveys

7. Management information and analysis

8. Program cost analysis

9. Web-based assessment tools

10. Annual campus performance report 11. NCA accreditation

4. Course evaluations

Planning & Budgeting 1. Mission, Vision, Goals developed 2. Unit goals aligned 3. Annual reports on web 4. Programs based on assessable goals,

with performance indicators

5. Biennial planning/budgeting hearings conducted

Implementation

(Everyone on campus implements goals)

Network Conference


Part II Summary

Assessing quality requires conceptual and contextual framesMeasures of quality derive from and pertain to the assessment process

They do not have a life in and of themselves

Accountability for quality may have as much or more to do with demonstration of process than of particular outcomes

Network Conference


Part III

How do we align approaches to quality

assessment across levels?

Network Conference


Aligning Approaches to Quality Assessment Across Levels

How can programs, schools, colleges, universities, systems, states,…each define terms of quality that…

Are appropriate to local contexts and conditions?Fit within higher level organizations goals and objectives?

What mechanisms promote fit among and between levels?How does mission differentiation fit in?

Network Conference


Mission Differentiation v. Uniformity

Do current measurement models and accountability systems promote 4-year institutions to strive to be either doctoral extensive or elite liberal arts?

or promote 2-year institutions to become 4-year institution?

Do systems and states provide appropriate incentives for defining excellence in a variety of ways?Is or can faculty culture be amenable to multiple dimensions of excellence?

Network Conference


Promoting Diversity of Excellence

U.S. News has a range of ranking categoriesNational, liberal arts, comprehensive, best value, business, engineering, programs that enhance learning

Institutions like Alverno, Truman State, Portland State, and (I’d like to think) IUPUI have ‘made the map’ on the basis of attention to student learningThese institutions are influencing their states’ conceptions and funding of higher education

Network Conference


Guidelines for Assessing HE Quality

Promote development of standards of excellence at each levelDirect accountability at each level for outcomes directly impacted by that level’s processesIndirect accountability for sub-level outcomes through demonstration of quality assurance processes

Network Conference


Guidelines (2)Collaborative review of how it all fits togetherOnce standards for excellence are negotiated and accepted, they should be accommodated in higher level goals and objectivesMake it possible that while no institution contributes as well to every outcome, every institution has a place to contribute significantlySimplification, which is necessary for effective management, should be balanced by rich detail between the lines (or within the hyperlinks)

Network Conference


Guidelines (3)Distinguish between outcome measures and measures that reflect contributions to outcomes

State-Level Measure: Access to Postsecondary EducationContributing measures (and who is most likely to contribute best)

Geographic diversity across state (selective, residential universities)Enrollment of low-income/place-bound students (regional colleges and universities)Enrollment by members of underserved populations (minority serving institutions, urban universities)Access by students with academic deficiencies (Community Colleges)

Network Conference


Concluding QuestionsHow does what we measure relate to what we hope to achieve?How does how we measure contribute to goal attainment? How do our goals and related measures align with

The goals and measures of component organizations and units?The goals and desired outcomes of constituent users and stakeholders?

Network Conference


Using Accountability to Support Quality

Documents