Program Evaluation and Outcomes Assessment: Channels to Quality Assurance February 9, 2008 ACCA 4 th National Conference Savannah, GA PRESENTER Laura A.

Program Evaluation and Outcomes Assessment:

Channels to Quality AssuranceFebruary 9, 2008

ACCA 4th National Conference

Savannah, GA

PRESENTER

Laura A. Dean, Ph.D., NCC

University of Georgia

ACCA Representative to CAS

Determining Quality in Higher Education

Rankings and Ratings Accreditation Licensure – Certification Follow-Up Studies Self-Studies Program Review/Evaluation Outcomes Assessment

Assessment/Evaluation/Research

Assessment is a snapshot of current data – what is happening?

Evaluation involves a judgment – is that good or does it need improvement?

Assessment & evaluation are focused on improvement of local situations

Research is designed to be generalizable to other settings

Program Evaluation

Is the program or service functioning effectively to achieve its mission?

What evidence is available to support the determination?

How is evidence used to make program decisions?

CAS as one tool for self-assessment of programs and services

Outcomes Assessment What is the effect of our work on students? How are they different as a result of

interacting with our programs or services? How do we demonstrate their learning? Need to develop measurable learning

outcomes Sources include CAS standards, Learning

Reconsidered, locally identified goals & outcomes

Comprehensive assessment plan

A comprehensive plan includes assessment of programs and services and of student outcomes

Outcomes are one element of a program evaluation

Why use CAS for program evaluation?

Nationally developed and endorsed standards

Approved by consortium of 35+ professional associations in higher ed.

Don’t reinvent the wheel – the work has been done for you

Materials support the process

CAS Standards Format 13 Component Parts

1. Mission 2. Program 3. Leadership 4. Organization & management 5. Human resources 6. Financial resources 7. Facilities, technology, & equipment 8. Legal responsibilities 9. Equity and access

10. Campus and community relations

11. Diversity12. Ethics13. Assessment & evaluation

Counseling Services Standard

& Guideline Example

STANDARDBOLD TYPE; AUXILIARY VERBS MUST & SHALLCounseling services must be a) intentional, b) coherent, c) based on theories and knowledge of counseling, learning, and human development, d) reflective of developmental and demographic profiles of the student population, and e) responsive to the needs of individuals within a higher education setting.

GUIDELINE LIGHT- FACED TYPE; AUXILIARY VERBS SHOULD & MAYCounseling services should provide consultation, supervision, and in-service professional development for faculty members, administrators, staff and student staff members, and

paraprofessionals.

The CAS SAG: A Self-Assessment Guide Translates CAS standards into an

effective workbook format

Promotes program self-assessment and development

Informs on program strengths and weaknesses

Supports professional staff development

Leads to enhanced student learning and development

Conducting the Self-Assessment Process: Team

Actions Decide whether to include guidelines or other

measures that go beyond the standards Gather and analyze relevant quantitative and

qualitative data Individuals rate each and every criterion measure Obtain additional documentary evidence if

required to make an informed team decision

Complete the assessment, ratings, & action plan worksheets

Criterion Measure Rating Scale

ND 1 2 3 4 NRNot Done Not met Minimally Well Fully Not Rated

Met Met Met

Using this scale, consider each criterion statement and decide…

•The extent to which each criterion measure has been met by the Counseling Service

Assessment Criteria Example

Part 6: FINANCIAL RESOURCESCounseling Services (CS) must have adequate funding to accomplish its mission and goals.

Funding priorities must be determined within the context of the stated mission, goals, objectives, and comprehensive analysis of the needs and capabilities of students and the availability of internal or external resources.

CS must demonstrate fiscal responsibility and cost effectiveness consistent with institutional protocols.

ND 1 2 3 4 NR Not Done Not Met Minimally Met Well Met Fully Met Not Rated

PART 6. FINANCIAL RESOURCES (Criterion Measures) Rating Scale NOTES6.1 The program has adequate funding to accomplish its mission and goals. ND 1 2 3 4 NR

6.2 Funding priorities are determined within the context of program mission, student needs, and available fiscal resources. ND 1 2 3 4 NR

6.3 The program demonstrates fiscal responsibility and cost effectiveness consistent with institutional protocols. ND 1 2 3 4 NR

Part 6: Financial Resources Overview Questions

A. What is the funding strategy for the program?

B. What evidence exists to confirm fiscal responsibility and cost-effectiveness?

Examples of Evaluative Evidence

STUDENT RECRUITMENT MATERIALS: Brochures & other program information Participation policies & procedures

PROGRAM DOCUMENTS: Mission statements; program purpose & philosophy statements Catalogs & related materials Staff & student manuals; policies & procedures statements

ADMINISTRATIVE DOCUMENTS: Organization charts; student & staff profiles Financial resource statements & budgets Annual reports

STAFF ACTIVITY REPORTS: Curriculum vitae & resumes; professional activity Service to other programs, departments, or community

STUDENT ACTIVITY REPORTS: Portfolios, developmental transcripts, resumes Reports of student service

RESEARCH & EVALUATION DATA: Needs assessments & self-studies Program evaluation; graduate & follow-up studies

Qualitative Data

Focus group information

Written summaries of responses to open-ended questions in interviews and on evaluations

Client satisfaction surveys, self-reports, and written comments, both solicited and unsolicited

Quantitative Data

Summaries of objective responses on a questionnaire or program evaluation

Statistics about use by students and various population subgroups

Needs assessments, follow-up studies, and self-study reports

Institutional research reports and fact books

Complete Each Work Form

Work Form A: Assessment, Ratings, and Significant Items

Work Form B: Follow-Up Actions

Work Form C: Summary Action Plan

Action Plan Components

Program Action Plans Identify responsible parties to

complete the action steps & timeline Summary action plan creates an

overview of the work to be done

PLAN For the Future

Closing the loop Purpose & scope of study is defined Self-study team is trained Self-study is conducted Evaluative evidence is collected Discrepancies are identified Appropriate actions are determined Special actions for program enhancement

are recommended Action plan is developed and communicated

Outcomes Assessment May be various kinds of outcomes Learning outcomes are the primary

emphasis of the field currently, driven by pressures for accountability

If goal is reaching more students, then tracking participation may be useful

Important to map desired outcomes to institutional mission, division mission & goals, department mission & goals

Map the linkages

Developing learning outcomes

Define the broad goal Define the objective Define the criteria Identify strategies Assess the results

Learning Outcomes Students will develop intrapersonal competence Students will demonstrate enhanced self-

esteem Students will recognize and demonstrate

appropriately assertive behavior Workshop on assertive behavior will be offered

twice this semester; attendance goal is 30 students total; attainment of the objective is defined as 80% of participants correctly identifying examples of assertiveness and being able to role-play successfully

continued

Assessing Outcomes Assessment:

Formative - recording % of participants who correctly identify examples and role-play successfully, as observed by workshop presenter

Summative - responses to an evaluation form that includes self-report of learning

Follow-up - observations, self-reports, or later survey

Program: Learning and Development Outcome Domains

Intellectual growth Effective communication Enhanced self-esteem Realistic self-appraisal Clarified values Career choices Leadership development Healthy behavior Meaningful interpersonal

relationships

Independence Collaboration Social responsibility Satisfying and

productive lifestyles Appreciating diversity Spiritual awareness Personal and educational

goals

Identify areas of focus for the program or service

Learning Reconsidered Outcome Domains Cognitive complexity Knowledge acquisition, integration,

& application Humanitarianism & interpersonal

competence Civic engagement Intrapersonal competence Practical competence

Frameworks for Assessing Learning and Development Outcomes

“FALDOs” – companion to CAS standards book

Chapters for each CAS learning outcome domain with a theoretical description of the domain (e. g., leadership development, social responsibility, career choices), assessment examples, list of possible instruments, and additional resources

Published as book & CD set

Where to start?? Start small Start doable Start narrow Start targeted Start.

One program, one workshop, one desired outcome, one measurement tool

Mapping Outcomes

In a more developed approach, a department or division may adopt a set of desired outcomes

Each specific department, program, etc. plays a role in contributing to the overall achievement

A chart can be developed to map where each of the desired outcomes will be targeted and assessed

The goal is not that everyone individually does everything; the goal is that everyone collectively addresses everything

Another approach is to develop a sequence of outcomes across several years, so that all are addressed in that span of time

Additional CAS Resources

www.cas.edu Standards book – revised in 2006

Prologue & Context with “How to” information CAS Statement of Shared Ethical Principles Characteristics of Individual Excellence Glossary & FAQs

FALDOs – book & CD formats as package SAGs – CD or individually