Program Review for Academic and Administrative Departments Kristina Kauffman,

Program Review for Academic and Administrative Departmentshttp://www.rcc.edu/administration/academicaffairs/effectiveness/review.cfm

Program Review for Academic and Administrative Departmentshttp://www.rcc.edu/administration/academicaffairs/effectiveness/review.cfm

Kristina Kauffman, Associate Vice Chancellor Institutional Effectiveness

David Torres, District Dean Institutional Research

Kristina Kauffman, Associate Vice Chancellor Institutional Effectiveness

David Torres, District Dean Institutional Research

Program Review GoalsProgram Review Goals

Extrinsic Resource allocation Improvement of student learning through the

provision of systematic feedback Intrinsic

Strengthening programs through self-improvement and self-determination

Strengthening the bonds within the college community and fostering cooperation between units

Generating continuous and ongoing dialogue about how student learning can be enhanced through program and service improvements

Evaluating the unit’s contribution to achieving the district goals and strategic initiatives

Extrinsic Resource allocation Improvement of student learning through the

provision of systematic feedback Intrinsic

Strengthening programs through self-improvement and self-determination

Strengthening the bonds within the college community and fostering cooperation between units

Generating continuous and ongoing dialogue about how student learning can be enhanced through program and service improvements

Evaluating the unit’s contribution to achieving the district goals and strategic initiatives

Types of Program ReviewTypes of Program Review

InstructionalAnnualComprehensive

Student Services

Administrative

InstructionalAnnualComprehensive

Student Services

Administrative

Types of Research Support


Data = Numbers Information Services

Raw data from Datatel Reports organized around regular requests such

as registration statistics Institutional Reporting

Current term enrollments Current term FTES Distributions Current term assignments Scheduling patterns

Institutional Research Past term(s) data including all in Institutional

Reporting’s list above CCCCO Referential MIS files

Data = Numbers Information Services

Raw data from Datatel Reports organized around regular requests such

as registration statistics Institutional Reporting

Current term enrollments Current term FTES Distributions Current term assignments Scheduling patterns

Institutional Research Past term(s) data including all in Institutional

Reporting’s list above CCCCO Referential MIS files



Information = Data in a meaningful context, organized for decision making Institutional Reporting

Summaries or organized reports about the current term Institutional Research

Reports on Employment in the Region Summaries or organized reports about past terms Matriculation reports Reports on Student Engagement and/or satisfaction

Institutional Effectiveness Reports to the State, Federal Government and

Accrediting Agencies Reports such as the McIntyre Environmental Scans

Discipline Program Review Documents Information used to “lobby” administration

Information = Data in a meaningful context, organized for decision making Institutional Reporting

Summaries or organized reports about the current term Institutional Research

Reports on Employment in the Region Summaries or organized reports about past terms Matriculation reports Reports on Student Engagement and/or satisfaction

Institutional Effectiveness Reports to the State, Federal Government and

Accrediting Agencies Reports such as the McIntyre Environmental Scans

Discipline Program Review Documents Information used to “lobby” administration



Analysis = Projections or qualitative observations often resulting in recommendations aligned with the college mission, community need or transfer requirements

Institutional Research Employment projections

Institutional Effectiveness Scenarios such as those found in the McIntyre Report Statewide and local economic and demographic projections Analysis of trends in Education law and politics Analysis of trends in Accreditation requirements

Units undergoing Program Review Each unit must determine for itself which data, information and the

analysis from other respected sources is relevant for its review. Ultimately the unit must prepare its own analysis of its unit and its

future as a capstone to data, information and analysis provided by others.

Units are not expected to generate their own data or information. They are expected to be able to identify additional data or information needs and advise Institutional Effectiveness and Research of those needs. As resources permit units will be provided with data and information or Institutional Effectiveness and Research will collaborate with units in collection of data.

Analysis = Projections or qualitative observations often resulting in recommendations aligned with the college mission, community need or transfer requirements

Institutional Research Employment projections

Institutional Effectiveness Scenarios such as those found in the McIntyre Report Statewide and local economic and demographic projections Analysis of trends in Education law and politics Analysis of trends in Accreditation requirements

Units undergoing Program Review Each unit must determine for itself which data, information and the

analysis from other respected sources is relevant for its review. Ultimately the unit must prepare its own analysis of its unit and its

future as a capstone to data, information and analysis provided by others.

Units are not expected to generate their own data or information. They are expected to be able to identify additional data or information needs and advise Institutional Effectiveness and Research of those needs. As resources permit units will be provided with data and information or Institutional Effectiveness and Research will collaborate with units in collection of data.

Instructional Program Review


Comprehensive Goals – focuses on each discipline’s courses,

pedagogy, assessment plan, and future goals and objectives related to the improvement of student learning (4 year cycle).

Forms Data

Annual Goals – Resource allocation and assessment

update Forms Data

Comprehensive Goals – focuses on each discipline’s courses,

pedagogy, assessment plan, and future goals and objectives related to the improvement of student learning (4 year cycle).

Forms Data

Annual Goals – Resource allocation and assessment

update Forms Data



Data for Comprehensive (con’t) Demographic information (ethnicity,

gender, age group, enrollment status)

Course Enrollments by time of day offered

Counts of sections offered Student Retention by course Student Success rates (two

measures)

Data for Comprehensive (con’t) Demographic information (ethnicity,

gender, age group, enrollment status)

Course Enrollments by time of day offered

Counts of sections offered Student Retention by course Student Success rates (two

measures)



Data for Comprehensive (con’t) FTES generated by course Weekly Student Contact Hours by

course Full-Time Equivalent Faculty by

course Percentages of student enrollments

taught by Full-Time and Part-Time faculty by course

Data for Comprehensive (con’t) FTES generated by course Weekly Student Contact Hours by

course Full-Time Equivalent Faculty by

course Percentages of student enrollments

taught by Full-Time and Part-Time faculty by course



Data for Comprehensive (con’t) Class enrollment by course and

term Grade point averages and grade

distributions by course and term Student success rates (two

measures) by Demographics by term

Degrees and certificates awarded, if applicable


















Instructional Program Review, Data ExamplesInstructional Program

Review, Data Examples























Data for Annual Class enrollment by course and

term Retention and Success Rates for

each course FTEF, WSCH for each course

Data for Annual Class enrollment by course and

term Retention and Success Rates for

each course FTEF, WSCH for each course

Student ServicesStudent Services

Goals Resource allocation State program goals and align future goals with the

College’s mission and goals. Collect and analyze data on key performance indicators,

service area outcomes and student learning outcomes, program activities, and accomplishments.

Examine and document the effectiveness of student support services.

Develop recommendations and strategies concerning future program directions and needs (e.g. budget, staffing, and resources).

Assure the accuracy of program information. Comply with Accreditation Standards, Federal and State

law, Title 5, Student Equity, VTEA, matriculation (including prerequisite and co-requisite standards), ADA (American with Disabilities Act), and other legal or certification requirements.

Forms Data – Developed by units

Goals Resource allocation State program goals and align future goals with the

College’s mission and goals. Collect and analyze data on key performance indicators,

service area outcomes and student learning outcomes, program activities, and accomplishments.

Examine and document the effectiveness of student support services.

Develop recommendations and strategies concerning future program directions and needs (e.g. budget, staffing, and resources).

Assure the accuracy of program information. Comply with Accreditation Standards, Federal and State

law, Title 5, Student Equity, VTEA, matriculation (including prerequisite and co-requisite standards), ADA (American with Disabilities Act), and other legal or certification requirements.

Forms Data – Developed by units

Administrative UnitsAdministrative Units

Goals – Same as Student ServicesFormsData - First year the departments

were asked to develop assessment plans and begin compiling data for reasonable measures of their department’s effectiveness

Goals – Same as Student ServicesFormsData - First year the departments

were asked to develop assessment plans and begin compiling data for reasonable measures of their department’s effectiveness

Focus Group AnalysisFocus Group Analysis

Findings The review process was useful because it helped units

take stock of effectiveness and what resources were needed.

It moved units to be more comprehensive about identifying resources and tying those resources to programmatic needs.

Need more training on budget and data analysis Believe improvements are being made in teaching and

learning as a result of program review Believe that the resource allocation process is critical to

the fate of some improvements Changes as a result

Vice Presidents for Business Services engaged in working with units on annual review and budget plans

More data given at the beginning of the process Enhanced training on analysis Increasingly simplified forms

Findings The review process was useful because it helped units

take stock of effectiveness and what resources were needed.

It moved units to be more comprehensive about identifying resources and tying those resources to programmatic needs.

Need more training on budget and data analysis Believe improvements are being made in teaching and

learning as a result of program review Believe that the resource allocation process is critical to

the fate of some improvements Changes as a result

Vice Presidents for Business Services engaged in working with units on annual review and budget plans

More data given at the beginning of the process Enhanced training on analysis Increasingly simplified forms

Integration with Budget and Planning

Integration with Budget and Planning

Campus Strategic Planning

Executive Committee*

Instructional Programs

Subcommittee

Student Support Services

Subcommittee

Library ResourcesSubcommittee

Human Resources Subcommittee

Physical Resources

Subcommittee

Technology Resources

Subcommittee

Institutional Mission &

EffectivenessSubcommittee

Campus Strategic Planning

Committee(Academic Senate

appoints faculty membership)

Sample Campus Riverside City Strategic Planning Process

With Feedback Loops

* Membership - Vice President Academic Affairs, Vice President

Student Services, Vice President Business Services, member of CSEA, and the Co-Chairs of Academic Planning Council Purpose – Disseminate program review data to the appropriate strategic planning subcommittees

FinancialResources

Subcommittee

District Strategic Planning

Committee

Campus Based Annual Instructional Program

ReviewDocuments

District-Wide Comprehensive


Documents (completed every 4 years)

Campus Administrative Unit Annual Program Review Documents

Campus Student Services Annual Program Review

Document

District Wide Program Review

Committee Facilitates

Development of and Organizes

Program Review Documents

District Assessment

Committee works collaboratively with PRC to assure all units are engaged

in SLO or AUO assessment

Vice PresidentDistributes Documents From

PRC to appropriate Campus Committees

Academic Planning Council

Reviews Requests for Faculty Positions

Campus President ForwardsAppropriate Recommendations to DSPC

Norco Campus Ranking Criteria

Norco Campus Ranking Criteria

Criterion Description Points

Enrollment Growth Potential for new position to contribute to enrollment growth as evidenced by high ratio of WSCH:FTEF or other documentation

20

Student Persistence

Potential for new position to contribute to increased term-to-term persistence, as evidenced by high current persistence rates (or proxy measures, such as student success rates) in discipline or potential for improvement in persistence associated with adding a full-time position. Also applicable to positions in counseling, basic skills and support services.

20

Increased Student Success

Potential for new position to contribute to increased student success as evidenced by high student success rates in discipline or the potential for a new faculty position to contribute to higher success rates.

20

Student Learning Outcomes

Maximum points awarded to disciplines (service areas) that have both identified SLOs and begun to measure their achievement. Mid-range points awarded to those that have defined SLOs but not yet begun to measure them. Zero points to disciplines and service areas that either have not defined SLOs or have not turned in program reviews.

10

Improved Quality of Student Experience

A somewhat subjective criterion designed to address disciplines and service areas for which quantitative data pertaining to above criteria are not readily available. Points awarded based on the extent to which requesting unit has demonstrated that adding the position requested will enhance the quality of the student experience at Norco.

10

Current Full-time Staffing

Extent to which department or service area depends upon part-time positions to deliver services. Maximum points awarded for disciplines with no full-timers; near maximum points awarded for those with only one FT position currently.

20

Accreditation ResponseAccreditation Response

Continue with current program review, add SLOs for all programs (we have them for all courses and GE)

Need better integration with budget (Norco process was in first year of cycle)

Continue with current program review, add SLOs for all programs (we have them for all courses and GE)

Need better integration with budget (Norco process was in first year of cycle)

Attitude Change: Empowerment

Attitude Change: Empowerment

Units get advice about lobbying for resources

Units are provided extensive support while going through the process, particularly in the beginning

Every round of program review utilizes increasing analytical skills Units analyze their own data to improve

student learning and enrollment management Units can confirm or refute operating

assumptions

Units get advice about lobbying for resources

Units are provided extensive support while going through the process, particularly in the beginning

Every round of program review utilizes increasing analytical skills Units analyze their own data to improve

student learning and enrollment management Units can confirm or refute operating

assumptions

Program Review for Academic and Administrative Departments Kristina Kauffman,

Documents

organized reports

datatel reports

terms data

institutional reportings

types of research support

additional data

collection of data

administration information