Conducting an Assessment Audit of General Education Courses for Purposes of Identifying Embedded Assignments Dr. Blase Scarnati Chair, Liberal Studies.

Conducting an Assessment Audit of General Education

Courses for Purposes of Identifying Embedded

AssignmentsDr. Blase Scarnati

Chair, Liberal Studies CommitteeAssociate Professor, Musicology, School of Music

Dr. Cynthia ConnAssociate Director, Office of Academic Assessment

Clinical Instructor, Educational Technology Program

Dr. Bruce FoxChair, Liberal Studies Committee Assessment

SubcommitteeDirector, Honors Program

Agenda Overview Definition of Skills Pilot Study Content Analysis Group Discussion: Ideas, Projects, and

Challenges

Flagstaff, Arizona State supported Doctoral intensive Significant distance learning

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Northern Arizona University

19,000 total students Approximately 6,000 graduate

students (3,800 FTE) Approximately 5,500 students

statewide

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Northern Arizona University

Approximately 11,400 undergraduates on the Flagstaff campus

Students come to us in a variety of ways

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Northern Arizona University

Approved Spring 1998 Fully implemented Fall 1999

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Liberal Studies Program

Coherent program Essential skills necessary for

citizenship Key themes

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Citizenship

35 credit total requirement Foundation requirements Distribution courses Courses embedded within the

academic major

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Liberal Studies Program Requirements

English composition course (4 credits)

Mathematics foundation course (3 credits)

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Foundation Requirements

Aesthetic and Humanistic Inquiry (3-6 credits)

Cultural Understanding (3-6 credits)

Laboratory Science (4 credits) Science and Applied Science

(3-6 credits) Social and Political Worlds (3-6

credits)

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Distribution Requirements (25 Credits)

Environmental Consciousness Technology and Its Impact Valuing the Diversity of

Human Experience

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Thematic Foci

Effective oral communication Critical reading Effective written communication Quantitative analysis and/or spatial

reasoning Creative thinking Critical thinking Scientific inquiry Ethical reasoning Effective use of technology

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Essential Skills

Courses embedded within the academic major

Diversity requirement Required freshman colloquium

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

University Requirements

Junior level writing course (3-4 credits)

Senior capstone experience (3 credits minimum)

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

In the Academic Major

U.S. Ethnic Diversity (3 credits) Global Awareness (3 credits)

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Diversity Requirement

Discontinued effective Fall 2004 Redesigned

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Required Freshman Colloquium (3 Credits)

Faculty body Six colleges Student representative Faculty Senate representative Administrative and programmatic

representatives

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Liberal Studies Committee (LSC)

Liberal Studies Committee Assessment Subcommittee

Liberal Studies Program Assessment Plan (Fall 2004)

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Assessment

Assess the achievement of NAU students

Effective writing Effective oral communication Critical reading

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Assessment Plan Goals (1)

Create a more cohesive curriculum of study

Analytical procedures Inform adjustments to the

curriculum

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Assessment Plan Goals (2)

To what extent are courses and embedded assignments that state they teach effective writing, critical reading, and/or effective oral communication aligned with the outcomes defined during the 2003-2004 academic year by the Liberal Studies Committee?

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Research Question 1

How should issues of non-alignment between courses and defined outcomes be addressed?

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Research Question 2

What expectations should be established for course content, assessment, and the broader curriculum?

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Research Question 3

Specifically, to enhance the cohesiveness of the curriculum, what changes should be made to the existing expectations (for faculty, departments, etc.) for submission and acceptance of courses into the Liberal Studies Program?

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Research Question 3

Courses and embedded assignments

Aligned with the outcomes Pilot Project

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Assessment Audit

Defining Skills Working definitions created by LSC

subcommittee Working definitions tentatively

approved by entire LSC Definitions “vetted” through

consultation with internal constituents, primarily departments

Consensus definitions approved by LSC

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Advantages:– Create buy in from other campus

units– Liberal Studies process seen as

open and participative– Better definitions!

Key strategy: visit folks personally. Do NOT rely solely on electronic communication

Agenda

Overview

Skills

Pilot Study

ContentAnalysis

GroupDiscussio

n

Defining SkillsAgenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Students will be able to:– Demonstrate steps in the writing

process– Tailor writing to a specific audience– Focus writing on a specific purpose– Provide logically coherent pieces of

written work– Apply general writing standards

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Effective Writing

Students will be able to: Summarize accurately and comprehensively Recognize the most significant textual and

visual elements Incorporate/use specific examples and/or

quotations Recognize patterns and valid comparatives Demonstrate an

understanding/comprehension of the material

Demonstrate evidence of an ability to analyze and evaluate texts

Demonstrate the ability to explicitly identify criteria and use these criteria accurately to evaluate a piece of written work

Critical ReadingAgenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Students will be able to:– Organize and deliver content based

on audience, location, allotted time, and purpose

– Speak with appropriate oral communications techniques

– Listen and respond thoroughly and thoughtfully to questions

– Create appropriate professional supplemental materials that reinforce the presentation

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Oral Communication

(aka “Methods of Formal Analysis”)

(aka “Quantitative Literacy”)

Suggested titles: More inclusive?

Agenda

Overview

Skills

Pilot Study

ContentAnalysis

GroupDiscussio

n

Quantitative & Spatial Analysis

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Methods of Formal Analysis Application of numerical, visual or

symbolic perspectives for the purpose of drawing inferences, understand phenomena or making predictions in situations that involve natural phenomena or real-world decision making

Agenda

Overview

Skills

Pilot Study

ContentAnalysis

GroupDiscussio

n

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Pilot Project Testing of Protocols ITERATIVE Foundation Principles

– Based on embedded assignments– Actively involves faculty– Supplements direct evidence with

indirect date sources

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Pilot Project Process

– Identify faculty teaching courses that include desired skill

– Recruit these faculty as sources of embedded assignments

– Use faculty consultants to develop and evaluate rubrics

– Use rubrics to assess level of skill mastery

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Pilot Project Process

– Collect assessment data for a stratified sample of courses

– Compile and analyze data– Triangulate using indirect data

sources– Evaluate protocols

Agenda

Overview

Skills

Pilot Study

Content Analysis

Group Discussion

Pilot Project Informing and Involving the Wider

Campus Community- Communications Plan Strategy to

inform entire campus community of assessment process in a deliberate and on-going manner

- Create a Steering Committee of on and off campus constituencies

Content Analysis Purpose

- Identify the alignment between Liberal Studies skills as they are now defined and syllabi of current Liberal Studies courses.

- Identify embedded assignments that could be used for assessment purposes

Agenda

Overview

Skills

Pilot Study

Content Analysis

Small Discussion

Full Group Discussion

Agenda

Overview

Skills

Pilot Study

ContentAnalysis

GroupDiscussion

How will findings be used?- Identify potential faculty for

participation as consultants for Pilot Study

- Target courses with strongest alignment for first phase of data collection

- Recommend policy changes regarding content of syllabi and expectations related to assessment

Agenda

Overview

Skills

Pilot Study

ContentAnalysis

GroupDiscussion

Content Analysis

Ideas: What works well at your institution with respect to assessing General Education student learning outcomes?

Challenges: What challenges concerning General Education and Assessment do you face at your institution?

Group DiscussionAgenda

Overview

Skills

Pilot Study

ContentAnalysis

GroupDiscussion

Walvoord, Barbara E. (2004). Assessment Clear and Simple : A Practical Guide for Institutions, Departments, and General Education. San Francisco: Jossey-Bass.

Neuendorf, Kimberly A. (2002). The Content Analysis Guidebook. Thousand Oaks: Sage Publications.

ReferencesAgenda

Overview

Skills

Pilot Study

Content Analysis

Small Discussion

Full Group Discussion

Contact Information

Cynthia Conn, Ph.D.Associate Director, Office of Academic Assessment

Clinical Instructor, Educational Technology ProgramP.O. Box 4091

Flagstaff, Arizona [email protected]

(928) 523-7624

Bruce Fox, Ph.D.Chair, Liberal Studies Committee Assessment Subcommittee

Director, Honors ProgramP.O. Box 5689

Flagstaff, Arizona [email protected]

(928) 523-6636

Blase Scarnati, Ph.D.Chair, NAU Liberal Studies Committee

Associate Professor, School of MusicP.O. Box 6040

Flagstaff, Arizona [email protected]

(928) 523-6537