CARTA Assessment Retreat

CARTA Assessment Retreat

May 4, 2012

Office of Academic Planning and Accountability (APA)

Florida International University

Introduction

• Susan Himburg, Director of Accreditation, APA

• Mercedes Ponce, Director of Assessment, APA

• Katherine Perez, Associate Director of Assessment, APA

• Barbara Anderson & Claudia Grigorescu, GAs, APA

Assessment in FIU

Goal Setting Activity

What would you like to learn and accomplish in

this retreat?

What would you like to accomplish within your

unit to enhance your current assessment

processes?

Retreat Goals Unit Goals

Retreat Agenda

9:00 a.m. – 9:15 a.m. IntroductionContinental BreakfastWelcome

9:15 a.m. – 10:00 a.m. Assessment in FIUActivity: Retreat Goal SettingOverview of AssessmentSLOs, POs, ALCs, CCOs, Global LearningTracDat Basics

10:00 a.m. – 11:00 a.m. Matrixes I: Effective OutcomesStreamlining Outcomes with Program GoalsTying Outcomes to CurriculumActivity: Writing SLOs and POs

11:00 a.m. – 12:00 p.m. Matrixes II: Effective MethodsChoosing InstrumentsIntroduction to RubricsActivity: Creating a RubricUsing Results for ImprovementsActivity: Developing Improvement Strategies

12:00 p.m. – 1:00 p.m. Q & A SessionLunch

Overview of Assessment: Structure

• Assessment Cycles

• Assessment Review Process

• Types of Assessments

– Programs

– Colleges/Schools

– Administrative Units

• Continuous Improvement

– Assessment-based improvement strategies

– Documenting improvement strategies

– Coding and analyzing improvement strategies

Overview of Assessment: Cycle

Outcomes

Assessment Methods

Data Collection

Analysis of Results

Improvement Strategies

• Use program mission and goals to help identify outcomes• Use SMARTER Criteria for creating outcomes

Step 1: Identify Specific Outcomes

• Determine how to assess the learning outcomes within the curriculum (by curriculum mapping)

Step 2: Determine Assessment Methods

• Collect evidence

Step 3: Gathering Evidence

• Organize and process data• Discuss significance of data and report it

Step 4: Review & Interpret Results

• Collaborate with faculty to develop improvement strategies based on results

Step 5: Recommend Improvement Actions

• Follow-up on improvement strategies, implement them, and report progress

• Restart the cycle to assess the impact of actions

Step 6: Implement Actions and Reassess

Overview of Assessment: Timeline

Deadline Task Due

Fall2012 Oct 15 2010 – 2012 Complete Reports

2012 – 2014 Plans

Summer2013 May 15 2012- 13 Interim Reports

(results only)

Fall2014 Oct 15 2012 – 2014 Complete Reports

2014 – 2016 Plans

Cycle A

Overview of Assessment: Assessment Committee

• The University Assessment Committee is composed of two branches

– Academic

– Administrative

• Role– Represent their academic and administrative units – Provide their unit with assessment guidance

• Best practices• Deadlines• Connecting faculty & staff to appropriate assessment resources

– Enhance the culture of assessment at FIU– Engage in dialogue with fellow experts to improve

assessment practices

Overview of Assessment: Types of Assessment

• Academic Programs• Student Learning Outcomes (SLOs)

• All academic programs• Miami-based programs• On-site, off-shore, and distance learning programs

• Academic Learning Compacts (ALCs)• Program Outcomes (POs)

• All academic programs• Miami-based programs• On-site, off-shore, and distance learning programs

• Core Curriculum Outcomes (COs)• Global Learning (GL)

• Administrative Assessments for Academic Units• Assessment Units Included

• Dean’s Offices• Centers/Institutes

Institutional Assessment of Learning

Program Level Outcomes

Student Learning

Outcomes (SLOs)

Skills Assessed:Content

Knowledge,Critical Thinking,Communication

Skills,Technology

Academic Learning Compacts (ALCs)

Required by the FLBOG for each baccalaureate

degree program

Curriculum Maps

Course Level Outcomes

Core Curriculu

m Outcomes

(CCOs)

Outcomes aligned to one of the core

standards

Global Learning Outcomes (GL)

Specialized GL approved courses

Outcomes aligned to each of the 3

learning outcomes

Student Learning Outcomes (SLOs)

• Student Learning Outcomes (SLOs) is a program related outcomes• SLOs focus on students’ knowledge and skills expected upon completion of an

academic degree program• “A learning outcome is a stated expectation of what someone will have

learned” (Driscoll & Wood, 2007, p. 5)• “A learning outcome statement describes what students should be able to

demonstrate, represent, or produce based on their learning histories” (Maki, 2004, p. 60)

• “A learning outcome describes our intentions about what students should know, understand, and be able to do with their knowledge when they graduate” (Huba & Freed, 2000, p. 9-10)

• What should my students know or be able to do at the time of graduation?

Global Learning Outcomes (GL)

Global Learning for Global Citizenship = FIU's Quality Enhancement Plan (QEP)• Multi-year initiative enables students to act as engaged global citizens • Integrated global learning curriculum and co-curriculum • Minimum of two GL-designated courses for Undergraduate students

These are the GL outcomes: Global Awareness: Knowledge of the interrelatedness of local, global,

international, and intercultural issues, trends, and systems Global Perspective: Ability to develop a multi-perspective analysis of local,

global, international, and intercultural problems Global Engagement: Willingness to engage in local, global, international,

and intercultural problem solving

Program Learning Outcomes (POs)

• Program Outcomes (POs) focus on expected programmatic changes that will improve overall program quality for all stakeholders (students, faculty, staff)• Program outcomes illustrate what you want your program

to do. These outcomes differ from learning outcomes in that you discuss what it is that you want your program to accomplish. (Bresciani, n.d., p. 3)

• Program outcomes assist in determining whether the services, activities, and experiences of and within a program positively impact the individuals it seeks to serve.

• Emphasizes areas such as recruitment, professional development, advising, hiring processes, and/or satisfaction rates.

• How can I make this program more efficient?

Administrative Assessment (AAs)

• Administrative Areas Dean’s Office Centers/Institutes

• Outcomes aligned to: Unit mission/vision Annual goals University mission/vision Strategic plan

• Outcomes focus on each of the following areas (all 4 required for Dean’s Office):

Administrative Support Services Educational Support Services Research Community Service

• Student learning is also assessed for units providing learning services to students (e.g., workshops, seminars, etc.)

Assessment Tracking

Microsoft WordForms

TracDat: http://intranet.fiu.edu/tracdat/

http://intranet.fiu.edu/tracdat/

http://intranet.fiu.edu/tracdat/

Matrixes I:Effective Outcomes


SMARTER Criteria• Specific – Is the expected behavior and skill clearly

indicated?• Measureable – Can the knowledge/skill/attitude be

measured?• Attainable – Is it viable given the program courses and

resources?• Relevant – Does it pertain to the major goals of the

program?• Timely – Can graduates achieve the outcome prior to

graduation?• Evaluate – Is there an evaluation plan?• Reevaluate – Can it be evaluated after improvement

strategies have been implemented?


Student Learning Outcomes (cognitive, practical, or affective)

1. Can be observed and measured2. Relates to student learning towards the end of the program (the graduating student)3. Reflects an important higher order concept

Formula: Who + Action Verb + What

Theater majors will analyze and compare the relationships among the elements of theatrical performance: writing, directing, acting, design, and the audience function.


• Theater majors will analyze and compare the relationships among the elements of theatrical performance: writing, directing, acting, design, and the audience function.

• Students will be able to calculate equivalent exposures, using F-stop and shutter speeds.

• Students will critically analyze building designs and conduct post occupancy evaluation studies.

• Graduates will understand and evaluate interpretations of any of a variety of man-made forms in Western civilization from pre-history to Imperial Rome.

Strong Examples

• Students will be able to demonstrate skills from their art form. • Musicians tend to be creative, in tune with their minds, bodies, and emotions.• Appreciate the social, political, religious, and philosophical contexts of art

objects.• Graduates will demonstrate a basic knowledge of artistic media and

performance styles from both western and non-western traditions.• Develop an awareness of the cultural and historical dimensions of the wide

variety of man-made forms in the period covered by this class.

Weak Examples

Program Outcomes (POs)

Program Outcomes (efficiency measures)

1. Can be observed and measured2. Related to program level goals that do not relate to student learning (e.g., student services, graduation, retention, faculty productivity, and other similar

Formula: Who + Action Verb + What

Full-time students will graduate from the Art History program within 6 years of program admission.

Program Outcomes (POs)

• The department’s advising office will schedule student appoints within 2 weeks of initial contact.

• Students will be satisfied with services provided by the career placement office in the Department of Architecture.

• Faculty in Music will be involved with a minimum of 2 public events per semester.

Strong Examples

• Graduation rates will increase. • Surveys will be used to assess student satisfaction.• Career services will work with student placements.

Weak Examples

Activity

Writing SLOs and POs

Streamlining Outcomes with Program Goals

Goals

Accreditation

Course Outcomes

Program Mission and Goals•Question: Do the mission and goals match the knowledge/skills expected for graduates?

•Task: Break down mission and goals; Verify these are reflected in the outcomes.

Accreditation Principles•Question: What are the competencies required for assessment and how do they match my program mission/goals?•Task: Review required competencies for accreditation or other constituencies; Streamline requirements and outcomes.

Course Outcomes•Question: How are the program’s learning outcomes reflected in the courses?•Task: Review course syllabi and outcomes to check for alignment; Develop a curriculum map.

Tying Outcomes to Curriculum: Curriculum Maps

Curriculum maps help identify where within the curriculum learning outcomes are addressed and provide a means to determine whether the

elements of the curriculum are aligned.

Planning

Curriculum Learning Outcomes

Identifying

Gaps ImprovementAreas Measures


Make Changes as AppropriateIf there are any gaps in teaching or assessing learning outcomes

Create a Curriculum MapCourses in one axis and learning outcomes in the other

Identify Major Assignments within CoursesDiscuss how accurately they measure the learning outcomes

Collaborate with Faculty and Staff MembersDelineate where the learning outcomes are taught, reviewed, reinforced, and/or evaluated within each of the required courses

Collect All Relevant or Required InformationEX: Course syllabi, curriculum requirements, and major learning competencies


Competency/Skill Introductory Course

Methods Course

Required Course 1

Required Course 2

Required Course 3

Required Course 4

Capstone Course

Content SLO 1 Introduced Introduced Reinforced Reinforced Mastery/Assessed

Content SLO 2 Introduced Reinforced Introduced Reinforced Mastery/Assessed

Content SLO 3 Introduced Introduced Reinforced Mastery/Assessed

Critical Thinking SLO 1 Introduced Introduced Reinforced

Critical Thinking SLO 2 Introduced Introduced Mastery/Assessed

Communication SLO 1 Introduced Reinforced Mastery/Assessed

Communication SLO 2 Introduced Mastery/Assessed

Integrity / Values SLO 1 Introduced Reinforced Reinforced Mastery/Assessed

Integrity / Values SLO 2 Introduced

•Introduced = indicates that students are introduced to a particular outcome•Reinforced = indicates the outcome is reinforced and certain courses allow students to practice it more•Mastered = indicates that students have mastered a particular outcome•Assessed = indicates that evidence/data is collected, analyzed and evaluated for program-level assessment

*Adapted from University of West Florida, Writing Behavioral, Measurable Student Learning Outcomes CUTLA Workshop May 16, 2007.

Matrixes II:Effective Methods

Choosing Assessment Measures/Instruments

1. Identify Assessment Needs

• What are you trying to measure or understand? Every thing from artifacts for student learning to program efficiency to administrative objectives.

• Is this skill or proficiency a cornerstone of what every graduate in my field should be able know or do?

2. Match Purpose with Tools

• What type of tool would best measure the outcome (e.g., assignment, exam, project, or survey)? • Do you already have access to such a tool? If so, where and when is it collected?

3. Define Use of Assessment Tool

• When and where do you distribute the tool (e.g., in a capstone course right before graduation)? • Who uses the tool (e.g., students, alumni)?• Where will the participants complete the assessment?• How often do you use or will use the tool (e.g., every semester or annually)?

Understanding Types of Measurements

• Direct versus Indirect Measures• Direct Measure: Learning assessed using tools that measure direct observations of

learning such as assignments, exams, and portfolios; Precise and effective at determining if students have learned competencies defined in outcomes

• Indirect Measure: Learning assessed using tools that measure perspectives and opinions about learning such as surveys, interviews, and evaluations; Provide supplemental details that may help a program/department understand how students think about learning and strengths/weaknesses of a program

• Program Measures versus Course Measures• Program Measure: Provides data at the program level and enables department to

understand overall learning experience; Includes data from exit exams and graduation surveys

• Course Measure: Provides data at the course level and enables professors to determine competencies achieved at the end of courses; Includes data from final projects/presentations and pre-post exams

• Formative Measures versus Summative • Formative Measures: Assessing learning over a specific timeline, generally

throughout the academic semester or year • Summative Measures: Assessing learning at the end of a semester, year or at

graduation

Examples of Measures/Instruments

Course Level•Essays•Presentations•Minute papers•Embedded questions•Pre-post tests

Program Level•Portfolios•Exit exams•Graduation surveys •Discipline specific national exams

Direct Measures

• Standardized exams• Exit examinations• Portfolios • Pre-tests and post-tests• Locally developed exams• Papers• Oral presentations• Behavioral observations• Thesis/dissertation• Simulations/case studies• Video taped/audio taped assignments

Indirect Measures

• Surveys or questionnaires• Student perception• Alumni perception• Employer perception

• Focus groups• Interviews• Student records

Institution-Level Assessments

NSSEFSSE

Proficiency Profile

(Kuh & Ikenberry, 2009, p. 10)

1. Graduating Master’s and Doctoral Student Survey

2. Graduating Senior Survey3. Student Satisfaction Survey4. Global Learning Perspectives Inventory

Alumni Survey

Case Response

Assessment

Introduction to Rubrics

Definition

• Rubrics are tools used to score or assess student work using well-defined criteria and standards.

Common Uses

• Evaluate essays, short answer responses, portfolios, projects, presentations, and other similar artifacts.

Benefits

• Learning expectations clear for current and future faculty teaching the course• Transparency of expectations for students • Providing meaningful contextual data as opposed to only having grades or scores• Providing students with clearer feedback on performance (if scored rubrics are handed back to students)• Useful for measuring creativity, critical thinking, and other competencies requiring deep multidimensional skills/knowledge• Increase of inter-rater reliability by establishing clear guidelines for assessing student learning• Possibility of easy, repeated usage over time• Inexpensive development and implementation

Steps for Developing Rubrics

1. Identify Competencies •Narrow down the most important learning competencies you are trying to measure. Ask yourself what you wanted students to learn and why you created the assignment. •List the main ideas or areas that would specifically address the learning competencies you identified.

2. Develop a Scale •Think of the types of scores that would best apply to measuring the competencies (e.g., a 5 point scale from (1)Beginning to (5)Exemplary). •Scales depend on how they would apply to the assignment, the competencies addressed, and the expectations of the instructor.

3. Produce a Matrix•Using the information gathered from the previous two steps, you can create a matrix to organize the information. •Optional: describe the proficiencies, behaviors, or skills each student will demonstrate depending on the particular criterion and its associated performance scale ranking or score.

Rubric Template

List All of the Competencies Measu

red

Performance Scale

1

Unacceptable2

Acceptable3

Excellent POINTS

Criterion 1

Competency not demonstrated

Competency demonstrated

Competency demonstrated at an

advanced level

Criterion 2 Competency not demonstrated



advanced level

Criterion 3 Competency not demonstrated



advanced level

AVERAGEPOINTS

Reporting Results

Summary of Results

Format Narrative Tables or charts

Analysis/Interpretation of results Explain results in a narrative form

by interpreting results or using qualitative analysis of the data.

Every student learning outcome must have at least:

One set of results One student learning improvement

strategy (use of results)

Reporting Results

Non-Examples:1. Our students passed the dissertation defense on the first attempt. 2. All the students passed the national exam.3. Criteria met.

Examples:1. 75% of the students (n=15) achieved a 3 or better on the 5 rubric

categories for the capstone course research paper. Average score was: 3.45

2. Overall, 60% of students met the criteria (n=20) with a 2.65 total average. The rubric’s 4 criteria scores were as follows:

o Grammar: 3.10 (80% met minimum criteria)o Research Questions: 2.55 (65% met minimum criteria)o Knowledge of Topic: 2.50 (55% met minimum criteria)o Application of Content Theories: 2.45 (60% met minimum criteria)

Reporting Results

Frequency of Student Results for all Four Categories of the Research Paper (N=20 Students)

1 NOVICE

2APPRENTICE

3PRACTITIONER

4EXPERT

TOTAL MEETING CRITERIA

Grammar N=2 (10%)

N=2 (10%) N=8 (40%) N=8 (40%)

3.10 average (62 points)80% (n=16) met criteria

Essay Structure

N=4 (20%)

N=3 (15%) N=11 (55%) N=2 (10%)


Coherence of Argument

N=2 (10%)

N=7 (35%) N=10 (50%) N=1 (5%)


Research Based Evidence

N=3 (15%)

N=5 (25%) N=12 (60%) N=0 (0%)


AVERAGE TOTAL

2.65 average score65% (n=11) met criteria

Reporting Results: Formulas

N = 20 (students) 1 2 3 4 TOTAL MEETING CRITERIA

Grammar N=2 (10%)2/20 = .10.10 (100) = 10%

N=2 (10%)2/20 = .10.10 (100) = 10%

N=8 (40%)8/20 = .40.40 (100) = 40%

N=8 (40%)8/20 = .40.40 (100) = 40%

3.10 average (62 points)2(1) + 2(2) + 8(3) + 8(4) = 62 62/20 = 3.1080% (n=16) met criteria40% + 40% = 80% (8+8=16)

Essay Structure N=4 (20%)4/20 = .20.20 (100) = 20%

N=3 (15%)3/20 = .15.15 (100) = 15%

N=11 (55%)11/20 = .55.55 (100) = 55%

N=2 (10%)2/20 = .10.10 (100) = 10%

2.55 average (51 points)4(1) + 3(2) + 11(3) + 2(4) = 5151/20 = 2.5565% (n=13) met criteria55% + 10% = 65% (11+2=13)

Coherence of Argument

N=2 (10%)2/20 = .10.10 (100) = 10%

N=7 (35%)7/20 = .35.35 (100) = 35%

N=10 (50%)10/20 = .50.50 (100) = 50%

N=1 (5%)1/20 = .05.05 (100) = 5%


Research Based Evidence

N=3 (15%)3/20 = .15.15 (100) = 15%

N=5 (25%)5/20 = .25.25 (100) = 25%

N=12 (60%) 12/20 = .60.60 (100) = 60%

N=0 (0%)0/20 = 00 (100) = 0%


AVERAGE TOTAL 3.10 + 2.55 + 2.50 + 2.45 = 10.610.6/4 = 2.65

2.65 average score

80% + 65% + 55% + 60% = 260260/4 = 65%16 + 13+ 11+ 12= 4343/4 =10.75 = 11

65% (n=11) met criteria

Using Results for Improvements

DO DON’T•DO focus on making specific improvements based on faculty consensus.

•DON’T focus on simply planning for improvements or making improvements without faculty feedback.

•DO focus on improvements that will impact the adjoining outcome.

•DON’T focus on improvements that are unrelated to the outcome.

•DO use concrete ideas (e.g., include specific timelines, courses, activities, etc.).

•DON’T write vague ideas or plan to plan.

•DO state strategies that are sustainable and feasible.

•DON’T use strategies that are impossible to complete within two years considering your resources.

•DO use strategies that can improve the curriculum and help students learn outside of courses.

•DON’T simply focus on making changes to the assessment measures used.

• Mandate or create new courses• Eliminate/merge course(s)• Change degree requirements

Curriculum Changes

• Change course descriptions• Change syllabi to address specific learning outcomesCourse Objectives

• Add new assignments to emphasize specific competencies• Increase time spent teaching certain content• Change themes, topics, or units

Within Course Activities

Using Results for Improvements: Student Learning

Using Results for Improvements: Student Learning

• Use outside resources to enhance student learning (e.g. refer students to the Center for Academic Excellence)University Resources

• Publish or present joint papers• Provide feedback on student work, advising, office hours• Disseminate information

(e.g. distributing newsletters, sharing publications, etc.)

Faculty- Student Interaction

• Create/maintain resource libraries(e.g. books, publications, etc.)

• Offer professional support or tutoring• Provide computer labs or software

Resources for Students

Using Results for Improvements: Program Outcomes

•Obtain financial resources: funding, grants, etc.•Hire new faculty/staff•Reduce Spending

Financial

•Change recruiment efforts/tactics•Increase enrollmentEnrollment

•Change policies, values, missions, or conceptual frameworks of a program or unitPolicy Changes

•Add or expand services to improve quality•Add or expand processes to improve efficiencyServices

Using Results for Improvements: Program Outcomes

•Conduct research•Gather and/or disseminate information•Produce publications or presentations

Research and Information

•Create professional development opportunities•Attend professional conferences or workshops

Professional Development

•Establish collaborations across stakeholders or disciplines•Provide services or establish links to the community Engagement

•Acquire new equipment, software, etc.•Provide resources to specific groupsResources

Activity

Developing Improvement Strategies

Q & A Session

Thank you for attending.

Contact Us:Katherine [email protected]

Departmental Information:[email protected] 112

mailto:[email protected]

mailto:[email protected]

CARTA Assessment Retreat

Documents

culture of assessment

assessment practices

cycle8overview of assessment

current assessment processes

learning outcome statement

learning histories maki

program goalstying outcomes

retreat agenda9