Program Annual Assessment Report - Morningside College · the externship/internship/research project, experience challenges reflection paper, capstone research paper, and capstone

2019 AAFS Programmatic Assessment - 1

Program Annual Assessment Report

Program: Applied Agricultural and Food Studies (BS and BA) Date: July 1, 2019

Please use a separate copy of the table and three prompts below for each outcome assessed this year.

Outcome: Program-Specific Outcome # 1

Students will demonstrate an understanding of

Midwestern agricultural systems.

Reviewers:

Dr. Thomas H. Paulsen

Dr. Annie Kinwa-Muzinga

Mr. Richard Crow

Mr. Dan Witten

Ms. Melissa Nelson

Ms. Dee McKenna

Number of Artifacts

Collected:

39 Written Papers

13 Oral Presentations

Number of Artifacts

Reviewed:

39 Written Papers

13 Oral Presentations

How were the artifacts reviewed selected:

A census of graduating senior Externship Supervisor

Written Evaluations were reviewed for baseline data in the

department’s second year of Programmatic Outcome

Assessment. The department garnered a 100% response

rate from graduating senior program completers.

Describe how the artifacts were used to assess student learning. (If a rubric was used please attach it to the

report.):

Introduction

Established in 2014 through a generous donation from Regina Roth, Siouxland philanthropist and cofounder/

executive at Beef Products, Inc., now empirical Foods Inc., the Applied Agricultural and Food Studies (AAFS)

curriculum was designed by an interdisciplinary cross-section of Morningside College faculty under the advisement

of agricultural industry leaders to meet the needs of the modern agricultural industry. A unique feature of the

curriculum is the full-time, academically integrated, working semester (the Externship) which typically occurs in the

fall of the senior year through collaboration with area, regional (and eventually national and international)

agricultural businesses for students desiring the BA and BS degree in AAFS.

As designed and approved in the program assessment plan, a culminating capstone paper is completed in AGRI 480,

the program’s capstone course, where students address the following Institutional Assessment requirements: Written

Communication/Writing Proficiency, Oral Communication, Integration/Synthesis in the program, Critical Thinking

in the program. Additionally the capstone paper addresses the following Essential Skills in the program: Writing in

the discipline Literature Review (Type 1) and Descriptive Writing (Type 2), Integration/Synthesis in the discipline,

and Critical Thinking in the program. Finally, the capstone paper is used as one of the artifacts for two AAFS

Program-specific Outcomes: 1) demonstrating an understanding of Midwestern agricultural systems, and 2)

formulating and critiquing business/management plans for real world situations. Externship Coordinator written

evaluations serve as the artifact for student modeling of “work habits and knowledge for common agriculture work

conditions”. Ethical reasoning in the program has previously been assessed through an artifact produced in PHIL

308: Applied Agricultural Ethics. The purpose of this report is to provide the baseline AAFS Program Annual

Assessment Report for Outcome #1: Students will demonstrate an understanding of Midwestern agricultural

systems.


Procedures

Artifacts

The AAFS Outcomes Assessment Plan calls for the Senior Capstone Paper and the recorded Senior Capstone

Presentation developed, submitted, and graded as part of AGRI 480: Capstone in Applied Agricultural and Food

Studies course to be used as the artifacts for measuring Programmatic Outcome #1.

The Senior Capstone Paper and recorded Senior Capstone Presentation artifacts are also used for the following

Institutional Outcomes (Essential Skills):

1. Writing in the Discipline (Type I) Literature Review

2. Writing in the Discipline (Type II) Descriptive

3. Integration/Synthesis in the Discipline

4. Critical Thinking in the Program

To measure the aforementioned Program-Specific Outcomes, the AAFS Outcomes Assessment Plan prescribes that

the Senior Capstone Paper contain several components listed as follows:

1. Review of the Externship Experience

2. Overview of a Research/Investigative Project implemented during the Externship Experience

3. Description of Midwestern agricultural systems

4. Formulated, analyzed, and critiqued Business/Management plan

5. Business Management Decision analysis and critique considering multiple factors and perceptions

Based upon recommendations from previous years programmatic assessment reports, the Capstone paper has been

divided into three specific assignments: 1) Externship/Internship/Research Reflection paper at the end of the

experiential learning activity, 2) Externship/Internship/Research Challenges paper developed as the first capstone

writing assignment, 3) Capstone Research paper, and 4) Capstone research presentation.

Each faculty and staff member assessed four of the thirteen senior student Capstone Artifacts (N = 52) using the

newly created rubric (attached) as none had been previously developed for this AAFS program-specific outcome.

Seven criterion were rated on a five-point scale to assess student attainment of the program-specific outcome. The

seven criteria included:

From the “Experiential Learning Component” Artifacts (Reflection Paper and Challenges Paper)

1. Background evidence of the experiential learning component of the program (Externship, Internship, or

Research)

2. Attainment of the learning objectives specified in the experiential learning component of the program

3. Challenges encountered through the experiential learning component of the program

From the “Capstone Research Component” Artifacts (Research Paper and Presentation)

1. Relevance of the research to the agricultural industry

2. Literature review

3. Research conclusions

4. Use of discipline-specific vocabulary

The five-point scale rubric used the following ordinal scale:

0 = Missing

1 = Ineffective

2 = Slightly Effective

3 = Somewhat Effective (Proficient level)

4 = Moderately Effective

5 = Effective (Capstone level)


Reliability

Since this is the first year of data collection to assess program completers (graduating Seniors) attainment of

Outcome #1, a rubric was developed requiring rater norming (Jonsson & Svingby, 2007). To determine inter-rater

reliability (IRR) of the newly developed program-specific outcome rubric, faculty initially discussed the newly

developed rubric, asked specific questions, and reviewed examples. AAFS faculty then assessed the Senior

Capstone artifacts submitted by the same student to determine Inter-rater Reliability (IRR) of responses on

their scaled responses individually. Responses that were not whole numbers were rounded up prior to determining

reliability. Percent agreement was then calculated between the raters. Ratings that varied between the following

dyads were considered different relative to reliability: 0-1, missing and ineffective; 2-3, slightly and somewhat

effective; and 4-5, moderately effective and capstone.

Percent agreement on the sample student was 85.71% and considered Acceptable (Norcini, 1999). Following initial

ratings using the newly developed program-specific outcome rubric, faculty further discussed each criterion, areas of

concern, etc. relative to the student’s attainment of the programmatic outcome. Once norming activities were

completed, faculty were placed into teams of two to review the remaining 12 students’ capstone artifacts. Faculty

teams were assigned four students each, rating each student on each criterion individually and then discussed

differences with their faculty peer. If faculty were in disagreement across dyads, a third rater was used. Table 1

displays results from the norming activity.

Conclusions of the review regarding student learning in the outcome with rationale supported by evidence

from the artifacts:

Findings

Individual student outcome attainment of the standard was determined by calculating the Grand Mean (GM) of the

faculty ratings for the seven aforementioned rubric criteria in the reviewed artifacts (reflection summary report from

the externship/internship/research project, experience challenges reflection paper, capstone research paper, and

capstone research paper/presentation) on a 5-point scale as follows (0 = Missing, 1 = Ineffective, 2 = Slightly

Effective, 3 = Somewhat Effective, 4 = Moderately Effective, 5 = Effective). Students who achieved a score GM ≥

3.00 on the 5.00 scale were considered Proficient. All thirteen students (N=100%) were considered Proficient. Table

2 displays the outcome attainment of AAFS seniors enrolled in the capstone agriculture course (AGRI 480) by

criteria and overall outcome attainment.


Table 2

Student Attainment of AAFS Programmatic Outcome #3: Students will demonstrate an understanding of Midwestern

agricultural systems

AAFS Departmental Outcome #1 Attainment (Met/Not Met) Reviewers:

Artifacts Reviewed:

Student Outcome #3 Criteria Faculty 1 Faculty 2 Mean Std. Dev. Grand Mean Grand Std. Dev. Outcome Met?

1 Background for Experience 3 3 3.000 0.000

1 Learning Objectives of Experience Attained 3 3 3.000 0.000

1 Challenges from Experience Identified 2.5 2.5 2.500 0.000

1 Research Topic Relevance 4 3.5 3.750 0.354

1 Literature Review 4 3.5 3.750 0.354

1 Research Conclusions 3.5 3 3.250 0.354

1 Discipline-specific Vocabulary 3.5 3.5 3.500 0.000 3.250 0.470 Yes

2 Background for Experience 4 3.5 3.750 0.354

2 Learning Objectives of Experience Attained 3.5 4 3.750 0.354

2 Challenges from Experience Identified 4 3 3.500 0.707

2 Research Topic Relevance 3.5 4 3.750 0.354

2 Literature Review 4 4 4.000 0.000

2 Research Conclusions 4 4 4.000 0.000

2 Discipline-specific Vocabulary 3 3 3.000 0.000 3.679 0.421 Yes




3 Research Topic Relevance 5 5 5.000 0.000

3 Literature Review 5 4.5 4.750 0.354



4 Background for Experience 2.5 2.5 2.500 0.000




























Fro

m t

he

Cap

sto

ne

Res

earc

h

Art

ifac

ts

Fro

m t

he

Cap

sto

ne

Res

earc

h

Art

ifac

ts

Fro

m t

he

Exp

erei

nce

Art

ifac

ts

Fro

m t

he

Cap

sto

ne

Res

earc

h

Art

ifac

ts

Fro

m t

he

Exp

erei

nce

Art

ifac

ts

Fro

m t

he

Cap

sto

ne

Res

earc

h

Art

ifac

ts

Fro

m t

he

Exp

erei

nce

Art

ifac

ts

Fro

m t

he

Cap

sto

ne

Res

earc

h

Art

ifac

ts

Fro

m t

he

Exp

erei

nce

Art

ifac

ts

Fro

m t

he

Cap

sto

ne

Res

earc

h

Art

ifac

ts

Fro

m t

he

Exp

erei

nce

Art

ifac

ts

Fro

m t

he

Cap

sto

ne

Res

earc

h

Art

ifac

ts

Fro

m t

he

Exp

erei

nce

Art

ifac

ts

Student will Demonstrate an Understanding of Midwestern Agricultural Systems

Experience (Research/Internship/Externship) Reflection Paper, Challenges Paper, Capstone Research Paper, Capstone Research Presentation

Ratings0 = Missing, 1 =

Ineffective, 2 =

Slightly Effective, 3

= Somewhat

Effective, 4 =

Moderately

Effective, 5 =

Capstone Level

≥ 3.00 = Proficient

Fro

m t

he

Exp

erei

nce

Art

ifac

ts


_______________________________________________________________________________________

Note: Student 13 served as the norming activity for the faculty.
















10 Learning Objectives of Experience Attained 3.5 4 3.750 0.354


10 Research Topic Relevance 3.5 4 3.750 0.354


















Fro

m t

he

Ca

psto

ne

Re

se

arc

h

Arti

fac

ts

Fro

m t

he

Ex

pe

re

inc

e

Arti

fac

ts

Fro

m t

he

Ca

psto

ne

Re

se

arc

h

Arti

fac

ts

Fro

m t

he

Ex

pe

re

inc

e

Arti

fac

ts

Fro

m t

he

Ca

psto

ne

Re

se

arc

h

Arti

fac

ts

Fro

m t

he

Ex

pe

re

inc

e

Arti

fac

ts

Fro

m t

he

Ca

psto

ne

Re

se

arc

h

Arti

fac

ts

Fro

m t

he

Ex

pe

re

inc

e

Arti

fac

ts

Fro

m t

he

Ca

psto

ne

Re

se

arc

h

Arti

fac

ts

Fro

m t

he

Ex

pe

re

inc

e

Arti

fac

ts


Conclusions/Discussion

Findings of this assessment indicate that program completers of the AAFS curriculum in the 2018-2019

academic year (N=13) were able to appropriately demonstrate an Understanding of Midwestern Agricultural

Systems. Previous outcome assessments in the first three years of the program provided significant recommendations

to assist AAFS faculty in in improving the scope and sequence of the required coursework in the program—

especially the connection and flow between the Workforce Preparation course (fall of the junior year); the

experiential learning component (Externship/Internship/and Research) and the formal supervision and reflection

process integrated within it; and the culminating Capstone course and its required research project.

The following specific areas of improvement identified from the assessment of Programmatic Outcome #2 in the

2017-2018 academic year have been implemented and helped students internalize the outcome addressed in this

report:

1. Additional structure within AGRI 380: Preparation for Externship should be established to include a formal,

written externship proposal. The proposal should include a specific career area of interest, background

research on several companies of interest, which provide that career area, and a list of several projects that

could serve as the foundation for developing a business management plan, analysis, and decision relevant to

the student extern’s future career. – Fall 2018

2. Specific activities to develop awareness of and skills in the measured outcomes related to the outcome

assessment criteria for the externship should be implemented in AGRI 380. Activities could include a

formal employer career panel, mock interviews, and other appropriate activities. – Fall 2018

3. A formal protocol for acquiring, gaining approval, and engaging in an externship experience (AGRI

480) should be developed and include a collaborative process for developing student learning outcomes

relevant to the employer’s needs and the student extern’s long-term development. – Fall 2018

4. A formal protocol for student reflection in the externship (weekly or bi-weekly) should be developed and

implemented which covers at a minimum the following items: - Summer 2018

Overview of activities assigned and completed

Major challenges or obstacles encountered and how they were addressed

New learning opportunities to be implemented over the next one-two weeks

5. A formal protocol for faculty supervision of the externship should be developed and implemented which

should include: - Summer 2018

Adequate resources to complete supervisor visits

A process for scheduling supervisor visits to include timing in relation to bi-weekly reflection

completion

A checklist of items to discuss with the employer and student during each visit including discussing,

adjusting, and assessing progress on the collaboratively developed learning objectives

6. A formal protocol for a mid-term employer evaluation of the student extern to allow for adjustments or

corrections in the student extern’s workplace behavior. – Summer 2018

7. A formalized one- to two-day professional development session should be held on campus at the end of the

summer (beginning of fall) and at the end of the fall semesters to reflect, debrief, receive feedback, and

finalize the formal written externship reflection paper and present the formal oral presentation. This will set

the stage for the spring capstone course and the development of a follow-up research paper to garner further

information needed for continued career development in the student extern’s career field of choice. – Fall

2018


8. Formally split the Externship Reflection Paper and Presentation from the Senior Capstone Paper and

Presentation assignment currently used in AGRI 480: Capstone in Agriculture.

9. Utilize the Externship Reflection Paper and Presentation as the culminating reflection assignment in AGRI

460: Externship in Agriculture to measure Outcome #3. – Fall 2018

Suggestions (if any) to improve student learning relative to this outcome. If no suggestions explain why not:

Although positive results in student attainment of Programmatic Outcome #3: Students will demonstrate an

understanding of Midwestern agricultural systems, have been noted in this report, continuous improvement of the

program through the following recommendations can occur.

A brief overview of these next steps is listed below:

.

1. Develop experiential learning activities in the freshman and sophomore year which include job shadowing

and internship experiences that are in clear alignment with Preparation for Externship, Externship, and the

Capstone course. (Fall 2019)

2. Develop formative activities that align and build to an assessment-ready Senior Capstone Project in the

Capstone course. (Fall 2019 – ongoing)

3. Develop an assessment plan to which provides formative feedback for instructors developing and improving

the foundational courses in the department. (Fall 2019)

4. Continue to improve the experiential learning supervision of learning and reflection process through the

improvement of bi-weekly reflection writing prompts and an enhanced supervision protocol that

intentionally utilizes faculty and industry mentors. (Summer 2019 – ongoing)

5. Work with students in the co-curricular clubs to improve programming (Leadership seminars, Career

seminars, Service-learning opportunities) that supports overall AAFS programmatic outcomes. (Fall 2019)

These specific recommendations, when implemented, will assist faculty in the AAFS department in overall program

improvement as it seeks to meet the mission of Morningside College.

Works Cited

Bartko, J. J. (1976). On various intraclass correlation reliability coefficients. Psychological Bulletin, 83(5), 762-765.

doi:http://dx.doi.org/10.1037/0033-2909.83.5.762

Fleiss, J. L., & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as

measures of reliability. Educational and psychological measurement, 33(3), 613-619.

Jonsson A., Svingby G. (2007). The use of scoring rubrics: reliability, validity and educational consequences.

Educational Research Review, 2(1) 130–144. doi: 10.1016/j.edurev.2007.05.002

Norcini, J. J. (1999). Standards and reliability in evaluation: when rules of thumb don't apply. Academic medicine:

Journal of the Association of American Medical Colleges, 74(10), 1088-1090.

Appendix

AAFS Program Outcome #1 Assessment Rubric.....................................................................................Page 8


Appendix 1: Programmatic Outcome #1 Assessment Rubric

Program Annual Assessment Report - Morningside College · the externship/internship/research project, experience challenges reflection paper, capstone research paper, and capstone

Documents