Evaluation Project

7/21/2019 Evaluation Project

1/69

THE STUDENT COMMUNITY BOARD

Program Evaluation

Proposal

Alison Reimel

Office of Student Conduct and Conflict Resolution

Loyola University Chicago


2/69

EVALUATION PROPOSAL 2

!"#$% '( )'*!%*!+

"#$%&$'()* +%$* ,

-./)0/'(1$% 20$3/4)05 6

+0)70$3 8)*'/9' :

;(


3/69


LMM/*=(9 FV 2)1&< F0)&M +0)')1)% 6J

LMM/*=(9 ?V 8)*


4/69


Evaluation Plan Introduction

The Student Community Board (SCB) at Loyola University Chicago (LUC) is an

undergraduate student organization sponsored, trained, and advised by LUCs Office of Student

Conduct and Conflict Resolution (OSCCR). SCB exists to hear cases of alleged student

misconduct on a peer-to-peer level, hold student accountable for policy violations, education the

student community on LUCs expectation and standards of behavior, and act as an advocate for

the student voice within the student conduct process. Like any other conduct administrator at

LUC, SCB is responsible for meeting with students, discussing an incident of alleged

misconduct, and determining if any violations of Community Standards occurred. Then, if any

violations did occur, SCB determines the sanctions, or required outcomes, that will aid in the

education, engagement, and behavioral change of those students in violation of LUC policies. In

addition to hearing cases of alleged misconduct, SCB also engages in a yearlong leadership

development experience aimed at increasing their efficacy and ability for leadership.

This project will evaluate the intended learning outcomes of the overall SCB experience,

which includes the skills and capacities necessary to carryout an administrative conduct hearing,

as well as the year-long learning goals of the leadership development experience. It is also

intended to better understand the SCB members experience outside the defined learning

outcomes. This proposal will further describe the context of theoretical framework for conduct

administrator skills and leadership development programs, the larger context of SCB, and the

key characteristic of the program. This information has informed my development of a mixed

methods evaluation plan intended to be implemented next Fall 2015. This plan also discussing

the rationale for each component of the evaluation and data analysis, the timeline for

implementation, and the future impact and steps stemming from the evaluation. Overall, the


5/69


evaluation was developed considering the resources available to OSCCR and has been designed

to be feasible and practitioner-friendly.

Theoretical Framework

As a student conduct administrator, I know first-hand the skill, competencies, and

practice is takes to become an effective hearing officer. It takes intention and continuous hard

work to continue to develop oneself as an effective conduct administrator. In the case of SCB,

we are expecting that undergraduate students possess and practice the skills necessary for this

role. Lancaster and Waryold (2008) identify many skills for effective practice, including a deep

awareness of self and others, practicing integrity, civility, collaboration, and attentiveness in

conversation. Supported by research these are in alignment with many of the expectations

OSCCR has of SCB functions and member development.

Beyond developing in SCB members the skills necessary be effective conduct

administrators, OSCCR also works to develop in them other skills and values needed for great

leadership. Reflected within the skills identified by Lancaster and Warywold, there are also

components of the Social Change Model of Leadership Development, (SCM) a leadership

development used in many colleges and university, including LUC. Specifically, SCM identifies

seven values necessary for student leadership. These competencies include a consciousness of

vales and beliefs, congruence with those values, conflict with civility, collaboration and

commitment, shared values, and citizenship (Cliente, 2009). Just as many of these are also found

in the essential skills identified by Lancaster and Warywold, they are also valued within the

context of student leadership development.

In developing SCB, OSCCR married all of these concepts in order to create an

experience where in members would develop the skills necessary to be effective in the role of


6/69


conduct administrator, as well as broader leadership skills that could be used inside and outside

of the SCB program. However, as of the current, OSCCR has no way of knowing whether or not

the program is developing students in this light. Therefore, this evaluation is essential to

ensuring that the SCB program is designed to develop the noted skills, values, and competencies.

Before further discussing the specifics of he evaluation plan, the following sections will more

thoroughly develop the context of SCB.

Program Context

As previously discussed, SCB is housed within the OSCCR, which is considered a

department within the larger LUC Division of Student Development. All of SCBs multiple

working parts are consistent with the mission of the OSCCR and reflective of the growth and

development of the department. Not only is SCB a relatively new component of the OSCCR,

but the student organization has also undergone a significant programmatic change within the

last year of its existence. Before discussing the evaluation plan, it is important to further details

the broader context of SCB, as well as the intricate details of the program that directly reflect that

evaluation design. This section first discusses the larger context for SCB including the mission

and values of OSCCR and SCB, the history of the organization, the key characteristics, and then

the key characteristics and goals of the program, as well as relevant stakeholders.

Mission and Values

First, it is important to understand the larger mission and values of OSCCR, as the

mission and values of SCB reflect the offices larger mission. The Office of Student Conduct

and Conflict Resolution (2014) states:

The Office of Student Conduct and Conflict Resolutions (OSCCR) is dedicated to

providing a safe environment for students by promoting responsible decision making and


7/69


a focus on self-awareness. We also strive to strengthen relationships between students

and their communities by encouraging students to take ownership of the choices they

have made, enforcing accountability and engaging in open dialogue with community

members.

Consistent with LUCs Jesuit tradition, OSCCR strives to uphold the Jesuit value of cura

personalis,or care for the whole person, which is reflected in the offices values. Specifically,

OSCCR (2014) cites the care, integrity, support, safety, and justice as the values that serve as

cornerstones to their work with students. Care speaks to demonstrating compassionate and

concern for all students in the LUC community. Integrity means modeling and holding students

accountable for their behavior as it relates to the Community Standards. Support refers to

responding to a students needs with dignity and respect. Safety speaks to the offices

responsibility to ensuring the personal safety of students as well as the safety of the larger

community. Lastly, justice is a commitment to a fair, consistent, and impartial process for all

students that honors the uniqueness of their identities.

As a function and component of OSCCR, SCB adopts the mission and values of the

office, but has a unique approach to acting on them in their interaction with students during a

hearing. Unlike any other conduct administrator at LUC, SCB consists of students. Because of

their unique position as peers to the students who come before them in hearing, the angle from

which they discuss misconduct and the values of the office and community is different. They

demonstrate care and support from a place of being equal peers rather than administration. SCB

has the unique opportunity to serve as a student role model for integrity and the Community

Standards. They incorporate justice into their work by ensuring that the student voice is

represented in LUCs conduct process. Though SCB has not always been a part of OSCCR,


8/69


since their inception they have had this unique way of carrying out the mission and values of

OSCCR.

History and Significant Changes

In order to learn more about the history of SCB, I spoke with stakeholder and director of

OSCCR, Dana Broadnax. It was not until 2005 that OSCCR created SCB out of a call from

students to have a voice in the student conduct process (D. Broadnax, personal communication,

September 18, 2014). At this time, many other institutions had student conduct boards that

functioned to do just this. Additionally, OSCCR staff recognized the opportunity for leadership

development for student members. Though there has not been consistent or detailed record

keeping of the exact ways in which SCB has evolved, current staff are still aware of the large

scale program changes over the last nine years. First, from 2005 to 2012, SCB consisted of of

two student boards of six members with alternate members of SCB who only heard cases if a

member of a board could not be present at a hearing. Members were trained in the on the

conduct process at the beginning of the year and there were monthly professional development

sessions that intended to continue to build skills and knowledge related to conduct hearings.

These sessions topics were chosen based upon the needs or perceived needs of the boards at that

particular time.

Over the course of the past two school years, SCB has undergone a few significant

changes. First, a six more student members were recruited and trained, which resulted in the

addition of a third hearing board within SCB. That year, SCB and OSCCR saw an increase in

interest during the member recruitment period (D. Broadnax, personal communication,

September 18, 2014). They determined that it was feasible to add more students to the

organization and did so by increasing the number of student selected for membership. Most


9/69


significantly, this school year (2014-2015), the professional development model used in the past

has been drastically altered. Through conversation with office professional staff, members

expressed not finding the monthly professional development sessions particularly useful or

engaging. At the same time, the director of the office expressed interest in increasing the

intentionality and impact of SCB to members development as student leaders on campus.

As a result of this feedback, I, as the current graduate assistant and advisor to SCB, spent

a significant amount of time in the spring of 2014 developing a more intentional developmental

program with the input of current SCB members. Discussed further in the next section, I

incorporated values of SMC to the monthly professional development sessions. Now considered

leadership development sessions, each session is based off of a leadership competency identified

from SCM. Considering the short history of the program, there has been little to no other

significant structural changes that current staff is aware of. With the history of the program in

mind, I now want to discuss the key characteristics of the SCB, as there are many working parts

to the current program.

Key Characteristics

This section is broken down into two basic parts, SCB as an hearing board and the

leadership development curriculum. First, SCB consists of fifteen undergraduate students.

Currently there are four sophomores, six juniors, and five seniors. Students are recruited, apply

for, and selected in January and February and are trained the following August. Hence, no first

year students are members of SCB. Within SCB, there are three hearing boards each consisting

of five members. Each board has a Board Chairperson, who is a returning member of SCB and

has demonstrated the skills and commitment needed to fulfill the role. SCB meets every

Wednesday from four to seven pm. In a given month, three of these Wednesday are dedicated to


10/69


hearing cases of alleged student misconduct and one Wednesday is dedicated to a leadership

development sessions. Therefore, typically, each board hears three cases per month and

participated in one leadership development session.

As stated, members are recruited in the winter months. Current SCB members lead the

recruitment efforts, encouraging students to apply using social media, the winter student

organization fair, and outreach to student organizations and campus partners. Students apply,

then are selected for an individual interview with the advisor and current members. Candidates

with high scores then participate in a group interview where they engage in group activities in

order to evaluate team work and consensus building ability. After this, the final selections are

made and selected candidates are asked to join for the next academic year. Typically, SCB

recruits seven to ten new members, as many members choose to return for a second or third year.

The following August prior to the start of school, participate in a two and a half day training

retreat created and led by OSCCR staff that is intended to educate SCB members on the student

conduct process at LUC, as well as develop the necessary competencies for the role. This is also

an opportunity for community building as an entire organization, as well as with their specific

boards.

As a vehicle for continued development throughout the coarse of the year, the second

basic component of the SCB program is the monthly leadership development sessions that are

attended by all members together. As stated, these sessions are based off of the Social Change

Model. According the Wagner (2009), SCM identifies seven values or competencies necessary

for leadership, including consciousness of self, values congruence, controversy with civility,

commitment, common purpose, collaboration, and citizenship. In employing these values,

people are groups are able to effect positive social change within their communities. With


11/69


feedback from SCB members that they wanted to be more present and engaged with their

campus community, I based each monthly leadership development session off of one of these

values, culminating with the implementation of a SCB-planned campus-wide service event.

As described in the syllabus (see Appendix A), the yearlong experience starts by

members engaging sessions that are focused on the self. More specifically, identifying there won

personal values and the ways in which they are congruent with those values. As SCM indicates,

a deep awareness of self is critical for the engagement with others when working toward creating

positive change (Wagner, 2009). Moving into fall and winter, the sessions focus on group

dynamics such as group conflict, identifying shared values and a common purpose, and

collaboration. In engaging with sessions focused on group values, SCB begins planning what

will be the culminating campus-wide service project at the end of the school year. The last of the

values identified by SCM is citizenship, which is a deep engagement with ones community

(Cliente, 2009). The service event, called Project Citizenship, is a way for SCB to engage

themselves and others with the LUC community in a way that honors the mission and values of

OSCCR.

Program Goals

Both of these program components, the conduct hearings and the leadership development

experience, have specific goals as stated in the SCB syllabus (see Appendix A). The training

retreat is aimed at developing specific outcomes identified by OSCCR as necessary for the role

of hearing administrators. Over the course of the year, the weekly hearings should continue to

develop these skills. What I will call hearing skills for the remainder of the proposal, include the

following competencies: active listening, effective questioning, consensus building, navigating


12/69


group dynamics, accountability to actions, critical thinking and analysis, openness to having

viewpoints challenged, and conflict resolution.

Additionally, the goal of the leadership development sessions are grounded in the values

of SCM. Specifically, once completed with all of the monthly sessions, students should have

developed in the following areas: exploring personal values and beliefs, practicing congruence

with those values, make and follow-through with commitments, develops a common purpose

with other SCB members based on shared values, practice collaboration by planning and

implementing a large scale program, engage in constructive conflict with others, and engage

more deeply with the LUC community. Overall, the hearing goals and the leadership

development goals work together to create a developmental experience rooted in SCM and the

mission and values of OSCCR.

Stakeholders

There are a variety of different groups and people who have an interest in the SCB

program and are invested in the members reaching the above goals. First, SCB members have an

interest in the program, as they are the people who are joining this organization for a given

purpose. That purpose might look very different depending on the member, but all are joining

with the intent to gain knowledge, experience, or build relationships in some way. The OSCCR

and larger Division of Student Development have a stake in the SCB, as SCB acts as conduct

administrators and student developers just as the professional staff. Because of this, an

inconsistent and undeveloped SCB compromises the integrity of the office and its mission, as

well as poses a risk to the larger division and the University. As a result, it is essential that

OSCCR and the larger division are confident that SCB members have the ability and necessary

skills to carry out the work, mission, and values of their office. Lastly, the undergraduate


13/69


students who are documented for possible misconduct and come before the SCB have an interest

in SCB. Students who come before SCB have a right to a fair and just process that is grounded

in the values of the OSCCR and the institution. A strong SCB makes for a consistent and fair

process, as well as a more developmental experience for the student.

Program Resources

In order to carry out the functions of SCB, a lot of OSCCR resources are allotted to the

program. A significant amount of monetary resources are funneled into SCB mostly used for

training and community building activities. SCB also utilizes campus partners, such as Campus

Ministry and Residence Life, to facilitate some of the leadership development sessions. Most

notably, the OSCCR graduate assistant, who is the primary advisor of SCB, dedicated about half

of her time to training, supporting, and advising the student organization. Together, these

resources are necessary to the functioning, health, and overall development of the student

members and whole organization.

Logic Model

In order to thoroughly design the evaluation, a logic model (see Appendix B) was created

to outline and organize the key components of the program. The logic model provides a concise

view of the different elements and how they work together to produce the desired outcomes. The

logic model is broken down into three parts. It first outlines the resources that are put into SCB,

called inputs. The inputs include the staff dedicated to training and advising SCB, the amount of

time they put into the program, the amount of money needed to run the program, and other

necessities for program functioning. The second part of the logic model is the outputs. The

outputs capture what SCB does and who is reached. The outputs include the training retreat,

weekly hearings, monthly leadership development sessions, the campus-wide service event, and


14/69


the recruitment and selection of member. Outputs also include the stakeholders as those reached

as a result of the SCB program. These include SCB members, students who come before SCB,

and office and division staff.

Following the outputs, the last component of the logic model is the program outcomes.

The program outcomes are divided into short term, medium term, and long-term goals. In this

case, the short-term goals refer to the intended outcomes of the training retreat and engagement

in hearings. The medium-term goals place more emphasis on the intended outcomes of the

monthly leadership development sessions. In this case, there are no specific long-term goals,

however it is possible that many of the short-term and medium-term goals continued to be

practiced far beyond students membership on SCB. It is also possible that as a result of the

evaluation, we are able to identify if there are long-term effects and what those might be.

Assumptions

In addition to the inputs, outputs, and outcomes, it is also relevant to discuss the

assumptions that factor into this evaluation. First, we are assuming that members of SCB find

their experience valuable in some way. We assume that students join this organization for a

purpose and are looking to get something out of it. This may look completely different

depending on the student, but overall we still assume that they are joining SCB in hope that it

will add value to their educational experience. We are also assuming that all SCB members are

open and have bought into the newly updated leadership development portion of the SCB

experience. The model also assumes that there will be cases of alleged misconduct appropriate

and challenging enough for the SCB to hearing. Whether or not these assumptions are true can

alter the outcome of the evaluation, so understanding them is key to appropriately measure and

analyze the evaluation data.


15/69


External Factors

The final component of the logical model outlines the external factors that may influence

the evaluation and outcomes of the program. First, it is important to consider SCB members

priorities and commitments outside of SCB, as this can effect how engaged students are with the

skills and experience. In the same light, the level of complexity to the conduct cases heard by

SCB also acts as a external factor. The cases are a primary outlet for SCB members to practice

many of the short-term and medium-term outcomes, such as active listening and consensus

building. Program funding and the amount of time and energy able to be committed by the

professional staff to advise SCB is also an external factor. Specifically, funding determines

much of the training resources for SCB. When the graduate assistant and staff members who

advise SCB become consumed with other aspects of their role, the same amount of attention,

feedback, and development cannot be paid to SCB. All of these external factors contribute to the

level at which outcomes are met. Though they can be hard to control for, it is important to

recognize the potential impact they could have on the outcome of the evaluation.

Purpose of Evaluation

The purpose of this evaluation is twofold. We are looking to evaluate the extent to which

student members of the SCB are meeting the intended outcomes of trainings and leadership

development sessions (see Appendix B), as well as gain an understanding of what students are

more broadly gaining from the SCB experience. Specifically, we are looking to answer the

question: what is the developmental experience of student as a result of membership in SCB?

We are interested in what skills and knowledge they are taking away from it, what they would

like to take away from it, and how we can improve the program in the future. With this in mind,

we will be taking an approach that combines formative and summative evaluation in order to


16/69


evaluate the SCB program. The evaluation is summative in that it is concerned with providing

information that will help us to make decision on program changes and continuation (Fitzpatrick,

1997). Is the also formative in that it will likely provide us with a deeper understanding of the

program, as well as how to improve upon it. The evaluation is intended to evaluate the intended

outcomes, as well as the overall experience, or process, of being an SCB member in order to

continuously improve the SCB experience. The outcomes approach will allow us to determine

whether or not students are meeting the intended learning outcomes of each leadership

development session and hearing skills (see Appendix B), and a process approach will allow us

to better understand the member experience and gain a deeper understanding of development.

Evaluation Format

Based on the purpose of the evaluation, as stated, we plan to use an outcomes-based

approach that again takes into account a summative and formative approach. The outcomes

approach is utilized in order to describe or explore the changes that that occur in participants as a

result of program participation (Fitzpatrick, 1997). Outcomes-bases approaches can be used to

measure specified learning outcomes or unintended outcomes and development from a program.

In this case, an outcomes approach is most appropriate for determining whether or not SCB

members develop in the programs intended outcomes, as well explore the outcomes beyond

these specified outcomes.

In order to measure the program goals, we will utilize a quantitative survey will measure

the development of competencies consistent with the program goals. In order to explore the

development of skills and competencies outside of intended outcomes, we will use focus groups.

These focus groups should aid in gaining a deeper understanding of SCB member development.

It should also allow us to gain insight on which parts of the program are most beneficial to the


17/69


development of the intended learning outcomes, therefore allowing us to make programmatic

changes that will enhance the developmental experience.

It is also important to note that the focus groups will be the primary source of data

collection and analysis for this evaluation. As discussed later, due to the small sample size and

focus on program improvement, the focus groups will provide us with the most valuable and

significant information.

Strengths and Weaknesses

Based on the purpose of the evaluation, the outcomes-based approach that incorporates

both formative and summative aspects of evaluation allows us to the opportunity to deeply

explore the developmental experience of SCB members. However, an inherent weakness to this

evaluation format is the vast difference in each individual students experiences. This

uniqueness may contribute to the inability to attribute the outcomes of the evaluation with

participation in the program, as we cannot control for differences in each members LUC

experience. To address this weakness, the focus group aspect of the evaluation is designed to ask

questions regarding the overall process of the SCB experience. These questions address what

parts of the program are students finding valuable to the development of particular competence,

as well as the pieces are they do not find valuable. This information will be essential to the future

improvement of the program. The last potential weakness of the evaluation format is the

possibility that self-reported skills and development may be difficult to measure, as well as be

skewed by what SCB members know are the program outcomes.

Evaluation Criteria

Given the intent to explore the outcomes of the SCB program, the evaluation approach is

defined by two main criteria, which will be used to guide the assessment of the value of the SCB


18/69


19/69


entire population receiving the intervention, or in our case, participation in the SCB program

(Wholey, Hatry, & Newcomer, 2010). Aiming for census, all fifteen students will be asked to

complete the surveys. The following section outlines implementation strategies to mitigate

coercive recruitment techniques, as well as how we plan to address the incredibly small sample

size. Since the approach is longitudinal in nature, the students will be pre-tested prior to the

training retreat, and then will be post-tested after the completion of the final leadership

development session. This approach was chosen in order to understand the student

developmental growth as they relate to the intended learning outcomes.

As consistent with a pre-experimental design, this evaluation will not utilize a control or

comparison group. Although a lack of comparison groups makes it difficult to attribute any

change or development to the SCB experience, it is not realistic to use one in this design. It

would take up an unreasonable amount of time to identify a comparison group and control for

other experiences that might contribute to the development of similar developmental outcomes.

It is important to take into account this design limitation when analyzing the data and impact of

the SCB experience. Another limitation of this type of pre-test and post-test survey concerns

inaccurate reporting. Student may sense what the evaluator is looking to find which could skew

the survey results. Fortunately, this will be mitigated and supplemented by the qualitative

approach, which will take place later.

Survey Instrument

Because of the expected high response rate and minimal cost of survey implementation, a

survey is a feasible option for assessing the intended outcomes of the SCB experience. In

attempting to use already existing survey instruments, it was difficult to find any that were

specific to the skills the SCB program is looking to encapsulate. For this reason, the survey


20/69


questions in the pre and post-test were created uniquely for the SCB program and as stated, are

informed by the intended learning outcomes of the experience (see Appendix C). These learning

outcomes are outlined in the syllabus and include but are not limited to listening and questioning

skills, effective communication, consensus building, personal values identification, conflict

resolution, and working in a team.

The pre-test survey (see Appendix D) is comprised of 6 questions, several with multiple

parts, and will take approximately three to five minutes to complete. The survey consists of

questions that ask students to identify their basic demographics including class standing and

standing within SCB. Because of such a limited sample size, participants will not be asked to

provide any other demographic information such as race, gender, religion, etc. This information

will be used to identify whether there is a difference in experience based on these factors. These

questions are in a checklist format. There is also a question regarding the motivation to join SCB

and what students are looking to gain from the experience, which are in Likert scale format. In

addition to these questions, the pre-test includes two other Likert scale questions that include

between multiple statements. These are statements reflected the intended outcomes and are

associated with specific goals represented in the logic model (see Appendix C). The scales range

from 3-point to 5-point in order to avoid neutral responses. Five-point and 4-point scales were

used, with 4-point scales being utilized in order to avoid a neutral response.

The post-test survey (see Appendix E) is comprised of three Likert scale questions, again

consisting of multiple statements. Like the pre-test survey, it is estimated to take four to five

minutes to complete. Because the evaluator is seeking to assess the change in the outcomes used

in the pre-test, the same questions will be asked. Likewise, students will be asked again what

they were hoping to gain from participating in SCB. This question is changed to reflect to what


21/69


extent they feel they have gained the given statements. In addition to the questions from the pre-

test, there is one additional Likert scale questions that addressed topics that are only applicable

once completing a year of the SCB experience. These include how they felt about a multitude of

different experiences within the SCB program.

Implementation Plan

For both the pre and post-test, the surveys will be distributed as a web-based survey upon

arriving to the training retreat and at the end of the last leadership development session. Students

will be asked to bring their laptop, as it has been confirmed that all student have access to a

laptop device. In both cases, the evaluator will leave the room and student will be asked to

spread out as not to coerce any students into completing the survey. In this case, the web-based

surveys allow for simplified evaluation and storage of the collected data.

Although not ideal, the students will be asked to use their name when completing the

survey as the evaluator needs to be able to march the pre and post surveys wen analyzing the

data. In order to maintain the privacy of the students, upon completion of the survey, the

evaluator will replace their name with a numerical code (e.g. 001) and then repeat the same upon

completion of the post-test.

Although the survey is new and the program has been reimaged and restructured in the past

year, a pilot study of the survey instrument is not feasible considering the already very small

sample size. With only fifteen participants, taking one or two from the sample has the ability to

significantly alter the data. However, other stakeholders and recently graduated SCB members

will be asked for feedback on the survey instruments to ensure that they are comprehensive to the

goals of the program. These stakeholders include the Director and the Coordinator of the

OSCCR.


22/69


Statistical Analysis Plan

Since the quantitative aspect of the evaluation plan is aiming to evaluate the outcomes of

the program, the evaluator will use two different tests to evaluate such outcomes. Specifically,

the evaluator will use central tendency measures and paired sample t-test. First, the descriptive

statistics test will measure the mean, median, and mode of the collected data across the Likert

type scales (Huck, 2012). This type of descriptive data allows the evaluator and stakeholders to

understand how the population as a whole met or did not meet the intended outcomes of SCB.

Additionally, this data will help us to understand the development of members based on their

year in school and role in the organization.

Next, as a precaution to the small sample size, we will conduct a power analysis in order to

determine the sample size needed for statistical significance. As described by McCrum-Gardner

(2009), when a sample size is too small, the data is limited to how significant, and therefore

valuable the findings are. It is our hope that using a power analysis, w can increase the validity

and usefulness of our data. In addition to the power analysis, the qualitative component of the

evaluation will also serve to counter the limitation of this small sample size.

Next, the paired sample t-test will allow for a more thorough understanding of the

development of SCB members based on the intended learning goals. Specifically, this test

determines whether or not there is a statistically significant change in the development of the

SCB members upon completion of the experience. Additionally, testing for change across a span

of time will also allow the evaluator to identify any significant differences in growth regarding

year in school and class rank.

Quantitative Results Presentation


23/69


The results of the quantitative portion of the pan will be compiled into a report that uses

graphs, charts, and written explanations of the evaluation findings. The report will be presented

to and discussed with key stakeholders, including the Director and Coordinator of OSCCR, in

conjunction with the findings of the qualitative portion of the report. The evaluator and named

stakeholders will then give a written report to the Associate Dean of Students who oversees the

OSCCR.

Qualitative Evaluation Approach

In addition to a quantitative approach, a qualitative approach adds depth and breadth to

the overall evaluation of the SCB. Specifically, a qualitative component to the evaluation of

SCB will allow the evaluator to gain an understanding of the process and personal experiences of

SCB members. It will provide the evaluators with additional anecdotal information beyond

whether or not intended learning outcomes are being met. A qualitative approach provides the

opportunity for the evaluators to explore growth and development that occurred outside the

constraints of learning outcomes. It can provide valuable feedback about how to improve the

program, making it more valuable for all stakeholders.

In this case, focus groups will be utilized in order to gather a substantial amount of

information from a small sample population. Focus groups reduce the amount of time and

resources needed for individual interviews, but allows for the same amount of participants to be

involved. The focus groups discussions will be based off of the questions asked in the

quantitative survey, but allow for expansion and deepened perspective from student experience.

The focus groups will also allow the evaluator to further explore themes drawn from the

quantitative evaluation (Schuh, 2009). Overall, the use of open ended and probing questions


24/69


should encourage open and honest conversation, resulting in rich data about the experience of

SCB members.

Focus Group Participants

Focus group participants, similar to the survey population, will be a census population,

meaning all current SCB members will be asked to participate. Considering the small number of

participants, the fifteen students will make up three focus groups of five students. First, the

evaluator will ensure that each of the three student conduct board chairpersons are split up

among the three groups. Then, by using maximum variation sampling, the evaluator can ensure

that there are a variety of experiences represented in each group. The intent is for each focus

group to have a mix of members from each of the three boards, year in school, and number of

years as an SCB member. This way, the evaluator is able to identify themes and trends across

the entire group rather than within individual boards.

SCB members will be asked to participate in the focus groups at the conclusion final

leadership development session. This way the evaluator has a captive audience in order to

explain the value of this portion of the SCB evaluation plan. The evaluator will encourage SCB

member participation by providing a free meal of the groups choosing during the focus group

meeting. However, it will be clearly stated that participation is voluntary, as ensure students do

not feel required to participate. As this is a tight knit group already largely invested in the SCB

program, it is expected that participation in the focus groups will be high. After this meeting,

SCB members will also be sent an email (see Appendix F) with more detailed information about

the focus groups, including a link to a scheduling tool called Doodle. This will allow the

evaluator to find a time and day that works for all members of a given focus group.

Focus Group Procedure


25/69


A Focus group protocol will be used in order to provide structure and direction to the

discussion (see Appendix G). The moderator for each of the three focus groups will be a

member of the Division of Student Development who is familiar with SCB, the purpose of the

group, and the new leadership development component of the program, but in no way directly

involved in the creating of the leadership development curriculum, recruiting, training, or

advising of the SCB. This way, SCB members do not have a pre-existing relationship with the

moderator based on their SCB experience that might bias responses or inhibit honesty. As

Creswell (2009) explained, if the evaluator is serving as the moderator, this could bias the

participants responses. The moderation will begin by giving an overview of the agenda for the

hour and a half discussion and further explaining the larger purpose of the focus group and

evaluation of SCB.

Next, the moderator will review the participants consent form (see Appendix H) and ask

the participants to sign it. Since all of the participants already know one another, there is no need

to do formal introductions. At this point, the moderator will begin the protocol questions.

Implementation Plan

In setting the stage for a focus group discussion, the evaluator has given much thought to

ensure that the participants feel that the environment is suitable for open and honest sharing of

their experiences. Keeping in the mind the purpose of this portion of the evaluation plan, the

following sections outline the key components to ensuring that SCB members are able to share

their thoughts, feelings, reflections, and feedback on the SCB program.

Moderator.

The focus group moderator will be Jen Kosciw, a Residence Life staff member within the

Division of Student Development who is familiar with SCB and its purpose on campus, and the


26/69


intended outcomes of the leadership development curriculum. This choice was made in order to

mitigate the potential pressure that student might feel if the moderator was the evaluator or

another member of OSCCR who they frequently work with. The evaluator recognizes that her

presence during this focus group could skew SCB members answers, as she has a close working

and mentoring relationship with many of them. With this in mind, the moderator will be a

neutral party that has experience in conducting focus groups, asking probing questions, and

setting a comfortable space for open and honest feedback. Since the curriculum is not graded

and all identifying information will be excluded, there should be little to no reason for biased

feedback during the discussion.

Recording procedures.

In order to ensure accuracy the information provided, the recording procedures will be

two-part. First, the discussion will be audio recorded using one of OSCCRs recording devices.

There will also be a graduate assistant within the Division of Student Development taking notes

on body language, tone, volume, how many times each person spoke, and other process-oriented

notes. This will act as supplemental information to the interview recording, providing valuable

information that cannot be provided through an interview transcription. Combined, this method

of recording is meant to capture the full group discussion experience. In addition to being audio

recorded, the recording will then be transcribed in full by the graduate assistant taking process

notes. Although transcribing takes a lot of time, the small number of focus group discussions

makes this task more manageable. Additionally, transcribing ensures that all feedback is

captured accurately and identifying information can be discarded.

Logistics and scheduling.


27/69


As touched on earlier, in order to increase participation and create a comfortable

experience for participants, a Doodle (online scheduling tool) will be used to ensure that the

focus group discussion is held during a time that fits within all participants schedules. The

focus groups will be held during the weeks after midterms and prior to reading week and finals

during the spring semester. This is again in an attempt to increase participation and mitigate

factors that might add stress to students experience.

The groups will be help in the same room as the leadership development sessions. This

way, participants are comfortable and familiar with the space, but do not experience the

evaluator or other OSCCR staff members in the space. This is a space where students are used to

holding deeper discussion and reflections, therefore it might be easier for them to get into a

reflective mindset. As stated, a full meal will be provided during the discussion as an incentive,

therefore the evaluator will try to schedule around a meal-time hour if participants schedules

allow.

Analysis plan

The audio recording transcription will be the primary data source to be coded. Since the

focus group protocol has been split into segments based on the key components of the SCB

program, it is logical that the coding uses the same components. Based off of these components,

an initial coding rubric has been developed (see Appendix I). The coding rubric consists of six

different codes, their meaning, and example themes of topics related to that code. Though the

evaluator is looking to gain a deeper understanding of the information gathered from the

quantitative survey, the questions are only loosely reflective of the specific outcomes. The intent

behind this is to also gather information on the process of SCB, allowing the students to speak to

what feels relevant to them in regards to their experience. Although this is not as focused of an


28/69


approach as the quantitative survey, it is essential look beyond learning outcomes especially in

the pilot year of a major change in the program.

Coding procedure.

Using the six codes from the coding rubric, descriptive coding will be applied to the

transcripts using the comment capability in Microsoft Word. The evaluator will then carefully

apply the codes to the whole transcript. This also allows for the evaluator to apply codes

possibly not included in the six codes included in the coding rubric as a way to accurately

capture any unexpected information that arises during the focus group discussions. After each

transcription is coded, the evaluator to will pull the text from each code into separate documents.

This way, it is easier to then identify themes and concepts within each topic. The evaluator and

another staff member in OSCCR will be responsible for coding each focus group discussion as a

way to ensure inter-rater reliability, which increases the validity and rigor of methods and

findings (Wholey et al., 2010).

Validity.

In addition to ensuring inter-rater reliability, the evaluator will also use member-checking

as a way to increase the validity of the methods and findings (Wholey et al., 2010). Immediately

after the transcripts are coded, a summary of the findings will be drafted and sent to the

respective members of each focus group. Members will be encouraged to provide additional

feedback on the themes and concepts pulled from each discussion. This strategy is employed to

ensure that the evaluators accurately captured intended messaged within each group discussion.

As addressed previously, the focus group facilitator will not be a stakeholder in the SCB

program. This is another attempt to ensure that the student feedback is honest and accurate.

Limitations.


29/69


Though the facilitator and transcriber will not be OSCCR staff, the coders are members

of OSCCR and heavily invested in the program. It is important to address this as a possible

limitation to the qualitative evaluation method. Up until this point in the evaluation, the

evaluator and other members of OSCCR staff have been excluded as a way to ensure student

feedback is not skewed. However, the intent in having the evaluator and OSCCR staff code is in

order for them to know and understand the data on a deep level. Also, the personal relationships

between SCB members may also affect the data. It is possible that their relationships with one

another hinders them from sharing honest feedback during the focus group discussions.

However, by using maximum variation sampling, the evaluator hopes to spread out board

members across groups, minimizing the closer relationship within each group. Lastly, using

pre-determined codes may allow for the evaluator to miss or misidentify themes outside of the

six in the rubric. Knowing this, the evaluator is aware of the ability to create more codes if

necessary.

Timeline

In order to ensure that the evaluation process moves forward, a timeline (see Appendix J)

was created to ensure that all steps of the evaluation are carried out in an timely and efficient

manner. Although it is ideal that a similar evaluation plan be implemented each year, this plan

outlines from July of 2015 to June of 2016, beginning just before the next academic calendar and

ending just after. Beginning in July of 2015, I will continue to work with stakeholders in order

to finalize any last detail or adjustments to the evaluation plan. In August, one day before SCB

goes on their training retreat, the pre-test web survey will be distributed. In early September, I

will analyze the data collected from the pre-test survey using SPSS. From October to April, no

tasks need to be completed for the plan. Once April, the tasks increase significantly. The post-


30/69


survey will be distributed early to mid April, along with the invitation to participate in the focus

group interviews. Focus groups will be held late April as to not interfere with final exams in

May. In May, we will analyze the quantitative data, as well as transcribe, code, and analyze the

qualitative data. Finally, we will pull together the report and sharing it with stakeholders in June.

This leaves time over the summer for OSCCR staff to make changes and improvements to the

SCB program based on the results of the evaluation. Although this timeline is in place in order

to keep the evaluation moving forward, we do have the ability to make adjustments to the

timeline if needed.

Budget

In addition to the timeline, a detailed budgeted is necessary to ensure that the evaluation

is feasible and fiscal responsibility is upheld. The budget (see Appendix K) outlines the

expected total costs, as well as a cost breakdown, for an entire phase of the evaluation. Overall,

the costs are fairly minimal. The majority of spending is in the dinner incentive for focus group

participation. Although this cost is not directly related to carrying out the evaluation procedures,

I do not anticipate that stakeholders, particularly the director of OSCCR, will consider this costs

unreasonable since students are giving up a significant amount of time for the programs benefit.

Additionally, without this incentive, we run the risk of an even smaller sample population, which

in turn further decreases the value of the data.

The higher cost of focus group incentives is countered by the extremely minimal cost of

other resources. Because of the nature of their work, OSCCR already owns an audio recording

device, a cost that may have been significant. Also, as a part of LUC, OSCCR has access to the

statistical analysis software, SPSS, needed to analyze that data. Most notably, OSCCR will not

have any extra cost for the staff resources, as the evaluator and focus group moderator,


31/69


transcriber, and coder are graduate assistants or staff within the division. Overall, the low cost of

the evaluation makes it feasible to continue for many years to come.

Next Steps

As this is the first time SCB will be formally evaluated, there are several future steps to

consider in order to continue developing the SCB program. First and foremost, the results of the

evaluation will be used to inform programmatic changes intended to strengthen the

developmental experience of SCB members. It is essential that the data from the evaluation be

used to create a clear and purposeful plan for improving the organization and student experience.

Given the timeline, this plan can be created over the summer months and implemented come the

start of the 2015-2016 school year.

In looking to the future, we also consider improvements to the evaluation itself. As

stated, the purpose of this first evaluation is determine whether learning outcomes are being met

and to gain a deeper understanding of the student experience on SCB. This is a basic evaluation

aimed at understanding the student developmental experience. With this said, there is

opportunity in future to further develop the evaluation plan in order to account for more of the

complexities within the SCB student experience. For example, it is relevant to understand how

the Board Chairpersons development differs from the general member. It is also necessary to

consider members who return for a second and their year, as their development can be tracked

over the course of multiple years.

Finally, we are interested in considering other means evaluating the SCB program in

order to get a full understanding from stakeholders. Specifically, this evaluation does not

incorporate the experience of students who come before SCB in hearings. This student

perspective is essential to understanding not only how developed the skills are of SCB members,


32/69


but also the impact that SCB has on their peers through interaction in the conduct process.

Overall, the hope is the continue to develop a program in which SCB members are achieving the

desired learning outcomes through developmental experiences including conduct hearing and

leadership development activities, resulting in a positive impact to their Loyola community.


33/69


References

Cilente, K. (2009). An overview of the social change model of leadership development. In S. R.

Komives, W. Wagner, & Associates (Eds.),Leadership for a better world:

Understanding the social change model of leadership development(pp. 43- 78). San

Francisco, CA: Jossey-Bass.

Creswell, J. (2009).Research Design: Qualitative, Quantitative, and Mixed Methods

Approaches(3rd ed). SAGE Publications: Thousand Oaks, CA.

Fitzpatrick, J., Sanders, J., & Worthen, B. (2003). Program evaluation: Alternative approaches

and practical guidelines

(3rd ed.) New York: Longman.

Fitzpatrick, J. L., Sanders, J. R. & Worthen, B. R. (1997). Program evaluation: Alternative

approaches and practical guidelines(2nd ed.). New York, NY: Addison Wesley

Longman.

Huck, S.W. (2012).Reading statistics and research (6th

edition). Boston, MA; Pearson Maki,

P.L. (2010).Assessing for learning: Building a sustainable commitment across the

institution (Second Edition). Sterling, VA: Stylus.

Lancaster, J. M., & Warywold, D. M. (2008). Student conduct practice: The complete guide for

student affairs professionals.Sterling, VA: Stylus

McCrum-Gardner, E. (2010). Sample size and power calculations made simple.International

Journal of Therapy and Rehabilitation (17)1, 10-14.

Office of Student Conduct and Conflict Resoluion (2014).Mission. Retreived from

http://www.luc.edu/osccr/about/mission/

Schuh, J.H. & Associates (2009).Assessment methods for student affairs. San Francisco, CA:

Jossey-Bass


34/69


Wagner, W. (2009). What is social change? In S. R. Komives, W. Wagner, & Associates (Eds.),

Leadership for a better world: Understanding the social change model of leadership

development(pp. 7- 42). San Francisco, CA: Jossey-Bass.

Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (Eds.) (2010).Handbook of Practical Program

Evaluation(3rd ed). Jossey-Bass: San Francisco, CA.


35/69


Appendix A

!"#"!$ &'()*+,(-# ./(.$0""11(.* "1 ,-&2*'- ."'2&.- $'2 ."'1!(.- +*,"!&-("'

345673458

,.9 545: ,.9 ;?@;A .B;


36/69


,.Z ?>R FDSD< ]DG ?>


37/69


1;?@A@J;J>Q: 2;;=IDF:

7 (=DJBDQFd WBD< ?>QD JBD U>;Q=


38/69


G%H'-E'# =I> II@JIDQ=DQ J> ?QD;JD O>F@J@SD ?B;IIPR JB;J ?>IIPII@JJD= ;PJ =>@V -B@F FDFF@>< R>?PFDF >< G>PQ ?>II@JID II>< \PQO>FD

2DF?Q@OJ@>II>< OPQO>FD @F DFFD

CQ>PO JQPFJ ;FD FDQSDF ;F ; FB;QD=

DAAD?J@SDAG J>W;Q= ; S@F@>< >Q C>;AV -B@F

FDFF@>< W@AA R>?PF >< =DSDA>O@II>< OPQO>FD JB;J CP@=DF >PQ

DRR>QJF J>W;Q= JBD R@[D?JV

1;?@A@J;J>Q: $A@ +D@IDA

( AA;U>Q;J@>AA;U>Q;J@>JB W@JB@< CQ>PO ;POL

IPAJ@OA@DF ; CQ>POdF DRR>QJF WBD< W>Q]@W;Q=F ; ?>II>< OPQO>FDV

.>AA;U>Q;J@>< ?;O@J;A@YDF >< JBD =@RRDQD


39/69


JBD@Q >W< ?>AA;U>Q;J@SD IDDJ@


40/69


7 $W =@= (^WD @IOQ>SD JBD FJ;JPF eP>g />W

=@= WD ?QD;JD ; UDJJDQ !&. ?>IIPW =@= (^WD

=DI>IR>QJ;U@A@JG W@JB JBD ;IU@CP@JG JB;J

;??>IO;G IG >W< S;APDFL

F]@AAFL ; JBD ?>II>< OPQO>FDg

7 (=D


41/69


Appendix B

SCB Program Logic Model


42/69


Appendix C

Construct Map and Item Matrix

Pre-Test

Logic

Model

Component

SQ

# Question

Answer

Type

Answer

Option

General

Information 1 Name Text box N/A

2

Please indicate you class standing. Select one

Sophomore

Junior

Senior

3Please indicate you role on SCB this

year.

Check all

that apply

New member

Returningmember

Board Chair

4Please indicate how important the

following were in your decision to join

SCB using the below scale.

Likert Scale

Check box

3-point scale

Very important

to Not

important

7Please indicate how important it is for

you to gain the following from you SCB

experience

Likert Scale

Check box

3-point scale

Very important

to Not

important

aExperience that will enhance my

resume

bA way to positively impact my

community

cFind a sense of community and support

at Loyola

d Building my capacity in communication

e Improving listening skills

f Working in groups

g Improving conflict resolution skills

h Engaging in consensus building

iHow do hold myself and others

accountable


43/69


Short Term

Outcomes

5How would you rate yourself in the

following skills using the following scale

Likert Scale

Check box

5-point scale

Major strength

to Major

weakness

a Listening skills

b Effective questioning skills

c Overall communication skills

d Consensus building

e Navigating difficult group dynamics

f Accountability

g Critical thinking and analysis

hOpenness to having personal views

challenged

i Leadership ability

j Conflict resolution

Medium

Term

Outcomes

6 Since starting at Loyola, how often do

you engage in the following

Likert Scale

Check box

5-point scale

Very often to

Never

a Explore my personal values and beliefs

b Practice congruence with my beliefs

cPractice commitment and follow-

through

dDevelop a common purpose with others

based on values

eCollaborate with peers to create a

change

f Engage in conflict constructively

gParticipate in service or community

engagement activities


44/69


Post-test

Logic

Model

Component

SQ

# Question

Answer

Type

Answer

Option

General

Information 1 Name Text box N/A

General

Information

2

Please indicate you class standing. Select one

Sophomore

Junior

Senior

3Please indicate you role on SCB this

year.

Check all

that apply

New member

Returning

member

Board Chair

6

Please indicate the extent to which you

have gained the following from you SCB

experience:

Likert Scale

Check box

3-point scale

Very much to

Not at all

a Experience that will enhance my resume

bA way to positively impact my

community

cFind a sense of community and support

at Loyola

d Building my capacity in communication

e Improving listening skills

f Working collaboratively in groups

g Improving conflict resolution skills

h Built consensus building skills

iHow do hold myself and others

accountable

Short Term

Outcomes:

Hearing

skills

4How would you rate yourself in the

following skills using the following scale

Likert Scale

Check box

5-point scale

Major strength

to Major

weakness

a Listening skills

b Effective questioning skills

c Overall communication skills

d Consensus building


45/69


e Navigating difficult group dynamics

f Accountability

g Critical thinking and analysis

h

Openness to having personal views

challenged

i Leadership ability

j Conflict resolution

Medium

Term

Outcomes:

Leadership

Dev

5 Since starting at Loyola, how often do

you engage in the followingLikert Scale

Check box

5-point scale

Very often to

Never

a Explore my personal values and beliefs

b Practice congruence with my beliefs

c Practice commitment and follow-through

dDevelop a common purpose with others

based on values

eCollaborate with peers to create a

change

f Engage in conflict constructively

gParticipate in service or community

engagement activities


46/69


Appendix D

Student Community Board: Pre-Test Survey

You are being invited to complete this survey as a member of the Student Community Board.

Your participation is voluntary, but we hope that you will aid in our evaluation of the StudentCommunity Board as a way to continue improving members experience.

This survey is the first component of the evaluation procedure. At the end of the school year, youwill be invited to take a follow-up survey. Although you will be asked for your name, the results

of this survey will not be shared with anyone outside of the evaluation team.

Should you have any questions, please contact: Dana Broadnax, Director of Student Conduct and

Conflict Resolution at =W0)=*$9Z%&1[/=&


47/69


Student Community Board: Student Pre-test Survey

1. Name [text box]

2.Please indicate you class standing (choose one):

7 Sophomore7 Junior7 Senior

3. Please indicate your role on SCB this year (choose all that apply):

7 New member7 Returning member7 Board Chair

4. Please indicate how important the following were in your decision to join SCB using thebelow scale:

1- Very important 2-somewhat important 3- Not important

a)SCB aligns with my career and/or academic interestsb)I was looking for a way to get involved on campusc)I had been through the conduct process before which prompted my interest in

SCB

d)Other (if you select this option, please explain in the box below)[text box]

5. How would you rate yourself in the following skills using the following scale:

5- Major Strength 4-Strength 3- Average 2- Weak 1- Major Weakness

a) Listening skillsb) Effective questioning skillsc) Overall communication skillsd) Consensus buildinge) Navigating difficult group dynamicsf) Accountabilityg) Critical thinking and analysish) Openness to having personal views challengedi) Leadership abilityj) Conflict resolution

6. Since starting at Loyola, how often do you engage in the following:

5- Very Often 4- Often 3- Sometimes 2- Seldom 1- Never

a) Explore my personal values and beliefsb) Practice congruence with my beliefsc) Practice commitment and follow-throughd) Develop a common purpose with others based on values


48/69


e) Collaborate with peers to create a changef) Engage in conflict constructivelyg) Participate in service or community engagement activities

7. Please indicate how important it is for you to gain the following from you SCB experience:

3- Very important 2-somewhat important 1- Not important

a) Experience that will enhance my resumeb) A way to positively impact my communityc) Find a sense of community and support at Loyolad) Building my capacity in communicatione) Improving listening skillsf) Working in groupsg) Improving conflict resolution skillsh) Engaging in consensus buildingi) How do hold myself and others accountable

Thank you for completing the SCB Pre-test survey!


49/69


Appendix E

Student Community Board: Post-test Survey

You are being invited to complete this survey as a member of the Student Community Board.:

Your participation is voluntary, but we hope that you will aid in our evaluation of the StudentCommunity Board as a way to continue improving members experience.

This survey is a follow- up to the survey you took at the beginning of the school year. Like thatsurvey, all information you provide will only be shared with the evaluation team.

Should you have any questions, please contact: Dana Broadnax, Director of Student Conduct and

Conflict Resolution at =W0)=*$9Z%&1[/=&


50/69


Student Community Board: Post-test Survey

1. Name [text box]

2[ Please indicate you class standing (choose one):

7 Sophomore7 Junior7 Senior

3. Please indicate your role on SCB this year (choose all that apply):

7 New member7 Returning member7 Board Chair

4. How would you rate yourself in the following skills using the following scale:

5- Major Strength 4-Strength 3- Average 2- Weak 1- Major Weakness

a) Listening skillsb) Effective questioning skillsc) Overall communication skillsd) Consensus buildinge) Navigating difficult group dynamicsf) Accountabilityg) Critical thinking and analysish) Openness to having personal views challengedi) Leadership ability

j) Conflict resolution

5. Since starting at Loyola, how often do you engage in the following:5- Very Often 4- Often 3- Sometimes 2- Seldom 1- Never

a) Explore my personal values and beliefsb) Practice integrity and congruence with my beliefsc) Make a commitment and follow-through with itd) Collaborate with peers to create a changee) Engage in organized conflictf) Participate in service or community engagement activities

6. Please indicate the extent to which you have gained the following from you SCB experience:3- Very Much 2-Somewhat 1- Not at all

a) Experience that will enhance my resumeb) A way to positively impact my communityc) Find a sense of community and support at Loyolad) Built my capacity for effective communication


51/69


e) Improved listening skillsf) Worked collaboratively in groupsg) Improved conflict resolution skillsh) Built consensus building skillsi) How to hold myself and others accountable

Thank you for completing the SCB Post-test survey!


52/69


Appendix F

Qualitative Introduction Email

Hi ______,

Recently at our last leadership development session, I talked to you all about participating

in a focus group discussion about your experience as an SCB member. As a valuable member of

SCB, I would highly encourage you to participate in one of the three, one-hour discussion. Like

I said during out meeting, a meal of your groups choice will be brought in for you all.

Your feedback is essential to ensuring that SCB continues to be a valuable and positive

experience for members. Considering all of the changes to SCB this year, we would really like to

know how it went and how it could be better.

The link below will prompt you to select your availability over the next two weeks. The

discussion will be scheduled during a time that works for you and you get free food. What more

is there to ask for!

link

Please let me know if you have any more questions about the focus group.

Best,

Ali Reimel

Graduate AssistantOffice of Stduent Conduct and Conflict Resolution

[email protected]


53/69


Appendix G

Focus Group Protocol

Preliminaries:

Introduction: Hi everyone, my name is __________. My role is at Loyola is _________. I amreally looking forward to our discussion today. For the next hour to hour and a half we will betalking about your experience as an SCB member over the past year.

Purpose: The purpose of this focus group is to gain a deeper understanding of your experience as

SCB members. I will be providing questions that will guide our conversation, but this is meant tobe a space for you all to provide open and honest feedback about your experience. Please feel

free to share as much or as little as you feel comfortable with. I will be recording this discussionand then transcribing it, so all of the identifying information will be excluded. This information

will be used to evaluate SCB as a program and leadership development experience. Do you haveany questions so far?

Now, I will direct your attention to the consent form in front of you. Please take a couple minutes

to read it and let me know if you have any questions or concerns. If not, please sign it and pass itin to me. If you havent already gotten your food or would like more, feel free to get some more

and we will start shortly.

Focus Group Discussion:

First, why did you choose to join SCB?

Segment One (overall experience):1. Can you share with me your general feeling about your experience on SCB this year?

a. What did you like? What did you not like?b. How did you feel about the format and structure of SCB?c. How did you find SCB beneficial?d. How could SCB continue to be improved in the future?e. Was there anything you wanted from your SCB experience that you did not get?f. Was there anything that you gained from SCB that you did not expect?

Segment Two (hearing skills development):

1. What skills were needed to be successful in hearing cases of alleged student misconduct?2. How were these skills developed over your time in SCB?

a. Did you feel prepared to hear cases of alleged student misconduct?b. What about the training retreat was most beneficial?c. How could the training retreat be improved?

3. How will you utilize these skills beyond SCB?

Segment Three (leadership development):

1. What are you general feeling about the leadership development curriculum?


54/69


a. For returning members, how would you compare last year with this year inregards to the curriculum?

b. What parts of the curriculum did you like?c. What parts of the curriculum did you dislike?d. How could the curriculum be improved?

2. What skills or insight did you gain from the leadership development sessions?a. Was there any session or topic that were extremely beneficial? Not beneficial?b. How will you utilize these skills beyond your SCB experience?

3. How do you feel that you impacted the LUC and great community?a. How did you effect change through the hearing piece of the SCB experience?b. How did you effect change through the culminating service day, Project

Citizenship

Conclusion:We have reached the end of my questions. Does anyone want to add any feedback, additional

questions, or comments that they would like to add?

Again, thank you for your participation in the focus group. You input and feedback is invaluablein continuing to improve SCB in the years to come. I hope that you have also found this

conversation to be an opportunity to reflect on your own experience and how you can keeputilizing all that you have gained in order to continue making a positive impact in your

community. Please feel free to contact me or Ali Reimel if you have further questions.


55/69


Appendix H

Consent Form

Project Title: Student Community Board Focus Group

Evaluator: Ali Reimel

Introduction: You are about to take part in a focus group to help the OSCCR gain a better

understanding of your SCB experience. You are being asked to participate in this focus groupbased upon your completion of the follow-up survey conducted.

Please read through the following information carefully. If you have any questions regarding the

information below, please ask the moderator prior to deciding whether to participate.

Purpose: The purpose of this focus group is to gain information about SCB membersexperiences.

Procedures: Once you agree to participate in the focus group, you will be asked a series of

questions pertaining to your perspective on SCB. Please respond openly and honestly, as there isno right answer. Please know that you are not required to respond to any of the questions and

should only respond based on your comfort level

Risk/Benefits: There are no known risks involved in participating in this study. Benefits toparticipating in this study may include further improvement to ACB and a deepened

understanding of learned outcomes experienced as a result of participation the studentorganization.

Confidentiality: By participating in the focus group, your name will be omitted from any

responses you provide. Any information provided will be compiled in a report and basic themeswill be shared with your prior to finalizing it. The information shared today will only be shared

with OSCCR staff.

Voluntary Participation: Participation in this focus group is voluntary. At any point in the focusgroup, you are free to withdraw from participation. Please know that you are not required to stay

for the entire time.

Contacts and Questions: If you have any questions about the focus group, you may contact AliReimel at [email protected]

Statement of Consent By signing below, you will be agreeing to the information provided around

the focus group. Additionally, you have had a chance to ask any concerning questions and agreeto participate in the following interview. A copy of this document will be provided for your

records.

_______________________________________Participant's Signature


56/69


_____________

Date_______________________________________

Evaluators Signature


57/69


Appendix I

Coding Rubric

)%0&/#"+/ )%N' L'30*0; 12'-'

/+0 /D;Q@IID BD;Q@IIPIIP


58/69


Appendix J

Evaluation Timeline

2015 2016

Task July Aug Sept Oct Nov Dec Jan Feb March April May June

Finalize

evaluationplain with

OSCCR

x

Administerpre-test

(day prior to

trainingretreat

x

Enter pre-test data in

SPSS

x

Administerpost-test

survey

(immediately

after final

leadership

development

session)

x

Invite SCB

members toparticipate in

focus groups

x

Enter post-

test datainto SPSS

x

Condcut

focus groupsx

Run analysisof

quantitativedata in SPSS

and organize

x

Listen to and

transcribe

focus groupinterviews

x


59/69


Code and

analyzequalitative

data

x

Prepare final

reportx

Share report

with

stakeholders

x


60/69


Appendix K

Evaluation Budget

Activity Item Cost per item Quantity Total $

Survey

Administration

Google form $0.00 30 $0.00

Focus Groups

Audio Recording Device $0 (pre-owned) 1 $0.00

Dinner $12 15 $180.00

Soda $2.00 3 $6.00

Plate, cups, utensils $1.00 15 $15.00

Room reservation $0 3 $0.00

Statistical Analysis

SPSS $0 (pre-owned) 1 $0.00

Reporting

Spiral bound reports $5 5 $25.00

Power point $0 1 $0.00

Total

cost: $226.00


61/69


Appendix L

Presentation


62/69



63/69



64/69



65/69



66/69



67/69



68/69



69/69


Evaluation Project

Documents