Evaluation Group Project

Running head: EVALUATION PLAN FOR BRIDGE TO LOYOLA 1

Evaluation Plan for Bridge to Loyola

Marlena Yang & Awatif “Ruth” Elias

Loyola University Chicago

EVALUATION PLAN FOR BRIDGE TO LOYOLA

2

Table of Contents

OVERVIEW OF BRIDGE PROGRAM 3 Introduction of Institution and Bridge to Loyola 3 History of Bridge to Loyola 4 Evaluation Purpose 6 Introduction of Evaluation Description 7 Bridge to Loyola’s Summer Program 8 Stakeholders 12 Logic Model 13 Assumptions and External Factors 15 Previous Evaluation 17

QUANTITATIVE APPROACH 17 Survey Description 18 Implementation and Administration 19 Statistical Analysis 20 Data Presentation 20

QUALITATIVE APPROACH 21 Participants 22 Focus Group Procedure & Implementation 22 Qualitative Data Analysis 24 Presentation Results 25

BUDGET 26 TIMELINE 26 NEXT STEPS 26 REFERENCES 28 APPENDICES 29

Appendix A: Logic Model 30 Appendix B: Survey Construct Map 31 Appendix C: Survey 32 Appendix D: Focus Group Email 36 Appendix E: Consent Form 37 Appendix F: Focus Group Protocol Script 38 Appendix G: Budget 40 Appendix H: Focus Group Sign-up Sheet 41 Appendix I: Timeline 42


3

Overview of Bridge Program

Postsecondary institutions enroll approximately 40% to 60% students who are in need of

additional support and further transition from high school to college (NCES, 2012). Universities

have instituted summer bridge programs in an effort to support these students in adjusting to

college life (Sablan, 2010). Summer bridge programs vary from one college or university to

another depending on the student population they serve. Bridge programs are implemented for

students needing additional support such as first-generation, low-income students,

underrepresented minority students or those who test as remedial (Kallison & Stader,

2012). Students identified to participate in these summer bridge programs either have low high

school grade point averages (GPAs), low scores in standardized tests such as the American

College Testing (ACT), the Scholastic Aptitude Test (SAT), and those majoring in a field that is

of focus for the program like science, technology, engineering and math (STEM) (Kezar, 2000,

as cited in Sablan 2010).

Bridge programs are intended to help students adjust to college socially and academically

by offering activities that include tutoring, study skills, advising, academic instruction and

general information about navigating college (Gullatt & Jan, 2003). We hope that by conducting

a process-based evaluation plan for a summer bridge program at Loyola University Chicago, we

will gain insights about the program’s effectiveness in the delivery of activities (workshops,

academic course, group service project, group extracurricular activities, & study hall) it offers its

students.

Introduction of Institution and Bridge to Loyola

Loyola University Chicago (LUC) is a private Catholic Jesuit institution founded in 1870

and is located in Chicago, Illinois. Loyola University Chicago is the largest Jesuit University in


4

the country. It has a total enrollment of 15,957 students with 10,168 undergraduate

students. This institution has six campuses: Lakeshore campus, Water Tower campus, Water

Tower campus, Health Sciences campus located in Maywood, John Felice Rome Center in Italy,

Beijing campus in China, and Ho Chi Minh City campus in Vietnam. Loyola’s mission focuses

on a diverse community that seeks God in all things and works to expand knowledge in the

service of humanity through learning, justice and faith.

The Bridge to Loyola program is a one-year (summer, fall, and spring), academic support

program providing services that include an intensive three-week summer program, academic

workshops, specialized academic advising and seminar class in both the fall and spring

semesters. The focus of this evaluation will be the summer portion of Bridge, which is a three-

week residential program all Bridge students are required to take before being fully admitted to

Loyola. Further details and information will be provided in later sections.

History of Bridge to Loyola

The Bridge Program at Loyola has taken different forms since its creation. The first

Bridge Program, the Learning Enrichment for Academic Progress (LEAP) was created in 1984.

LEAP started out as a summer bridge program that took place between one to two weeks during

the summer season. LEAP admitted students who had not met the admission criteria based on

ACT scores a chance to be fully admitted to Loyola once they participated in the summer

program. Participants in LEAP were required to attend lectures in which they were quizzed on

material covered including workshops that addressed academic and social aspects of being a

Loyola student. LEAP was later extended to a full academic year to continue supporting

students in their transition to college. Students took a university seminar, a three-hour credit


5

course in the fall, as well as a spring UNIV 102, a one-hour credit course to further build upon

their academic skills and adjustment to Loyola.

Success Through Educational Preparation (STEP) is another separate Bridge Program

from LEAP that has been offered at this institution. This program offered conditional admission

to students who did not meet Loyola’s admission requirement based on their high school grade

point averages (GPAs). STEP sessions often took place over a six-week period. Unlike LEAP,

STEP students were graded on attending study workshops, social activities, and on the academic

course. Students had to obtain a passing grade in the academic course in order to be admitted to

Loyola. However, in 2010, LEAP and STEP were combined to become the new Bridge to

Loyola Program due to reasons about funding and staff.

The Office of First and Second Year Advising (FSYA) operate the new Bridge to Loyola

program. Currently, Bridge to Loyola serves students who do not meet Loyola’s ACT/SAT

score requirement, but have the potential to succeed based on their high school grades. Bridge

serves students from all diverse backgrounds that fall under this certain category. It is designed

to help students who either lack motivation, have personal circumstances that inhibit their

academic performance, as well as students who are highly motivated, but come from

academically less rigorous high schools and therefore need additional support. Bridge to Loyola

was initiated to serve this student population at this specific institution. As Garcia (1991) stated,

a university or college may choose to require first-generation students, low-income students,

students of color, or students who have lower than average admissions tests and high school

GPAs to participate in a summer bridge program to raise the chances that these applicants will be

retained.


6

Evaluation Purpose

For the purpose of our evaluation plan, we will solely focus on the process of Bridge to

Loyola’s summer program versus the program as a whole, which includes the summer program

as well as a fall and spring portion. This process-oriented approach will guide us in describing

how effective Bridge to Loyola Summer Program services are delivered (Fitzpatrick, Sanders, &

Worthen, 2011). Our evaluation will be formative since its main purpose is to provide

information for program improvement (Fitzpatrick et al., 2011).

Ultimately, the overarching question we hope to address through the evaluation process is

how effectively is Bridge to Loyola’s summer program delivering its program content through

the services (academic seminars, summer course, community based learning project, study

groups, community programs, and living on campus) it offers? By effective, we want to measure

how useful the services are in meeting the needs of the students. For each of the services, we are

eager to learn if the content is applicable to the students’ personal and academic

needs. Discovering the effectiveness of each service will help the stakeholders tailor the

program content when necessary. Essentially, this evaluation plan is meant to be the starting

point for a potential process-oriented approach evaluation plan. Some overriding questions we

hope to answer after the evaluation is complete are as follows:

• Academic seminars: How effective stakeholders are in facilitating the content of the

following workshops: Learning styles, time management, and study and test preparation

skills.

• Community based service learning: Is there a reflective component post-volunteer

experience? If so, how are students expanding their knowledge?


7

• Academic course: How are students’ retaining content material within the intense three-

weeks of the program?

• Community programs: how are the activities designed, delivered and in what type of

environment and whether they are inclusive

Introduction of Evaluation Description

Given what we hope to tap into, our evaluation plan is intended to be a formative

evaluation. This type of evaluation will help to uncover ways to better deliver a program

(Wholey, Hatry, & Newcomer, 2010). By doing so, the evaluation approach will focus on the

process more so than the outcome, which former evaluation plans have generated in the past. By

referencing our evaluation question: “how effective is Bridge to Loyola’s summer program in

delivering its six program service components (academic success seminars, three-credit

academic course (CIEP 111), community based learning project, study hall sessions, community

programming, and live on campus experience) it offers?

By effective, we want to understand and fully grasp if the program is delivered according

to their learning outcomes and summer program benefits. We believe this process approach will

help us accomplish most of our program objectives. Since the focus is on the process, we hope

to gather information from students’ perspectives, observations of student engagement and make

alterations based on the information gathered to improve subsequent services. The process

approach is important because it will give insights about what is happening to students’

experiences while the services are being delivered, as well as inform if it is accomplishing the

intended outcomes set out in the program content.

Like every evaluation conducted, our program has potential strengths and weaknesses.


8

An area of strength for our process evaluation approach is it allows us to gather information

through observations and collect feedback from students, which will be used to make changes

while the summer program is still in session. This approach gives us the opportunity to focus

both on the program content, which has often been paid less attention to and on students. We are

also able to have insight on what parts of the program work well and which do not.

One potential weakness is overlooking student perspectives in shaping the program

services. The more varying perspectives present, the more they contribute in the creation of its

program services. However, since our main focus is centered on program content, we can easily

miss that student perspective. In addition, as evaluators, we do not have a direct relationship

with Bridge, therefore we might assume we know what Bridge needs but as is not true, what we

think they need might not align with their actual needs. We acknowledge that our subjective

perspectives might influence our evaluation.

Bridge to Loyola’s Summer Program

As mentioned in the introduction, the Bridge to Loyola program is a one-year, academic

support program providing services that include an intensive three-week summer program,

academic workshops, specialized academic advising and seminar class in both the fall and spring

semesters. However, the focus of this evaluation will be on the summer program, which is a

three-week residential program that all Bridge students are required to take and pass with a C- or

higher in order to be fully admitted to Loyola. In this section, we highlight the details of Bridge

to Loyola’s summer program in introducing the benefits, learning outcomes, and program

specifics.

The entirety of Bridge to Loyola’s summer program is through six service components:

academic success seminars, a three-credit academic course (CIEP 111), community building


9

programs, study hall sessions, and a community based learning project. It is through these areas

that Bridge students benefit personally, socially, and academically. The program is intended to

create a well-rounded college experience of what its like to be college student. It is designed to

mirror a typical first-year student schedule with classes, study time, and co-curricular activities.

In addition, students get to live on campus in one of the university residence halls with an

assigned roommate. In regards to the program learning outcomes, the overarching goal is to

develop self-efficacy and personal growth for students to be successful while transitioning to

their first year in college (Hill & Teetsov, 2014).

Bridge to Loyola strives to align their learning outcomes through the six service

components aforementioned. In doing so, the program kicks off with the mandatory orientation

program, which all incoming first-year students are required to attend. All Bridge to Loyola

students will arrive to campus ready to settle in the residence hall for three weeks. Loyola’s

first-year orientation program runs for two consecutive days welcoming students and their

families to the Loyola community. Soon after orientation is complete, the official Bridge to

Loyola program begins immediately with a warm welcome reception. After the welcome

ceremony, students’ depart directly to Loyola University’s Recreation and Ecology Center

(LUREC) for a weekend retreat filled with team-building activities, opportunities for self-

reflection, and workshops. In respective order, we will describe six of the service components

that make up the summer Bridge program: academic success seminar, academic course (CIEP

111), community based learning project, community programming, study hall, and living on

campus.


10

Academic Success Seminars

These are academic strategy workshops provided for Bridge students throughout the

summer. The stakeholders and campus partners facilitate all workshops with themes that include

reading and note-taking strategies, learning styles, time management, and studying and test

preparation skills. Each workshop session takes place for an hour during the weekday and is

mandatory for all students to attend.

CIEP 111 Academic Course

Bridge students are required to take a three-credit introductory course titled Sex and

Gender in Anthropological and Psychological Perspective. This course is intended to provide

students with a scientific foundation upon which anthropological and psychological concepts can

be built. The following are examples of a few of the learning goals presented by Grauer and

Leon (2013), which students will be introduced and expected to understand:

• Processes of human development

• Biological mechanisms of evolution

• How scientific conclusions and theories are developed

Community Based Learning Project

The stakeholders work in conjunction with community partners in finding service

learning opportunities for all Bridge students. This is the student’s opportunity to branch outside

of Loyola’s campus and explore different parts of Chicago. Students volunteer every Saturday

morning for a couple hours during the three weeks of the program.

Community Programming

The peer mentors and the graduate intern spearhead the community programs. These

programs are social activities intended to a build a sense of community. The activities are what


11

students look forward to after a long and intensive week of academics. The following are

examples of weekend programs from this past summer: target run, on-campus scavenger hunt,

movie night, Navy Pier fireworks, arts and crafts, pajama party karaoke night, and study break

game night. Each activity is intended for students to build community and develop relationships

with their peers.

Study Halls

Each night Sunday through Thursday, there is a study hall session from 6:30 p.m. to 8:30

p.m. Each study session is mandatory and expected for all student participants to take part in.

Students are provided with academic support and are surrounded by peers creating a supportive

and encouraging learning environment.

Living on Campus

Students get to first hand experience of what its like to live on a college campus. As part

of the three-week summer program, students live on campus in one of Loyola’s residence halls.

The peer mentors serve as resident assistants and the graduate intern’s role is similar to an

assistant residence director.

Post Bridge to Loyola Summer Program

By the end of the program, students who have successfully completed the summer

program will continue with Bridge to Loyola in the fall and spring completing the one-year

program. Bridge students will take a one-credit First-Year Seminar course (UNIV 101), which is

a university requirement for all first-year students. The primary stakeholders (discussed in the

next section) teach the course as well as academically advise Bridge to Loyola students. In the

spring, students will then take a CIEP 112 Skills Strategies course also instructed by the primary

stakeholders.


12

Stakeholders

There are a number of campus partners, community partners, as well as professional

faculty and staff who contribute to the operation of Bridge. In this section we will introduce the

primary stakeholders and other campus partners who help make this program possible and

function properly.

The primary stakeholders for Bridge to Loyola are academic advisors China Hill and

Natasha Teetsov from the Office of First and Second Year Advising (FSYA). Once the

Admissions Office (another stakeholder) determines who and how many students are selected

into Bridge, China and Natasha are given student information to coordinate the program that will

accommodate the selected group of students. Interestingly, China and Natasha do not participate

in the selection process, which is an important perspective they hope to gain for the upcoming

Bridge program. With that said, each summer is a different summer to look forward to, as the

number of students will determine the dynamic of the program. Last year, there were

approximately 100 student participants and 36 this past year. However, Natasha has indicated

that for the following summers, the numbers will remain around 40 students (Natasha Teetsov,

Personal Communication, November 18, 2014). A graduate intern, supervised by Natasha, also

works closely with the primary stakeholders in developing and executing programs before,

during, and after the summer portion of Bridge. The intern works directly in supervising,

training, and advising peer mentors in preparation for the students’ arrival and all social

programs. Aside from the primary stakeholders, a graduate intern, and the admissions office,

faculty members from the Department of Psychology and the Department of Anthropology are

also involved in the three-week summer program. Both faculty members co-instruct the Bridge

to Loyola summer CIEP 111 three-credit hour academic seminar course.


13

In terms of individuals and groups who are directly and indirectly affected by the

program, students are the ones personally and academically benefitting as the program is solely

created to support them. Meanwhile, we recognize the Loyola community, Bridge parents, and

community partners are groups who indirectly derive benefit based on the improvement and

success of the program. Bridge is aimed to amplify students’ academic performance, social

integration, as well as their ability to persist throughout their educational journey. The Loyola

community will gain students who will most likely know how to navigate campus and reach out

for support when needed. Retention rate is intended to improve, which is a good representation

of the university. Parents of Bridge students become more aware of Loyola resources and will

encourage their students to take advantage of those opportunities. The growth of their students is

in and of itself an affective benefit. As part of the service-learning component of the program,

community partners will gain volunteers to support their organization. All in all, individuals,

groups, and even society, are positively affected in some way.

Logic Model

A logic model is a visual representation that is useful for conceptualizing, planning and

communicating with stakeholders about our planned work and intended outcomes about the

Bridge to Loyola Summer Program (Kellogg, 2004). We will be utilizing a logic model (see

Appendix A) to illustrate inputs, outputs and outcomes. Inputs represent the activities and

resources the Bridge to Loyola summer session invests in. People tend to be the main focus of

inputs (Kellogg, 2004). The summer portion of the Bridge Program includes one staff from the

undergraduate admissions office that is responsible for recruiting students for this program. Two

academic advisors from FYSA are responsible for meeting and advising students, while two

faculty members, one from the Psychology Department and another from the Anthropology


14

Department work collaboratively to create a syllabus for the seminar course to be taught by them

during the summer. Lastly, one graduate intern is available to assist staff in any way needed, and

four peer mentors are present as resources for incoming students in any capacity they need such

as navigating campus, and asking relevant questions.

Additionally, campus partnerships are important in contributing to the success of the

program. The summer session teams up with the university orientation team, academic services

(i.e., tutoring), and residence life since Bridge requires its students to live in campus housing for

the duration of the program. Time is also an important asset spent training staff, reserving

locations where sessions will take place, as well as purchasing and gathering materials. These

materials include bus passes, pens, papers, day-to-day agenda of events and meals.

The activities offered to incoming first-year Loyola Bridge students encompass two main areas:

academic support and programming. Academic support is comprised of academic advising, a

summer course, and workshops about time management, test preparations, and reading

strategies. In programming, Bridge offers activities to help students build community by

participating in the welcome reception, retreat, exploring Chicago and taking part in a service-

learning project.

The short-term changes we expect will result after the completion of each particular

session or sessions. The first two outcomes can be grouped to represent study skills acquired

from attending academic workshops. We hope students can discover different learning styles

and apply them accordingly to the study material and establish effective study habits. The

following three can be grouped to represent a general understanding and application of those

concepts to their own lives. For example, we hope students are able to learn by navigating LUC

and taking initiative in exploring the city of Chicago, enhancing their self-efficacy for academics


15

such as posing questions, and gaining an awareness of college socially and academically. Short-

term changes are intended to lead to medium-term outcomes, which will take place after the

completion of the three-week residential summer program.

Some of the results of the medium-term outcomes include students’ confidence in

reaching out for support when needed, familiarization with where resources are located,

established connections and relationships with peers, and a smoother acclimation to college

during the first week of classes. Students will also likely apply the academic strategies they

learned from summer to their fall classes. The long-term outcomes take place during a student’s

enrollment at Loyola and beyond graduation. We hope students will be able to grow personally

and professionally by developing confidence and self-efficacy, as well as discerning their career

choices. We also hope students are able to take agency of their own learning via asking questions

in class, or establishing study groups with peers to better understand academic content while still

seeking support if needed. We expect them to be motivated to continue their education until

graduation and pursue their career aspirations. After participating in the program, we anticipate

them possibly giving back to the program by becoming mentors, sharing Bridge experiences with

prospective students, or in any other way they find fitting.

Assumptions and External Factors

We also took assumptions and external factors into consideration when developing our

logic model. We considered four assumption areas and four external factors that have a close

relationship with one another.

First, we assume the summer program outcomes set up first-year students for academic,

social and personal success after completion. To describe what we mean by this, Bridge to

Loyola is a well-rounded program aimed to prepare students to have a smooth transition to


16

college, which simply explains what the academic, social, and personal success refers to. This

also puts them at an advantage in navigating campus before the start of school. Next, we assume

first-year students are prepared for engagement and self-reflection, as the program is set-up with

opportunities to do so. Third, we assume campus partners are fully equipped and prepared to

facilitate academic and leadership workshops. Primary stakeholders work collaboratively with

campus partners to present on materials in which students will benefit from. Not only are

students exposed and introduced to opportunities at Loyola, but campus partners get to promote

their programs and through that, active student participation. Finally, our last assumption is that

the residence hall is a comfortable environment to build community. As mentioned earlier,

students are required to live on campus as part of their participation in Bridge to Loyola. Again,

going back to the well-rounded college experience addressed in our first assumption. We

recognize these assumptions connect with our established external factors. With that regard,

demographics, location, student expectations, and different student academic experiences are

identified external factors.

Students arrive to the program with unique experiences and different backgrounds. There

is a diversity of students in terms of social class, racial/ethnic identity, personalities, and first-

generation students. Not only that, but students may enter Bridge to Loyola’s summer program

from a rural area where they are not familiar with Chicago’s fast-paced environment and the

multicultural communities within Chicago. We also have to recognize that students may have

certain expectations prior to starting Bridge based on what they may have heard from peers who

went through the program. Overall, assumptions and external factors are important parts of our

logic model to highlight in our evaluation plan.


17

Previous Evaluation

In regards to previous evaluation, there has not been formal evaluation completed on the

Bridge to Loyola program. No assessment has been created on college statistical data besides

anecdotal data (China Hill, personal communication, September 26, 2014). Anecdotal data is

recorded to take note on successes among the students as well as information about the

circumstances under which students are not persisting. Prior to Natasha and China’s

involvement with Bridge, the former Assistant Director for FSYA worked with two graduate

students in the Higher Education program at Loyola to create an evaluation plan for Bridge to

Loyola. Although we wish to culminate a plan as well, our evaluation approach is different as

we aim to focus on the process of the summer program in terms of the effectiveness of program

delivery, which we describe in the next section. Meanwhile, the previous evaluation plan was an

outcome-based approach solely focusing on if students were successful and transformative by the

end of the program. Given no formal evaluation has been created, academic achievement data

has been collected and analyzed for retention and academic success levels. In 2010, Bridge to

Loyola program saw a 93% retention rate, which was the highest in recent years (“Bridge to

Loyola,” 2011)

Quantitative Approach

For the quantitative part of the evaluation plan, we will study student participants in the

Bridge to Loyola Summer Program. Since we plan to sample the entire student population of

approximately 40 students, the approach we intend to use is the census-sampling frame (Wholey,

et al., 2010). We will not use a comparison group because we currently recognize no relevance

for it. The program tends to make changes in terms of the recruitment process, therefore,

producing fluctuating numbers of student participants each year. For instance, during the


18

summer of 2013, the primary stakeholders had to coordinate the Bridge to Loyola summer

program for 130 students. However, summer of 2014 had 36 student participants. For the

upcoming year, Natasha emphasized that the student numbers will remain around 40.

We are anticipating a 100% student response rate, as there will be time during the

program for students to complete the paper-based survey. Cross-sectional design is the type of

research design we plan to use. It makes sense to utilize this design since our approach is

process-based and because we are distributing the survey at a single point in time for the entire

student population and a variety of program elements. Although a cross-sectional design is

necessary for our approach, it does reveal limitations. There is a higher chance for data-entry

error to occur. To prevent that high chance, we will keep the paper copy of the surveys and

double check responses.

Survey Description

Our survey (see Appendix B) has a total of eleven main questions with sub-questions for

the first seven. The last four questions (Q8 through Q11) are general demographic questions,

which are best asked at the end of the survey. These questions include racial/ethnic identity,

emphasis on parental academic background, campus familiarity, and geographic residence. Each

of these different types provides us with important context to collect when analyzing the data

results. It helps to simply get a general idea where students are coming from.

We sorted the variety of questions into three different categories: personal, academic,

and social. As indicated in our survey introduction and our evaluation program description, there

are six imperative service components of the summer program process we wish to

assess: academic success seminars, academic course (CIEP 111), community programming,

community based learning project, study hall sessions, and live on campus experience. We


19

decided to break up the questions into three categorical areas because we recognized a theme in

the delivery of each of these service components. The method of breaking up the sections into

three will help students fully grasp the intentions of the Bridge to Loyola’s summer program,

which is to create an academic, social, and personal growth experience.

Implementation and Administration

After formulating questions for our survey, we will use pilot testing to help us identify

which questions are confusing or need rewording, to determine the time needed to complete the

survey as well as organizational issues that emerge (Wholey et al., 2010). We will work with our

primary stakeholders Natasha and China, the primary academic advisors for Bridge to Loyola to

reach out to former Bridge participants who are still current Loyola students since the Bridge

Program only lasts for one year. We will be utilizing dual outreach to seek feedback from Bridge

alumni participants through e-mail and in person contact. We encourage all students interested in

helping us out to come look at our survey questions during business hours from 9-5pm before the

spring 2015 semester ends. We also understand that not everyone will be able to come in person,

therefore, those who would like to participate but are unable to come in person will be e-mailed

our quantitative survey and they can respond with recommendations, gaps, and suggestions.

Having multiple students take a look at our survey will give us many perspectives of how they

understand the survey and if the questions are relevant.

Natasha and China will be administering our paper-based survey after a mandatory

workshop event at the beginning of the third week. Since we are expecting about forty students

to complete the survey, it will be feasible to manually enter the data into a database. We will not

have an incentive to allow equal treatment for all participants. Since the academic workshops are


20

a requirement, we assume almost all students will be present and will be able to complete the

survey in the allotted 10-15 minutes timeframe, or however long they take.

Statistical Analysis

For the statistical analyses in assessing the process of Bridge to Loyola Summer Program,

we will use simple descriptive statistics to answer our evaluation questions, which describes the

characteristics of these variables in terms of their central tendency and their dispersion (Schuh,

2009). This method will help us to determine areas of strengths and weaknesses within the

program when looking at the traditional tendencies of mean, median, and mode. Then we will

employ inferential analyses to see whether there are significant differences between demographic

groups or whether particular items on the survey relate to each other.

Furthermore, we will examine results across demographic segments of race and gender

using t-test to identify mean differences on the Likert-scale questions. In examining these

segments, we can interpret whether gender differences exist as well as if different racial groups

experience the Bridge Program services differently. Even though we realize that results will be

useful for the Bridge to Loyola Summer Program, delving into the specific gender and racial

demographics can help us understand how students differ in those aspects and how to implement

changes in those particular areas.

Data Presentation

We will present our data using tables and charts. By using tables, we will describe what

students thought about the delivery of the program in terms of their academic, social, personal,

and live-in experiences. Creating tables will help us measure the extent to which students agree

with the effectiveness of particular program components. After that, we will be using a histogram

and a pie chart. The histogram would be more useful in seeing what activities garner more


21

student interest and by how much. The pie chart will be more useful in presenting demographics

such as race, gender, and type of activities (academic, social and personal).

Qualitative Approach

Qualitative research can utilize different forms such as transcripts, notes from interviews,

observational field notes, videos, and e-mails to gain a deeper understanding of a subject matter

(Wholey et al., 2010). For the qualitative approach of our evaluation, we will use a focus group

approach to help us better understand our evaluation question, and to obtain a richer narrative of

students experiences (Creswell, 2009). Our evaluation question aims to find out how effective

Bridge to Loyola Summer program is in delivering its program content through the following

services it offers: workshops, summer course, service learning, study groups, and social

activities. Schuh (2009) states that focus groups can help evaluators when they “need more

information to help shed light on quantitative data already collected” (p. 69). Conducting a

quantitative survey provided important results, but only general knowledge of responses to those

questions. Thus, carrying out a focus group will enable us to build upon questions from the

quantitative survey by probing for clarity by inquiring why and how events happened.

One of the reasons we chose to employ the focus group approach as opposed to other

methodologies is to elicit ideas, attitudes, feelings or perceptions regarding the process of content

delivery and student learning as perceived through students’ own perspectives. In addition, this

approach enables us to see ideas emerge from the group (Schuh, 2009). For example, when one

participant shares an idea or experience, it is likely that other students will recall similar

experiences they identify with and expand on them, thus generating common themes. Finally,

focus groups provide specific feedback and insights about their experiences.


22

Participants

Bridge students who completed the Bridge to Loyola Summer Program and decided to

attend Loyola will be the only ones asked to participate in focus groups. Since the summer

Bridge program consists of about thirty-six students, we will be conducting two separate focus

groups of six students each. “This number provides for enough conversation while yet allowing

ample time for individuals to express their opinions” (Schuh, 2009, p. 91). Even though we only

need twelve students overall, we will have a Google sign up sheet (see appendix H) with fifteen

spots in case not everyone shows up. Each focus group will be divided into two by ensuring that

there is variation between gender and racial demographics.

The focus groups will be a follow up to the quantitative survey response. The focus group

will serve to illuminate some of the survey findings. The quantitative survey will be completed

during the summer while the focus group will take place at the beginning of fall semester in

2015. An e-mail (see appendix D) will be sent out to all students who have completed the Bridge

to Loyola Summer Program and have chosen to enroll in Loyola.

Focus Group Procedure & Implementation

The primary stakeholders, Natasha and China, and the graduate intern, will alternate

moderating and observing (assistant moderator) the intended two focus groups. For instance, the

first focus group will be led by Natasha and observed by the graduate intern. The following

focus group will be observed by the graduate intern and moderated by China. The primary

stakeholders want to provide their graduate intern an opportunity to make the most of their

internship experience. Therefore, the graduate intern will take the lead in moderating the second

group while Natasha assist. With available space in the Office of First and Second Year


23

Advising, where the primary stakeholders are located, one of the conference rooms will be

reserved for each focus group.

Bridge to Loyola summer program begins at the end of July and finishes early August.

Once the program is complete, students have a few weeks before the start of the semester to

determine if they wish to continue their college journey at Loyola. There are also students who

may not have passed the summer course, which as a result, does not admit to Loyola that

academic semester. Students who decide to attend Loyola will be the ones invited to participate

in the focus group. The first two days (Tuesday and Wednesday) after Labor Day is when the

focus groups will occur, with one happening on each day. In an effort to accommodate student

schedules and because a food and drink incentive will be provided, the times will occur during

lunch and dinner for approximately ninety minutes total (30 minutes for lunch/dinner, 60 minutes

for focus group). Below is an example of the focus group schedule:

• Focus Group #1 (Tuesday, 12pm – 1:30pm), Sullivan Center Room 290

o Lunch will begin at noon

o Focus Group begins promptly at 12:30pm, Sullivan Center Room 290

• Focus Group #2 (Wednesday, 5:30pm – 7pm)

o Dinner will begin at 5:30 pm

o Focus Group begins promptly at 6pm

The focus group is intended to be small and intimate for a rich conversation. Our hope is

to have at least six students per session with the anticipation of an audiotape present in the space.

However, we recognize that some students may be less candid and uncomfortable if they know

they are being audio recorded. With that, we will emphasize the confidentiality part as well as

the importance and reason behind the decision to audiotape the focus group. We want to ensure


24

we do not miss any relevant information that will allow us to improve our program. The

assistant moderator will also being taking notes to pick up on non-verbal messages and thematic

topics, which is necessary when analyzing the data.

Our protocol (see appendix F) demonstrates the entirety of the focus group process from

start to finish. We formatted the protocol in a way that would allow individuals who have no

affiliation with the program to lead the discussion. The dynamic we wish to create is a

comfortable and friendly space. With that, we included a short low-energy icebreaker after

announcing the ground rules. We also indicated the final ground rule as to “have fun!” to go

along with nature of the space we aim to get at.

Qualitative Data Analysis

Once the focus groups are completed, the audiotapes will be transcribed by one of the

primary stakeholders. Since the purpose is to learn the delivery effectiveness (process) of the

program components in Bridge to Loyola’s summer program, we have decided not to develop a

coding scheme prior to the interview. Instead, we are relying on an emergent approach to

develop codes, which are themes emerged as the data is being reviewed (Whole et al., 2010). In

doing so, the methods of categorization used will be descriptive and pattern as we are seeking to

discover recognizable themes and trending patterns from the data. To help with this process,

word repetition and pawing are two coding techniques necessary for this approach.

We have decided not to employ member checking due to the lack of scheduling time.

Fall semester for the primary stakeholders is a busy time in the office with teaching and

academic advising. However, testing for inter-rater reliability will be in done to ensure both

primary stakeholders are consistent with the results from coding. Since member checking is not

considered, information pertaining to personal biases will be incorporated, especially since data


25

analysis is derived only from the interpretation of the primary stakeholders. Given the number of

data collection findings obtained from survey results, observations, and focus groups, process of

triangulation is essential when finalizing the conclusion for ensuring trustworthiness and

credibility (Schuh, 2009). Triangulation affirms and validates the final results. It is a process of

comparing and contrasting collected data across the qualitative and quantitative approaches and

recognizing that there are potential errors to encounter.

There are certainly benefits and limitations for our intended approach. A limitation, as

recognized earlier, is not incorporating member checking. This incorporation is important to

consider, as it is a valuable process that will ensure personal biases do not filter participants’

narratives in ways that do not reflect student’s personal thoughts and stories shared in the focus

groups. Another limitation is not being intentional about selecting who is partaking in the focus

group. Our pool of students is a randomized selection process, mainly to accommodate our

student’s availability. However, reliance on inter-rater reliability and the process of triangulation

is how we hope to counter both limitations addressed.

Presentation Results

For the final report, our results will be presented using tables, which will include direct

quotes in the table. For instance, since we are assessing the effectiveness of five service

components (academic workshops, study halls, academic course, extracurricular activities, and

service project), the first column will entail a row for each of these component areas. The

second column will have the themes derived from each area. The third column will list direct

quotes pertaining to each service component.


26

Budget

We will utilize the FSYA and LUC libraries to minimize the cost for our budget (see

appendix G). Our incentive consists of food (pizza, salad, and water), which will be the only

costs. Other than that, we plan to use the FSYA conference room to conduct focus group

interviews, as well as using the FSYA office for printing and storing data using its USB drive

and storing it in a locked drawer. The personnel from the FSYA office will be the ones

facilitating, transcribing, and observing the focus group interview.

Timeline

A brief 2015 timeline has been created (see Appendix H) to project when the evaluation

plan will come to life. A meeting with the primary stakeholders should begin in May when the

logic model is developed and presented. After that initial meeting, the following month should

be when the quantitative survey questions are generated and pilot testing will take place. The

very end of July and early August is when the survey is distributed to students. Data analysis of

the survey occurs within the same month the focus group email invitation is sent. By September,

after Labor Day weekend, the conduction of the focus groups should happen as well as

transcription. The entire month of October should be reserved for transcription and data analysis

to prepare for November when the final report is complete and ready to present.

Next Steps

It is completely up to primary stakeholders, Natasha and China, to carry out

improvements from data collection and if they choose to share the final report with campus

partners and students. We recommend the primary stakeholders share, as the campus partners all

play an important role in the process of the Bridge to Loyola program. In addition, we

encourage a conduction of the same evaluation plan for the next few years to check progression,


27

which would allow the opportunity to compare and contrast results. Stress management is a

crucial component primary stakeholders should take into account next time they conduct a

similar evaluation. Most of the students in Bridge Programs are those in need of additional

academic and transitional support. The material for this particular summer program is a lot and

packed in such a short time. We hope these stakeholders are able to identify how students

manage stress in a brief, three-week intensive Bridge to Loyola Summer Program.


28

References

“Bridge to Loyola” (2011). Bridge to Loyola [Annual Report]. First and Second Year Advising,

Loyola University Chicago, Chicago, IL.

Creswell, J.W. (2014). Research design: Qualitative, quantitative, and mixed method

approaches (4th Edition). Thousand Oaks, CA: Sage Publications.

Fitzpatrick, J., Sanders, J., & Worthen, B. (2011). Program evaluation: Alternative approaches

and practical guidelines (4th Ed.) New York: Longman.

Garcia, P. (1991). Summer bridge: Improving retention rates for underprepared students.

Journal of the Freshman Year Experience, 3(2), 91-105.

Grauer, A., & Leon, S. (2013). Sex and gender in anthropological and psychological perspective

[Class handout]. Department of Anthropology and Psychology, Loyola University

Chicago, Chicago, IL.

Gullatt, Y., & Jan, W. (2003). How do pre-collegiate academic outreach program impact college-

going among underrepresented students? Boston, MA: Pathways to College Netwo

Clearinghouse.

Hill, C., & Teetsov, N. (2014). Bridge to Loyola [Class handout]. First and Second Year

Advising, Loyola University Chicago, Chicago, IL.

Kallison, Jr, M. J., &Stader, L.D. (2012). Effectiveness of Summer Bridge Programs in

Enhancing College Readiness. Community College Journal of Research and Practice,

36,340-357.

Sablan, R.J. (2014). The challenge of summer Bridge Programs. American Behavioral Scientist

Schuh, J.H. & Associates (2009). Assessment methods for student affairs. San Francisco, CA:

Jossey-Bass.


29

Wholey, J.S., Hatry, H.P., & Newcomer, K.E. (Eds.). (2010). Handbook of practical program

evaluation (Third Edition). San Francisco, CA: Jossey-Bass.

Evaluation Group Project

Documents

summer bridge programs

history of bridge

lowincome students

undergraduate students

appendix f

appendix e

appendix h

appendix g