Scholarship Universe Assessment Plan V1.2

University of Arizona

Scholarship Universe Assessment Plan A comprehensive guide to what SU hopes to achieve and how our work will be evaluated, assessed, and reported.

Chris Elsner 4/9/2014

1

Contents Overview ....................................................................................................................................................... 2

Mission .......................................................................................................................................................... 2

Purpose ......................................................................................................................................................... 2

Goals ............................................................................................................................................................. 2

Outcomes ...................................................................................................................................................... 3

Delivery of Outcomes ................................................................................................................................... 4

Measurement................................................................................................................................................ 6

Direct ......................................................................................................................................................... 6

Indirect ...................................................................................................................................................... 6

Measurement Tools .................................................................................................................................. 6

Implementation ............................................................................................................................................ 8

Implementation Timeline ......................................................................................................................... 8

Existing Resources ................................................................................................................................. 8

Future Resources .................................................................................................................................. 8

Annual Timeline ........................................................................................................................................ 9

Reporting .................................................................................................................................................... 10

Appendix ..................................................................................................................................................... 11

Measurement Tool Documentation ....................................................................................................... 11

Google Analytics Assessment Plan .......................................................................................................... 11

Social Media Assessment Plan ................................................................................................................ 11

2

Scholarship Universe Assessment Plan

Overview When Scholarship Universe (SU) was initially created, the focus and emphasis was on creating a service

for UA Students – little attention was paid, at the time, to how we would assess and evaluate Scholarship

Universe. The need to report on the outcomes of SU since that time have resulted in the creation of a

series of ad-hoc and partial assessment tools. While these tools have provided useful data and allowed

us to assess SU to some degree, they don’t provide all of the information we need to accurately and

efficiently assess and report on the outcomes and impact that Scholarship Universe is having.

This document will provide a review of our assessment methods to date, including where they have fallen

short and lessons learned so far. We will outline the goals, outcomes, and impacts we wish to measure.

And finally we will lay out a comprehensive assessment plan on how we will measure the above, what

we’ll need to do it, and a tentative timeline for the implementation of the assessment plan.

Mission The mission of Scholarship Universe is to provide a one-stop-shop for scholarships for UA students by

making it easier to find and apply for both external and internal scholarships, thus increasing the number

and value of scholarships that UA students receive each year.

Purpose The purpose of our assessment efforts will be twofold: to improve site functionality, usability, and

engagement and to demonstrate the effectiveness of SU in terms of impact on students, departmental

users, and the University of Arizona in general.

Goals 3-5 high level outcomes we wish to achieve.

1. Improve student scholarship attainment by reducing the time it takes to find a scholarship, matching

students to scholarships they’re eligible for, and increasing the volume of vetted scholarships in our

database.

2. Continually improve the functionality of the Scholarship Universe site to increase ease of use and user

engagement.

3. Help students, as a whole, maintain or increase external funding levels.

4. Improve the efficiency of departmental scholarship processes, increase applicant quality and quantity,

and reduce the amount of undisbursed scholarship funding.

3

Outcomes What are the measurable actions we can assess to demonstrate progress on the aforementioned goals?

Outcomes Measurement Tool(s)

Maximize impact of time students spend on scholarships (i.e. students should get the most out of their time searching and applying for scholarships).

Survey of SU and Non-SU users (need benchmark to compare)

Increase the avg. number of eligible scholarships for each student.

Reporting in SU scholarships vs. students, compared year on year

Increase the total number of scholarships in our database. Reporting in SU on Live/Achieved/Pending Scholarships – track year on year

Identify funding gaps based on scholarship eligibility requirements vs. UA student attributes.

Attribute and eligibility reports within Scholarship Universe on students and scholarships

Increase user engagement with SU. Google Analytics: page views and time on site, eventually include transactions and conversions

Improve user experience with SU. User feedback surveys and eventually built in survey/forms in SU

Increase overall amount of external funding students bring in each year.

SSSP Baseline report

Maintain/improve overall external funding level brought in by students.

UA Analytics – overall external funding amounts and debt levels relative to increasing tuition

Increase efficiency of internal scholarship processes. Survey of SUDA users (pre/post?)

Increase applicant quantity and quality for internal scholarships.

Survey of SUDA users, historical comparisons

Reduce the amount of undisbursed departmental scholarship money.

UAF Reports on unspent scholarship funding each year

Improve donor satisfaction with quality of applicants and awardees.

Survey of donors via foundation and/or individual departments

4

Delivery of Outcomes How will we / are we achieve(ing) those outcomes?

1. Maximize impact of time students spend on scholarships: We’ve already made searching for

scholarships much easier by matching students only to scholarships for which they’re eligible

based upon their profile question responses. We’ve also made it easier to apply for university

scholarships by allowing students, in many cases, to apply for multiple scholarships with one

application. To further reduce the time expenditure for students and maximize their return, we’ll

need to continue to improve the matching capabilities of SU, streamline the profile so it takes less

time to complete – or can be completed in sections, and tweak site functionality to improve user

experience. 2. Increase the avg. number of eligible scholarships for each student: By changing the way some

underlying functionality works with how we match students to scholarships, we’ll now be able to

measure the avg. number of scholarships a student is eligible for. To increase this, we’ll have to

continue searching for and adding external scholarships to SU in addition to brining on more UA

scholarships. 3. Increase the total number of scholarships in our database: Our student worker team will have to

continue to research and find external scholarships. We’ll also have to continue working with and

bring on new colleges/depts. to increase the number of internal scholarships which are available. 4. Identify funding gaps based on scholarship eligibility requirements vs. UA student attributes:

Recent changes to the structure of SU should enable us, for the first time, to analyze the eligibility

information of the student population as well as scholarships offered in SU. We will need to

develop and improve our reporting tools, but once those tools are in place we should be able to

more easily identify gaps between what scholarships are asking for and what students are

providing. 5. Increase user engagement with SU: With the addition of our Automatic Email System, which will

send automated updates to SU users, we’ll be able to provide students with notifications that

they have new scholarships that they’re eligible for, unanswered profile questions, or when

scholarship deadlines are approaching. This feature is expected to encourage regular usage of SU

since there currently aren’t a lot of reasons for students to log in frequently on their own. 6. Improve user experience with SU: First, we’ll continue to survey and solicit feedback from users

regarding the site and what’s working and what’s not. Secondly, we can analyze Google Analytics

data to identify when and where students leave the site to find bottlenecks or areas which

discourage use. 7. Increase overall amount of external funding students bring in each year: We provide an ever

expanding list of vetted external scholarships which students can be matched to. By doing so we

provide increased opportunities for students to find and apply for external scholarships. These

opportunities improve the chances that a student will be able to find a scholarship – the more

5

scholarships students have available, the more they can apply to, and the greater the likelihood

will be that they’ll bring in external funding. We will continue to search for, vet, and add external

scholarships to our system to maximize the opportunities available to students.

8. Maintain/improve overall external funding level brought in by students: By continuing to

provide as many scholarship opportunities to students as possible, we can help to maintain overall

external funding brought in by students relative to increasing tuition costs.

9. Increase efficiency of internal scholarship process: We are currently achieving this outcome by

making SUDA available to colleges/departments at UA. SUDA provides a unified tool to collect

and review scholarship applications, make awarding decisions, and track scholarship awards over

time. There are numerous features in SUDA which increase the efficiency of the scholarship

process compared to what many colleges/departments are currently doing. We’ll continue to

improve the functionality of SUDA through feedback from our current SUDA users and the

development of new features.

10. Increase applicant quantity and quality for internal scholarships: By providing students with

single place to search for internal and external scholarships, and by providing students with the

ability to apply for internal scholarships, we can increase the number of applicants for internal

scholarships. The idea is that SU will become the go-to place for UA students to find scholarships.

Further, by matching students only to scholarships for which they’re eligible we reduce the

number of ineligible applications that colleges/departments must sort through thus improving

the quality of the pool of applicants they do receive.

11. Reduce the amount of undisbursed departmental scholarship money: In providing numerous

tools for the management of the scholarship process in one place, SUDA reduces the overall

amount of work required to administer such a process. This means that administrators in the

various UA colleges/departments can more successfully implement their scholarship processes.

Furthermore, by increasing the number of applicants we can help ensure that a

college/department will always have a qualified student to award scholarships too.

12. Improve donor satisfaction with quality of applicants and awardees: Working in conjunction

with the UA Foundation and various development officers within partner colleges/departments,

we’ll develop and implement a survey specifically for donors to gauge their overall satisfaction

with scholarship applicant pools and awardees. Ideally this survey will be conducted

longitudinally and include results from donors associated with departments both pre and post

adoption of SU.

6

Measurement How will we measure outcomes?

We’ll measure our outcomes through a variety of means including user surveys, data collected from

UAccess, data generated internally by the SU site, Google Analytics data, and data collected via social

media.

Direct What are the things we can measure directly?

Much of the way students interact with and use the Scholarship Universe site can be measured directly

through internally generated data and Google Analytics data. We can also directly measure the total

amount of scholarship opportunities available through SU, the total value of external funding brought in

by students, and the value of external funding brought in by Scholarship Universe users. Finally we can

directly measure the amount of unspent scholarship funding that colleges/departments have left over

after each scholarship cycle.

Indirect What are the outcomes that will require indirect measurement?

From some of the direct measurements above, we can also indirectly measure things like overall level of

external funding relative to tuition costs. By talking with colleges/departments and looking at previous

applicant pool data (when available) we can also measure the quantity and quality of the applicant pool

in SU.

Measurement Tools What are the resources we currently have? Where are they located? What are their limitations?

Below are the primary measurement tools we have available to us (see appendix for more information):

Google Analytics (see appendix): this tool provides information relating to site usage. Currently,

it provides us with general information in the form of: number of site visitors, time on site, page

views, exit rates, source traffic, user devices, and location of users. In the future, we hope to

optimize Google Analytics to also provide information on usage patterns, flow of traffic, and to

capture the impact of email/social media campaigns. Google Analytics is an online tool – all data

is stored on and accessed from Google’s servers (raw data can be exported/downloaded). It’s

limited in that we cannot assess why people do what they do on the site or what they do after

they leave our site (for instance to apply for an external scholarship).

7

Internally Generated SU Data: quite a bit of data is generated as a consequence of regular site

usage. This includes user lists, profile information, metrics for individual users, scholarship data,

user generated feedback, and user interactions with tools on the site. This data is stored on our

servers in SASG. A current limitation of SU generated data, is a lack of access to

reporting/analytics tools. Right now reporting tools are limited – our only real option is to pull

raw data from the site then attempt to analyze it via a 3rd party application. However, in the

future it would be desirable to have some minimal built-in reporting/analytics tools to make this

information more accessible, easier to navigate, and easier to analyze.

User Surveys: we have access to and frequently use the Campus Labs survey toolset as our

primary means of survey collection. We’ve conducted numerous surveys of our core users in

addition to some peripheral user groups which are relevant the scholarship process. This is an

online resources made available to us through Student Affairs and data is primarily stored on

remote servers, though it must be downloaded in order to perform any sort of substantial

analysis. The primary limitation of surveys is participation. We’ve had some success in improving

response rate through monetary incentives, but this is cost prohibitive. Furthermore, we must be

very wary to use surveys sparingly to avoid survey fatigue among our respondents.

UAccess: this system provides us with financial aid information on every student at the UA. This

information provides big picture information about the flow of funding brought in by students

and helps us understand the impact of SU. All of the UAccess data are stored on UA servers. The

primary limitation of the data from UAccess is the lack of specificity relating to how external

awards are categorized in the system. In order to keep the system simple, the vast majority of

external scholarship awards are classified as ‘Donor Select’ which makes it attribution to a specific

scholarship impossible. However, by adding a step to the check processing process – we have

started linking external awards to Scholarship Universe so we can at least attribute it to SU, if not

to the exact scholarship in SU.

Social Media Analytics (see appendix): right now, our social media presences is somewhat

limited. It has not been a top priority and it’s not been necessary to achieve desired outcomes.

However, as time goes on it may be one of the most cost effective ways to reach students and

promote Scholarship Universe. Currently, we have Facebook and twitter accounts, however the

twitter account is somewhat underutilized. These resources are online, however data can be

exported to local machines for additional analysis. Our primary limitations are twofold: for one,

we somewhat lack the capacity and staff to fully take advantage of social media and two, the

information gained may not necessarily aid us in measuring out stated outcomes.

8

Implementation How will our measurement efforts be implemented?

Implementation Timeline Some of our evaluation and assessment efforts are already underway, others are currently in

development, and a few still remain on the drawing boards to be implemented in the future. Below will

be a summary of what has already been established, followed by a rough outline of when additional

assessment tools will come into effect.

Existing Resources

UAccess – we’ve already established what data we need from UAccess (primarily the SSSP

Baseline Query in RPT). We recently added a comment option to UAccess to capture scholarships

that can be potentially associated with Scholarship Universe when UA receives checks. This

should enable us to provide a more accurate indirect measure of the financial impact of SU.

Google Analytics – google analytics has been set up for the current site and provides basic

superficial usage statistics which are adequate for now.

Survey Data – we have data from previous surveys which could be useful in future reporting as

well as providing a template for additional surveys in the future.

SU Generated Data – there is a sizable amount of data on SU’s usage in terms of internal metrics.

However, we have to find appropriate uses for it and figure out easier ways to get it out of the

system.

Future Resources

Create annual report template based upon this plan. This will potentially include analyzing

UAccess data with new comment code complete initial draft by May 2014.

Create, implement, and analyze a survey for SUDA users. To be completed by July 2014.

Update and refine Google Analytics site tracking abilities. To be completed concurrent with

website redesign process by January 2015.

Create a standardized student user survey to be used annually. Start in November 2014 and

complete initial survey round by March 2015.

Analyze AY 2014-15 UAccess data – first full year of data with new ID comment. Completed as

part of annual assessment – May 2015.

Develop and improve reporting and analytics tools in both SU Admin and SUDA. Tentative start

date of January 2015 and completion date of May 2015.

Develop randomized internal user feedback collection on SU. Tentative start date of January 2015

and completion date of May 2015.

9

Annual Timeline Below is a general guide to our yearly assessment cycle. It includes rough time frames for when major

reports are required as well as rough time lines for regular data collection activities and reporting.

January February March April May June

Annual

Student User

Survey Opens

Revise/review

assessment

plan.

Social Media

Review and

Planning

Google

Analytics

Review

Assess/prepare

student survey

Annual

Student

Survey Closes

Student

Survey

Analysis

Google

Analytics

Review

SUDA User

Survey Opens

Collection &

Analysis of

UAccess Data

Collection of

Internal SU

Data for AY

Annual AY

Reporting:

Scholarship

Value, Users,

Trends

Google

Analytics

Review

July August September October November December

Annual SUDA

Review

SUDA Survey

Closes

Prep for

SSF/FF

Reports

SSF/FF

Progress

Reports

Google

Analytics

Review

Social Media

review and

planning

Google

Analytics

Review

Collection of

Data for

SSF/FF Report

SSF/FF

Progress

Reports

10

Reporting How will we communicate our assessment results in terms of frequency, audience, and medium?

Scholarship Universe Annual Report

As mentioned above, we will start to compile and provide an annual Scholarship Universe report. This

report will coincide with end of the school year, allowing us to look at an entire academic year and

compare its results with previous years. This report will contain key findings and figures which may be

then used in additional documents/reports. This report will be geared primarily toward upper

administration officials. It will be created and distributed electronically with hard copies available upon

request. Additionally, elements of the report may be used for marketing purposes and end up in other

places.

Google Analytics Report(s)

Our Google Analytics report will be compiled every other month. It will compare usage statistics with the

previous periods as well as the same period in the previous year (when possible). The primary audience

for this report will be immediate supervisors. The report will be generated electronically and 1-2 page

executive summaries will be provided at the appropriate meetings.

SSF/FF Progress Reports

The SSF and FF progress reports will include key performance indicators in the appropriate context. The

audience for each of these reports is the corresponding student board. These updates are provided bi-

annually.

11

Appendix The appendix items have not yet been completed, but below you can see what we’re planning on

including for each section.

Measurement Tool Documentation This section will contain detailed information on our various measurement tools. This will include the

following:

Step by step walkthrough for pulling data from each measurement tool,

Where and how the data is accessed from each measurement tool,

Who the contact person associated with each measurement tool is,

The process for verifying, organizing, and analyzing data for reports,

Any known limitations or issues with each measurement tool.

Google Analytics Assessment Plan This section will contain a detailed plan for our Google Analytics reporting. It will include:

The specific things we’re trying to measure in Google Analytics,

Specific Key Performance Indicators (KPI’s) for the things we’re measuring,

How Google Analytics will be structured to capture information on the outlined KPI’s,

A plan for what our Google Analytics dashboard should include,

Instructions for reporting data and statistics from Google Analytics.

Social Media Assessment Plan This section will match up with our forthcoming Social Media Plan and include information on:

Specific social media platforms currently utilized and potential areas of expansion,

How each social media platform interacts with and relates to SU and overall social media goals,

Instructions on how access analytical information from social media platforms in use,

Benchmarks from the past and goals for the future relating to user engagement with our social

media presence,

Ways to connect reported social media outcomes with overall SU outcomes.