Outcome Based Evaluation for Digital Library Projects and Services Kristine Brancolini, Director Digital Library Program Digital Library Brown Bag March.

Outcome Based Evaluation for Digital Library Projects

and Services

Kristine Brancolini, Director

Digital Library Program

Digital Library Brown Bag

March 25, 2005

Project Evaluation Formative evaluation during the project

Delivering collections/resources in a way that meets user needs

How well did you do it? Summative evaluation at the end of the project focused

upon the goals of the project Did we do what we promised? Did we finish on time and with the budget? Does it work?

Impact evaluation -> Outcome based evaluation Did the fact that you did it make a difference for

your audiences?

Measuring Success

Easy to measure Did we do what we promised? Did we finish on time and with the budget? Does it work?

More difficult to measure Delivering collections/resources in a way that

meets user needs Most difficult to measure

Intended impact

Outcome Based Evaluation [1]

Based upon a two-day training program developed by Performance Results, Inc.

Sponsored by the Institute of Museum and Library Services

All recipients of National Leadership Grants are required to attend

Since I attended in 2001, the focus has expanded from evaluation to planning and evaluation: OBP & E not just OBE

Outcome Based Evaluation [2]

Purpose of the workshop is to “engage participants in the clarifying the purpose of the project and desired collective and individual results.”

Focus on outcomes, not outputs alone

Definitions

Outcome based planning uses audience needs and hoped-for results as the foundation for programs and design decisions.

Outcome based evaluation (OBE) is a systematic way to assess the extent to which a program has achieved its intended results.

Key Questions in OBE

How has my program made a difference? How are the lives of the program participants better

as a result of my program?

The origins of the methodology are clear by these questions: social services and education.

Not the type of questions we usually ask ourselves about our work – but not a bad idea!

Benefits of OBP & E

Provide a logical framework for program development

Increase program effectiveness Communicate program value Generate information for decision-making

Note the use of the word “program,” not “project.” May be more useful for former than latter.

Limitations of OBE

OBE is not identical to formal research OBE suggests cause and effect; it doesn’t

intend to prove it. OBE shows contribution not attribution

What does this mean? My program contributed to a particular outcome but I cannot say that the outcome is specifically the result of my program.

Program Defined

Activities and services leading towards intended outcomes

Generally has a definite beginning and end We would call this a “project,” not a “program”

Designed to change attitudes, behaviors, knowledge, or increase skills and abilities based upon assumed need This is what makes it “outcome based” – focus on

impact of our work upon one or more audiences

Step 1.1 – Build: Needs

Audience needs – programs are developed as a result of assumptions about people’s needs

Need: A condition, want, deficit that is commons to a group of individuals

Solution: A program that will change or improve behaviors, knowledge, skills, attitudes, life condition, or status

Desired results: The change or improvement you expect to achieve

Step 1.2 – Build: Stakeholders

Stakeholders are individuals, agencies, funding sources, competition, community groups and professional affiliations. They influence: Type and nature of services Who is served Desired outcomes How results are communicated

Step 1.3 – Build: Program Purpose

Program purpose is driven by assumptions about need. It relates to the organization’s mission statement. It defines audience, services, and outcomes.

We do what, for whom, for what outcome or benefit?

The tricky part is the outcome. Remember: We want to change or improve behaviors, knowledge, skills, attitudes, life condition, or status. That’s the outcome. Digitizing photographs, for example, isn’t an outcome. What will be the result for – for our audiences -- of digitizing the photographs?

Step 2.1 – Evaluate: Logic model An outcomes logic model (evaluation plan) is a clear, graphic

representation of the links between program activities, the results these activities produce, and how the results will be measured. Elements of logic model: Outcome -- intended impact Indicators – observable and measurable conditions or

behaviors Data source – sources of information about conditions being

measured Applied to – the population being measured Data intervals – when data are collected Specific goals (also called Targets) – amount of impact

desired

Step 2.2 – Evaluate: Inputs

Resources dedicated to or consumed by the program Staff Funding Materials Existing infrastructure Facilities

Step 2.3 – Evaluate: Activities/Services

What are you going to do to accomplish your project goals?

Activities are management related Services directly involve end-users

These distinctions don’t hold up for digital library projects. Some of our project services involve project partners and some involve our digital library audience.

Step 2.4 – Evaluate: Outputs

A direct program product, usually measured in numbers

This definition doesn’t entirely hold up for digital library projects either. We often list our outputs software tools, research reports, conference presentations. However, we do specifically state number of images to be digitized or number of metadata records to be created.

Step 3.1 – Manage: Report

Reports summarize the results of the outcome data and include Participant characteristics Inputs, activities and services, outputs, and

outcomes Elements requested by “influencers” Comparisons of previous periods Interpretation of the data

What did we want to do? What did we do? So what?

Step 3.2 – Manage: Evaluate System

After one year of operation, evaluate the effectiveness of the outcomes system: Have audiences been sufficiently identified? Are outcomes clearly written? Are outcomes sufficient to describe what you

hope will happen? Are data collection methods cost efficient? Add:

Do they provide the data you want and need?

Using Outcome Based Evaluation in Digital Library Projects

Important and useful tool for planning and evaluation; we write better proposals because we are thinking about outcomes.

Model is most useful up to the point where we have to measure outcomes.

Very difficult to measure outcomes in an online environment. Methodology developed in social services/educational environment with direct user contact.

Writing Outcomes

Each outcome must address a specific target audience. For our projects, the intended audiences are often diverse,

due to the nature of the Web Our projects usually address a professional audience as

well as users of our resources Our projects often reach unintended audiences. One

recent example: Cushman and bloggers

Difficult to identify desired changes or improvements in skills, attitudes, knowledge, behaviors, status, or life condtion

Measuring Results

Important to find relevant indicators, which must relate to the target audience

Can be difficult to find appropriate data sources. We can see what people are doing, but not the impact on their lives.

“Applied to” refers to a sample, but can be difficult to reach a representative sample

“Targets” – expressed in numbers or percentages are arbitrary and do not seem to be helpful

LSTA Outcomes Plan

For the first time applicants for the Indiana State Library’s Digitization Grants had to submit an Outcomes Chart along with their proposals.

Many applicants were confused by this and most didn’t complete it correctly. Without learning about the methodology, difficult to focus on outcomes, identify indicators, and identify sources and methods.

Handouts

Take a look at handouts…

“Measuring Program Results” for our IN Harmony Project

“LSTA Outcomes Plan” for our proposed Online Indiana Authors and Their Books Project

Conclusions

Outcome Based Planning and Evaluation can be an extremely useful tool for digital library projects

It is difficult to apply the methodology exactly as described by the training workshop and materials

The emphasis on accountability at the state and federal level requires us to find ways to document the effectiveness of our work

Resources

IMLS publication "Perspectives on Outcome Based Evaluation for Libraries and Museums:" http://www.imls.gov/pubs/pdf/pubobe.pdf

Information about Outcome Based Evaluation on the IMLS web site, including a glossary of terms: http://www.imls.gov/grants/current/log_glossary.htm#intro

OBE FAQ on the IMLS web site: http://www.imls.gov/grants/current/crnt_outcomes.htm