Lessons Learned: How to Survive the Development of Outcome Measures Cherie McCraw, M.L.S., Library Consultant, Division of Library and Information Services.

Lessons Learned: How to Survive the Development of

Outcome Measures

Cherie McCraw, M.L.S., Library Consultant, Division of Library and Information Services

Linda B. Schrader, Ph.D., Florida State University

Overview

Overview of the four-year implementation process

Development of an outcome evaluation system for the State Library of Florida

Examination of lessons learned and challenges encountered

History 1993 Government Performance Results

Act – Requires federal agencies to establish

specific objective, quantifiable, and measurable performance goals for each program

1996 reauthorization of Library Services and Technology Act (LSTA)

LSTA Grant Background State-based program Focuses on information access through

technology and information empowerment through special services

Emphasizes public libraries, but encourages all types of libraries to apply

Emphasizes accountability and evaluation

Institute of Museum and Library Services (IMLS)

Turned to outcome-based evaluation for LSTA grants– United Way model– Looks at the effect of an institution's

activities and services on the people it serves

– Powerful tool for identifying successes and communicating their value

IMLS Plan Five states were selected to implement

outcomes evaluation of LSTA grants early and develop model programs– Florida– Minnesota– North Carolina– Maryland– Washington

Florida’s Four Year Implementation Plan

Development of outcome model and standardized outcomes

Marketing of outcomes evaluation process

Planning for staff training and technical assistance

Conducting outcome evaluations

Implementation Activities

Administrative and management tools Senior level buy in State Library and field training

1998 IMLS selects states for pilot projects1998 Consultant hired to help develop model1998 State Library staff receive United Way Outcomes Evaluation trainingOct 1998 Outcomes Evaluation concept presented at annual Public Library Directors’ Conference1999 Standardized outcomes for 6 LSTA categories developed1999 OPPAGA Review favorable toward outcome conceptOct 1999 Outcomes Evaluation program presented at annual Public Library Directors’ ConferenceNov 1999 Born to Read Capacity Building Workshop (outcome plans developed)July 1999 Florida’s “LSTA Outcome Evaluation Plan” publishedFall 1999 Workshops on outcome evaluation presentedJan 2000 LSTA Grants workshop featured new guidelinesMar 2000 LSTA applications required outcome plan (not scored)Sep 2000 “Workbook: Outcome Measurement of Library Programs” publishedOct 2000 Panel discussion of outcomes at annual Public Library Director’s ConferenceNov 2000 LSTA Capacity Building Workshop (outcome plans refined)Oct 2000 Outcomes based LSTA grants began2000 LSTA categories changed from 6 to 2/new standardized outcomes createdMar 2001 LSTA applications required outcome plan (scored)Jul 2001 Outcomes based evaluation workshops conductedNov 2001 LSTA Capacity Building Workshop (evaluation strategies and tools)Nov 2001 LSTA annual reports reflected outcomes evaluationDec 2001 LSTA Grants workshop included session on outcomes

State Library of Florida’s Outcomes Timeline

Development of Outcome Model

Hired consultant Adapted United Way Outcome Model Identified standardized state outcomes Revised rules, guidelines and forms

Outcome Model Components

Inputs Activities Outputs Outcomes Evaluation Indicators Evaluation Sources/Methods

Florida’s LSTA OUTCOMES PLANProject Name ______________________________ Library______________________

Project Summary/Program Purpose:

Inputs Activities Outputs Outcomes Indicators Sources/Methods

Standardized State Outcomes

Characteristics– Broad scope, relevant– Easily understandable– Focus on skills/behaviors– Can include quantitative and qualitative

indicators– Tailored to local programs

Standardized State Outcomes

Access for persons having difficulty using libraries– Persons having

difficulty using library services use services or information that were not previously available

Library Technology Connectivity and Services– Public uses technology

to get information

– Public learns to use technology

Access for persons having difficulty using libraries

Golden Gateways Library Family

Learning Centers Small Business

Information Resource Center

REACH: REmote ACcess to the Homebound

Parents Plus Born to Read Juniors to Seniors:

Hillsborough Remembers

Library Elderly Outreach (LEO) Project

Library Technology Connectivity and Services

Development of the Everglades Information Center

Electronic Library: Community Training and Outreach

Seniors Connect @ Jacksonville Public Library

Mobile Training Lab Osceola Internet

Improvement Library Automation:

Moving Legal Information Toward the Public

Mi Servicio de Biblioteca

Florida’s LSTA OUTCOMES PLANProject Name __________Family Literacy for Lifelong Learning____ Library_______________

Project Summary/Program Purpose: With support from schools and community partners, the library will continue withthe family literacy program. Services will be expanded at our four branches where the program is currently offered.Additionally, services will be initiated at a fifth branch. Elements to be offered: (1)One-on-one tutoring for children andsupport for parents in meeting the child’s reading needs, (2)literacy- or basic-skills support for adults through the libraryor its partners, (3)increased services for Spanish-speaking families.Inputs

Establishedlibrary systemwith full timestaff

45 literacytutors

Literacyresources(books, 75videos, 13Brainchilds, 44LearningPAKs)

16 partners

Grant funds

Local match

Activities

Hire literacycoordinator

Designmarketingcampaign

Visit otherliteracyprograms toexchange ideas

Attend literacyworkshops

Order literacymaterials

Recruit families

Recruit tutors

Collaborate withpartners

Outputs

# familiesserved

# childrentutored

# hourschildrentutored

# adultstutored

# hoursadults tutored

# parent/childinteractionmeetings

# PAKsloaned

# librarycards issued

Outcomes

1. Persons havingdifficulty using libraryservices useservices orinformation that werenot previouslyavailable

Indicators

1a. # and %children whosereading skillsimprove

1b. # and %increase in basicadult literacylevels.

1c. # and %parents who meettheir personalgoals.

1d. # and %parents who readto their children15 min/day (3times/week).

Sources/Methods

1a. Pre-and post-testing

Teacher’s reports

1b. Pre-and post-testing

1c. Interviews

1d. Interview orlog

Florida’s LSTA OUTCOMES PLANProject Name _Electronic Library: Community Training & Outreach Program____ Library_____

Project Summary/Program Purpose: To provide consistent training presence at seven electronic libraries operatedin the county. The personnel funded through this program will provide a regular training presence and will conducttraining classes for electronic library users. The classes will meet user demand for assistance in learning how touse the resources available in the electronic libraries. The project personnel will survey users and evaluate theresources and effectiveness of the e-libraries.Inputs

LibrariansLibrary

TechnicalAssistant

E-libraries

Computers

Software

Partneragency staff

Grant funds

Local match

Activities

Hire personnel

Conduct classes

Evaluate classes

Evaluateresources

Survey e-libraryusers

Write/send pressreleases

Visit communityschools/areacommunitycenters

Outputs

# classes

# trainingprograms

# attendees

# evaluationscollected

# surveyscollected

# schools &communitycenterscontacted

# pressreleases

Outcomes

Public learns to usetechnology

Public usestechnology to getinformation

Indicators

# of peopleindicating theyhave learned howto use theelectronicresources

# of people whoaccessinformation at thee-libraries

Sources/Methods

Surveys

Program records

Evaluation forms

Sign-in sheets

Program records

Surveys

Marketing the Outcome Process

State leadership vanguard for change– Paradigm shift in what gets measured

Senior level buy in of new evaluation process

Continuous communication throughout the four year period

Ongoing support

Staff Training Developmental

approach to training Technical assistance

provided by regional library consortiums and state library consultants

Statewide conferences

Conducting Outcome Evaluations

Phased in requirements for outcome evaluations in grant process

Continued training and support to field staff

Initial review of annual reports

Lessons Learned

A State Library assumed leadership roleB Implementation phased in over timeC Standardized the outcomesD Measurement tools could be the sameE Better communication between libraries

and grant partnersF Multiple training opportunities

ChallengesA Resistance from State and field staffB Cannot be implemented quicklyC Adapting local programs to the standardized

outcomesD Tailoring existing data collection tools E Libraries and grant partners interpret

outcomes evaluation differentlyF Different levels of knowledge about outcomes

Where Do We Go From Here?

Continue training field librarians Promote/pilot Outcomes Model for other

library programs Develop new Long Range Plan using

outcome measurement