LS 403 Evaluation of Information Services

Laura Saunders 1

LS 403Evaluation of Information Services

Introduction

Laura Saunders 2

Why Take this Class?… besides that it’s required.

Laura Saunders 3

Why Take This Class Take 3 minutes to answer these questions…

What do you expect/ hope to learn from this class?

What do you want to be able to do after this course? Why?

Why is this field (evaluation research) important? Or is it?

Laura Saunders 4

Background Edward Suchman observed that “one of the

most appealing ideas of our century [the 20th century] is the notion that science can be put to work to provide solutions to social problems” Substitute “social sciences” for “science” Expand “social problems” to problems faced by

organizations

Laura Saunders 5

Background: Evaluation vs. Social Research Evaluation Research

Uses many of the same methods/ practices Typically takes place within an organization-

with its own culture and politics Requires understanding/ ability to work with

groups, personalities, teams, etc. Requires sensitivity to issues and concerns of

various interests involved.

Laura Saunders 6

What and Why According to Peter Hernon

The purpose of evaluation is “to collect data useful for documenting the quality and effectiveness of existing programs and services, and to use that information for service improvement, demonstrating best practices, or providing evidence of accountability.

According to William Trochim Evaluation is the systematic acquisition and assessment of

information to provide useful feedback about some object

Laura Saunders 7

Purpose and Potential Evaluation compares “what is” with “what

should (could) be”- element of judgment Evaluation must have a purpose, rather than

be an end in itself. There is no need to evaluate without the

potential for action. Evaluation must go beyond description to

understand relationships. Joseph Matthews

Laura Saunders 8

Contexts Evaluation research may generate conflict

Often begins with problem identification Tends to involve groups of people, personalities,

etc. Competing priorities. Purpose is improvement- often results in change.

Laura Saunders 9

What and Why Unfortunately, according to James G. Neal,

“decisions are routinely not supported by the evidence of well-designed investigations,”

“research in the field is poorly communicated, understood and applied.”

Neal, J.G. (2006). “The research and development imperative in the academic library: Path to the future.” portal: Libraries and the Academy, 6: 1.

Laura Saunders 10

Current Developments Increased attention to utilization of evaluation results

Focus on Evidence-Based Practice Technology affects evaluation research

Topics to cover Software to analyze data collected Presentation of findings (e.g., word-processing and

graphics capability) New techniques for data gathering (e.g., Web surveys)

New skills and competencies needed

Laura Saunders 11

Evaluation AppliedWhat to study…

Laura Saunders 12

Evaluation Can Be Applied To Collections Staff Facilities Services Products Information Technology

Laura Saunders 13

Some Topical Areas of Evaluation Collections

People find what they want/need Quality/value Size Use patterns

Online Systems Complaint analysis Ideal page design Navigation/location of needed

information Tracking eye movement Use patterns

Reference Service Accuracy Complaint/compliment analysis Satisfaction Service quality (e.g.,

approachability) Technical Service

Efficiency (costs and time) Time: staff time to

complete task Time: time spent waiting

between process activities

Quality Accuracy (e.g., typographical

errors in bibliographic records) Workflow analysis

Laura Saunders 14

Evaluation Can Be Formative

Providing feedback during the development or offering of a program or service

Summative After the program or

service is over

Laura Saunders 15

Formative Evaluation How can the program or service be improved

or made more efficient or effective? To what extent are the goals, objectives, and

“performance measures” realized?

Laura Saunders 16

Summative Evaluation Is program or service X worth continuing or

expanding? How effective is it? What conclusions can be drawn about its

effects or various components? Did it meet its goals, objectives, and

performance expectations?

Laura Saunders 17

Measuring Things…How much, How well, How fast, How many…

Laura Saunders 18

Concept Measurements of Evaluation Extensiveness, or the amount of a service provided

in relation to the population served. This criterion is generally a measure of quantity rather than quality

Effectiveness Efficiency, or the appropriateness of resource

allocations Cost-effectiveness Cost-benefit Quality

Laura Saunders 19

Continuous Quality Improvement General Systems Model

Input (staff, budget, resources) Throughput (processes) Output (services, collections) Feedback to Input (evaluation)

All of above viewed within the context of the planning process

Laura Saunders 20

Uses of Evaluation Continue or discontinue (or improve) services

(programs) Improve, add, or drop practices and

procedures (strategies and techniques) Institute programs developed elsewhere Allocate resources among competing services

or programs Accept or reject a program approach or theory

Laura Saunders 21

Types of Evaluation Activities Program Planning: focuses on

“What is the extent and distribution of the target population?”

“Is the program or service designed in conformity with its intended goals?”

“Are the chances of successful implementation maximized?”

Program (Service) Monitoring: focuses on “Is the program or service reaching the persons,

households, or other target units to which it is addressed?” “Is the program or service providing the resources and

other benefits that were intended in the project design?”

Laura Saunders 22

Types of Evaluation Activities Impact assessment: focuses on

“Is the program or service effective in achieving its intended goals?” “Can the results of the program or service be explained by some

alternative process that does not include the program?” “Is the program or service having some effects that were

unintended?” Economic efficiency: focuses on

“What are the costs of delivering services and benefits to program participants?”

Does the program represent and efficient use of resources in comparison to alternative uses of the resources?”

Source: Rossi and Freeman, Systematic Evaluation

Laura Saunders 23

Evaluation QuestionsAll you need to do is ask.

Laura Saunders 24

Evaluation Centers on “How” Questions

Library-Centric View

Combination View

Customer-Centric View

How much?How many?How economical?How prompt?

How reliable?How accurate?How valuable?

How well?How courteous?How responsive?How satisfied?

Joseph Matthews

Laura Saunders 25

Examples of Evaluation Questions Are the services of the Libraries appropriate

considering the needs of the university, its academic programs, the faculty, and students?

Are the services effectively delivered? Are there important expectations or interests

of the academic programs that the Libraries do not meet?

Laura Saunders 26

Examples Cont’d What are the specific strengths and weaknesses in

libraries services at XXX? Are there good lines of communication within the

Libraries? Is staff morale good? Are there adequate provisions for professional

development of staff? Is there adequate succession planning within a

framework of future organizational structures? Are there adequate and appropriate avenues for users

to communicate needs and desired services back to the Libraries?

Laura Saunders 27

Examples Cont’d Is appropriate and sufficient attention being paid to

anticipating future IT needs and trends? Are there healthy and productive relationships with

other academic service units on the campus? Are staff adequately trained in and familiar with new

technological directions? What will be the most important qualities sought in

the next director of libraries?

Laura Saunders 28

If it’s so easy…Why isn’t everyone doing it?

Laura Saunders 29

Barriers & Obstacles The belief that everything is fine in my library The library might apply evaluation inappropriately

or merely serve organizational or administrative self-interests

Justify a weak or bad program or service by deliberately selecting for evaluation only those aspects that “look good” on the surface. Appearance replaces reality

Cover up program or service failure or errors by avoiding objective appraisal. Vindication replaces verification

Laura Saunders 30

Barriers Cont’d “Torpedo” or destroy a program or service

regardless of its effectiveness. Politics replaces research

Treat evaluation as a “gesture” of objectivity or professionalism. Ritual replaces research

Delay needed action by pretending to seek the “facts.” Research replaces decision making

Perceived lack of resources to support the evaluation process

Etc.

Laura Saunders 31

Barriers Cont’d Lack of staff Space Budget Conflicting perceptions of library Limited staff skills Physical location Limited bibliographic control Apathy of clientele and stakeholders about library Poor organizational structure, planning, and decision

making

Laura Saunders 32

Preparing for Research

Laura Saunders 33

Overview of the Evaluation Process Phase One: Preparation

Establish library goals and objectives Train staff Understand evaluation Put in place team or however the study will be done Decide on the problem

Phase Two: Evaluation Research Set up and execute the study

Phase Three: Assessing and, as decided, applying the results