New measures for new services Indicators for quality, cost, and impact of electronic library services Roswitha Poll Münster LIDA 2007 LIDA 2007.
Post on 16-Jan-2016
216 Views
Preview:
Transcript
New measures for new servicesIndicators for quality, cost, and impact
of electronic library services
Roswitha PollMünster
LIDA2007LIDA2007
The new electronic library services
• catalogue databases• portals• online ordering and delivery• personalized services• online reference• digitized collections• electronic publishing• online self-paced training• Internet access via the library
LIDA2007LIDA2007
Reasons for evaluation of quality and impact
accountability
special funding needed: evidence of positive effectsresults-based budgetingjustifying investment into change
resource management
rising expenses for new servicesincreasing demand for new servicesnew allocation of resources necessary
LIDA2007LIDA2007
promotion of the library‘s role
communication of benefits by using library services
What library standards want to achieve
to spread the knowledge about existing tested methods and practices
to standardize
with the aim of comparing and aggregating results
terminology and definitionsterminology and definitions
methods of data collectionmethods of data collection
methods of analyzing the datamethods of analyzing the data
LIDA2007LIDA2007
ISO Technical Information and committee 46: documentation
Subcommittee 8: Quality, statistics and performance evaluation
WG 2: International library statistics
ISO 2789 (2006) International library statistics4th edition
WG 4: Performance indicators for libraries
ISO DIS 11620 (2006) Library performance indicators2nd edition
WG 7: Quality measures for National Libraries
ISO TR 281180 Performance indicators for National Librariesin preparation
LIDA2007LIDA2007
Stages of an ISO standard
Approved New Work Item
Working Draft
Committee Draft (CD)
Draft International Standard (DIS)
Final Draft International Standard (FDIS)
Standard
votingrequired
LIDA2007LIDA2007
Problems of standardizing data for electronic services
quick changes in media and information ways changes not always predictable no clear boundaries of documents and
localities data must be collected from many sources
(vendors, suppliers, institutional servers)
State of technical
development varies in libraries
LIDA2007LIDA2007
In terms of• collection size• number and type of users• use of services• staff• space• equipment• training
LIDA2007LIDA2007
Assessingquantity
(ISO 2789)
problemof countingproblem
of counting
can be counted manually
can be counted manually
OPACOPACOPACOPAC
web-web-sitesite
web-web-sitesite
electronic electronic collectioncollectionelectronic electronic collectioncollection
Internet Internet accessaccess via the via the librarylibrary
Internet Internet accessaccess via the via the librarylibrary
electronic document delivery
electronic document delivery
online online referencereference
online online referencereference
user training on electronic
services
user training on electronic
services
Electronic library services: counting use
LIDA2007LIDA2007
Electronic collection
Electronic collection
databasesdatabases electronicserials
electronicserials
digital documents
digital documents
full textfull textabstract
andindexing
abstractand
indexing otherother electronicpatents
electronicpatents
networkedAV-docu-
ments
networkedAV-docu-
mentsotherothereBookseBooks
ISO 2789
computerfiles
computerfiles
LIDA2007LIDA2007
Forms of useForms of use
sessionssessions rejectedsessionsrejectedsessions
contents or
recordsdownload
contents or
recordsdownload
searchessearches InternetsessionsInternet
sessionsvirtualvisits
virtualvisits
sessiontime
sessiontime
inside the libraryinside the libraryoutside the library,
but inside the institution/population
outside the library,but inside the
institution/population
from outsidethe institution/
population
from outsidethe institution/
population
ISO 2789
LIDA2007LIDA2007
ISO 2789: Definitions of electronic use
LIDA2007LIDA2007
searchspecific intellectual query
sessionsuccessful request of a database or the online catalogue
rejected sessionunsuccessful request exceeding the simultaneoususer limit
session time
period between log-inand implicit or explicitlog-off
content unit downloaded
content unit beingsuccessfully requested
record downloaded
descriptive record being successfully requested
Internet access by a user from inside the library
Internet session
a user's request on the website from
outside the library premises
a user's request on the website from
outside the library premises
web browservisit
web browservisit
IP visitIP visit home pagevisit
home pagevisit
ISO 2789: virtual visits
LIDA2007LIDA2007
Counting electronic usage in German academic Counting electronic usage in German academic libraries: Percentage of libraries delivering datalibraries: Percentage of libraries delivering data
2005 2006
OPAC sessions 45.3 % 39.8 %
Database sessions
28.0 % 31.1 %
E-journal sessions
25.8 % 32.0 %
Digital document sessions
21.8 % 17.0 %
Downloads from databases
7.1 % 9.2 %
Downloads from e-journals
9.3 % 13.1 %
Virtual visits 36.0 % 34.5 % LIDA2007LIDA2007
In terms of
• user-orientation• reliability• currency• usability/accessability
• competence• responsiveness• speed• cost-effectiveness• perceived quality
LIDA2007LIDA2007
Assessingquality
(ISO 11620)
ISO 11620: Library performance indicators
resources, access, infrastructure
use
efficiency
potentials and development
New structure: Balanced scorecard
LIDA2007LIDA2007
Indicators for "hybrid" services
ISO 11620
resources, access,
infrastructure
• percentage of required titles in the collection• speed of interlibrary lending
use • library visits per capita (including virtual visits)• percentage of population reached• user satisfaction
efficiency • cost per library visit• cost per user• correct answer fill rate
potentials,development
• attendances at training lessons per staff member
LIDA2007LIDA2007
Indicators for electronic services
ISO 11620
resources, access,
infrastructure
• percentage of rejected sessions• public access workstations per capita
use
• number of content units downloaded per capita• percentage of information requests submitted electronically• workstation use rate
efficiency • cost per database session• cost per content unit downloaded
potentials,development
• percentage of acquisitions expenditure spent on the electronic collection• percentage of library staff providing and developing electronic services
LIDA2007LIDA2007
Possible performance indicator for website quality
Direct access from the homepage
Measured by• number of clicks neccessary to find relevant information• adequacy of terms used on the homepage Example: renewalsDirectly on the homepage: user account, my accountUnder broader terms: circulation, loan service
LIDA2007LIDA2007
Performance indicators for costs ISO 11620
costs includecosts include• total recurrent expendituretotal recurrent expenditurecost per loan
cost per database session
costs includecosts include• contractual costs of a databasecontractual costs of a database
cost per content unit downloaded
costs includecosts include
• contractual costs of an electronic contractual costs of an electronic resourceresource
LIDA2007LIDA2007
Cost issues
Changing from print to electronic collections
What changes for the library?
decrease of traditional services (loans, visits, decrease of traditional services (loans, visits, copying)copying)
increase of electronic useincrease of electronic use
In use
in costs
less costs for storage, bindingless costs for storage, binding
higher systems and staff costshigher systems and staff costs
better qualified staff neededbetter qualified staff needed
lower cost per uselower cost per use
overall:overall:increase in useincrease in use
LIDA2007LIDA2007
Cost issues
Changing from print to electronic collections
What changes for the users?
access from the working-place or homeaccess from the working-place or home
access at onceaccess at once
24-hours access24-hours access
searching options, linkssearching options, links
In use
in costs
time saved !time saved !
no ways, no fetching of materials,no ways, no fetching of materials,
no waiting in queuesno waiting in queues
additional additional valuevalue
academics spend half a day academics spend half a day per week on seeking and per week on seeking and
procuring informationprocuring information
LIDA2007LIDA2007
Cost issues
Journals: cost per use
Print onlysubscription
costs
usage cost per use
E-onlysubscription costs
usage cost per use
257.737 € 13.795 18.86 € 223.647 € 64.491 3.47 €
Users accessed online Users accessed online versions 10 times versions 10 times more than print more than print versionsversions
Users accessed online Users accessed online versions 10 times versions 10 times more than print more than print versionsversions
Only subscription costs are considered!Only subscription costs are considered!
Example 1: Medical library MüsterExample 1: Medical library Müster
LIDA2007LIDA2007
Journals: cost per use
Cost typeElectronic Journals
Print Journals (current)
Print Journals (bound)
Space $ 5,000 $40,000 $205,000
Systems $ 10,000 $ 2,500 $ 2,400
Supplies & Services $ (2,000) $ 600 $ 8,000
Staff $125,000 $46,000 $ 42,000
TOTAL OPERATIONAL COSTS $138,000 $90,000 $258,000
Operational costs of print and electronic journals
Montgomery, Carol Hansen, Donald W. King: Comparing libraryMontgomery, Carol Hansen, Donald W. King: Comparing library and user related costs of print and electronic journal collections.and user related costs of print and electronic journal collections.
D-Lib Magazine October 2002D-Lib Magazine October 2002Drexel University PhiladelphiaDrexel University Philadelphia
Example 2: Drexel UniversityExample 2: Drexel University
LIDA2007LIDA2007
Staff costs by function (Drexel)
E-journals
Print (current)
Print (bound)
Director/Administration $ 22,000 0 0
Communications (among staff) $ 7,000 $ 2,500 $ 800
Collection development $ 18,000 $ 6,000 0
Acquisitions $ 8,000 $ 4,000 0
Physical handling: bindery, labeling, reshelving 0 $10,000 $22,000
Record creation and maintenance $ 16,000 $10,000 $ 6,000
Public relations $ 11,000 $ 1,000 $ 600
Reference $ 37,000 $ 7,000 $ 4,000
Teaching (users) $ 6,000 $ 1,000 $ 600
Other 0 $ 4,000 $ 6,000
$125,000 $46,000 $42,000 LIDA2007LIDA2007
Cost per use by journal type (Drexel)
recorded use
subscription cost per use
operational cost per use
total cost per use
print current
15.000 $ 2.50 $ 6.00 $ 8.50
print bound
9.000 ---- $ 30.00 $ 30.00
total print 24.000 $ 2.50 $ 15.00 $ 17.50
total E-journals
335.000 $ 1.40 $ 0.45 $ 1.85
LIDA2007LIDA2007
Journals:cost per use
Problems in measuring cost per use
package deals for journalspackage deals for journals
consortia with "additional access"consortia with "additional access"
publishers insist on "print + electronic"publishers insist on "print + electronic"
For subscription pricesFor subscription prices
For measuring useFor measuring use
reshelving counts may under-estimate use reshelving counts may under-estimate use
E-journal use data may differ between publishersE-journal use data may differ between publishers
For assessing operational costsFor assessing operational costs
assessing staff time will be time-consumingassessing staff time will be time-consuming
long-term preservation costs for E-media still long-term preservation costs for E-media still questionablequestionable
High acceptance and use
High acceptance and use
of E-media will evidently result in lower
of E-media will evidently result in lower
cost per usecost per use
LIDA2007LIDA2007
In terms of
knowledge
information literacy
academic or professional success
social inclusion
individual well-beingchanges in skills, competences, attitudes, behaviour
Assessingimpact/outcome
LIDA2007LIDA2007
Methods for assessing impact
Use statistics as measures of impact
electronic media• frequency of use
• forms of use (sessions, searches, downloads)
• range of resources used
change of use in traditional services
change in remote use
number of new users• that had not used the library
• that had not used the same service in traditional form
percentage of the primary user group using E-services
Users are familiar with the use of
electronic resources and
services
LIDA2007LIDA2007
Qualitative methods
Surveys (postal, face-to-face, telephone, online)
Interviews
Focus groups, discussion groups
Self assessment of users
Data and „stories“ must be brought in a systematic
order in order to make trends visible
The "anecdotal evidence" supports
the statistics
Methods for assessing impact
LIDA2007LIDA2007
Quantitative methods
Tests of user skills before and after training
Observation by trained observers
Data mining
Analysis of bibliographies in publications of users
Comparing library use to academic success
Problem:Influences on individuals are manifold. The library‘s influence not always clear
Methods for assessing impact
LIDA2007LIDA2007
Example: Example:
Changing the journal collection to electronic formChanging the journal collection to electronic form
Statistical resultsStatistical results Survey to facultySurvey to faculty
Assessing the outcome of electronic services
online versionswere accessed10 times more
usage of an online article 5.4 times
cheaper high decline in print
usage
remote use increased
physical libraryvisits decreased
• In 1998 a number ofIn 1998 a number of academics were sceptical academics were sceptical as to E-journalsas to E-journals
• In 2002 it was exceptional forIn 2002 it was exceptional for faculty staff not to use the faculty staff not to use the electronic version electronic version
LIDA2007LIDA2007
change of change of attitudes attitudes
and and behaviourbehaviour
Projects of assessing outcome
eVALUEd: in "evidence base", University of Central England: Toolkit for evaluating electronic information serviceshttp://www.evalued.uce.uk/index.htm
IMLS (Institute of Museum and Library Services): outcome-based evaluation of projectshttp://www.imls.gov/index.htm
ARL New Measures Initiative: mehrere Projekte - Learning outcomes - Higher education outcomes research review - MINES (Measuring the impact of networked electronic services)http://www.arl.org/stats/newmeas/index.html
SCONUL and LIRG (Library and Information Research Group): impact initiative and mailing listhttp://www.sconul.ac.uk/activities/performance/impact.html
LIDA2007LIDA2007
Standardization of impact measures?
projects still on the wayIFLA Section Statistics and Evaluation has instituted a working groupBibliography of projects and literature:
http://www.ulb.uni-muenster.de/outcome
LIDA2007LIDA2007
Experience Experience is is the mother the mother of of sciencescience
Traditional ProverbTraditional Proverb
LIDA2007LIDA2007
top related