1GreekAcadLibs17 JED 2008 © J.E. Davies
What does success look like? What does success look like? Evidence–based assessment of the Evidence–based assessment of the
academic libraryacademic library
Dr. J. Eric Davies
Honorary Visiting Research Fellow
Loughborough University
17th Conference Greek Academic Libraries
2GreekAcadLibs17 JED 2008 © J.E. Davies
Outline:Outline:
• Management environmentManagement environment• Success factors, mission and strategySuccess factors, mission and strategy• Performance evaluation and enhancementPerformance evaluation and enhancement• Evidence typesEvidence types• Culture and competencesCulture and competences• Future trends speculations Future trends speculations
17th Conference Greek Academic Libraries
3GreekAcadLibs17 JED 2008 © J.E. Davies
Management environment:Management environment:
Change –Change –
• TechnologyTechnology
• Scholarly Communication and InformationScholarly Communication and Information
• Attitudes/Expectations/DemandsAttitudes/Expectations/Demands
• Services, Information Skills and SupportServices, Information Skills and Support
• Finance/Resources/AccountabilityFinance/Resources/Accountability
• Innovation/DevelopmentInnovation/Development
17th Conference Greek Academic Libraries
4GreekAcadLibs17 JED 2008 © J.E. Davies
Success factors, mission and strategy:Success factors, mission and strategy:
• Success Framework/CriteriaSuccess Framework/Criteria
• Mission/Vision/ValuesMission/Vision/Values
• Aims/Objectives/StrategiesAims/Objectives/Strategies
• Service goals/TargetsService goals/Targets
• Performance IndicatorsPerformance Indicators
• Evidence Based ManagementEvidence Based Management
17th Conference Greek Academic Libraries
5GreekAcadLibs17 JED 2008 © J.E. Davies
Framework for Performance Measurement:-Framework for Performance Measurement:-
Integration with institutional goals Integration with institutional goals
User satisfaction User satisfaction
Effectiveness (delivery) Effectiveness (delivery)
Efficiency Efficiency
EconomyEconomyFollett Report – academic libraries –1993Follett Report – academic libraries –1993
17th Conference Greek Academic Libraries
6GreekAcadLibs17 JED 2008 © J.E. Davies
Special Libraries Association –Special Libraries Association –
For special librarians, evidence-based practice For special librarians, evidence-based practice
refers to consciously and consistently making refers to consciously and consistently making
professional-level decisions that are based on professional-level decisions that are based on
the strongest evidence of what would work best the strongest evidence of what would work best
for our clients.for our clients.
17th Conference Greek Academic Libraries
7GreekAcadLibs17 JED 2008 © J.E. Davies
Types of evidence –Types of evidence –
• Performance EvidencePerformance Evidence
• Research EvidenceResearch Evidence
17th Conference Greek Academic Libraries
8GreekAcadLibs17 JED 2008 © J.E. Davies
Systematically gathered, structured and Systematically gathered, structured and contextualised information relating to local contextualised information relating to local operations and needsoperations and needs
Performance EvidencePerformance Evidence
17th Conference Greek Academic Libraries
9GreekAcadLibs17 JED 2008 © J.E. Davies
Performance Evidence –Performance Evidence –
Services Services
ProcessesProcesses
QuantitativeQuantitative
QualitativeQualitative
17th Conference Greek Academic Libraries
10GreekAcadLibs17 JED 2008 © J.E. Davies
Inputs Inputs
Quantitative DataQuantitative Data OutputsOutputs
Qualitative Evidence Qualitative Evidence OutcomesOutcomes
User ResponsesUser Responses
ImpactsImpacts
Performance
17th Conference Greek Academic Libraries
11GreekAcadLibs17 JED 2008 © J.E. Davies
User Responses –User Responses –
• SurveysSurveys
• Focus groupsFocus groups
• Open DaysOpen Days
• LibQUALLibQUAL++TMTM
• Rodski/InsyncRodski/Insync
17th Conference Greek Academic Libraries
12GreekAcadLibs17 JED 2008 © J.E. Davies
User Surveys –User Surveys –
• User Practice [frequency of visits, types of use]User Practice [frequency of visits, types of use]• User Capabilities [ IT use, vision, mobility]User Capabilities [ IT use, vision, mobility]• User Motivations [reasons for use/non-use]User Motivations [reasons for use/non-use]• User Preferences [service development]User Preferences [service development]• User Needs/Wants [service priorities]User Needs/Wants [service priorities]• User Opinions [perceptions of service]User Opinions [perceptions of service]• User Satisfaction [with various features]User Satisfaction [with various features]
17th Conference Greek Academic Libraries
13GreekAcadLibs17 JED 2008 © J.E. Davies
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Feb-02
Mar-03
Feb-04
Mar-05
Feb-02
Mar-03
Feb-04
Mar-05
Sa
tisfa
ctio
n w
ith s
taff
Sa
tisfa
ctio
n w
ithse
rvic
es/
faci
litie
s
Excellent
Good
Fair
Poor
British Library reader satisfaction survey ratings
14GreekAcadLibs17 JED 2008 © J.E. Davies
LibQUAL+LibQUAL+TMTM
Developed by ARL/Texas A&M UnivDeveloped by ARL/Texas A&M Univ
Based on SERVQUAL - Gap Analysis Based on SERVQUAL - Gap Analysis
Minimum, Desired, Perceived user view of quality Minimum, Desired, Perceived user view of quality
relating to -relating to -
•Access to InformationAccess to Information
•Affect of ServiceAffect of Service
•Library as PlaceLibrary as Place
•Personal ControlPersonal Control
17th Conference Greek Academic Libraries
15GreekAcadLibs17 JED 2008 © J.E. Davies
4
5
6
7
8
9
AI-1
AI-2
AI-3
AI-4
AI-5
AS-1
AS-2
AS-3
AS-4
AS-5
AS-6
AS-7AS-8AS-9
LP-1
LP-2
LP-3
LP-4
LP-5
PC-1
PC-2
PC-3
PC-4
PC-5
PC-6
Perceived greater than desired Perceived less than desired
Perceived greater than minimum perceived less than minimum
Example of data display of LibQUAL+TM
Results
16GreekAcadLibs17 JED 2008 © J.E. Davies
XXX
4
5
6
7
8
9
AI-1
AI-2
AI-3
AI-4
AI-5
AS-1
AS-2
AS-3
AS-4
AS-5
AS-6
AS-7
AS-8
AS-9
LP-1
LP-2
LP-3
LP-4
LP-5
PC-1
PC-2
PC-3
Perceived greater than desired Perceived less than desired
Perceived greater than minimum perceived less than minimum
Example of data display of LibQUAL+TM
Results
17GreekAcadLibs17 JED 2008 © J.E. Davies
Example of Gap Analysis from LibQUAL+TM Results
0
1
2
3
4
5
6
7
8
9
Access toInformation
Affect of Service Library asPlace
Personalcontrol
Range ofminimum todesired
Range ofminimum toperceived
18GreekAcadLibs17 JED 2008 © J.E. Davies
The amount of information and a manager’s ability to use it have The amount of information and a manager’s ability to use it have
increased explosively not only for internal processes but also for increased explosively not only for internal processes but also for
the engagement of customers. The more a company knows about the engagement of customers. The more a company knows about
them, the better able it is to create offerings they want, to target them, the better able it is to create offerings they want, to target
them with messages that get a response, and to extract the value them with messages that get a response, and to extract the value
that an offering gives them. The holy grail of deep customer insight that an offering gives them. The holy grail of deep customer insight
– more granular segmentation, low-cost experimentation, and mass – more granular segmentation, low-cost experimentation, and mass
customization – becomes increasingly accessible through customization – becomes increasingly accessible through
technological innovations in data collection and processing in technological innovations in data collection and processing in
manufacturing.manufacturing. James M. Manyika, Roger P. Roberts and Kara L. Sprague ~ James M. Manyika, Roger P. Roberts and Kara L. Sprague ~
McKinsey Q.(1) 2008. McKinsey Q.(1) 2008.
17th Conference Greek Academic Libraries
19GreekAcadLibs17 JED 2008 © J.E. Davies
Service Service
DomainDomain Community ProfilingCommunity Profiling
DemographicsDemographics
User DataUser Data
Lapsed, Non-Lapsed, Non-User, Potential User DataUser, Potential User Data
Information EnvironmentInformation Environment
Performance
17th Conference Greek Academic Libraries
20GreekAcadLibs17 JED 2008 © J.E. Davies
Comparative Benchmarking –Comparative Benchmarking –
Evaluate the level of performance of Evaluate the level of performance of various services within an institutionvarious services within an institution
Compare performance over timeCompare performance over time Compare with other institutionsCompare with other institutions Compare with recognised standardsCompare with recognised standards
17th Conference Greek Academic Libraries
21GreekAcadLibs17 JED 2008 © J.E. Davies
Comparative Benchmarking –Comparative Benchmarking –
STRATEGIC STRATEGIC
macro performance metrics macro performance metrics and qualitative evidenceand qualitative evidence
PROCESS PROCESS
detail cost performance datadetail cost performance data
17th Conference Greek Academic Libraries
22GreekAcadLibs17 JED 2008 © J.E. Davies
Comparative Benchmarking –Comparative Benchmarking –
To make changes that lead to quantum and To make changes that lead to quantum and continuous improvements in products, continuous improvements in products, processes and services that result in total processes and services that result in total customer satisfaction and competitive customer satisfaction and competitive advantage.advantage.
[Motorola + D.E.C. + Xerox][Motorola + D.E.C. + Xerox]
17th Conference Greek Academic Libraries
23GreekAcadLibs17 JED 2008 © J.E. Davies
Percentage of requests met in 7,15 & 30 days
0%
20%
40%
60%
80%
100%
98/9 99/0 00/1 01/2 02/3 03/4
XXX 30
XXX15
XXX 7
Std 30
Std 15
Std 7
24GreekAcadLibs17 JED 2008 © J.E. Davies
0
1
2
3
4
5
6
7
8
9
10
1997-98 1998-99 1999-00 2000-01 2001-02 2002-03 2003-04
XXXXX
Group
CURL
SCONUL
University Library - FTE students per seat
25GreekAcadLibs17 JED 2008 © J.E. Davies
0
5
10
15
20
25
30
35
40
45
1991-92 1992-93 1993-94 1994-95 1995-96 1996-97 1997-98 1998-99 1999-00 2000-01
£
Library X Comparators New SCONUL
University Library - Book expenditure per FTE student
26GreekAcadLibs17 JED 2008 © J.E. Davies
Average Weekly Opening Hours
0
10
20
30
40
50
60
70
80
90
94-5 95-6 96-7 97-8 98-9 99-0 00-1 01-2 02-3 03-4 04-5
xxxxxxxxx
Benchmark
Old
Old + CURL
SCONUL
27GreekAcadLibs17 JED 2008 © J.E. Davies
Re-shelving and tidying performance
Site A
Site B
Site C
Site D
Site E
Site F
Site G
0%
20%
40%
60%
80%
100%
0 2 4 6 8 10 12 14 16 18 20 22 24
Number of hours
28GreekAcadLibs17 JED 2008 © J.E. Davies
Research based intelligence and macro data Research based intelligence and macro data that informs service design and decisionsthat informs service design and decisions
Research EvidenceResearch Evidence
17th Conference Greek Academic Libraries
32GreekAcadLibs17 JED 2008 © J.E. Davies
Performance Evidence Culture –Performance Evidence Culture –
Service design and delivery are informed Service design and delivery are informed by appropriate internal and external by appropriate internal and external evidenceevidence ‘ ‘Think Evidence’ attitude permeates the Think Evidence’ attitude permeates the organisationorganisation Skills, knowledge, attitudes in using Skills, knowledge, attitudes in using evidence are embedded at ALL levels of evidence are embedded at ALL levels of the organisationthe organisation
17th Conference Greek Academic Libraries
33GreekAcadLibs17 JED 2008 © J.E. Davies
What will success look like – What will success look like – tomorrow??? tomorrow???
• Optimal Management Optimal Management
• Maximised Value AddedMaximised Value Added
• Academic Culture/EnvironmentAcademic Culture/Environment
17th Conference Greek Academic Libraries
34GreekAcadLibs17 JED 2008 © J.E. Davies
How will we measure success – How will we measure success – tomorrow??? tomorrow??? • Contingent valuation analysisContingent valuation analysis• Economic impact studiesEconomic impact studies• Critical incident studies Critical incident studies • Scenario planning Scenario planning • Lean managementLean management• Six sigmaSix sigma
17th Conference Greek Academic Libraries
35GreekAcadLibs17 JED 2008 © J.E. Davies
‘‘Management in particular is not about the preservation Management in particular is not about the preservation of the status quo, it is about maintaining the highest of the status quo, it is about maintaining the highest rate of change that the organization and the people rate of change that the organization and the people within it can stand.’within it can stand.’
‘‘Whether change is comfortable or not, it is inevitable.’Whether change is comfortable or not, it is inevitable.’
[ Harvey-Jones, J. [ Harvey-Jones, J. Making it happen: Reflections on Making it happen: Reflections on leadershipleadership. London, Collins, 1988. ]. London, Collins, 1988. ]
17th Conference Greek Academic Libraries
36GreekAcadLibs17 JED 2008 © J.E. Davies
Thank you for your attention.Thank you for your attention.
Questions?Questions?
17th Conference Greek Academic Libraries