Page 1
Not always discovered: Phase two of a study of the effect of discovery systems on online (journal) usage
ER&LMarch 18, 2014
Michael Levine-Clark, University of DenverJohn McDonald, University of Southern California
Jason Price, SCELC Consortium
http://bit.ly/discovery-impact-erl2014
Page 2
“…a steep increase in full text downloads and link resolver click‐throughs suggests Summon had a dramatic impact on user behavior and the use of library collections during this time period.”
The Impact of Web-scale Discovery on the Use of a Library CollectionDoug Way (2010) http://scholarworks.gvsu.edu/library_sp/9/
Page 3
http://www.oclc.org/partnerships/econtent/solutions.en.html
Vendor marketing
Page 4
Does implementation of a discovery service impact usage of publisher-
hosted journal content?
Page 5
Publisher-hosted journals are only part of the picture
eBooks, pBooks, aggregator journal content, etc.publisher journal contentThe six publishers in this study
Page 6
What did we measure?
• Whether there is an effect
• NOT why that effect exists (that’s a future study!)
Page 7
• “Society will need to shed some of its obsession for causality in exchange for simple correlations: not knowing why, but only what”
• (Cukier & Mayer-Schonberger. 2013. Big data: A revolution that will
transform how we live, work, and think.)
Page 10
Data collection• List of libraries with discovery services
>Searched on lib-web-cats
• Surveyed Libraries>Discovery service Implemented>Implementation Date (month/year)>Search box location>Marketing effort
• 149 Libraries Gave Approval>33 libraries selected for this phase>6 for each of the 4 major discovery services and a
group of 9 libraries with no service
Page 11
Dataset• 33 Libraries
– 28 US, 2 CA, 1 each from UK, AUS, NZ
–WorldCat book holdings>Average: 1,114,193 ; Range: ~300k to ~2.6mil
• Implementation dates (Discovery Libraries): >2010 (3), 2011 (19), 2012 (2)
• 6 Publishers
• 9,206 Journals
• 163,545 Usable Observations
Page 12
Methodology
Compared COUNTER JR1 total full text article views for the
12 months before vs 12 months after implementation date
June
201
0St
art
Impl
emen
tatio
nM
ay 2
011
May
201
2En
d
Year 1 Year 2
Included implementation month in Year 1 to ensure that both periods included an entire academic year
Page 13
Examine Data for Outliers
Page 14
Observations by Library & Service
Page 15
Observations by Publisher
Page 17
Average Usage Change By Discovery & Publisher
Page 18
Analyzing Usage Change: % vs Total
Use 12 months before
Use 12 months
after% Change Total
Change
Journal A 500 600 20% 100
Journal B 5 15 200% 10
Which is the better measure?
Is it the same for publisher- & journal-level data?
Page 19
Reducing variation due to institution sizeCurrently converting to change per FTE
Values are shown as x 1,000 to bring the change metric back per journal-library combination to a minimum of 0.1
2013 JISC Discovery study took a similar approach
Page 20
Average Usage Change By Discovery & Publisher
Per Journal & Per 10,000 FTE
Page 22
Full Model
Including Discovery Service, Publisher, and Library
Including Discovery Service, Publisher, and Library
Page 23
Nested ANOVA Model
[all three factors – preliminary results]
Page 24
Does usage change vary across libraries?
Institution (sorted by Mean Change)
Page 25
Does usage change vary across libraries using the same service?
Library 10-15 Library 16-21 Library 22-27 Library 28-33Library 1-9
Page 26
Does usage change vary across publishers?
Publisher (sorted by Mean Change)
B
AAA A A
Page 27
Does usage change vary across discovery services?
A
BB
C
D
Page 28
Publisher
Does the effect of discovery service differ across publishers?
Page 29
ResultsCan we detect differences between Discovery Services, Publishers, and/or Libraries and/or their interactions? • Library – Yes• Publisher – No• Discovery Service – Yes
• Differential discovery service effect by publisher – Yes
Page 30
Next Steps• Design & test for effects of:
–Aggregator full text availability–Publisher Size–Journal Subject–Overall usage trends (Requires Disc Srvc ‘control’)–Configuration options in Discovery services
• Expand pool of libraries• Perhaps explore WHY
Page 31
Sharing Data• With participating libraries
–Customized reports for each library• With participating publishers
–Customized reports for each publisher–Presentations as requested
• With discovery vendors–Presentations as requested
• In publications and presentations–Maintaining anonymity of data
Page 32
Past/Future Presentations• Ithaka Sustainable Scholarship Conference (October 2013)• Charleston Conference (November 2013)• ER&L/Library Journal Webinar (December 2013)• Shangai Jiao Tong Univ / Beijing Univ Forum (Jan 2014)• SCELC Colloquium (March 2014)• ER&L (March 2014) http://bit.ly/discovery-impact-erl2014 • UKSG (April 2014)• Presentations posted on slideshare :
–http://visualcv.com/lpq4t1s
[email protected] | [email protected] | [email protected]