COUNTER: making statistics useful Peter Shepherd Director COUNTER January 2007
Mar 27, 2015
COUNTER: making statistics useful
Peter Shepherd
DirectorCOUNTER
January 2007
Background
Understanding usage Different approaches Role of usage statistics
Usage statistics Should enlighten rather than obscure Should be practical Should be reliable Are only part of the story Should be used in context
How can usage statistics help us measure… Success? Value? Behaviour?
COUNTERCodes of Practice
Definitions of terms used Specifications for Usage Reports
What they should include What they should look like How and when they should be delivered
Data processing guidelines Auditing Compliance Maintenance and development of the
Code of Practice Governance of COUNTER
COUNTER: current Codes of Practice
1) Journals and databases
Release 1 Code of Practice launched January 2003 Release 2 published April 2005 replacing Release 1 in January
2006 Now a widely adopted standard by publishers and librarians 60%+ of Science Citation Index articles now covered Librarians use it in collection development decisions Publishers use it in marketing to prove ‘value’
2) Books and reference works
Draft Code of Practice published in February 2005 for comment
Final version incorporating feedback was launched March 2006
Relevant usage metrics less clear than for journals Different issues than for journals
Direct comparisons between books less relevant Understanding how different categories of book are used is more
relevant
Journal and Database Code of Practice
Usage Reports
Journal Report 1 Full text article requests by month and journal
Journal Report 2 Turnaways by month and journal
Database Report 1 Total searches and sessions by month and database
Database Report 2 Turnaways by month and database
Database Report 3 Searches and sessions by month and service
Code of Practice for books
Book Report 1 Number of successful requests by month and title
Book Report 2 Number of successful section requests by month and
title Book Report 3
Turnaways by month and title Book Report 4
Turnaways by month and service Book Report 5
Total searches and sessions by month and title Book Report 6
Total searches and sessions by month and service
Journal Report 1Full text article requests by journal
Html and PDF totals reported separately
COUNTER: deriving metrics from Journal Report 1
Local metrics For libraries and library consortia At journal, collection and publisher level To compare the cost-effectiveness of journal
subscriptions To assess the value of Big Deals
Global metrics For authors, funding agencies, libraries and
publishers At article, journal, collection and publisher level To compare quality and value
COUNTER: ‘local’ metrics
JISC (UK Joint Information Systems Committee) Funded by UK higher education funding councils Supports higher education in the use of information and
communications technologies Access to information and communication resources Advice on creation and preservation of digital archives Implications of using ICT Network services and support Research to develop innovative solutions
National overview of online journal usage Develop a reliable, widely applicable methodology Use COUNTER Journal Report 1 ‘article full-text requests’
Local metrics: JISC Project
COUNTER data was analysed in relation to: usage range Price band Subject category
Metrics derived from this analysis Trend in number of full-text article requests Full text article requests per title Full text article requests per publisher package Full text article requests per FTE user Most requested titles Usage of subscribed vs.. unsubscribed titles Cost per full-text article request Cost per FTE user
Summary report available at:www.ebase.uce.ac.uk/projects/NESLi2.htm
Local metrics: JISC project
Growth in article downloads Publisher A: 12%- 208% Publisher B: 12%- 59% Publisher C: 23%- 154% Publisher D: 22%- 81%
Cost per download Publisher A: £0.97- £5.26 Publisher B: £0.70 - £2.91 Publisher C: £0.80 - £3.29 Publisher D: £0.45 - £2.26
COUNTER: ‘global’ metrics
Impact Factor Well-established, easily understood and accepted Endorsed by funding agencies and researchers Does not cover all fields of scholarship Reflects value of journals to researchers Over-emphasis on IF distorts the behaviour of authors Over-used, mis-used and over-interpreted
Usage Factor Usage-based alternative perspective Would cover all online journals Would reflect value of journals to all categories of user Would be easy to understood
Global metrics: UKSG Project
Assess the feasibility of developing and implementing journal Usage Factors
Level of support from author, librarian and publisher communities
Data from which UF would be derived COUNTER Journal Report 1? Article numbers Process for consolidation, calculation and reporting of UFs
Factors in the calculation Level of reporting Total usage Articles
Report in April 2007 Just completed set of 29 interviews with industry leaders Wider online survey will take place in February 2007
UKSG Project: feedback
Are the COUNTER usage statistics sufficiently robust? Frustration at lack of comparable, quantitative data on journals Should items covered by restricted to articles? Many journals still have significant usage in print Diversity of views on the factors in the calculation
Specified usage period Specified publication period
Usage data is more susceptible to manipulation Will the journal be a meaningful concept in the future? Two measures with different limitations are better than one,
and UF will be derived from a set of credible, understandable data
Usage data will be used as a measure of value, whether publishers like it or not
Current issues Interface and user behaviour effects on usage
statistics E.g. downloading HTML and PDF of the same article in one
session COUNTER is testing data filter solutions, but what does the
duplicate downloading signify? Reporting separately purchasable digital archive
usage Currently all usage for a journal is usually reported together Separately purchasable archives mean we need separate
reports for archival content, or a year of publication breakdown of usage
Usage in Institutional Repositories Growth in Institutional Repository (IR) content Need for credible IR usage statistics IR usage statistics already being collected, but no standards
What to count and what not to count:- How to deal with partial open access journals How to deal with journals whose content becomes free after
a fixed time period
New Development - SUSHI
Standardized Usage Statistics Harvesting Initiative (SUSHI)
No mechanism yet for automatically retrieving, combining, and storing COUNTER usage data from different sources
NISO-sponsored XML-based SUSHI aims to provide a means to do just this, via a standard model for machine to machine automation of statistics harvesting.
COUNTER and NISO have signed an agreement to work together on the development of SUSHI. More details of SUSHI can be found at:-
http://www.niso.org/committees/SUSHI/SUSHI_comm.html
Other technologies that may impact usage
reporting
Metasearch
Also known as: Federated search Broadcast search
User presented with a single search interface Searches multiple information sources at same
time Eliminates (or assists with) resource selection Retrieve, consolidates and ranks results A new challenge for statistics
Metasearch: a new challenge
What can be affected Session counts Search counts
Why? “Search all” option or automatic
selection/search of many resources Perform simultaneous activities Optimization techniques
Prefetching
Google, and other services are introducing prefetching of links to improve performance
How it works… A user performs a search that results in a
number of results. Google will introduce <link> tags for the
first few to cause the browser to fetch in the background
The prefetched pages are loaded into browser cache
Prefetching
If the results were links to publisher full text then: Each prefetch is a full text request When the user does click the link,
another header request is made, that could be considered yet another full text request
Without some kind of control, this activity could result in significant over-counting
Prefetching
The prefetch action is identified in the HTTP header of the request
Vendors can change systems to recognize prefetch and do one of: Simply return an error (don’t deliver
full text), the end user does not see this error, or
Fulfill the request, but do not count the transaction
What do usage statistics tell us about…
Success? Value? Behaviour?
Usage statistics: measuring value
JISC Project has identified some basic metrics derived from COUNTER data
Trend in number of full-text article requests Full text article requests per title Full text article requests per publisher package Full text article requests per FTE user Usage of subscribed vs. unsubscribed titles Cost per full-text article request Differences between subject fields
But…. Limited to data from COUNTER-compliant vendors Does not distinguish between different types of usage
Usage statistics: measuring success
Impact Factor Widely used as a measure of ‘success’ for a journal But…
Citation habits vary across different scientific fields Citation patterns depend on journal type Do not cover all journals
Usage Factor? An alternative measure Relevant in applied fields, where citation levels are
lower But….
Requires global usage statistics Requires universal adoption of the same standards
Usage statistics: measuring behaviour
Usage of different components of the journal TOC, abstract, full-text, references
Variations between fields Physics, medicine
Variations between institutes Academic
Teaching, research, etc Between departments
Industrial
COUNTER Membership
Member Categories and Annual Fees (2007)
Publishers/intermediaries: £530 Library Consortia: £355 Libraries: £265 Industry organization: £265 Library affiliate: £106 (non-voting
member)
Benefits of full membership Owner of COUNTER with voting rights at
annual general meeting, etc. Regular bulletins on progress Opportunity to receive advice on
implementation
http://www.projectcounter.orghttp://www.projectcounter.org
Apply for COUNTER membership
Apply for COUNTER membership