Finding Insights in ALMS for Research Evaluation Richard Cave, @richardcave I.T. Director, PLOS Charleston Conference 2013
Jan 24, 2015
Finding Insights in ALMSfor Research Evaluation
Richard Cave, @richardcave
I.T. Director, PLOS
Charleston Conference 2013
Article-Level Metrics at PLOS
• Article-Level Metrics (ALM) allow the comprehensive analysis of the post-publication activity around a paper, employing usage stats, citations and altmetrics.http://article-level-metrics.plos.org/alm-info/
• Started in 2009 at PLOS• Lots of data, tracking over a dozen sources
2
ALMs provide the flexibility to look at a wide range of numbers
3
Authors want to see the impact of their article
4
What about funders and institutions?
• Funders want to monitor the research that they support– Wellcome Trust spends approximately £650 million on
research per year. What’s the impact?– Médecins Sans Frontières – how do they know the
reach of their article?
• Institutions want to see how their researchers are making a change– Stanford University – do librarians know what research
was published by Stanford researchers?
5
Numbers are GREAT
but what’s the impact of the research?!?
6
How easy it for institutions and funders to track research impact?
• It’s not easy!• Aggregate data for all articles? Hard to do• Follow tweets? Not likely• News/blog stories? Hard to find correct sources
7
8
ALM Reports
almreports.plos.org
Launched June 18
Allows researchers, institutions & funders to: • Create a report of the ALMs for a
set of PLOS articles • View a summary of the metrics
along and data visualizations.
But more important, provides a way to provide context of ALMs with search
Search based on: • Keyword• Author name & country • Institutional affiliation• Publication date• Subject areas• Funder
9
10
For researchers at your institution, can show impact for all of their articles
For institutions and funders - Get specific!
(author_affiliate:"Stanford University") AND (financial_disclosure:"National Institutes of Health" )
(author_affiliate:"Stanford University") AND (financial_disclosure:"Wellcome Trust")
(author_affiliate:"Stanford University") AND (financial_disclosure:"National Cancer Institute")
(author_affiliate:"Stanford University") AND (financial_disclosure:"National Cancer Institute" AND financial_disclosure:"Doris Duke Foundation")
11
Page views and citations over time
Standford University + National Cancer InstituteTotal usage includes page views and downloads from PLOS and PMC. Bubble size correlates with Scopus citations and bubble color with the PLOS journal.
12
Page views and Mendeley bookmarks over time
Standford University + National Cancer InstituteTotal usage includes page views and downloads from PLOS and PMC. Bubble size correlates with Mendeley bookmarks and bubble color with the PLOS journal.
13
Rectangle size correlates with total views (HTML and PDF) from PLOS website. Color intensity correlates with number of Scopus citations.
Article usage and citations by subject category
Standford University + National Cancer Institute
Research collaborationsStandford University + National Cancer Institute
15
Wellcome Trust evaluation
Adam Dinsmore at Wellcome Trust:• Identified and exported the entire corpus of PLOS articles
associated with the Wellcome• Analysed the ALMs associated with them and benchmarked
against the sector average• Produced a consumer report
Findings:• Wellcome Trust associated PLOS articles are viewed, saved and cited
more often than the average for the biomedical research sector• They are discussed by the blogosphere more often than the sector
averagehttp://article-level-metrics.plos.org/files/2013/10/Dinsmore.pptx
16
For Publishers• Improve Business Intelligence
– Optimize publicity efforts with more targeted campaigns (Call For Papers, conference presence, etc.)
– Launch new collections of papers in areas of interest – Find new fields growing at the intersections between communities of
interest
• Deepen Editorial Capacities– Direct editorial resources to promote new research of interest in
growing fields– Identify promotional opportunities for papers and authors (writing
press releases, front matter content, editorial overviews, etc.)– Uncover potential new authors
• Deliver a Richer Product– Maximize front matter visibility – Create more targeted editorial overviews – Enhance press reviews and promotional content for high-impact
papers17
But beware of dragons
• Data issues = Garbage in, garbage out• Author names• Funder names
• “Welcome Trust” vs. “Wellcome Trust”• Institutional names
• A dozen ways to enter “University of California, Berkeley”
• Will be helped in the future by:• ORCID • FundRef
18
ALM info
• General information on ALM
http://article-level-metrics.plos.org/• ALM Reports
http://almreports.plos.org/– Future: ALM Reports will be open source but will have
limitations (no search functionality)
• ALM Application is Open Source (Ruby on Rails)https://github.com/articlemetrics/alm – Recently launched at Copernicus Publications– OJS has a beta trial
19