Measuring Research Impact Dr. Neeraj Chaurasia Indian Institute of Technology (IIT) Delhi [email protected]
Measuring Research Impact
Dr. Neeraj ChaurasiaIndian Institute of Technology (IIT) Delhi
Outlines
Research Impact
What we are Measuring ?
Why Assess Scholarly Impact of Research ?
How do we Measure the Impact of our Work?
Researcher Impact
Research Metrics
An Overview of available Metrics and Tools
Altmetrics
Role of Libraries and Librarians
What is Research Impact?
"Research impact is the demonstrable
contribution that excellent research makes to
society and the economy. Impact embraces all
the extremely diverse ways in which research-
related knowledge and skills benefit individuals,
organizations and nations including academic,
economic, and societal impacts...”
(Research Councils UK)
What we are
Measuring ?
Journal Quality
Article reach
Article impact/Influence
Dissemination
Researcher Impact
Why Scholarly Impact of Research need to be Measure ?
Strengthen CV for promotion or tenure
Showing individual or collective productivity
Quantify ROI on research for grant renewals and progress reports
For future funding by illustrating the value of your research
Identify who is using your work and confirm that it is appropriately credited
Identify collaborators within or outside of your discipline
Benchmarking - self, research group, department, or institution
Measuring the Citations
Showcase network of collaborators
How are we Measuring?
quantitative methods – such as - citation counts, the h-index,
and journal impact factors
there is no one tool or system that completely measures impact
of research
each database or tool uses its own measurement systems,
indices, data and authority files
difficult to use these tools to compare across disciplines that
have different research and publication practices.
Research Metrics
➢Quantitative analysis of scientific and scholarly research outputs andtheir impacts.
➢ Include a variety of measures and statistical methods for assessing thequality and broader impact of scientific and scholarly research, as wellas to track researcher impact
➢ Measure impact and provide insight into the influence of specificjournal publications, individual articles, and authors.
Research Metrics - How are we Measuring? –
Journal-Level Metrics
Measure quality of Journal using citation formulas, such as Impact Factor, Helps to track citation patterns within journals anddetermine which journals are highly-cited.
Article-Level MetricsArticle Metrics, or citation tracking, is used to determine if an article, book, journal, or particular author has been cited by another work.
Author-Level Metrics -Measure bibliographic impact of individual authors
-Measure the impact and productivity of a researcher.
AltmetricsMeasures and monitors reach and impact of research through online interactions like soial media mentions, data sets, websites, blog posts, and more…
Some of the more common metrics and tools you can use to measure research impact
Different Level of Metrics
Journal-Level Metrics
Some of common metrics and evaluation tools used to evaluate and measure a journal’s impact
Journal Impact Factor (JIF) CiteScore SCImago Journal
Ranking (SJR)
Source Normalized Impact
per Paper (SNIP)
Measure
of...
Attempt to quantify the
importance of a particular
journal in a field.
Frequency with which the
'average article' in a journal
has been cited in a particular
year or other defined time
period
Calculates the average
number of citations
received in a calendar
year by all items
published in that journal
in the preceding three
years
Measure of prestige;
accounts for number of
citations received by a
journal and importance
of journals that
citations came from
Normalizes for differences in
citation behavior between
subject fields.
Measures citation impact by
weighting citations based on
the total number of citations
in a subject field
Calculation A: Number of citations in the
current year to papers
published in the journal in the
previous two years
B: Total number of articles
published in the journal in the
previous two years
Impact Factor = A/B
Counts all documents
since they all have the
potential to attract
citations,
CiteScore is independent
of the document-type
classification
Citations from more
prestigious journals
(higher SJR) weighted
more than citations
from less prestigious
journals (lower SJR)
Citations from subject fields
in which citations are less
likely are weighted more
Where can
you get it?
Journal Citation Reports
Clarivate Analytics
Find CiteScore at
https://journalmetrics.sc
opus.com/ Elsevier
free from SCImago
Journal & Country Rank
Elsevier
Scopus OR free from CWTS
Journal Indicator
Journal Impact Factor (JIF)
Number of citations to a journal in a given year from articles occurring in
the past 2 years, divided by the number of scholarly articles published in
the journal in the past 2 years
For example, the 2019 impact factor for a journal would be calculated as follows:
A: Number of citations in the current year(2019) to papers published in the journal in the previous two years (2017 & 2018)
B: Total number of articles published in the journal in the previous two years (2017 & 2018)
2019 impact factor = A/B
e.g. 600 citations = 2
150 + 150 articles
Scopus-Based Metrics CiteScore
Calculates the average number of citations received in a calendar year by all items published in that journal in the
preceding 3 years
CiteScore counts all documents since they all have the potential to attract citations, and the Impact Factor counts
the documents considered most likely to attract citations.
CiteScore is independent of the document-type classification
Find CiteScore at https://journalmetrics.scopus.com/
SJR Indictor (SciMago) Measure of prestige;
Citations are weighted, depending on the rank of the citing journal
A citation from an important journal will count as more than one citation; a citation coming from a less important
journal will count as less than one citation.
Find at www.scimagojr.com
SNIP (Source Normalized Impact per Paper) SNIP weighted citations based on the total number of citations in a subject field.
Normalizes for differences in citation behavior between subject fields.
Corrects for subject-specific characteristics of the field someone is publishing in so any two journal can be compared
Find at CWTS Journal Indicators http://www.journalindicators.com/
Author Level Metrics
h-Index and i10-Index
h-Index i10-IndexMeasure : Measure both productivity and impact of publications
Meaningful when compare to others within the same discipline area
Calculation: A scholar with an index of h has published h paperseach of which has been cited by others at least htimes..
e.g. - If a researcher has 20 papers, each of which hasat least 20 citations, their h-index is 20.
e.g. - An h-index of 10 tells us that an author haspublished 10 papers which have each been cited atleast 10 times
It was introduced in 2011 by Google aspart of their work on Google Scholar,
i10 index refers to the number ofpaper with 10 or more citations.
Limitations:Inaccurate measure of early career researcher impact. Only used in Google Scholar
Where to Find h-Index
Value of the h-index may vary depending on the : source of
information, number of indexed publications, time span, etc.
Citation databases
Web of Science
Scopus
Google Scholar
Requires creation of Google Scholar profile before providing metrics
H-index tends to be higher than what is calculated by Web of Science
Definition
A scientist has index h if h of his/her Np papers have at least h
citations each, and the other (Np − h) papers have no more than
h citations each.
h-Index
A scholar with an index of h has published h papers each of which has been cited by others at least h times
20 papers
cited 20 times or more
Calculate Your h-index - Manually
To manually calculate your h-index, organize articles in descending order, based on the number of times they have been cited....
Web of Science, Scopus, and Google Scholar can also be used to calculate an h-index for that particular citation-tracking database.
In the below example, an author has 8 papers that have been cited 33, 30, 20, 15, 7, 6, 5 and 4
times. This tells us that the author's h-index is 6.
What does an h-index of 6 mean?
An h-index of 6 means that this author has published at least 6 papers that have each received at
least 6 citations.
H-index - Interpretation
14-05-2020
14-05-2020
In the below example, an author has 8 papers that have been cited 33, 30, 20, 15, 7, 6, 5 and 4 times. This tells us that the author's h-index is 6.
What does an h-index of 6 mean?•An h-index of 6 means that this author has published at least 6 papers that have each received at least 6 citations.
Interpreting
14-05-2020
In the below example, an author has 8 papers that have been cited 33, 30, 20, 15, 7, 6, 5 and 4 times. This tells us that the author's h-index is 6.
What does an h-index of 6 mean?•An h-index of 6 means that this author has published at least 6 papers that have each received at least 6 citations.
Interpreting
Web of Science (h-Index)
Web of Science (h-Index)
Google Scholar (h-Index)
Google Scholar - Metrics
H-Index Limitations
• h-index increases with age so comparing
productivity of younger researchers is
problematic
• Calculated in controlled databases but need
comprehensive citation report of all author’s
publications
• Different databases yield different h-index
scores
• Can comparing scientists working in the same
field;
• Citation conventions differ widely among
different fields.
Example - h-index of a
author :
Coverage is different :
Scopus
publications indexed = 10
H-index= 3
Google Scholar
publications indexed = 28
H-index = 6
Web of Science
publications indexed = 5
H-index = 1
Article-Level Metrics
Article-Level
Metrics
Citation-based and altmetric measures can show impact of individual research publication
How many times was an article cited
How is it tracking in social media
What is the geographic distribution of citing papers
What is the disciplinary distribution of citing papers
What is the impact outside of the scholarly Community
Altmetrics address
Ref : http://pitt.libguides.com/altmetrics/introduction
How many times something is downloaded
Who is reading the work
Has is it been covered by
news outlets
Who is commenting on
the work
How is it being shared
Which countries are looking at
my work
Altmetrics
Coined by Jason Priem in 2010
Also known as; Alternative Metrics/ Article Level Metrics (ALM) or ‘Alt-metrics’
Alternative to more traditional citation impact metrics, such as impact factor and h-index
Altmetrics include social media activity, coverage in media outlets, and inclusion in policy documents or scholarly commentary, among other types of activity.
Several publishers are providing such information to readers, including BioMed Central, Public Library of Science, Frontiers, Nature Publishing Group, Elsevier, Research Gate, academic.edu
Some Altmetrics Tools
Altmetrc
ImpactStory
ReaderMeter
ScienceCard
PLoS impact explorer
PLoS article level metrics
PaperCritic
Crowdometer
PlumX
Usage
HTML views, PDF/XML downloads (various sources – eJournals, PubMed Central,
FigShare, Dryad, etc.)
Captures
CiteULike bookmarks, Mendeley readers/groups, Delicio.us
Mentions
Blog posts, news stories, Wikipedia articles, comments, reviews
Social Media
Tweets, Google+, Facebook likes, shares, ratings
Citations
Web of Science, Scopus, CrossRef, PubMed Central, Microsoft Academic Search
Ref : Altmetrics Manifesto - http://altmetrics.org/about/
Altmetrics
Altmetrics data is aggregated from many sources
https://springerlink.altmetric.com/details/10218936
https://www.nature.com/nclimate/journal/vaop/ncurrent/nclimate3316/metrics
Researchers are communicators:
Within academia:
Presentations and seminars
Academic books
Journal articles and posters
Term papers and essays
Meetings and conferences
Within society:
Speaking at public events
Interviews and news mentions
Press, social media, blogs
We need to measure both?
Role of Libraries and Librarians
High quality content in the form of E-Resources; includes the citation databases and Research Supporttools, Data management tools, Reference management tools (Mendeley, Endnote, zotaro, etc),Grammarly, etc
Help in selecting a good impact factor journal to publish their research
Help in providing list of quality journals in respective disciplines to researchers
Help in Publishing in open access journals/platforms
Help in resolving APC related issues
Help in creating the ORCID profile, etc
Increase the visibility and enhance the impact of their research work - like IRINS
Academic/Creative Writing & Publishing
Awareness on Predatory publishing/ plagiarism
Bibliometrics support
Research Guides and E-Learning Support
Library Outreach activities
Acknowledgement and declaration by presenter
I would like express my sincere thanks to various internet
resources and also Authors of those Internet sources used to
prepare this presentation.
Wherever possible the links have been provided however any
omission is duly regretted.
The presentation is mainly prepared to create an awareness
amongst students and researchers about the various Research
Support Tools.
These slides have been/being used in my various talks and
presentations both online and offline