1 b b www.know-center.at Altmetrics Open Science – Technologie und Wissenschaft Wien, 13. Jänner Dr. Peter Kraker
1
b
b
www.know-center.at
Altmetrics
Open Science – Technologie und Wissenschaft
Wien, 13. Jänner
Dr. Peter Kraker
2
AN EXPONENTIALLY
EXPANDING UNIVERSE
Part 1
3
The Hitchhiker‘s Guide to the Galaxy
3 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
4
How to Stay on Top of the Universe
4 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
5
How to Stay on Top of Scientific Literature
5 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
6 6 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
“One of the diseases of this age is the
multiplicity of books;
they doth so overcharge the world that it is not
able to digest the abundance of idle matter that
is every day hatched and brought forth into the
world.“
Attributed to Barnaby Rich in 1613
7 7 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Price (1963)
8
SCIENTOMETRICS:
THE QUANTITATIVE
ANALYSIS OF SCIENCE
Part 2
9
Eugene Garfield: Pathways through Science
9 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Science Citation Index (Garfield 1955)
Web of Science/Web of Knowledge
An index of incoming citations
Outgoing citations
Incoming citations
10 10
Garfield et al.
(1964)
Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
11 11 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
12 12 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Van Eck and Waltman
(2010)
13
Evaluative Scientometrics: Impact Factor
13 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
A measure to quantify the relative importance of a scientific
journal
The average number of citations in a given year y to papers of a
journal in the years y-1 and y-2
𝐼𝐹𝐽𝑜𝑢𝑟𝑛𝑎𝑙 𝑋, 2015 =
𝐶𝑖𝑡𝑎𝑡𝑖𝑜𝑛𝑠 𝑖𝑛 2015 𝑡𝑜 𝑎𝑟𝑡𝑖𝑐𝑙𝑒𝑠 𝑝𝑢𝑏𝑙𝑖𝑠ℎ𝑒𝑑 𝑏𝑦
𝐽𝑜𝑢𝑟𝑛𝑎𝑙 𝑋 𝑖𝑛 2013 𝑎𝑛𝑑 2014
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑟𝑡𝑖𝑐𝑙𝑒𝑠 𝑝𝑢𝑏𝑙𝑖𝑠ℎ𝑒𝑑 𝑏𝑦 𝐽𝑜𝑢𝑟𝑛𝑎𝑙 𝑋
𝑖𝑛 2013 𝑎𝑛𝑑 2014
14 14 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Source: Thomson Reuters
15 15 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Journal X
Paper 1 Citations 15
Paper 2 Citations 17
Paper 3 Citations 14
Paper 4 Citations 18
Paper 5 Citations 15
Paper 6 Citations 15
Impact Factor 15.5
Median 15 2
Journal Y
100
2
1
2
1
2
18.0
Rank 2 1
16
Criticisms of the Impact Factor
16 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Source: Amin & Mabe (2000)
17
Antidotes
17 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
„Blockbuster“ papers can skew the IF
Use robust statistics (e.g. the median)
Report variation in the data (CWTS Journal Indicators)
Publication and citation behavior varies wildly between fields
Source-normalised impact factor (SNIP)
The IF window is too short
3- and 5-year Impact Factors
18
Criticisms of citation-based metrics
18 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Citations take very long to appear in meaningful quantities
Citation-based metrics are dependent on the corpus that is used
for calculation
Citations are mostly assigned to articles, leaving other products
of research out of the picture (data, code etc.)
Citation-based metrics are often intransparent and irreproducible
19
ALTMETRICS:
A NEW HOPE
Part 3
20
Setting the Stage for Alternative Metrics
20 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Open Science and the publication of more research products
Increased use of online services in the scientific community
Source: figshare
21
Altmetrics
21 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Altmetrics: metrics based on data generated in online systems
Seeing academic literature through the eyes of the readers (Rowlands
& Nicholas 2007)
Usage data (downloads, views, readership)
Links, likes and shares
Blogs and comments
Promises of altmetrics
Assess publications quicker and on a broader scale
Consider all outputs of research, not just papers
The altmetrics manifesto: http://altmetrics.org
22 22 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
http://zenodo.org/record/34079
23 23 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
http://www.altmetric.com/details/4822452
24 24 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Source:
https://impactstory.org/Carl
Boettiger
25 25 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
26 26 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
http://openknowledgemaps.org
Kraker et al.
(2014)
27 27 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
http://stellar.know-center.tugraz.at/umap
Evolution of a Knowledge Domain
Kraker et al.
(2014)
28
The Emerging Open Digital Science Ecosystem
Bibliographic data – bibliometric data – full texts – open source software
29
CAVEATS
Part 4
30 30 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
31 31 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
32 32 Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Kraker & Lex
(2014)
33
The Leiden Manifesto: Ten Principles for Research
Metrics (Hicks et al. 2015)
1) Quantitative evaluation should support qualitative, expert
assessment
…
4) Keep data collection and analytical processes open,
transparent and simple
5) Allow those evaluated to verify data and analysis
…
9) Recognize the systemic effects of assessment and indicators
10) Scrutinize indicators regularly and update them
34
Further Reading
Blog: http://blog.wikimedia.de/2015/09/15/freies-wissen-und-
wissenschaft-teil-4-ueber-die-notwendigkeit-fuer-mehr-offenheit-
und-transparenz-in-der-qualitaetssicherung-und-evaluierung-
open-peer-review-und-open-metrics/
Publication: Kraker, P., & Lex, E. (2015). A Critical Look at the
ResearchGate Score as a Measure of Scientific Reputation.
In ASCW’15 Workshop at Web Science 2015.
doi:10.5281/zenodo.35401
Source Code: https://github.com/pkraker/Headstart