Top Banner
Tweets and Mendeley readers Two different types of article level metrics [email protected] @stefhaustein Stefanie Haustein
23

Tweets and Mendeley readers: Two different types of article level metrics

Aug 19, 2014

Download

Education

Presentation at APE2014
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Tweets and Mendeley readers: Two different types of article level metrics

Tweets and Mendeley readers Two different types of article level metrics

[email protected] @stefhaustein Stefanie Haustein

Page 2: Tweets and Mendeley readers: Two different types of article level metrics

Overview •  Altmetrics

•  increasing use •  meaning?

•  Aim of the studies •  Data sets and methods •  Results

•  documents •  correlations •  disciplines

•  Conclusions & outlook

Page 3: Tweets and Mendeley readers: Two different types of article level metrics

Altmetrics: increasing use •  social media activity around scholarly articles growing by

5% to 10% per month (Adie & Roe, 2013)

•  Mendeley and Twitter largest altmetrics sources •  Mendeley

•  521 million bookmarks •  2.7 million users •  32% increase of users from 09/2012 to 09/2013

•  Twitter •  500 million tweets per day •  230 million active users •  39% increase of users from 09/2012 to 09/2013

Adie, E. & Roe, W. (2013). Altmetric: Enriching Scholarly Content with Article-level Discussion and Metrics. Learned Publishing, 26(1), 11-17. Mendeley statistics based on monthly user counts from 10/2010 to 01/2014 on the Mendeley website accessed through the Internet Archive Twitter statistics: https://business.twitter.com/whos-twitter and http://www.sec.gov/Archives/edgar/data/1418091/000119312513400028/d564001ds1a.htm

Page 4: Tweets and Mendeley readers: Two different types of article level metrics

Altmetrics: meaning? •  ultimate goals

•  similar to but more timely than citations Ø  predicting scientific impact

•  different, broader impact than captured by citations Ø  measuring societal impact

•  impact of various outputs Ø  “value all research products”

Piwowar (2013)

Piwowar, H. (2013). Value all research products. Nature, 493(7431), 159.

Page 5: Tweets and Mendeley readers: Two different types of article level metrics

Altmetrics: meaning? •  Altmetrics are “representing very different things”

(Lin & Fenner, 2013)

•  unclear what exactly they measure: •  scientific impact •  social impact •  “buzz”

Lin, J. & Fenner, M. (2013). Altmetrics in evolution: Defining and redefining the ontology of article-level metrics. Information Standards Quarterly, 25(2), 20-26.

Page 6: Tweets and Mendeley readers: Two different types of article level metrics

Altmetrics: meaning?

ad-hoc classifications need to be supported by research

Page 7: Tweets and Mendeley readers: Two different types of article level metrics

Altmetrics: meaning? scientist on Twitter tweeting scientific paper in non-scholarly manner: •  scientific impact? •  social impact? •  buzz?

Page 8: Tweets and Mendeley readers: Two different types of article level metrics

Altmetrics: meaning?

Page 9: Tweets and Mendeley readers: Two different types of article level metrics

Aim of the studies •  providing empirical evidence of Mendeley reader counts

and tweets of scholarly documents for a large data set •  generate knowledge about factors influencing popularity of

scholarly documents on Mendeley and Twitter •  analyzing the following research questions:

•  What is the relationship between social-media and citation counts? •  How do social-media metrics differ? •  Which papers are highly tweeted or highly bookmarked? •  How do these aspects differ across scientific disciplines?

Haustein, S., Peters, I., Sugimoto, C.R., Thelwall, M., & Larivière, V. (2014). Tweeting Biomedicine: An Analysis of Tweets and Citations in the Biomedical Literature. Journal of the Association for Information Sciences and Technology. Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (submitted). Tweets vs. Mendeley readers: How do these two social media metrics differ? IT-Information Technology.

Page 10: Tweets and Mendeley readers: Two different types of article level metrics

Aim of the studies •  large-scale analysis of tweets and Mendeley readers of

biomedical papers •  Twitter and Mendeley coverage •  Twitter and Mendeley user rates •  correlation with citations

•  discovering differences between: •  documents •  disciplines & specialties

Ø  providing an empirical framework to compare coverage, correlations and user rates

Page 11: Tweets and Mendeley readers: Two different types of article level metrics

Data sets & methods •  1.4 million PubMed papers covered by WoS

•  publication years: 2010-2012 •  document types: articles & reviews •  matching of WoS and PubMed

•  tweet counts collected by Altmetric.com •  collection based on PMID, DOI, URL •  matching WoS via PMID

•  Mendeley readership data collected via API •  matching title and author names

•  journal-based matching of NSF classification

Page 12: Tweets and Mendeley readers: Two different types of article level metrics

Data sets & methods: framework

Page 13: Tweets and Mendeley readers: Two different types of article level metrics

Data sets & methods: age biases Current biases influencing correlation coefficients

Ø  compare documents of similar age Ø  normalize for age differences

Page 14: Tweets and Mendeley readers: Two different types of article level metrics

Results: documents

Publication year

Twitter coverage

Papers (T≥1)

Spearman's ρ Mean Median Maximum

T2010 2.4% 13,763 .104** 2.1 1 237 C2010 18.3 7 3,922

T2011 10.9% 63,801 .183** 2.8 1 963 C2011 5.7 2 2,300

T2012 20.4% 57,365 .110** 2.3 1 477 C2012 1.3 0 234

T2010-2012 9.4% 134,929 .114** 2.5 1 963 C2010-2012 5.1 1 3,922

•  Twitter coverage is quite low but increasing •  correlation between tweets and citations is very low

Page 15: Tweets and Mendeley readers: Two different types of article level metrics

Results: documents

Article Journal C T

Hess et al. (2011). Gain of chromosome band 7q11 in papillary thyroid carcinomas of young patients is associated with exposure to low-dose irradiation PNAS 9 963

Yasunari et al. (2011). Cesium-137 deposition and contamination of Japanese soils due to the Fukushima nuclear accident PNAS 30 639

Sparrow et al. (2011). Google Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips Science 11 558

Onuma et al. (2011). Rebirth of a Dead Belousov–Zhabotinsky Oscillator Journal of Physical Chemistry A -- 549

Silverberg (2012). Whey protein precipitating moderate to severe acne flares in 5 teenaged athletes Cutis -- 477

Wen et al. (2011). Minimum amount of physical activity for reduced mortality and extended life expectancy: a prospective cohort study Lancet 51 419

Kramer (2011). Penile Fracture Seems More Likely During Sex Under Stressful Situations Journal of Sexual Medicine -- 392

Newman & Feldman (2011). Copyright and Open Access at the Bedside New England Journal of Medicine 3 332

Reaves et al. (2012). Absence of Detectable Arsenate in DNA from Arsenate-Grown GFAJ-1 Cells Science 5 323

Bravo et al. (2011). Ingestion of Lactobacillus strain regulates emotional behavior and central GABA receptor expression in a mouse via the vagus nerve PNAS 31 297

Top 10 tweeted documents: catastrophe & topical / web & social media / curious story scientific discovery / health implication / scholarly community

Page 16: Tweets and Mendeley readers: Two different types of article level metrics

Results: correlations

Spearman correlations between citations (C), Mendeley readers (R) and tweets (T) for all papers published in 2011 (A, n=586,600), for papers with respectively at least one citation (B, n=410,722), one Mendeley reader (C, n=390,190) or one tweet (D, n=63,800), one Mendeley reader and one tweet (E, n=45,229) and one citation, one Mendeley reader and one tweet (F, n=36,068). All results are significant at the 0.01 level (two-tailed).

PubMed papers covered by Web of Science (PY=2011)

Page 17: Tweets and Mendeley readers: Two different types of article level metrics

Results: disciplines PubMed papers covered by Web of Science 2010-2012

Page 18: Tweets and Mendeley readers: Two different types of article level metrics

Altmetrics: disciplinary biases x-axis: coverage of specialty on platform y-axis: correlation between social media counts and citations bubble size: intensity of use based on mean social media count rate

Page 19: Tweets and Mendeley readers: Two different types of article level metrics

Results: disciplines

Scatterplot of number of citations and number of tweets (A, ρ=0.181**) and Mendeley readers (B, ρ=0.677**), bubble size represents number of Mendeley readers (A) and tweets (B). The respective three most tweeted (A) and read (B) papers are labeled showing the first author.

General Biomedical Research papers 2011

Page 20: Tweets and Mendeley readers: Two different types of article level metrics

Results: disciplines

Scatterplot of number of citations and number of tweets (A, ρ=0.074**) and Mendeley readers (B, ρ=0.351**) for papers published in Public Health in 2011. The respective three most tweeted (A) and read (B) papers are labeled showing the first author.

Public Health papers 2011

Page 21: Tweets and Mendeley readers: Two different types of article level metrics

Conclusions & outlook •  uptake, usage intensity and correlations differ between

disciplines and research fields Ø  social media counts of papers from different fields are not

directly comparable •  citations, Mendeley readers and tweets reflect different

kind of impact on different social groups •  Mendeley seems to mirror use of a broader but still academic

audience, largely students and postdocs •  Twitter seems to reflect the popularity among a general public

and represents a mix of societal impact, scientific discussion and buzz

Ø  the number of Mendeley readers and tweets are two distinct social media metrics

Page 22: Tweets and Mendeley readers: Two different types of article level metrics

Conclusions & outlook •  before applying social media counts in information

retrieval and research evaluation further research is needed: Ø  identifying different factors influencing popularity of

scholarly documents on social media

Ø  analyzing uptake and usage intensity in various disciplines

Ø  differentiating between audiences and engagements

Page 23: Tweets and Mendeley readers: Two different types of article level metrics

Haustein, S., Peters, I., Sugimoto, C.R., Thelwall, M., & Larivière, V. (in press). Tweeting biomedicine: an analysis of tweets and citations in the biomedical literature. Journal of the Association for Information Sciences and Technology.

Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (submitted). Tweets vs. Mendeley readers: How do these two social media metrics differ? IT-Information Technology.

Stefanie Haustein

Thank you for your attention! Questions?

[email protected] @stefhaustein