Altmetrics journal club

Thelwall, Mike, et al. "Do altmetrics work? Twitter and ten other social web services." PloS

one 8.5 (2013): e64841.

Anne Madden

Journal Club 27th April 2015

• What are altmetrics?

• How are they calculated?

• Paper: Rationale and methodology

• Results?

• Discussion: do altmetrics “work”?

Priem (2010): definition

“No one can read everything. We rely on filters to make sense of the scholarly literature, but the narrow, traditional filters are being swamped.”From the altmetrics manifesto. (Priem 2010)

http://support.altmetric.com/knowledgebase/articles/83337-how-is-the-altmetric-score-calculated

Rationale for this study

“To what extent do the altmetric indicators associate with citation

counts?”

Building some form of evidence base to underpin altmetrics?

Methodology• PubMed articles with a non-zero altmetric

score

• Sign test: “..each article is compared only against the two articles published immediately before and after it….”

• Self-citations removed – why?

• Altmetric scores gathered between July 2011 – 1st Jan 2013

Definition of success

“..if the altmetric score is higher than the average altmetric score of the two adjacent articles and its citation score is higher than the average of the two adjacent articles OR the altmetric score is lower than the average altmetric score of the two adjacent articles and its citation score is lower than the average of the two adjacent articles.”

Results

1. Total no. of successes v no. of failures: overall success2. No. of journals where success exceeds failure: success3. Individual altmetric level: mixed results. Four

altmetrics “significantly and positively correlate with citations..the correlation for Twitter is significant and negative”.

• “..results provide strong evidence that six of the eleven altmetrics.. associate with citation counts. ..correlation effect size is unknown”

Experiment. Starting from non-zero citations: a very rough comparison

• PubMed papers, with citations

• Timeframe: mid to end 2013

• Search: “Influenza” [Title]

• 20 papers selected at random

• 14 had non-zero citations

Lowest altmetric score was for non-English paper (Chinese)

Cut-off point where association between altmetric score and citation count lose significance.

Boon 2014, IATUL Proceedings: Altmetrics is an Indication of Quality Research or Just HOT Topicshttp://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=2034&context=iatul

Discussion• Anonymity in most altmetric tools• Multiple accounts: the “Scholarly

Selfie”• “Gaming” – could affect both

altmetric and citation scores• Relative labour-intensity and

engagement with paper

GAMING

Groups doing it for themselves

“Early experience with JACR tweet chats demonstrates that organizing Twitter microblogging activities around topics of general interest to their target readership bears the potential for medical journals to increase their audiences and reach.”

Hawkins, C. Matthew, et al. "The Impact of Social Media on Readership of a Peer-Reviewed Medical Journal." Journal of the American College of Radiology 11.11 (2014): 1038-1043.

Topical, timely, authoritative, clear message, relevance to a broad audience

Appeals to a broader audience; the ready-made headline…

4 Wikipedia pages (aubergine bit)

Do altmetrics work?

“Not everything that counts can be counted, and not everything that can be

counted counts”

Neither altmetrics nor citation scores are

evidence of quality.

“Although association preferences documented in our study theoretically could be a consequence of either mating or shoaling preferences in the different female groups investigated (should we cite the crappy Gabor paper here?), shoaling preferences are unlikely drivers of the documented patterns ……”

From Ben Goldacre on YouTube

Hopewell, Sally, et al. "Publication bias in clinical trials due to statistical significance or direction of trial results." The

Cochrane Library (2009).

“Trials with positive findings are more likely to be

published and published quicker than trials with

negative findings.”

Failure to publish: “scientific misconduct” valid

expectation on the part of trial participants that

their participation will contribute to knowledge.

…which scholarly products are read, discussed, saved and recommended as well as cited.

“..provide an insight into things we have not measured before….” Tattersall 2015

“…analyzing the impact of outputs in different formats.. as opposed to the analysis of only journal papers.” (Costas 2014)

“…impact of scholarly material, with an emphasis on social media outlets…” (Priem 2014)

…more timely data, showing evidence of impact in days instead of years

..the impact of web-native scholarly products

…impacts on diverse audiences including scholars but also practitioners, clinicians, educators and the general public. (Piwowar 2013)

Scholarly journals or scholarly communication?

“For all new grant applications from 14 January, the US National Science Foundation (NSF) asks a principal investigator to list his or her research “products” rather than “publications” in the biographical sketch section. This means that according to the NSF, a scientist’s worth is not dependent solely on publications.”

Piwowar, Heather. "Altmetrics: Value all research products." Nature 493.7431 (2013): 159-159.

Djuricich, Alexander M., and Janine E. Zee-Cheng. "Live tweeting in medicine:'Tweeting the meeting'." International Review of Psychiatry 0

(2015): 1-7.

Cochran, Amalia, et al. "Use of Twitter to document the 2013 Academic Surgical Congress." Journal of Surgical

Research 190.1 (2014): 36-40.

Aslam, Anoshé A., et al. "The Reliability of Tweets as a Supplementary Method of Seasonal Influenza Surveillance." Journal of medical Internet research 16.11 (2014).

“….this study demonstrated increased accuracy in using Twitter as a supplementary surveillance tool for influenza as better filtering and classification methods yielded higher correlations for the 2013-2014 influenza season than those found for tweets in the previous influenza season, where emergency department ILI rates were better correlated to tweets than sentinel-provided ILI rates.”

• H-index outdated? Register for your ORCID Identifier (http://orcid.org). For more info see: http://libguides.ucd.ie/orcid

• Create an author profile on Google Scholar Citations (http://scholar.google.com/citations)

• Create an impactstory account for your biog. (http://impactstory.org/)

From: Rathemacher, Andrée J. "Article-Level Metrics and Altmetrics: New Ways to Measure the Impact of Your Research." (2014).

• When publishing ask for permission to store a copy in the Institutional Repository (http://www.lenus.ie)

“Do altmetrics work?” Has Thelwall proven his case?

• Altmetrics were never intended to replace / anticipate / mimic citation counts.

• Altmetrics have a far wider scope, with more focus on unpublished born-digital material and other audiences. Therefore, to prove they “work”, more multi-faceted research required.