Research assessment: New metrics? More metrics? No metrics? Anne Cambon-Thomsen CNRS & Université de Toulouse Epidemiology and Public health, Inserm, UMR 1027 Metrics and Impact factors of what? The example of BRIF, Bioresource Research Impact Factor 27/07/2016 1
105
Embed
Research assessment: New metrics? More metrics? No metrics? · Anne Cambon-Thomsen CNRS & Université de Toulouse Epidemiology and Public health, Inserm, UMR 1027 Metrics and Impact
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Research assessment: New metrics? More metrics? No metrics?
Anne Cambon-Thomsen CNRS & Université de Toulouse
Epidemiology and Public health, Inserm, UMR 1027
Metrics and Impact factors of what?
The example of BRIF, Bioresource Research
Impact Factor
27/07/2016 1
What is science producing and needing?
• Publications
• Infrastructures
• Databases and datasets
• Collections of biological samples and data attached
• Specialised software and methods
• Bioinformatics tools
• …..
Among thoses which ones are measured, evaluated, valued?
27/07/2016 ESOF 2016, Manchester, 24-27/7/2016 2
Human bioresources are key
components of biomedical research.
Yet, their role is underestimated and the
work provided to setting up and
maintaining a valid bioresource is not
recognized. Cambon-Thomsen et al. Nat Genet 2003, 34:25–26
• biological samples with
associated data (medical/epidemiological, social),
• databases independent of
physical samples
• other biomolecular and
bioinformatics research
tools
BIORESOURCES (BR)
27/07/2016 ESOF 2016, Manchester, 24-27/7/2016 3
Why are such resource sharing important….
Much biomedical/epidemiological research is based on usingbioresources / approx. 300 million of tissue samples stored in the USA and 20 million of biological resources in Europe, for research and market use.
• Their access to all relevant researchers is essential
• Promoting their sharing is crucial, but does not mean « just » putting files on the web!
• It requires work…. Poorly recognised
There are today principles but few tools and ~ no incentive / tools to that.
ESOF 2016, Manchester, 24-27/7/201627/07/2016 4
and poorly done?
ESOF 2016, Manchester, 24-27/7/2016
• BR not visible enough
• difficult to trace
• not acknowledged adequately
• difficult to assess their usage reliably
due to:
lack of a unique BR identification system to trace them precisely
lack of standards for BR citation in the scientific literature
lack of indicators describingefficient usage and management of BR
27/07/2016 5
Anne Cambon-Thomsen, leader
Laurence Mabile, project manager
The Bioresource
Research Impact
Factor initiative
ESOF 2016, Manchester, 24-27/7/201627/07/2016 6
The BRIF initiative
• HOW? • By creating a set of adequate
standardized tools:
• standards for citation / acknowledgement of bioresourcesin scientific articles in order to trace their use on the web
• BRIF indicator: a tool to establish frequency of BR use and evaluate their impact based on metrics and on the use of a unique digital resource identifier
Work in progress, currently developing a framework for recognising the specific contribution of bioresources to Research (in scientific literature)
• Final objective:
• To create tools that will:
- promote a philosophy of sharing in the biomedical community
- facilitate the practice of sharing policies for data and samples
ESOF 2016, Manchester, 24-27/7/2016
www.gigasciencejournal.com/content/2/1/7
27/07/2016 7
Working subgroups
‘BRIF & Digital Identifiers’co-chaired by G. A. Thorisson, University of Leicester, UK and P.A. Gourraud, University of California
‘BRIF dissemination’chaired by L. Mabile, Inserm UMR1027, Tlse, FR [email protected]
ESOF 2016, Manchester, 24-27/7/201627/07/2016 8
CoBRA : Citation of Bioresources in Research Articles. A milestone developed
by the BRIF Journal Editors’ subgroup
Sensitizing editors and their associations about BR issues (targeted surveys) Dissemination of BRIF in international Science Edition and
other Conferences Organize restricted workshops addressed to Journal editors
and experts (Rome, June 21, 2013; Toulouse, Oct 9, 2016)Work out a guideline for citation of bioresources Launching an open access journal for describing bioresources
with re-use potential
ESOF 2016, Manchester, 24-27/7/201627/07/2016 9
ESOF 2016, Manchester, 24-27/7/201627/07/2016 10
The Open Journal of Bioresources (OJB) features peer-reviewed short papers helping researchers to locate and cite bioresources with high reuse potential.
Making bioresources more openly discoverable has enormous benefits not only for the research community and the wider public, but for the producers of the bioresources as well.
Both the resources and the OJB papers are citable and this will be tracked to provide authors with metrics on reuse and impact.
http://openbioresources.metajnl.com
Collaboration with Ubiquity Press
Launch of an open access data journal dedicated to the publication of description of bioresources
Aim: - Increase the visibility of bioresources by offering the possibility of an open access “marker paper” , according to an established template of description- Provides a bioresourcewith a DOI
• A guideline that is not implemented is of no use!
• What mechanisms: endorsement at various levels• Institutional (Universities, national institutes,
infrastructures...)
• Scientific (Scientific consortia, scientific and professionalsocieties…)
• Administrative : Inclusion of the reference to use in MTA
• Educational : good practices taught to PhD students usingbioresources
• Editorial (instruction to authors, to reviewers; incentives to use EQUATOR’s references guidelines…)
ESOF 2016, Manchester, 24-27/7/201627/07/2016 14
Conclusion
• By providing an operational guideline to cite in a harmonised way the bioresources used or referred to in an article, CoBRA fills a gap, indispensible towards BRIF, to measure impact of such resources.
• Its implementation requires joint efforts from several stakeholders, in particular editors.
• It opens the possibility to attribute impact • To infrastructures and their use• To individuals contributing to such resources through link to ORCID or other
personal Unique Identifiers
• It empowers patients and research participants for following themselves what is done with their samples and data.
References used• Napolitani F et al. Treat the poison of invisibility with CoBRA, a systematic way of citing
bioresources in journal articles. Biopreservation and biobanking, 2016, in press
• Bravo E et al. Developing a guideline to standardize the citation of bioresources in journal articles (CoBRA). BMC Med. 2015;13(1):266.
• Bravo E et al. Citation of bioresources in biomedical journals: moving towards standardization for an impact evaluation. European Science Editing 2013;39(2): 36-38.
• De Castro P et al. Open Data Sharing in the Context of Bioresources. Acta Inform Med. 2013, 21(4): 291-292.
• Mabile et al. Quantifying the use of bioresources for promoting their sharing in scientific research. GigaScience 2013, 2:7.
• Cambon-Thomsen A et al The role of a Bioresource Research Impact Factor as an incentive to share human bioresources. Nat Genet. 2011, 43(6):503-4.
• Kauffmann F., Cambon-Thomsen A. Tracing biological collections: between books and clinical trials. JAMA, 2008;299(19): 2316-2318.
• Cambon-Thomsen A. Assessing the impact of biobanks. Nat Genet, 2003, 34, (1) 25-26.27/07/2016 ESOF 2016, Manchester, 24-27/7/2016 17
Beyond notions of ‘misuse’ in
debates on metrics
Sarah de RijckeCentre for Science and Technology StudiesLeiden University
• If more transparency is the solution, what precisely is the problem?
'Misuses' and 'unintended effects’?
Is the JIF 'misused'?
• Arguments against the JIF often cite its technical shortcomings
• "Single numbers conceal skew of distributions and variation incitations received by published papers"
• In line with principle 8 Leiden Manifesto: avoid false precision
But also: too optimistic mode of 'implementation'
Researchers as passive, relatively powerless, actors in science system
Obscures much more fundamental issue about 'constitutive effects'
9. Recognize the systemic effects of assessment and indicators10. Scrutinize indicators regularlyand update them
- selecting useful information from the overwhelming amounts of literature they could potentially read
- settling discussions over whom to collaborate with and when
- how much - additional - time to spend in the laboratory
- deciding on how much time and effort to put into article peer review
How do researchers use indicators?
Does the JIF 'mislead'?
1. Not all embedded uses are grounded in naïve assumptions
2. Who misleads whom? Researchers are active users of metrics; not simply evaluated subjects
Different embedded uses of metrics ‘trickle back up’ into assessments
“Auditors are not aliens. They are versions of ourselves.” (Strathern 1997, 319)
• Calls for researchers and evaluators to ‘drop’ the JIF in assessments are actually calls for quite fundamental transformations in howresearchers currently produce scientific knowledge
• This transformation of the epistemic process is the primary task
• When and how exactly do metrics reshape research? What is at stakein this situations? What kind of research do we want?
Key question
• Constitutive effects of assessment and indicators, affecting the kinds of knowledge researchers consider viable and interesting
• How can we let this work for instead of against us?
‘universities should be encouraged to sign-up to the San Francisco Declaration on Research Assessment.This requires that metrics are used appropriately in the evaluation of individual researchers in order to remove distortion in the market by privileging certain publication routes.’
scooping protection; one round of revision; preprint publication
dissemination
research finding
quality control
validation
Preprint servers – accelerating discovery
Journals:Peer review
Preprint: Discovery time stamp
time
relia
bili
ty
• Minimal delay to disseminating findings• Global reach• Community comments & collaborations• Time stamp: document priority
Preprints + DORA may Transform Research Assessment
Funders increasingly consider preprint posts for grants and fellowships
Research Assessment: New metrics? More metrics? No metrics?
Ismael Rafols
Ingenio (CSIC-UPV), Univ. Politècnica de València
SPRU (Science Policy Research Unit), Univ. Sussex
Re-shaping design and use of indicators• Indicators may be harming research
• Current indicators are only (partially) appropriate for some types of science.
• Biases against and potential suppression of creative and valuable types of research (agro-, health,…). Threat to diversity.
• Not only more, but other types of indicators needed• Making visible other contributions (e.g. IDR) and other types of research
(e.g. action research, co-creation)• Enhancing visualisation of metrics for “opening up” perspectives rather
than facilitating “closing down”
• Towards different uses of indicators• New embedding in assessment or policy context• Indicators used to pluralise (opening up) perspectives, as tools
for interpretation and deliberation, not a substitute for judgement
Uses of indicators: Pressing demands of research management and evaluation --- Can indicators help?
Yes, indicators can help make decisions…
Reduce time and costs
Increase transparency and sense of objectivity
Reduce complexity, accessible to managers
but do they lead to the “right” decisions?
Evaluation gap (Wouters):
“discrepancy between evaluation criteria [implicit in indicators] and the social and economic functions of science”