The dos and don'ts in individudal level bibliometrics

’

1, 2

1Centre for R&D Monitoring and Dept MSI, KU Leuven, Belgium2Centre for Science and Technology Studies, Leiden University, The Netherlands

Introduction

In the last quarter of the 20th century, bibliometrics evolved from asub-discipline of library and information science to an instrument forevaluation and benchmarking (G, 2006; W, 2013).

• As a consequence, several scientometric tools became used in a context forwhich they were not designed (e.g., JIF).

• Due to the dynamics in evaluation, the focus has shied away from macrostudies towards meso and micro studies of both actors and topics.

• More recently, the evaluation of research teams and individual scientistshas become a central issue in services based on bibliometric data.

• The rise of social networking technologies in which all types of activities aremeasured and monitored has promoted auto-evaluation with tools such asGoogle Scholar, Publish or Perish, Scholarometer.

G W, The dos and don’ts, Vienna, 2013 2/25

Introduction

In the last quarter of the 20th century, bibliometrics evolved from asub-discipline of library and information science to an instrument forevaluation and benchmarking (G, 2006; W, 2013).

• As a consequence, several scientometric tools became used in a context forwhich they were not designed (e.g., JIF).

• Due to the dynamics in evaluation, the focus has shied away from macrostudies towards meso and micro studies of both actors and topics.

• More recently, the evaluation of research teams and individual scientistshas become a central issue in services based on bibliometric data.

• The rise of social networking technologies in which all types of activities aremeasured and monitored has promoted auto-evaluation with tools such asGoogle Scholar, Publish or Perish, Scholarometer.


Introduction

There is not one typical individual-level bibliometrics since there aredifferent goals, which range from the individual assessment of proposals orthe oeuvre of applicants over intra-institutional research coordination to thecomparative evaluation of individuals and benchmarking of research teams.

As a consequence, common standards for all tasks at the individual leveldo not (yet) exist.

☞ Each respective task, the concrete field of application requires a kindof flexibility on the part of bibliometricians but also the maximum ofprecision and accuracy.

In the following we will summarise some important guidelines for the useof bibliometrics in the context of the evaluation of individual scientists,leading to ten dos and ten don’ts in individual level bibliometrics .


Introduction

There is not one typical individual-level bibliometrics since there aredifferent goals, which range from the individual assessment of proposals orthe oeuvre of applicants over intra-institutional research coordination to thecomparative evaluation of individuals and benchmarking of research teams.

As a consequence, common standards for all tasks at the individual leveldo not (yet) exist.

☞ Each respective task, the concrete field of application requires a kindof flexibility on the part of bibliometricians but also the maximum ofprecision and accuracy.

In the following we will summarise some important guidelines for the useof bibliometrics in the context of the evaluation of individual scientists,leading to ten dos and ten don’ts in individual level bibliometrics .


Ten things you must not do …

1. Don’t reduce individual performance to a single number

• Research performance is influenced by many factors such as age,time window, position, research domain. Within the same scholarlyenvironment and position, interaction with colleagues, co-operation,mobility and activity profiles might differ considerably.

• A single number (even if based on sound methods and correct data)can certainly not suffice to reflect the complexity of researchactivity, its background and its impact adequately.

• Using them to score or benchmark researchers needs to take theworking context of the researcher into consideration.



2. Don’t use IFs as measures of quality

• Once created to supplement ISI’s Science Citation Index, the IFevolved to an evaluation tool and seems to have become the“common currency of scientific quality” in research evaluationinfluencing scientists’ funding and career (S, 2004).

• However, the Impact Factor is by no means a performance measureof individual articles nor of the authors of these papers (e.g., S,1989, 1997).

• Most recently, campaigns against the use of the IF in individual-levelresearch evaluation emerged on the part of scientists (who feelvictims of evaluation) and bibliometricians themselves (e.g.,B HL, 2012; B, 2013).

◦ The San Francisco Declaration on Research Assessment (DORA) hasstarted an online campaign against the use of the IF for evaluation ofresearchers and research groups.



3. Don’t apply (hidden) “bibliometric filters” for selection

• Weights, thresholds or filters are defined for in-house evaluation orfor preselecting material for external use.Some examples:

◦ A minimum IF might be required for inclusion in official publicationlists.

◦ A minimum h-index is required for receiving a doctoral degree or forconsidering a grant application.

◦ A certain amount of citations is necessary for promotion or forpossible approval of applications.

This practice is sometimes questionable: If filters are set, they shouldalways support human judgement and not pre-empt it.

☞ Also the psychological effect of using such filters might not beunderestimated.



4. Don’t apply arbitrary weights to co-authorshipA known issue in bibliometrics is how to properly credit authors for theircontribution to papers they have co-authored.

• There is no general solution for the problem.

• Only the authors themselves can judge their own contribution.

• In some cases, pre-set weights on the basis of the sequence ofco-authors are defined and applied as strict rules.

• The sequence of co-authors as well the special “function” of thecorresponding authors do not always reflect the amount of their realcontribution.

• Most algorithms are, in practice, rather arbitrary and at this levelpossibly misleading.



5. Don’t rank scientists according to 1 indicator

• It is legitimate to rank candidates who have been short-listed, e.g.,for a job position, according to relevant criteria, but ranking shouldnot be merely based on bibliometrics.

• Internal or public ranking of research performance without anyparticular practical goal (like a candidateship) is problematic.

• There are also ethical issues and possible repercussions of theemerging “champions-league mentality” on the scientists researchand communication behaviour (e.g., G D, 2003).

• A further negative effect of ranking lists (as easily accessible andready-made data) is that those could be used in decision-making inother contexts than they have been prepared for.



6. Don’t merge incommensurable measures

• This problematic practice oen begins with output reporting by thescientists them-selves.

◦ Citation counts appearing in CVs or applications are sometimes basedon different sources (WoS, SCOPUS, Google Scholar).

• The combination of incommensurable sources combined withinappropriate reference standards make bibliometric indicatorsalmost completely useless (cf. W, 1993).

• Do not allow users to merge bibliometric results from differentsources without having checked their compatibility.



7. Don’t use flawed statistics

• Thresholds and reference standards for the assignment toperformance classes are proved tools in bibliometrics (e.g, foridentifying industrious authors, uncited or highly cited papers).

◦ This might even be more advantageous than using the originalobservations.

• However, looking at the recent literature one finds a plethora offormulas for “improved” measures or composite indicators lackingany serious mathematical background.

• Small datasets are typical of this aggregation level: This mightincrease the bias or result in serious errors and standard(mathematical) statistical methods are oen at or beyond their limithere.



8. Don’t blindly trust one-hit wonders

• Do not evaluate scientists on the basis of one top paper and do notencourage scientists to prize visibility over targeting in theirpublication strategy.

◦ Breakthroughs are oen based on a single theoretical concept or wayof viewing the world. They may be published in a paper that thenaracts star aention.

◦ However, breakthroughs may also be based on a life-long piecingtogether of evidence published in a series of moderately cited papers.

☞ Always weight the importance of highly cited papers versus thevalue of a series of sustained publishing. Don’t look at topperformance only, consider the complete life work or the researchoutput created in the time windows under study.



8. Don’t blindly trust one-hit wonders

• Do not evaluate scientists on the basis of one top paper and do notencourage scientists to prize visibility over targeting in theirpublication strategy.

◦ Breakthroughs are oen based on a single theoretical concept or wayof viewing the world. They may be published in a paper that thenaracts star aention.

◦ However, breakthroughs may also be based on a life-long piecingtogether of evidence published in a series of moderately cited papers.

☞ Always weight the importance of highly cited papers versus thevalue of a series of sustained publishing. Don’t look at topperformance only, consider the complete life work or the researchoutput created in the time windows under study.



9. Don’t compare apples and oranges

• Figures are always comparable. And contents?

• Normalisation might help make measures comparable but only likewith like.

• Research and communication in different domains is differentlystructured. The analysis of research performance in humanities,mathematics and life sciences needs different concepts andapproaches.

◦ Simply weighting publication types (monographs, articles, workingpapers, etc.) and normalising citation rates will just cover up but noteliminate differences.



10. Don’t allow deadlines and workload to compel you to dropgood practices

• Reviewers and users in research management are oen overchargedby the flood submissions, applications and proposals combined withtight deadlines and lack of personnel.

◦ Readily available data like IFs, gross citation counts and the h-indexare sometimes used to make decisions on proposals and candidates.

• Don’t give in to time pressure and heavy workload when you haveresponsible tasks in research assessment and the career of scientistsand the future of research teams are at the stake and don’t allowtight deadlines to compel you to reduce evaluation to the use of“handy” numbers.


Ten things you might do …

1. Also individual-level bibliometrics is statistics

• Basic measures (number of publications/citations) are importantmeasures in bibliometrics at the individual level.

• All statistics derived from these counts require a sufficiently largepublication output to allow valid conclusions.

• If this is met, standard bibliometric techniques can be applied butspecial caution is always called for at this level:

◦ A longer publication period might also cover different careerprogression and activity dynamics in the academic life of scientists.

◦ Assessment, external benchmarking and comparisons require the useof appropriate reference standards, notably in interdisciplinaryresearch or pluridisciplinary activities.

◦ Special aention should be paid to group authorship (groupcomposition and contribution credit assigned to the author).



2. Analyse collaboration profiles of researchers

• Bibliometricians might analyse the scientist’s position amonghis/her collaborators and co-authors. In particular, the followingquestions can be answered.

◦ Do authors preferably work alone, work in stable teams, or preferoccasional collaboration.

◦ Who are the collaborators and are the scientists rather ‘junior’, ‘peers’or ‘senior’ partners in these relationships.

• This might help recognise the scientist’s own role in his/her researchenvironment but final conclusion should be drawn in combinationwith “qualitative methods”.



3. Always combine quantitative and qualitative methodsAt this level of aggregation, the combination of bibliometrics withtraditional qualitative methods is not only important but indispensable.

• On one hand, bibliometrics can be used to supplement thesometimes subjectively coloured qualitative methods by providing“objective” figures to underpin, confirm arguments or to makeassessment more concrete.

• If discrepancies between the two methods are found try toinvestigate and understand what the possible reasons for thedifferent results could be.☞ This might even enrich and improve the assessment.



4. Use citation context analysis

The concept of “citation context” analysis was first introduced in 1973 byM and later suggested for use in Hungary (B, 2006).

• Here citation context does not cover the position, where a citation isplaced in an article, or the distance from other citations in the samedocument. It covers the textual and contentual environment of thecitation in question.

• It is to be shown that a research results is not only referred to but isused indeed in the colleagues’ research and/or is scholarlydiscussed. ⇒ The context might be positive or negative.

“Citation context” represents an approach in-between qualitative andquantitative methods and can be used in the case of individual proposalsand applications.



4. Use citation context analysis

The concept of “citation context” analysis was first introduced in 1973 byM and later suggested for use in Hungary (B, 2006).

• Here citation context does not cover the position, where a citation isplaced in an article, or the distance from other citations in the samedocument. It covers the textual and contentual environment of thecitation in question.

• It is to be shown that a research results is not only referred to but isused indeed in the colleagues’ research and/or is scholarlydiscussed. ⇒ The context might be positive or negative.

“Citation context” represents an approach in-between qualitative andquantitative methods and can be used in the case of individual proposalsand applications.



5. Analyse subject profiles

Many scientists do research in an interdisciplinary environment. Eventheir reviewers might work in different panels. The situation is evenworse for “polydisciplinary” scientists.

In principle, three basic approaches are possible.

1. Considering all activities as one total activity and “define” anadequate topic for benchmarking

2. Spliing up the profile into its components (which might, of course,overlap) for assessment

3. Neglecting activities outside the actual scope of assessment

It depends on the task, which of the above models should be applied.More research on these issues is urgently needed.



5. Analyse subject profiles

Many scientists do research in an interdisciplinary environment. Eventheir reviewers might work in different panels. The situation is evenworse for “polydisciplinary” scientists.

In principle, three basic approaches are possible.

1. Considering all activities as one total activity and “define” anadequate topic for benchmarking

2. Spliing up the profile into its components (which might, of course,overlap) for assessment

3. Neglecting activities outside the actual scope of assessment

It depends on the task, which of the above models should be applied.More research on these issues is urgently needed.



6. Make an explicit choice for oeuvre or time-window analysis

The complete oeuvre of a scientist can serve as the basis of the individualassessment. This option should rather not be used in comparativeanalysis.

• The reason is different age, profile, position and the complexity of ascientists career.

Time-window analysis is more suited for comparison, provided, of course,like is compared with like and the publication period and citationwindows conform.



6. Make an explicit choice for oeuvre or time-window analysis

The complete oeuvre of a scientist can serve as the basis of the individualassessment. This option should rather not be used in comparativeanalysis.

• The reason is different age, profile, position and the complexity of ascientists career.

Time-window analysis is more suited for comparison, provided, of course,like is compared with like and the publication period and citationwindows conform.



7. Combine bibliometrics with career analysis

This applies to the assessment on the basis of a scientist’s oeuvre.

• Bibliometrics can be used to zoom in on a scientist’s career. Here theevolution of publication activity, citation impact, mobility andchanging collaboration paerns can be monitored.

• It is not easy to quantify the observations and the purpose is not tobuild indicators for possible comparison but to use bibliometric datato visually and numerically depict important aspects of the progressof a scientist’s career.

Some preliminary results have been published by Z G (2012).



8. Clean bibliographic data carefully and use external sources

Bibliometric data at this level are extremely sensitive. This implies thatalso input data must be absolutely clean and accurate.

• In order to achieve cleanness, publication lists and CVs should beused if possible. This is important for two reasons:

◦ External sources help improve the quality of data sources.◦ Responsibility with the authors or institutes is shared.

• If the assessment is not confidential, researchers themselves mightbe involved in the bibliometric exercise.

• Otherwise, scientists might be asked to provide data according to agiven standard protocol that can and should be developed ininteraction between the user and bibliometricians.



8. Clean bibliographic data carefully and use external sources

Bibliometric data at this level are extremely sensitive. This implies thatalso input data must be absolutely clean and accurate.

• In order to achieve cleanness, publication lists and CVs should beused if possible. This is important for two reasons:

◦ External sources help improve the quality of data sources.◦ Responsibility with the authors or institutes is shared.

• If the assessment is not confidential, researchers themselves mightbe involved in the bibliometric exercise.

• Otherwise, scientists might be asked to provide data according to agiven standard protocol that can and should be developed ininteraction between the user and bibliometricians.



9. Even some “don’ts” are not taboo if properly applied

There is no reason to condemn the oen incorrectly used Impact Factorand h-index. They can provide supplementary information if they areused in combination with qualitative methods, and are not used as theonly decision criterion.

Example:

• Good practice (h-index as supporting argument):“The exceptionally high h-index of the applicant confirms his/herinternational standing aested to by our experts.”

• estionable use (h-index as decision criterion):“We are inclined to support this scientist because his/her h-indexdistinctly exceeds that of all other applicants.”



9. Even some “don’ts” are not taboo if properly applied

There is no reason to condemn the oen incorrectly used Impact Factorand h-index. They can provide supplementary information if they areused in combination with qualitative methods, and are not used as theonly decision criterion.

Example:

• Good practice (h-index as supporting argument):“The exceptionally high h-index of the applicant confirms his/herinternational standing aested to by our experts.”

• estionable use (h-index as decision criterion):“We are inclined to support this scientist because his/her h-indexdistinctly exceeds that of all other applicants.”



10. Help users to interpret and apply your results

At any level of aggregation bibliometric methods should bewell-documented. This applies above all to level of individual scientistsand research teams.

• Bibliometricians should support users in a transparent manner toguarantee replicability of bibliometric data.

• They should issue clear instructions concerning the use andinterpretation of their results.

• They should also stress the limitations of the validity of these results.


Conclusions

• The (added) value of or damage by bibliometrics in individual-levelevaluation depends on how and in what context bibliometrics isapplied.

• In most situations, the context should determine which bibliometricmethods and how those should be applied.

• Soundness and validity of methods is all the more necessary at theindividual level but not yet sufficient. Accuracy, reliability andcompleteness of sources is an absolute imperative at this level.

• We recommend to use individual level bibliometrics always on thebasis of the particular research portfolio. The best method to do thismay be the design of individual researchers profiles combiningbibliometrics with qualitative information about careers andworking contexts. The profile includes the research mission andgoals of the researcher.


Conclusions

• The (added) value of or damage by bibliometrics in individual-levelevaluation depends on how and in what context bibliometrics isapplied.

• In most situations, the context should determine which bibliometricmethods and how those should be applied.

• Soundness and validity of methods is all the more necessary at theindividual level but not yet sufficient. Accuracy, reliability andcompleteness of sources is an absolute imperative at this level.

• We recommend to use individual level bibliometrics always on thebasis of the particular research portfolio. The best method to do thismay be the design of individual researchers profiles combiningbibliometrics with qualitative information about careers andworking contexts. The profile includes the research mission andgoals of the researcher.


Acknowledgement

The authors would like to thank I R and J G fortheir contribution to the idea of a special session on this important issueas well as the organisers of the ISSI 2013 conference for having given usthe opportunity to organise this session.

We also wish to thank L W and R C for theiruseful comments.