The Infrastructure for Alternative Metrics

The Infrastructure for Alternative Metrics

What do we need to compare digital apples to digital apples?

!Todd Carpenter, Executive Director, NISO

AAAS Annual Meeting Chicago, IL | February 16, 2014

We hate discussing infrastructure

Photo: Minneapolis Collegeof Art and Design Library

• Non-profit industry trade association accredited by ANSI with 190+ members

• Mission of developing and maintaining technical standards related to information, documentation, discovery and distribution of published materials and media

• Volunteer driven organization: 400+ spread out across the world

• Responsible in various wasy for standards like ISSN, DOI, Dublin Core metadata, DAISY digital talking books, OpenURL, MARC records, and ISBN

About

Photo by dumbledad on Flickr http://www.flickr.com/photos/dumbledad/298650884/!

But what is the “infrastructure” for

assessment?

We stand on the shoulders of giants...

The Mighty Citation

Impact factor

Thomson-Reuters IMPACT FACTOR

Two Stanford Students

Two Stanford Students

They wrote a paper

They started a company in someone’s garage

Photo: AP

Citations are awesome, aren’t they?

A perfect solution to all the world’s problems, no?

A simple conclusion can be drawn

Back to Sir Isaac Newton“What Des-Cartes did

was a good step. You have added much several ways, & especially in

taking ye colours of thin plates into philosophical consideration. If I have seen further it is by

standing on ye shoulders of Giants."

Citations aren’t always

what we think they

are

Caveat emptor

Another problem with citations

So where to from here?

Define altmetrics, please

Social Media Metrics

Image: Danny Brown, The State of Social Media Marketing 2012

Usage Based Metrics

Click Map Analysis

Clickstream data yields high-resolution maps of science

Johan Bollen, Herbert Van de Sompel, et al. PLoS One, February 2009

Network Analysis

Clickstream data yields high-resolution maps of science

Johan Bollen, Herbert Van de Sompel, et al. PLoS One, February 2009

Network Analysis

Behavioral Metrics

Sentiment Analysis

Image: Domenico, Caron, Davis, et al.

Non-traditional content

Assessing impact of new forms of communication isn’t as simple as one might think.

Image: Domenico, Caron, Davis, et al.

Non-traditional content

Assessing impact of new forms of communication isn’t as simple as one might think.

We’ve got what, now how?

Image: Flickr user karindalziel

What can be measured?

How do we measure things?

At what granularity do we measure things?

Consistency in what is counted

Source:  Scott  Chamberlain,  Consuming  Article-­‐Level  Metrics: Observations  And  Lessons  From  Comparing  Aggregator  Provider  Data,  Information  Standards  Quarterly,  Summer  2013,  Vol  25,  Issue  2.

Which measurements are valuable?

How do we ensure the validity of those measurements?

How to (or even if we should) deal with other issues

such as gaming?

What infrastructure do we need for

alternative metrics?

Basic Definitions !

(So we are all talking about the same thing)

Comparison across providers

Source:  Scott  Chamberlain,  Consuming  Article-­‐Level  Metrics: Observations  And  Lessons  From  Comparing  Aggregator  Provider  Data,  Information  Standards  Quarterly,  Summer  2013,  Vol  25,  Issue  2.

Authorship -----

Disambiguation &

Contributor Roles

Element Identification

Open exchange of component data

TRUST

=

Where does NISO fit in?

Alterna(ve  Assessment  Ini(a(ve  

!

Phase  1  Mee(ngs October  9,  2013    -­‐  San  Francisco,  CA December  11,  2013  -­‐  Washington,  DC

January  23-­‐24  -­‐  Philadelphia,  PA !

Phase  1  report  expected  in  May  2014

Just  a  few  ideas  from  in-­‐person  mee(ngs  Disambigua)on  problems  

Create  a  central  repository  for  data  Define  provenance  for  metrics  

Foster  culture  of  DOI  use  in  mass  media  Establish  defini)ons  for  metric  types  Define  data  extract/gathering  methods  

Standardize  units  of  measure  Define  levels  of  transparency/privacy  

!

!

Alterna(ve  Assessment  Ini(a(ve   !

Phase  2 Presenta(ons  of  report  (June  2014) Priori(za(on  Effort  (June  -­‐  Aug,  2014)

Project  approval  (Sept  2014) Working  group  forma(on  (Oct  2014)

Consensus  Development  (Nov  2014  -­‐  Dec  2015) Trial  Use  Period  (Dec  15  -­‐  Mar  16)

Publica(on  of  final  recommenda(ons  (Jun  16)

May 15, 2013 74

Thank you! Todd Carpenter, Executive Director

tcarpenter@niso.org @TAC_NISO

!National Information Standards Organization (NISO)

3600 Clipper Mill Road, Suite 302 Baltimore, MD 21211 USA

+1 (301) 654-2512 www.niso.org