Alternative assessment metrics initiateve meeting 1 opening

Do We Need Standards for

Alternative Metrics?

What are we hoping to accomplish?

Todd Carpenter, Executive Director, NISONISO Alternative Assessment Initiative

Discussion Forum #1 San Francisco, CA | October 9, 2013

Friday, October 11, 13

• Non-profit industry trade association accredited by ANSI with 150+ members

• Mission of developing and maintaining technical standards related to information, documentation, discovery and distribution of published materials and media

• Volunteer driven organization: 400+ spread out across the world

• Responsible in various wasy for standards like ISSN, DOI, Dublin Core metadata, DAISY digital talking books, OpenURL, MARC records, and ISBN

About

Friday, October 11, 13

We stand on the shoulders of giants...

Friday, October 11, 13

The Mighty Citation

Friday, October 11, 13

Impact factor

Friday, October 11, 13

Thomson-Reuters IMPACT FACTOR

Friday, October 11, 13

Two Stanford Students

Friday, October 11, 13

They wrote a paper

Friday, October 11, 13

They started a company in someone’s garage

Photo: APFriday, October 11, 13

Friday, October 11, 13

Citations are awesome, aren’t they?

A perfect solution to all the world’s problems, no?

A simple conclusion can be drawn

Friday, October 11, 13

“What Des-Cartes did was a good step. You have added much several ways, & especially in taking ye colours of thin plates into philosophical consideration. If I have seen further it is by standing on ye shoulders of Giants."

Back to Sir Isaac Newton

Friday, October 11, 13

Citations aren’t always what we think they are

Caveat emptor

Friday, October 11, 13

Another problem with citations

Friday, October 11, 13

So where to from here?Friday, October 11, 13

Define altmetrics, please

Friday, October 11, 13

Usage Based Metrics

Friday, October 11, 13

Clickstream Analysis

Friday, October 11, 13

Social Media Metrics

Image: Danny Brown, The State of Social Media Marketing 2012

Friday, October 11, 13

Clickstream data yields high-resolution maps of science

Johan Bollen, Herbert Van de Sompel, et al. PLoS One, February 2009

Clickstream network analysis

Friday, October 11, 13

Clickstream data yields high-resolution maps of science

Johan Bollen, Herbert Van de Sompel, et al. PLoS One, February 2009

Clickstream network analysis

Friday, October 11, 13

Behavioral Metrics

Friday, October 11, 13

Multivariate analysis

Friday, October 11, 13

Sentiment Analysis

Friday, October 11, 13

Image: Domenico, Caron, Davis, et al.

Citation of non-traditional content

Citations to new forms of communication aren’t as simple as one might think.

Define a use of this?

Friday, October 11, 13

We’ve got what, now how?

Friday, October 11, 13

Image: Flickr user karindalziel

What can be measured?

Friday, October 11, 13

How do we measure things?

Friday, October 11, 13

At what granularity do we measure things?

Friday, October 11, 13

Which measurements are valuable?

Friday, October 11, 13

How do we ensure the validity of those measurements?

Friday, October 11, 13

How to (or even if we should) deal with other issues

such as gaming?

Friday, October 11, 13

Issues we face with alternative

metrics

Friday, October 11, 13

Basic Definitions

(So we are all talking about the same thing)

Friday, October 11, 13

Comparison across providers

Source:  Scott  Chamberlain,  Consuming  Article-­‐Level  Metrics:Observations  And  Lessons  From  Comparing  Aggregator  Provider  Data,  Information  Standards  Quarterly,  Summer  2013,  Vol  25,  Issue  2.

Friday, October 11, 13

Consistency in what is counted

Source:  Scott  Chamberlain,  Consuming  Article-­‐Level  Metrics:Observations  And  Lessons  From  Comparing  Aggregator  Provider  Data,  Information  Standards  Quarterly,  Summer  2013,  Vol  25,  Issue  2.

Friday, October 11, 13

Authorship-----

Disambiguation&

Contributor Roles

Friday, October 11, 13

Open exchange of component data

Friday, October 11, 13

TRUST

Friday, October 11, 13

What is NISO trying to do?Friday, October 11, 13

Friday, October 11, 13

Friday, October 11, 13

Alterna(ve  Assessment  Ini(a(ve  

Phase  1  Mee(ngsOctober  9,  2013    -­‐  San  Francisco,  CADecember  11,  2013  -­‐  Washington,  DC

January  23-­‐24  -­‐  Philadelphia,  PA

Phase  1  report  expected  in  May  2014

Friday, October 11, 13

Alterna(ve  Assessment  Ini(a(ve  Phase  2

Presenta(ons  of  report  (June  2014)Priori(za(on  Effort  (June  -­‐  Aug,  2014)

Project  approval  (Sept  2014)Working  group  forma(on  (Oct  2014)

Consensus  Development  (Nov  2014  -­‐  Dec  2015)Trial  Use  Period  (Dec  15  -­‐  Mar  16)

Publica(on  of  final  recommenda(ons  (Jun  16) Friday, October 11, 13

Criticism

Friday, October 11, 13

Why now?Isn’t it too soon?

Can NISO to do this?

Friday, October 11, 13

How soon is now?

Friday, October 11, 13

These things take time

Friday, October 11, 13

It will be hard to changeFriday, October 11, 13

Friday, October 11, 13

“All I see now is blonde, brunette,

redhead”

Friday, October 11, 13

One Very Last Thing…

Image from Will Lion: http://www.flickr.com/photos/will-lion/2646213692/

Friday, October 11, 13

Thank you!

Todd Carpenter, Executive [email protected]

National Information Standards Organization (NISO)3600 Clipper Mill Road, Suite 302

Baltimore, MD 21211 USA+1 (301) 654-2512

www.niso.org

Friday, October 11, 13