Return On Contribution (ROC) ECSCW 2009 Muller Et Al

Return On Contribution (ROC):

A Metric for Enterprise Social Software

Michael Muller, Jill Freyne*, Casey Dugan,David R Millen, & Jennifer Thom-Santelli

IBM Research & IBM Center for Social Software

Cambridge MA USA

*Jill Freyne is now at Tasmanian ICT Center, CSIRO, Australia

1

Agenda

• How to measure the benefits of social software in

organizations?

• Return On Contribution (ROC)

• Applying ROC to

– Enterprise social software applications

– Types of resources in social software applications– Types of resources in social software applications

– Points of articulation in social software applications

– Individual users (with a few notes on privacy)

• Conclusion and Next Steps

2

Benefits of Social Software

• Informal arguments are known

– Better knowledge-sharing

– Better personal effectiveness

– Improved ability to manage one’s reputation

– In organizations, better satisfaction and retention

• There are few strong studies to support those claims• There are few strong studies to support those claims

• Management’s desire: Return On Investment

– Has been shown for niche social software applications

• Customer-support operations

• Customer communities

– Has been shown for advertising opportunities

• Social networking sites

3

Return On … What?

• Return On Investment

– ROI = Benefit / Cost � € / € (unitless economic ratio)

– We hope for ROI >> 1.0

• Social software benefits – and even costs – are

difficult to measure

– Is the purpose of social software to increase productivity?– Is the purpose of social software to increase productivity?

– How do you calculate ROI of a telephone? an IM product?

– How do you calculate ROI of a relationship?

4

Return On Contribution (ROC)

• A social ratio

– ROI = € / €

– ROI = Benefit / Cost

– ROC = Beneficiaries / Contributors (unitless social ratio)

= Consumers / Producers

• Return On Contribution• Return On Contribution

– A measure of social effectiveness – do more people benefit

(or consume) than contribute (or produce)?

– Rational Choice theory (Pirolli, 2007)

• Over time, people’s work-oriented decisions are beneficial to them

• Measure those choices and summarize them as a metric

5

• Beehive

– “Social networking

behind the firewa

– Overall usagell”

– Common goods

• Photos

ROC for Two Enterprise Services

• Dogear

– “Social bookmarking

behind the firewall”

– Overall usage

– Common goods

• Bookmarks Photos

• Lists (HiveFives)

• Events

• (Person-summaries)

Bookmarks

• Tags

• (Person-summaries)

6

• Beehive

– “Social networking

behind the firewall”

– Overall usage

– Common goods

• Photos

ROC for Two Enterprise Services

• Dogear

– “Social bookmarking

behind the firewall”

– Overall usage

– Common goods

• Bookmarks Photos

• Lists (HiveFives)

• Events

• (Person-summaries)

Bookmarks

• Tags

• (Person-summaries)

7

• Beehive

– “Social networking

behind the firewall”

– Overall usage

– Common goods

• Photos

ROC for Two Enterprise Services

• Dogear

– “Social bookmarking

behind the firewall”

– Overall usage

– Common goods

• Bookmarks Photos

• Lists (HiveFives)

• Events

• (Person-summaries)

Bookmarks

• Tags

• (Person-summaries)

8

• Beehive

ROC for Two Enterprise Services

• Dogear

9

• Beehive

ROC for Two Enterprise Services

• Dogear

10

• Beehive

– “Social networking

behind the firewall”

– Overall usage

– Common goods

• Photos

ROC for Types of Contributions

• Dogear

– “Social bookmarking

behind the firewall”

– Overall usage

– Common goods

• Bookmarks Photos

• Lists (HiveFives)

• Events

• (Person-summaries)

Bookmarks

• Tags

• (Person-summaries)

11

• Beehive

– “Social networking

behind the firewall”

– Overall usage

– Common goods

• Photos

ROC for Types of Contributions

• Dogear

– “Social bookmarking

behind the firewall”

– Overall usage

– Common goods

• Bookmarks

Monthly Social-Networking ROC for three media types

6

8

10

12

14

16

18

20

ROCC

Photo

List

Event

Data range of Figure 1

Monthly Social-Networking ROC for three media types

6

8

10

12

14

16

18

20

ROCC

Photo

List

Event

6

8

10

12

14

16

18

20

ROCC

Photo

List

Event

Photo

List

Event

Data range of Figure 1

Photos

• Lists (HiveFives)

• Events

• (Person-summaries)

Bookmarks

• Tags

• (Person-summaries)

12

0

2

4

6

Jun-07 Jul-07 Aug-07 Sep-07 Oct-07 Nov-07 Dec-07 Jan-08

Date

Data range of Figure 1

0

2

4

6

Jun-07 Jul-07 Aug-07 Sep-07 Oct-07 Nov-07 Dec-07 Jan-08

Date

0

2

4

6

Jun-07 Jul-07 Aug-07 Sep-07 Oct-07 Nov-07 Dec-07 Jan-08

Date

Data range of Figure 1

• Beehive

– “Social networking

behind the firewall”

– Overall usage

– Common goods

• Photos

ROC for Specific Contributions

• Dogear

– “Social bookmarking

behind the firewall”

– Overall usage

– Common goods

• Bookmarks Photos

• Lists (HiveFives)

• Events

• (Person-summaries)

Bookmarks

• Tags – specific tags

• (Person-summaries)

13

• Beehive

– “Social networking

behind the firewall”

– Overall usage

– Common goods

• Photos

ROC for Specific Contributions

• Dogear

– “Social bookmarking

behind the firewall”

– Overall usage

– Common goods

• Bookmarks Photos

• Lists (HiveFives)

• Events

• (Person-summaries)

Bookmarks

• Tags – specific tags

• (Person-summaries)

14

Tagging for audiences (Thom-Santelli

et al., 2008)

• Publishers

• Evangelists

• Beehive

– “Social networking

behind the firewall”

– Overall usage

– Common goods

• Photos

ROC for Specific Contributions

• Dogear

– “Social bookmarking

behind the firewall”

– Overall usage

– Common goods

• Bookmarks Photos

• Lists (HiveFives)

• Events

• (Person-summaries)

15

Bookmarks

• Tags – specific tags

– Publisher: podcast tag “Tag-City”

Tag ROC = 7.41 readers/contributor – Personal ROC = 63.00

– Evangelist: tag “web2.0”

Tag ROC = 1.95 readers/contributor – Personal ROC = 1245.00

• (Person-summaries)

• Beehive

– “Social networking

behind the firewall”

– Overall usage

– Common goods

• Photos

ROC for Specific Contributions

• Dogear

– “Social bookmarking

behind the firewall”

– Overall usage

– Common goods

• Bookmarks Photos

• Lists (HiveFives)

• Events

• (Person-summaries)

16

Bookmarks

• Tags – specific tags

– Publisher: podcast tag “Tag-City”

Tag ROC = 7.41 readers/contributor – Personal ROC = 63.00

– Evangelist: tag “web2.0”

Tag ROC = 1.95 readers/contributor – Personal ROC = 1245.00

• (Person-summaries)

ROC for Other Social Applications

Service Beneficiaries Contributors ROC C

Dogear 10896 4213 2.59

Beehive 21453 8397 2.55

Wiki server 238838 36377 6.57

Discussion server 150000 23000 6.52

Person-tagging 20973 3102 6.76

File-sharing 68762 11276 6.19

17

Implications for Design or Potential Use

• Track the development of organizational value of an application

over time

– Does it increase? Does it stabilize?

• Compare the organizational value of different types of

contributions over time

• Compare the organizational value of specific contribution

instancesinstances

• Assist the development of individual contributors, especially in

assigned roles such as “evangelist” or “publisher”, by providing

private views of her/his personal ROC

• Monitor, on an anonymous basis, the development of social

capital through aggregate, summary ROC measures across all

beneficiaries and contributors

18

Unanswered Questions about ROC

• Are there characteristic “signature” ROC values for

different types of applications?

• How to determine “stabilization” of ROC over time?

• What should the “target” ROC be for a discussion

forum?

– ROC >> 6.0 for some applications looked very nice– ROC >> 6.0 for some applications looked very nice

– However, ROC=1.0 suggests full democratic participation

– When are different values of ROC desirable?

• What should the “target” ROC be for a type of object,

or a particular object (e.g., a tag)?

• Can ROC help to show the value of “lurkers”? When is

it permissible (under privacy rules) to study “lurking”?19

Summary of Contributions

• Lurkers as non-public participants (Nonnecke and

Preece, 2001) and as altruists (Takahashi et al., 2003)

– Employees in some jobs are “paid to lurk”

– Lurkers’ “consumption” of shared objects is a test of the

organizational value of those objects

• ROC provides a “social value” metric for• ROC provides a “social value” metric for

– Social software applications (Dogear, Beehive)

– Types of contributions (Photos, Lists, Events in Beehive)

– Specific contributions (Tags in Dogear)

– (where permitted) specific contributors (Taggers in Dogear)

• ROC can help organizations and researchers to assess

and study the value of social media20

Thank you!

Slides may be found on slideshare.net

michael_muller@us.ibm.com

21

• Beehive

ROC for Two Enterprise Services

• Dogear

22

• Beehive

ROC for Two Enterprise Services

• Dogear

Two ways to think about benefit

• All users (including contributors)

ROC C = AllUsers/Contributors

• Lurkers only

ROC = Lurkers/Contributors

23

ROC L = Lurkers/Contributors

• Beehive

ROC C and ROC L

• Dogear

Two ways to think about benefit

• All users (including contributors)

ROC C = AllUsers/Contributors

• Lurkers only

ROC = Lurkers/Contributors

24

ROC L = Lurkers/Contributors

• Beehive

ROC for Two Enterprise Services

• Dogear

25