SharePoint "Moneyball" - The Art and Science of Winning the SharePoint Metrics Game

SharePoint Moneyball – The Art of Winning the SharePoint Metrics Game

Susan Hanley – President, Susan Hanley LLC

2

Now that’s just like an engineer.

We’re looking for HEIGHT …

… and she gives us LENGTH!

3

What have YOU done for

me lately??

4

5

Why Measure? – The Four “F” Words

Feedback

Funding

Follow-on

Focus

Measurement throughout the life-cycle

Before

Make the business case

During

Provide a target

Make tradeoffs

Tune the implementation process

After

Develop benchmarks

Develop lessons learned

6

7

Make your case for the solution SharePoint enables

8

Measurement Process

3. Who are the metrics

STAKEHOLDERS?

6. What do the metrics TELL us about what we

need to CHANGE?

1. What are the BUSINESS

OBJECTIVES?

2. How should the solution be

DESIGNED to meet the objectives?

5. How can we COLLECT the

metrics?

4. What are the METRICS and how

should we PRESENT them?

Aid decision making

Modify the

process or tool

Modify the

measures

9

1. What are the BUSINESS OBJECTIVES?

Without a critical business initiative …

… career limiting move

…“Career limiting move”

10

Be the main event

11

It’s easy to go for the “motherhood” objectives …

More innovative products and services

More effective marketing

Better access to knowledge

Lower cost of doing business – reduction in travel and other operational costs

Higher revenues

Improved employee, customer, and partner satisfaction

12

It’s better to Get SMART!

Measurable (quantifiable, comparable)

Achievable (feasible, actionable)

Realistic (consider resources)

Time-bound (deadline driven)

Specific (concrete and well-defined)

13

SMART objective for a proposal library

Reduce the average amount of time it takes to produce complex proposals by 10% in the next year

Specific

Measurable

Time-bound

Achievable

Realistic

Reduce the average amount of time it takes to produce complex proposals by 10% in the next year

Reduce the average amount of time it takes to produce complex proposals by 10% in the next year

14

2. How should the solution be DESIGNED to meet these objectives?

Site Architectur

e

Technical Infrastructure Features

Customization Security

Governance Roles and Responsibilities

Training and Communications

15

Your business case is personal

16

3. Who are the metrics STAKEHOLDERS?

They’re at all levels - especially in the middle

They care about different things

17

For each key stakeholder, ask …

What counts?

What keeps you up at night?

What do you already use?

What do I need to tell you?

Focus on the outcomes, then work backwards to figure out

how you will measure that outcome

18

4. What are the METRICS and how should we PRESENT them?

Identify the type

• Quantitative• Qualitative

Consider the life-cycle

Establish a baseline

Gain commitment

about targets

Decide the best way to

communicate

19

Good metrics come in multiple types … plan on both

QuantitativePerformance between pointsSpot trends

QualitativeProvide contextUsed when numbers aren’t easy (storytelling)Used at early project stages (future scenarios)Richer (“serious anecdotes”)

20

Return on Investment

Benefit > Cost

Be careful: whoever controls the spreadsheet and the assumptions can make an ROI that can justify anything.

Resources: Total Economic Impact™ of Microsoft SharePoint Server 2010 (Forrester) http://bit.ly/cWfeyN

Best for platform investment, less helpful for individual solutionsCloud vs. On Premise Calculator (Andrew McAfee and Google Analytics) http://bit.ly/R6jlsZ (for small to medium businesses)

21

ROI is only PART of the story

Good metrics are:

Related to outcome

Relevant to stakeholders

Collected at low cost

Balanced

22

Consider two types of quantitative metrics

BUSINESS METRICS

SYSTEM METRICS

23

Sample Business Metrics

Hours per week to execute a process

Number of Proposals/Contracts per year

Number of “[My Organization]-All” emails

Number of email attachments

Call center or support call deflection

Average application training costs

Cost savings to retire a legacy application

24

Business Metrics Example: “Support Call Deflection” +Business

Goal• Increase member satisfaction with HQ

by providing better access to helpful resources any time/any place

• Reduce the number of support calls to be able to serve increasing numbers of members with same staff

• Reduce cost of “authorship” by HQ due to “crowd-sourced” content

Approach• Number of member-generated resources• Number of member-generated resources

with > x “likes”• Number of posts and comments in forums• Number of downloads of HQ and member

content

System Metrics

• Membership survey to seek out specific re-use cases, membership satisfaction

• Number of support request calls to HQ• Reduction in simple requests• Potential increased % of complex requests

Business Metrics

25

Business Metrics Example: Process Improvement

Approach

Business Goal

• Allocate limited SharePoint Resources for Process Improvement Projects

• SMART Objective Example – Reduce the amount of time for a task by x% in 90 days

MeasureBaseline Target

x x x

T = Time on task (in minutes)

E = Number of employees performing that task

N = Number of times per year task is performed

S = Average employee loaded salary cost per minute

26

Qualitative Metrics – the stories that drive it home

Keep it real

In the storyteller’s

words

Serious Anecdotes

27

Serious Anecdote | Consulting

I joined the organization on March 16 without previous experience. After one week of training, I joined a project team.

After one day of training on the project, I was assigned a task to learn a particular technology that was new to everyone on the team. I was given a bunch of books and told that I had three days to learn how to create a project using this technology.

In my first week of training, I learned about the company’s intranet where people described their expertise. I sent an email to four people I found with a search for that technology asking for their help. One of them sent me a link to a document containing exactly what I needed. Instead of three days, my task

was completed in 4 hours.

28

Serious Anecdote | Pharma – The Need

A scientist with Thrombotic & Joint Diseases in Germany began a project to isolate and culture macrophages and needed some help.

Meanwhile, two scientists in the US had deep experience in protocols for this area.

29

Serious Anecdote | Pharma – The Result

Benefit: The German scientist was able to leverage existing internal expertise and, in the process, reduce his research effort by four weeks.

Both scientists quickly responded with assistance. One helped him with culturing protocols and the other helped him with information on magnetic cell sorting.

The German scientist consulted the expertise directory to find that expertise existed within the company and contacted the two US scientists he found in his search.

30

How do you spell success? Have a Baseline and Target

31

Presenting Metrics

Balanced Scorecard

Dashboard

“Report Card”

32

Balanced Scorecard Dimensions

Capabilities

Business Value

Health

33

Balanced Scorecard Example | Expertise Location

Business Value:

Health:

Capabilities & Culture:

Metric Target Pilot Outcome

# searches/user/week .25 .58

Usefulness rating 3.5 out of 5 3.6 out of 5

% of users who say “Don’t take it away”

66% 83%

Usability/friendliness rating 3.5 out of 5 4.1 out of 5

# Anecdotes (repeat metric) 10 serious 22

% of participants attending training 75% 85%

# of Anecdotes 10 serious 10 serious + 12 transactional

Minimum $ value/anecdote $X $2X

Estimated time saved X months X + 3 months

34

5. How can we collect the metrics?

Try not to over-achieve – balance counting with “doing”

Automate where possible

Get creative when it comes to qualitative metrics

AskSurveyUsability TestingActive ListeningSeekSend out a “journalist”

TrackClassify by typeKeep storyteller value metrics – what was the benefit to you?

35

Example Survey Questions

If given the choice, would you KEEP it?

How does this COMPARE?

How EASY was it to …?

Don’t Take It Away

“User-Friendliness” Rating

“Intuitiveness” Rating

36

6. What do the metrics tell us about how we need to change?

Are we doing the right thing?

What areas are most successful?

What areas should we be promoting?

In which areas should we be investing?

Which initiatives should we discontinue?

37

Keep in mind

Metrics alone won’t make your program successful

You need someone whose job it is to

monitor them

You need someone who is accountable for making changes based on analysis

It’s as important to have a plan for acting on metrics as it is to have a plan for collecting

them!

38

Call to Action

Develop a plan to capture quantitative and qualitative

metrics.

Make sure metrics are part of someone’s

job.

Identify baseline measures – and gain commitment on targets – before you start!!

Develop a library or list to capture and

categorize qualitative metrics.

Develop an approach to produce and promote

metrics.

About Me

• Governance• User Adoption• Metrics• Information Architecture• Knowledge Management• Portals• Collaboration Solutions

• President, Susan Hanley LLC• Led national Portals, Management

Collaboration, and Content practice for Dell• Director of Knowledge Management at

American Management Systems

susanhanley

[email protected]

39

August 2013!

Susan HanleySusan Hanley LLC

[email protected]

301 469 0770 (o)

301 442 0127 (m)

@susanhanley

www.susanhanley.comhttp://www.networkworld.com/community/sharepoint

Thank you for your attention!

This presentation will be available on the Toronto

SharePoint Summit web site a few days after the event.

Please rate this session!Fill out the survey and get a chance to win a Surface

Extras

43

44

White Paper

For a white paper that explains the concepts in this presentation in more detail – with lots more examples, please go to http://www.susanhanley.com.

45

Examples of Intranet System Measures

Key Measure Objective Metric

Which features of the intranet are most important?

Knowing which pages are most used can help to prioritize which pages should be improved or developed.

You can also see which business units are the biggest intranet users and which business unit’s content is used the most.

Page Hits “Dwell” Time (Time on

Page/Site)

Which features are not being used?

If certain pages have low usage numbers, it is an indication that either the page is not very popular—and therefore should be a lower priority to develop—or that people are just not aware of its existence (which might be a communications or "promotion" problem).

Page Hits Document Downloads

Is the site navigation effective?

A high number of hits on a page that is not easily accessible from the main page indicate that the popular page should be moved up in the hierarchy.

Search results with no hits present opportunities to both promote content and add best bets.

Page Hits on pages deep in the hierarchy

Which team sites should be archived or deleted?

Sites that have not been accessed in the past 12 months might be candidates for archival or deletion if the content is no longer useful.

Page Hits

What are the peak/low usage times?

Monitoring usage trends helps identify patterns or problems and potentially alerts the Exchange Business Owner and Portal Administrator of potential user or performance issues – ideally, before they become a problem.

Usage by time

How is usage trending?

Trending reports are available for a limited period of time within SharePoint 2010. Third-party tools are required to do multi-year detailed trend analysis.

Number of users and number of unique users over time

46

Sample System Metrics (“out of the box” SharePoint 2010)

Metric Objective

Number of Unique Users (month to month)

• Provides a proxy for adoption, which is a loose proxy for value.

Most Viewed Pages/Sites

• Provides a proxy for the most valuable content.• Sites not being used help identify content that might either need to be promoted or deleted.

Top Queries (search)

• Identifies “trending “ content.• Top queries can also provide insights about what content should be promoted to the home page.

Failed Queries / No Results Queries

• Identifies candidates for best bets or synonyms and identifies emerging business terms or concepts.

Best Bet Suggestion Report

• Helps the business owner improve user outcomes by identifying URLs as most likely results for search queries.

Best Bet Usage • Helps identify which best bets are adding value and as an input to determine new best bets or best bets that need updating.

47

Additional Useful System Metrics (third-party for 2010)

Metric Objective

Most Viewed Documents Provides a proxy for the most valuable content.

Document Contribution/Editing Analysis

Provides a way to measure sustained adoption from the perspective of employee engagement.

Team Site Summary Information Total Number of Team Sites Viewed in Past 30 days Modified in Past 30 days Sites with no access in past

12 months Trend of the number of team

sites created

Provides a way to understand which sites are actively being used to monitor the health of the collaborative team sites.

Can be used to identify which sites are no longer being used and might be able to be deleted or archived.

Provides a proxy for whether or not team sites are adding value.

My Site Summary and Trends Total number of My Sites Viewed in past 30 days Modified in past 30 days Average size

Identifies adoption of people-to-people collaboration features.

Proxy for employee engagement.

48

System Metrics in SharePoint 2013 Online

49

One sample metric for each stage (more in White Paper)

Life-cycle Stage

Example Quantitative Metric

Example Qualitative Metric

Sources

Planning • Time to perform current process

• "Day in the life" future stories

• Work measurement studies

• Interviews of key stakeholders

Start up • N/A • Immediate term “day in the life” stories

• Employee surveys

Pilot Conclusion

• Same metrics you used for baselines

• Usage anecdotes –specific examples from pilot

• Follow up work measurement studies

• Surveys and follow up interviews

Ongoing • Additional metrics relevant to the business problem available with new process

• Usage anecdotes with a “serious” punch line that you collect and catalogue on an ongoing basis

• New solution system metrics

• Employee surveys and follow up interviews

50

Objectives Critical Success Factors Source Sample Metrics

Gain frequent and sustained adoption of solution

• High volume of needs that can’t be met through existing channels

• Positive impact on existing workload or work processes

• System metrics• User Surveys

• # of searches per week• # of average users per week• # unique users per week• # of “hits” on key pages/sites• “Usefulness rating” from user surveys• % of users who say “don’t take it away” at the

end of the pilot

Provide reliable, easy-to-use technology that can be incorporated into work processes

• Solution user-friendliness and intuitiveness

• Solution reliability• Integration of the solution with work

processes and existing tools

• System metrics• User Surveys• Direct measurement

• “Usefulness rating” from user surveys• # of searches per week• # of average users per week• # unique users per week• # of “hits” on key pages/sites• # Help Desk calls/week

Ensure users understand objectives and how to leverage the solution

• User training• Effective help resources• Persistent, clear communications• Active, sustained management support• Incorporation of collaboration into

performance objectives and evaluations

• System metrics• User Surveys• Direct measurement

• % of users trained• % of pilot milestones achieved• # of communications events/activities

Demonstrate clear value with respect to the business strategy

• Tangible, quantifiable examples of reductions in process cycle time

• “Serious” Anecdotes collected via surveys

• Estimates and/or direct measurement of cycle time

• # of anecdotes• $ value of anecdotes• Cycle time improvement (in hours)

Business Value

Solution Health

Capabilities

Balanced Scorecard Framework

51

Survey Example: Post-Pilot Usability

Usability Question Metric

If presented the choice, do you want to keep the solution?

“Don’t Take it Away”

Don't take it away

Take it away 

Usability/friendliness - how does the usability of this solution compare to other solutions you use on a regular basis?

“User Friendliness Rating”

Much easier to use

Easier to use About the

same

Harder to use Much harder to use 

How easy and intuitive was the solution to use for each of the following [specific task]?

“Intuitiveness Rating”

Very easy Easy Moderate

Difficult Very Difficult 

52

Other Resources

How to Measure Anything: Finding the Value of Intangibles in Business by Douglas Hubbard

Jakob Nielen’s Alert Box - Current Issues in Web Usability: http://www.useit.com/alertbox/

Determining the Value of Social Business ROI: Myths, Facts, and Potentially High Returns: http://www.idc.com/getdoc.jsp?containerId=225497

Social Software for Business Performance: The missing link in social software: Measureable business process performance: http://www.deloitte.com/assets/Dcom-UnitedStates/Local%20Assets/Documents/TMT_us_tmt/us_tmt_socialsoftwareexecsummary_021411.pdf

SharePoint Lifecycle Management Solution with Project Server 2010: http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=17058