Open Analytics: Building Effective Frameworks for Social Media Analysis

Building Effective Frameworks for Social Media Analysis

Open Analytics NYC – 11/08/2012

Agenda

• Social Media: An Intelligence perspective• Common Analytic Pitfalls• An Analytic Framework• Case Study: Brand Management– Problem Definition– Source Selection– Data Capture– Data Reporting– Data Analysis

• Ways Forward, Future Analysis• Questions?

Intelligence

• Intelligence is information that has been transformed to meet an operational need

Data Intelligence

Operational Lens

Intelligence Cycle

• No matter what methodology you use…

intelligence analysis is an iterative process.

Collect

Store

Analyze

Distribute

Social Media: Intelligence Perspective

HUMINT

SIGINTOSINT

• Social Media Intelligence is a combination of the best and worst features of:– HUMINT– OSINT– SIGINT

Social Media Analysis Goals

• Provide value to the organization – turn data into intelligence using an “operational lens”

• Ensure cyclical feedback occurs during collection, processing, analysis, and consumption

• Validate that a particular network is the right source of data for the questions you need answered

Common Misconceptions

• Social media is not a panacea– Not everyone uses social media– Users of social media use it unevenly– User behavior changes based on situations

• Just because people can talk about anything does not mean they talk about everything all the time.

Common Pitfalls

• Analyzing What Instead of Why:The important thing is often not what people are saying… but why they are saying it.

• Using the Wrong Analysis Tools:Reporting tools rarely help dig into the why. Many common tools, reports, and metrics are actually misleading:– Word clouds atomize message context– Sentiment metrics are often highly inaccurate– Information in aggregate hides more than it reveals

Pitfalls: An Example of the Challenge

Pitfalls: An Example of the Challenge

Dangers of Disintegration

Source: Matthew Auer, Policy Studies Journal, Volume 39, Issue 4, pages 709–736, Nov 2011

Analytic Framework

• Data Capture (DC)• Data Reporting (DR)• Data Analysis (DA)– What to measure– What the data is saying– What should be done based on the data

Source: Avinash Kaushik, Occam’s Razor Blog http://www.kaushik.net/avinash/web-analytics-consulting-framework-smarter-decisions/

Capture

ReportAnalyze

Choosing a Platform

• Social media, and the ways that it is used, is relatively new and evolving rapidly:– Static approaches to social media are flawed from

the outset– No one metric or set of metrics will always let you

know what is happening• Platforms need to be open and highly

adaptable to facilitate data capture, reporting, and analysis

Case Study: Brand Management

• Industry: Gaming– Experiencing 10% growth annually– Overall revenue expected to exceed $80 billion by

2014• In May, Zenimax Online Studios announced

Elder Scrolls Online– Elder Scrolls V: Skyrim 2nd largest game of 2011

Problem Definition

• Question: How can brand managers use social media to track and understand public attitudes toward a product?

• Challenge: Capture relevant information for social media sources.– Query too large = false positives– Query too small = miss potential information

Twitter

• Twitter has excellent analytical potential:– Enormous volume, 400 million+ tweets per day– Large user base, 140 million+ accounts– Open API

• But its not without its limitations:– 140 characters– Limited historical (lookback) capacity without

using a 3rd party provider like DataSift or GNIP

Data Capture: Initial Query

• Twitter search for “Elder Scrolls Online”– Simplest possible way to access information– RSS feed for 10 days (Jun 27 – July 6 2012)

Data Capture: Entities & Associations

Hashtag TwitterHandle URL

Unstructured Keywords

Time / Date Stamp

WhoTwitterHandle

WhatHashtags, Keywords, URLs

WhenTime, Date

WhereGeo (if Available)

Data Reporting

Data Reporting

Data Analysis

• Analysis needs to be rooted in the operational need:

“How can I use social media to track and understand public attitudes toward my product”

• Emphasis on hypothesis generation, testing, and experimentation

Data Analysis: Hashtags

• Top hashtags were almost all generic or abstract– Undermines tracking and understanding– Top hashtags tied to franchise, not to the game

Hashtags#ElderScrolls #concept#games #nerd#online #geek#MMO #gamer#skyrim #ScreenShot

Data Analysis: Expanding the Query

• Hash tags from an initial subset of Tweets fed back into the initial query

Twitter Stream

Initial Query Results

Expanded Query Results

Data Analysis: Sentiment

• Sentiment analysis on small snippets of text like Tweets is generally poor

• Follow and convert linked URLs into derivative sources

• Larger text sources offer potential value with sentiment analysis that tweets alone cannot offer

Data Analysis: Sentiment

• Top negative and positive sentiment scores can provide a glimpse into aggregate attitudes

• Provide starting points for additional analysis

Next Steps: Shape the Conversation

• Create and promote hashtags that help shape the conversation and make it easier to collect and analyze the Twitter stream

Next Steps: Segment the Data

• Segment, or cluster, your data by:– User name or handle– Hashtags– Keywords– Geographic region

to explore patterns and trends at the micro level versus the entire dataset

Next Steps: Segment the Data

Next Steps: Graph Analysis

Lessons Learned

• Don’t:– Try drinking from a fire hose, sometimes less

really is more;– Use metrics you can’t tie to actions;– Use visualizations or reports that strip the data

from its context.

Lessons Learned

• Do:– Segment data rather than attempting to work in

the aggregate;– Look for the why behind the message;– Always return to the source material;– Explore alternative explanations;– Always consider the ultimate goal.

Thank You!

Craig Vitter

[email protected]

github.com/ikanow/Infinit.e