People as sensors - mining social media for meaningful information

People as Sensors

Tom Raftery

ThingMonk,

London Dec 2013.

1

2

• Lead analyst, energy and sustainability practice, RedMonk

• GreenMonk.net

• twitter.com/tomraftery

• [email protected]

• +34 677 695 468

• SlideShare.net/TomRaftery

Tom Raftery

3

Mobile data

http://www.flickr.com/photos/traftery/8551389911/

Mobile phones with GPS, accelerometers, compass, M7, barometers, moisture, etc

4

Mobile data

http://www.zeit.de/datenschutz/malte-spitz-data-retention

Mobile phones with GPS and data retention lawsGreen party sued Deutsche Telecom for dataPut it onlineSource of concern

5

Social Media

Mobile is not the only data we’re broadcasting

6

Facebook

http://newsroom.fb.com/Key-Facts

1.19 billion monthly active users (MAU) as of September 30, 2013

727 million daily active users (DAU) in Sept 2013

874 million mobile MAUs as of September 30 2013

7

Facebook

http://techcrunch.com/2013/05/17/facebook-growth/

Most recent usage data is from May 2013 - so well out-of-date

4.75 billion content items shared per day (status updates + wall posts + photos + videos + comments)

4.5 billion Likes per day

8

Twitter

http://hide.dyndns.info/tweetcounter/index-en.cgi

Between 4,000-8,500 tweets per second per dayAvg over 6,000 tweets per sec, all stored in perpetuityTweets drop off at midnight ET, start picking up again at 6am ET!

9

Twitter

http://hide.dyndns.info/tweetcounter/index-en.cgi

Increasing tweets per sec over the last year (max value per day used)Two peaks on the right-hand-side

10

Twitter

http://hide.dyndns.info/tweetcounter/index-en.cgi

Increasing tweets per day over the last year

11

Twitter

https://blog.twitter.com/2013/new-tweets-per-second-record-and-how

Most recent numbers from Twitter:Twitter avg - >500m TPDTwitter avg - 5,700 TPSAug 2nd had a 143k TPS record (>28x the average) - no blip to service

12

Data for sale…

http://online.wsj.com/news/articles/SB10001424052702304441404579118531954483974

Twitter in its IPO filings disclosed it is making $47.5m from selling access to its data

13

Google+

http://googleblog.blogspot.com.es/2013/10/google-hangouts-and-photos-save-some.html

Google+ 540m MAU300m active in stream1.5bn photos per month

14

Sina Weibo

http://sg.finance.yahoo.com/news/sina-weibo-passes-500-million-151054944.html

China’s Sina Weibo is growing with 74% year on year user growth (http://www.chinadaily.com.cn/bizchina/2013-02/21/content_16243933.htm)

Has 220m ‘active users’ while Twitter has 170m ‘active users’ http://blogs.wsj.com/chinarealtime/2013/03/12/how-many-people-really-use-sina-weibo/

15

Waze

https://www.waze.com

Waze had an estimated 50m users in June 2013

16

Waze

https://www.waze.com

Waze had an estimated 50m users in June 2013

17

Use Cases

Some positive use cases of the data

18

Crowd-Sourcing

http://csce.uark.edu/~tingxiny/courses/5013spring13/readingList/crowdsource.pdf

Academic study on feasibility of using Twitter to crowdsourced data

19

Meteorology

http://uksnowmap.com/#/

UK Snow Map by Ben MarshTweet #uksnow, postcode and x/10 rating

20

Utilities

http://greenmonk.net/2012/11/01/sustainability-social-media-and-big-data/

GE’s Grid IQ Insight can mine social media for mentions of outagesGives early notifications of an outage in an areaIf geotagged and/or includes images/video can confirm cause of outage and speed up time to resolution

21

Utilities

http://greenmonk.net/2012/11/01/sustainability-social-media-and-big-data/

Utilities are aware of reason for outage, speeds up time to resolution (reduces need for investigatory truck roll)

22

Risk Analysis

http://europeandcis.undp.org/blog/2012/11/16/social-media-and-political-risk-analysis/

United Nations Development Program & their Recorded Future project

Using publicly sourced data looking for signs of disruption or unrest

23

Risk Analysis

http://europeandcis.undp.org/blog/2012/11/16/social-media-and-political-risk-analysis/

The same graph turned into media sources- who is talking about Georgia in this period of time

24

Risk Analysis

http://europeandcis.undp.org/blog/2012/11/16/social-media-and-political-risk-analysis/

Mentions turned into social network analysis- who is talking to whom, who is meeting whom

“next phase will focus on conducting a regional political risk analysis and forecasting for South Eastern Europe and Central Asia”

UNDP slides courtesy of Milica Begovich (aka @elim????)

25

Automotive

http://www.magazine.pamplin.vt.edu/fall12/vehicledefects.html

Social Media monitoring tool developed by Pamplin College of BusinessInitial version worked from automotive fora and blogs, now expanding to take in Twitter and Facebook“Robust” way to discover and classify vehicle defects from social media posts across multiple automotive brandsFaster than reporting back up through the dealer chain

26

Finance

http://arxiv.org/PS_cache/arxiv/pdf/1010/1010.3003v1.pdf

Academic paper from University of Manchester and Indiana University shows that Twitter can predict the Dow Jones Industrial Average with 87.6% accuracy

27

Finance

http://www.caymanatlantic.com/investment-management/4574471088

UK Firm Derwent Capital Markets signed an exclusive deal with the authors to create a hedge fund - became Cayman Atlantic

28

Law Enforcement

http://support.sas.com/resources/papers/proceedings12/309-2012.pdf

SAS produced an interesting white paper on this space and bought UK firm Memex - definitely chasing this space

Citing use cases like - finding individuals - analysing their social graph to find accomplices/gang structureAlso identify precursor activity to events like riots

29

Law Enforcement

http://www.policemag.com/blog/technology/story/2012/09/social-media-analytics-in-law-enforcement.aspx

“Social media is a huge network of informants—and one you don't have to pay for.”Law Enforcement use cases (information distribution, fake profile creation, etc.)Helps first responders gain situational awareness prior to having feet on the groundHelps Emergency Operations Centres gain information in the event of natural disasters, for exampleSunday’s train crash in NY, for example

X

Law Enforcement

http://www.huffingtonpost.com/2012/09/04/web-surveillance-social-media_n_1854750.html

Other vendors in this space outlined in this article3i-Mind - http://www.3i-mind.com/HMS Technologies - http://www.hmstech.com/Visible Technologies - http://www.visibletechnologies.com/ Attensity - http://www.attensity.com/home/CrowdControlHQ - http://www.crowdcontrolhq.com/index.phpandAs well as Law Enforcement use cases (information distribution, fake profile creation, etc.)

30

Law Enforcement

http://www.lexisnexis.com/government/investigations/

Good infographic on Law Enforcement use of social media - based on a LexisNexis Risk Solutions survey of 1,200 law enforcement professionalsFull report is available at http://solutions.lexisnexis.com/forms/GV12LEOMPSoMeSurveyforLE9677

31

Law Enforcement

Needs to be approached sensitively - the way some of this is reported often prompts visions of ‘pre-crime’ and Minority Report

32

Smart Cities

https://itunes.apple.com/us/app/boston-citizens-connect/id330894558

Graffiti, Pothole, & LA school district apps

33

Healthcare

http://www.google.org/flutrends/intl/en_us/

Google use frequency of certain search terms as a way to estimate flu activityAlso have one for Dengue FeverSearch data is increasingly mobile

34

Healthcare

http://www.nature.com/nature/journal/v457/n7232/full/nature07634.html

Google wrote this up as an academic paper and it was published in Nature

35

Healthcare

http://www.ajtmh.org/content/86/1/39.abstract

A group led from Harvard Medical School studied viability of using social media for predicting cholera outbreakFound that the data from Twitter closely corresponded with government data, was available up to two weeks earlier

The paper concludes informal media could be used to study the activity of other disease outbreaks around the worldFinancial support was provided by Google.org

36

Healthcare

http://propellerhealth.com

Formerly asthmapolis - wireless asthma puff data - where/whenCan map where asthma outbreaks occur - people sensitive can avoid triggers

37

Healthcare

http://www.huffingtonpost.com/keith-runyon/louisville-chooses-asthma_b_4086297.html

Rolled out in conjunction with city of Louisville KyResidents experience as much as a 13-year gap in life expectancy depending upon where they liveFindings eagerly anticipated - only rolled out in Oct

38

CRM

http://www.ft.com/intl/cms/s/0/bd5a5ce2-aa57-11e1-899d-00144feabdc0.html#axzz2OAlD9lav

T-Mobile in US analysed its 33m customer data records, web logs, billing data and social media information to predict customer defections

It halved customer defections in 3 months

39

Brand Management

http://www.youtube.com/watch?v=VaJjPRwExO8

Nestle were Greenpeace’d because palm oil in Kit Kat came from Sinar Mas - company involved in deforestation

40

http://greenmonk.net/2010/03/19/can-corporate-social-responsibility-affect-your-companys-bottom-line/

Brand Management

In the social media storm which followed Nestlé made every mistake in the bookNestlé received over 200,000 protest emails and their share price was negatively affectedSo they decided to work with Greenpeace to fix their supply chain andTo initiate a social media strategy for the organisation

41http://uk.reuters.com/article/2012/10/26/uk-nestle-online-water-idUKBRE89P07Q20121026

Brand Management

Set up a social media command centre, staffed by their Digital Acceleration Team

42http://www.reputationinstitute.com/thought-leadership/global-reptrak-100

Brand Management

In 2013 Nestle entered the Reputation Institute’s Global top 10 for the first time.

43

Transportation

https://twitter.com/ehn/status/396307684661530624

Waze data now being incorporated in Google Maps

44

Transportation

http://www.bbc.co.uk/news/technology-22357748

2.5bn anonymised call records from 5m Orange phone users in Ivory Coast Looked at patterns of people’s movements in Abidjan - capital city of Ivory CoastRealised they could reduce travel times of ppl by 10%

45

Looking Ahead

http://www.flickr.com/photos/35468133931@N01/8699901706

Google Glass

46

Looking Ahead

http://www.instabeat.me

Instabeat gives swimmers stats in their goggles as they swimAnd subsequent download

47

Looking ahead

http://www.fitbit.com/force

Fitbit force, Nike+ Fuelband, Jawbone UpCan see a situation where sports players are broadcasting vital stats similar to F1

48

Data and data sources are increasing exponentially - go hack that data for good.

Conclusion

Thanks!Contact information:

Tom Raftery

Principal Analyst, Energy & Sustainability, RedMonk

[email protected], GreenMonk.net,

Twitter.com/tomraftery

+34 677 695 468

No tweets were hurt in the making of this presentation

49