2013 csi interchange_pietro_leo - ex

Big Data, the latest updates

1

Plenary session – 14 June 2013

Pietro LeoExecutive Architect - IBM GBS Italy Big Data Analytics LeaderGlobal Technology Progam Manager - IBM Academy of TechnologyEmail: [email protected]

@pieroleo www.linkedin.com/in/pieroleo

mailto:[email protected]

http://www.linkedin.com/in/pieroleo


BIOGRAPHY



Executive Analytics Architect with 20 years professional experience in Research & IT Services

IBM Academy of Technology Core management Team Member and Global Technology Program Manager

Extensive experience on Content Analytics, Big Data Analytics, Social Media Analytics, Knowledge Management, Knowledge and Data Integration, Very Large Mining and Search Engines, Semantic Search, Bioinformatics areas helping clients in complex (multi-millions) and mission-critical projects

Technical leader as well as chief architect and scientist in a number of analytics projects whose overall effort size is over 150 years/man

Social Business Passionate from disparate angles: Member of IBM Service Corps working in Ghana and strong #Socbiz and #innovation expert and #startup hunter

Multidisciplinary education background: Higher artistic degree in Oboe, Computing science degree, Master of Science by Research degree in applied artificial intelligence, Master's degree in public funding management.

Keynote speaker and author or co-author of more than 70 scientific and technical publications and and as well as co-author of two IBM books edited by IBM Readbooks.

Received more than a dozen of IBM special awards for high technical achievements including also the mention into the IBM Corporate Technical Award Book 2010 edition, the IBM IT Architect Worldwide Technical Leadership Excellence Award in 2006 and the IBM Academy President's Award 2012.





Agenda

Defining Big Data

Big Data as a macro-trend and the State-of-the-Art

The business impact of Big Data & Deep Dive on selected Big Data Experiences

A Big Data IT Perspective

Wrap-up & organizing for Big Data: Next “best action”



Measuring Big Data by using Big Data

Source: Google Trends

Big Data

Business Intelligence

Data Warehouse

Business Analytics

Correlated Query Terms





Conventional Definitions of “Big Data”

Never before possibleSocial Data

Large volumesUnstructured Data

Valuable insight, but difficult to extractBasically an ETL environment

….....

These are partial and

or wrong definitio

ns


7@pieroleo www.linkedin.com/in/pieroleo

Big Data enables us to see with new eyes....



“The real voyage of discovery consists not in seeking new lands, but in seeing with new eyes....”

Marcel Proust, A la recherche du temps perdu, 1913/27

Salvador Dalì - Impresiones de África y Afgano invisible con aparición sobre la playa del rostro de

García Lorca en forma de frutero con tres higos, 1938 Dog Head

Fruit Bowl Waterfall

Table

BridgeDog Collar

Dog Muzzel

Hill

Beach

River

Point of View 1Point of View 1 Point of View 2Point of View 2



Big Data enables us to see with new eyes....Salvador Dalì - Impresiones de África y Afgano invisible con aparición sobre la playa del rostro de García Lorca en forma de frutero con tres higos, 1938


@pieroleo www.linkedin.com/in/pieroleo10

>80% Unstructured Data

+ External Data“Untouched” Data+ Stream of Data

Enterprise Data Machine Data People Data

Big Data metaphor



Data is there and we need to make the best out of it



We produce and consume Data for a specific purpose


@pieroleo www.linkedin.com/in/pieroleoSource: A statue representing Janus Bifrons in the Vatican Museums

Big Data as a new Business Concept and as a new Technology Concept



Big Data as a new business concept: New values and opportunities for a number of stakeholders

Chief Marketing Officerhow to improve customer focus?...could predict the right offer for the right customer at the right time and improve customer value and intimacy or prevent churn?

Chief Product Designer...how we can innovste? … could

we improve our product channels/design offering??

Chief Finance Officer

...could streamline compliance and understand risk

exposure across businesses and

regions?

Chief Risk Officer...uses anti fraud predictive analytics to detect and prevent rapid fire anomalous transactions or wire transfers identified as high probability of fraud?

Chief Executive Officer...could make better business decisions using accurate data across all company/system dimensions and across time horizons: past, present and future?

Chief Information Officer ...could analyze oceans of machine generated logs to

predict which components or equipment in the datacenter are likely to fail and thereby avert a disruption

during critical quarter end? How we can support Zero high risks or manage crisis?

Big Data

Analytics



Big Data as a new technology concept: We need to combine internal and external data, utilized and under-utilized data, structured and unstructured data... and cross-link organization knowledge & data silos

CRM• emails• claims• call center scripts• Chats with customers• …

Transactional Info.:• Transactions• Orders• consultancies• …

Legal Info:• Contracts• Complaints• Reports• Legal Actions• Fraud Data• …

Knowledge Management•Manuals, wikis, couses•Projects Data•Market Analysis•RSS Business Feeds•Data feed: Bloomberg reuters• …

IT SystemsSystem LogsApplication logs: web, vending machines, mobileVideoSensor Networks, RFID• …

Social Media:• Global Social Networks: tweeter, facebook, etc.• Small communities: blogs, muros corporativos,• Internal Social Networks (employees)• News • …

Big Data

Analytics



Source: Cornell University - Maize kernal infected with Aspergillus flavus, which produced aflatoxin.http://www.plantpath.cornell.edu/labs/milgroom/Research_aflatoxin.html And http://www.special-clean.com/special-clean/en/mold/mold-lexicon-1.php

For science, Big Data is the microscope of the 21st century


http://www.plantpath.cornell.edu/labs/milgroom/Research_aflatoxin.html


For Science, Big Data is the microscope of the 21st century

Wine DNA Tracing



Just ONE Transaction path goes to the end in thousands and to complete that path tens of decision points were considered. Right now we store and analyze in our transactional systems just the end points:

BuyerFail!

Fail!

Fail!

Fail

Fail!Fail!

Fail!

Fail!

Fail!

Fail!Fail!Fail

Fail!

Fail!Fail!

….Win!!!

Buying DecisionCloud

Yes!

For Business, Big Data is the answer and the need of the new emerging sub-transactional era



SocialData from and about People

PhysicalSensors & Streams

Terabytes to exabytes of existing data

to process

Streaming data, milliseconds to seconds to

respond

Structured, Semi-structured Unstructured,

text & multimedia

Uncertainty from inconsistency,

ambiguities, etc.

Volume

Velocity

Variety

Veracity

DataContent

>80%

<20%

Traditional Enterprise Data

Big data embodies new data characteristics created by today’s digitized marketplace

BiologicalDNA Sequencers


@pieroleo www.linkedin.com/in/pieroleo20 20

Glo

bal

Dat

a V

olu

me

in E

xab

ytes

Sens

ors

(Inte

rnet

of T

hing

s)

Multiple sources: IDC,Cisco

100

90

80

70

60

50

40

30

20

10

Agg

rega

te U

ncer

tain

ty %

VoIP

9000

8000

7000

6000

5000

4000

3000

2000

1000

0

2005 2010 2015

By 2015, 80% of all available data will be uncertain: Veracity

Enterprise Data

Data quality solutions exist for enterprise data like customer, product, and address data, but

this is only a fraction of the total enterprise data.

By 2015 the number of networked devices will be double the entire global population. All

sensor data has uncertainty.

Social Media

(video, audio and text)

The total number of social media accounts exceeds the entire global

population. This data is highly uncertain in both its expression and content.



“Big Data is the set of technical capabilities,

management processes and

skills for converting vast, fast, and varied data into Right Data to produce useful

knowledge”

Source: Definition discussed during the work of the Word Summit on Big Data and Organization Design Paris – 2013 and Adapted from: Beacon Report – Big Data Big Brains – 2013

In summary, what is Big Data?



What is New and Different?

A lot more data and different kinds of data.Historically most data was structured data – rows and columns

Today it is unstructured data like aerial photos, audio from call centers, video from surveillance cameras, e-mails, texts, diagrams.

A shift in focus from data stocks to data flows.Historical information was stored in data warehouses and analyzed by data mining.

Streaming data arrives in real time allowing us to influence events as they happen. We can prevent some bad events from ever happening at all.

Shift in the power structure of the company. Many companies have analog establishments. We need to shift power to those who can draw valuable insights from data and analytics and implement them.

Shift from periodic to real time or continuous decision making. We need an increase in the clock speed of every process in the company.

There is a potential for “Big Data” to become a fundamental center for the company. Is it a new dimension of structure?

Organization Design IssuesTechnology Issues

Source: Jay R. Galbraith, from Word Summit on Big Data and Organization Design Paris – 2013



Agenda

Defining Big Data







IBM Institute for Business Value and the Saïd Business School partnered to benchmark global big data activities

24

IBM Global Business Services, through the IBM Institute for Business Value, develops fact-based strategies and insights for senior executives around critical public and private sector issues.

Saïd Business School University of Oxford

IBM Institute for Business Value

The Saïd Business School is one of the leading business schools in the UK. The School is establishing a new model for business education by being deeply embedded in the University of Oxford, a world-class university, and tackling some of the challenges the world is encountering.

www.ibm.com/2012bigdatastudy



Big Data Analytics has evolved from business initiative to business imperative

63%

58%

37%

2012

2011

2010 70% increase

Source: 1 2010 and 2011 datasets © Massachusetts Institute of Technology. 2 Analytics: The real-world use of big data. 2012 Study conducted by IBM Institute for Business Value, in collaboration with Säid Business School at the University of Oxford.

3.6x

Likelihood of organizations competing on analytics to outperform their peers2

Percentage of respondents who cited a competitive advantage from the use

of information and analytics1,2



Three out of four organizations have big data activities underway; and one in four are either in pilot or production

26

Total respondents n = 1061Totals do not equal 100% due to rounding

Big data activities

Respondents were asked to describe the state of big data activities within their organization.

Early days of big data era Almost half of all organizations surveyed

report active discussions about big data plans

Big data has moved out of IT and into business discussions

Getting underway More than a quarter of organizations have

active big data pilots or implementations Tapping into big data is becoming real

Acceleration ahead The number of active pilots underway

suggests big data implementations will rise exponentially in the next few years

Once foundational technologies are installed, use spreads quickly across the organization



Five key findings highlight how organizations are moving forward with big data

27

Big data is dependent upon a scalable and extensible information foundation2

The emerging pattern of big data adoption is focused upon delivering measureable business value5

Customer analytics are driving big data initiatives1

Big data requires strong analytics capabilities4

Initial big data efforts are focused on gaining insights from existing and new sources of internal data3



Key Findings: Customer analytics are driving Big Data initiatives

Big dataInfrastructure

Big dataSources

Analytics capabilitiesTotal respondents n = 1061

Big data objectives

Top functional objectives identified by organizations with active big data pilots or implementations. Responses have been weighted and aggregated.

Customer-centric outcomesOperational optimizationRisk / financial management

New business model

Employee collaboration

Big Data areas of work



Big data leadership shifts from IT to business as organizations move through the adoption stages

29

CIOs lead early efforts Early stages are driven by CIOs once

leadership takes hold to drive exploration

CIOs drive the development of the vision, strategy and approach to big data within most organizations

Groups of business executives usually guide the transition from strategy to proofs of concept or pilots

Business executives drive action Pilot and implementation stages are

driven by business executives – either a function-specific executive such as CMO or CFO, or by the CEO

Later stages are more often centered on a single executive rather than a group; a single driving force who can make things happen is critical

Leadership shifts

Respondents were asked which executive is most closely aligned with the mandate to use big data within their organization. Box placement reflects the degree to which each executive is dominant in a given stage.

Total respondents n = 1028



Agenda

Defining Big Data







Utilities Weather impact analysis on

power generation Transmission monitoring Smart grid management

Retail 360° View of the Customer Click-stream analysis Real-time promotions

Law Enforcement Real-time multimodal surveillance Situational awareness Cyber security detection

Transportation Weather and traffic

impact on logistics and fuel consumption

Traffic congestion

Financial Services Fraud detection Risk management 360° View of the Customer

IT System log analysis Cybersecurity

Telecommunications CDR processing Churn prediction Geomapping / marketing Network monitoring

What can you do with Big Data?

Health & Life SciencesEpidemic early warningICU monitoringRemote healthcare monitoring



• Advanced client segmentation• Leveraging customer sentiment analysis • Reducing customer churn

• Optimizing the supply chain • Deploying predictive maintenance capabilities• Transform threat & fraud identification processes

Operations

• Enabling rolling plan, forecasting and budgeting• Automating the financial close process • Delivering real-time dashboards

Finance

• Making risk-aware decisions• Managing financial and operational risks• Reducing the cost of compliance

Risk

Examples:

Customers / Clients

Another perspective: let's focus on ROI in core business areas for Big Data

• Advanced client segmentation• Leveraging customer sentiment/opinion analysis • Reducing customer churn



Big Data for Customer Analytics challenge: build a 360°Integrated Customer View … and more!

SINGLE VIEWBusiness Data,

Social Data, Interactive data

360°Integrated Customer View

Marketing

Cust. Care

Sales

Risk, Fraud

Customers / Clients



Big Data for Customer Analytics challenge: build a 360°Integrated Customer View … and more

SINGLE VIEWBusiness Data,

Social Data, Interactive data


Marketing

Cust. Care

Sales

Risk, Fraud

Customers / Clients

How?How?Why?Why?

Who?Who? What?What?



Big Data for Customer Analytics challenge: build a 360°Integrated Customer View … and more


Customers / Clients

How?How?Why?Why?

Who?Who? What?What?Project Example 1TV Broadcaster

Project Example 2Media and Entertainment

Project Example 3Hair care manufacturer

Big Data Analytics Project Space



• Social media analysis is a new and increasingly relevant way to become more competitive in consumer-driven markets. Mediaset wanted to increase its market share as well as launch new services and digital-content distribution. s marketing campaigns and better

• Television content and services are becoming increasingly consumer driven, and the media outlet that can capture and use customer sentiment to its benefit gains a competitive advantage. This media provider in Italy applied an advanced analytics solution to analyze more than 1.6 million unstructured data points from Web 2.0 sources to gain an understanding of its customers’ attitudes, opinions and preferences.

Challenge

Benefits

Solution

Customer Quote

“Big data is a great opportunity for TV innovation in the next years. TV viewing is transforming into a multiplatform and participative experience; the better we know and understand our viewers, the better we can serve them”.

36

A TV broadcaster analysed Big Data Analytics to collect Customer longitudinal point of views from Web 2.0 and correlate them with internal data

• Television content and services are becoming increasingly consumer driven, and the media outlet that can capture and use customer sentiment to its benefit gains a competitive advantage. This media provider in Italy applied an advanced analytics solution to analyze more than 1.6 million unstructured data points from Web 2.0 sources to gain an understanding of its customers’ attitudes, opinions and preferences.

• Analyzed more than 1.6 million data points on social media outlets to discover public sentiment and correlations with customer satisfaction

• Helped Mediaset to discover and measure viewer sentiments expressed in Web 2.0 contents related to its TV contents and ad campaigns



Big Data Analytics to expand knowledge about customer profiles and measuring marketing campaign

• Analysis of social media messages for large Media and Entertainment company to determine reaction to movie commercials aired during the SuperBowl

• Insights based on 30M+ social media consumer profiles created by analyzing over a Billion messages• Real-time evolution of insights correlated with the airing of the commercial

• Analysis of social media messages for large Media and Entertainment company to determine reaction to movie commercials aired during the SuperBowl

• Insights based on 30M+ social media consumer profiles created by analyzing over a Billion messages• Real-time evolution of insights correlated with the airing of the commercial

Key Business Questions:

How many people are talking about the film ?• Intention to see the movie, Impact of SuperBowl commercial

Who are they ?• Demographics, Influencers, avid movie goers

What is the reaction ?• Categorized sentiment (plot, characters, …)• Comparison with competitive movies

Key Business Questions:

How many people are talking about the film ?• Intention to see the movie, Impact of SuperBowl commercial

Who are they ?• Demographics, Influencers, avid movie goers

What is the reaction ?• Categorized sentiment (plot, characters, …)• Comparison with competitive movies

Jan 1

5pm 6pm 7pm 8pm

Super Bowl

Monitoring Period Feb 5th

Golden Globes NFC Championship

9pm 10pm 11pm

• Buzz and sentiment• Gender, Location and Occupation• Avid movie-goers, comic book fans• Intent to see specific films• Specific attributes of the film/trailer



■ Their earlier analysis of Google search requests suggested that hair problems formed a significant part of what consumers care about…

■ … but Big Data Analytics showed that people rarely chatted about their hair problems when discussing and comparing hair care products

The marketing messages were re-focused in line with this more nuanced insight – promoting what customers want for their hair to harmonize with the social media agenda

On another perspective an Hair care manufacturer finds out what consumers really chat about



360-degree Consumer Profiles from Social Media

Personal Attributes• Identifiers: name, address, age, gender, occupation…• Interests: sports, pets, cuisine…• Life Cycle Status: marital, parental

Personal Attributes• Identifiers: name, address, age, gender, occupation…• Interests: sports, pets, cuisine…• Life Cycle Status: marital, parental

Products Interests • Personal preferences of products• Product Purchase history• Suggestions on products & services

Products Interests • Personal preferences of products• Product Purchase history• Suggestions on products & services

Life Events• Life-changing events: relocation, having a baby, getting married, getting divorced, buying a house…

Life Events• Life-changing events: relocation, having a baby, getting married, getting divorced, buying a house…

Monetizable intent to buy products Life Events

Location announcementsIntent to buy a house

I'm thinking about buying a home in Buckingham Estates per a recommendation. Anyone have advice on that area? #atx #austinrealestate #austin

I'm thinking about buying a home in Buckingham Estates per a recommendation. Anyone have advice on that area? #atx #austinrealestate #austin

Looks like we'll be moving to New Orleans sooner than I thought.Looks like we'll be moving to New Orleans sooner than I thought.

College: Off to Stanford for my MBA! Bbye chicago!College: Off to Stanford for my MBA! Bbye chicago!

I'm at Starbucks Parque Tezontle http://4sq.com/fYReSjI'm at Starbucks Parque Tezontle http://4sq.com/fYReSj

I need a new digital camera for my food pictures, any recommendations around 300?

I need a new digital camera for my food pictures, any recommendations around 300?

What should I buy?? A mini laptop with Windows 7 OR a Apple MacBook!??!

What should I buy?? A mini laptop with Windows 7 OR a Apple MacBook!??!

Timely Insights• Intent to buy various products • Current Location• Sentiment on products, services, campaigns• Incidents damaging reputation• Customer satisfaction/attrition

Timely Insights• Intent to buy various products • Current Location• Sentiment on products, services, campaigns• Incidents damaging reputation• Customer satisfaction/attrition

Relationships• Personal relationships: family, friends and roommates…• Business relationships: co-workers and work/interest network…

Relationships• Personal relationships: family, friends and roommates…• Business relationships: co-workers and work/interest network…



AMEX Example: Business Models based on connecting Virtual and Real Worlds

American ExpressSmart Offer

A portal that collects special offers and discounts from retailers and detail about the customer segment that is target

Marketing segmentation engine that evaluate customer profiles and select the best coupon to propose

Moble app and connection with Twitter, Facebook e Foursquare to communicate with the customers and enable viral effects

Just virtual Coupons are managed! Customers activate the coupon and receive on montly basis on the credit card account the equivalent of the coupon discounts after that transactions were registred

API



What Data AMEX Sync acquires from Facebook, Twitter e Foursquare? New CRM Data...

American ExpressSmart Offer



Twitter Inc. is experimenting with becoming a shopping mall.

Twitter and American Express Co. said Monday they struck a partnership to allow Twitter users to buy products for the first time directly on the short messaging service.

American Express card holders who connect their card numbers to their Twitter accounts can post on Twitter to trigger a purchase of select products, including discounted American Express gift cards, Kindle Fire tablets from Amazon.com Inc. and jewelry from designer Donna Karan. The program will roll out over the next few days.

The arrangement hints at a potential new source of revenue for Twitter, which has largely been reliant on advertising for revenue. Neither Amex nor Twitter will discuss financial terms of their partnership, but Twitter wouldn’t rule out taking a cut of future e-commerce sales. The American Express partnership also is a way for Twitter to prove the link between marketing activity on Twitter and a ringing cash register.

API - S

ervi

ces

Twitter, Amex Launch Pay-By-Tweet Service

Source: http://blogs.wsj.com/digits/2013/02/11/twitter-amex-to-collaborate-on-e-commerce-sales-on-twitter/



external traits

intrinsic traits

Omni Profileof each individual

…..Further Hyper-Personalized

360°Integrated

Customer View

+Personality

OpennesConscientiousnessExtraversion Agreeableness Neuroticism Perception

Fundamental needsIdeal

Liberty

Love

Structure

Social behavior

Responsiveness

Temporal patterns of activities

Social relationships to othersSimilarityTie strength

FrequencyRecencyIntensityReciprocityIntimacy

Big Data for Customer Analytics challenge: build a

360°Integrated Customer View … and more!Customers / Clients



Big Data enabled doctors from University of Ontario to apply neonatal infant monitoring to predict infection in ICU 24 hours in

advance

Performing real-time analytics using physiological data from neonatal babies

Continuously correlates data from medical monitors to detect subtle changes and alert hospital staff sooner

Early warning gives caregivers the ability to proactively deal with complications

“Customer Analytics” in

some Industry means safe life



Agenda

Defining Big Data







Data WarehouseOperational Analytics

Structured, analytical, logical

Big DataAd Hoc Analytics

Creative, holistic thought, intuition

UnstructuredExploratoryIterativeBrand sentimentProduct strategyMaximum asset utilization

StructuredRepeatable

LinearMonthly sales reports

Profitability analysisCustomer surveys

Big Data IT Perspective: augmenting traditional IT investments



Manage & store huge volume of any data

Hadoop File System

MapReduce

Manage Streaming Data

Stream Computing

Analyze Unstructured Data Text Analytics Engine

Data WarehousingStructure and control data

Integrate and govern all data sources

Integration, Data Quality, Security, Lifecycle Management, MDM

Understand and navigate federated big data sources

Federated Discovery and Navigation

From an IT perspective leveraging Big Data and Big Data Analytics requires multiple platform capabilities



BI / Reporting

Exploration / Visualization

FunctionalApp

IndustryApp

Predictive Analytics

Content Analytics

Analytic Applications

IBM Big Data Platform

Systems Management

Application Developmen

t

Accelerators

BigInsights

Volume, Variety

Cost-effectively process and analyze any type of data

Visualization & Discovery

Visibility

Understand, find, and navigate federated big data

Data Warehouse

Volume, Structured

Purpose-built offeringsHigh-performance appliances and software

Information Integration & Governance

Veracity

Trusted informationParallel processing for high-volume integrationBest practices

Stream Computing

Building a Big Data Platform: The IBM perspective

Velocity

Analyze data-in-motion to produce insights in micro-seconds

Agile, multi-tenant shared infrastructure

BIG Performance

Option of an optimized low-latency MapReduce implementation fully compatible with open-source Hadoop



Agenda

Defining Big Data







Business-centric Big Data enables you to start with a critical business pain and expand the foundation for future requirements.... start with the most critical one!



Using twitter?

Dear delegate, we value the feedback provided through feedback forms, but we would like to encourage you to use the twitter hashtag #IBMCSII for your:

Findings on the eventLogisticsSuggestionsNetwork dinerSpeakersContentWeather ...

@pieroleo

www.linkedin.com/in/pieroleo

You can find me here:



Grazie!






2013 csi interchange_pietro_leo - ex

Technology

big data big data

big data source

data integration

external data untouched

accurate data

selected big data experiences

vatican museums big

ibm books