Sunz2013 vince morder

Vincent Morder

Analytics Manager, Loyalty New Zealand

Insights on Loyalty’s Approach

To Big Data

SUNZ February 2013

Loyalty Vision: To create, maintain and

motivate loyal customers for our Participants

End to End Marketing

4

History of data and analytics In the beginning….

• Data • Small, manual, fixed

• Systems • Static, relational

• Analytics • Sophistocated statistics and mathematics

• Retrospective

• Hand- crafted variables

• Static view

• Long time lags

5

History of data and analytics Over time….

• Data

• Bigger data, coming from various systems, more timely

• Systems

• Operational systems

• More storage required

• Relational databases

• Computing power increased

• Analytics

• Applying stats to business, banks, retail, telecommunications, etc…

• Same statistical principles applied, but now on hormones with software

• Sample size no longer an issue.

6

Today: Big Data

• Sheer magnitude of data captured by digital world:

• Web

• Social

• Free form text

• Retail Transactions

• Mobile device activity

• Software logs

• Cannot be managed by traditional data management tools

• Unstructured to a large degree

• Data is distributed, so difficult to perform traditional queries for analysis

• Applications and needs are for real time.

7

Big sets of Data

• Historically loyalty has a lot of data that is big

• Transactions

• SKU Transactions

• Campaign history

• But it has not been Big Data

• Here are some examples….

8

Transactions

• SKU transactions still fit into structures and can be managed

• But it’s big – over 1,000,000,000 items and 100,000 products.

• Challenged to make data manageable

• Models help to key dimensions to make actionable

• E.g., Clusters using Ideal Dimensions (Morder, SUNZ 2012)

• Boil down products and customers to 20 key dimensions.

9

Campaigns

• Campaign history still fit into structures and can be managed

• But it’s big – over 500,000,000 records and 000’s of campaigns.

• Models on campaign history have to boil down data to key dimensions to make actionable.

• E.g., SAS MO uses these models to optimise campaign performance.

• Boil down 70 response models (one per partner).

10

Where the world is heading

• Digital data will continue to grow faster than traditional data

• Digital representation of all our transactions and activity

• Retail environment craves more info to keep up with competition

• ……And our customers expect us to use it instantly

11

Big Data at Loyalty

• Last 12 months we have put a new web-server called Harry.

• Postgres System

• Captures web activity

• Integrates with Service Centre.

• Plans are for Harry expand to incorporate

• Core Fly Buys rewards, transactions, and points processing

• Smart phone app behaviour

• Real time recommendations

• … And this is Big Data.

12

CAP Theorem or Brewer’s Triangle

• Consistency: all clients always have the same view of the data

• Availability: each client can always read and write

• Partition Tolerance: operation continues despite physical network partitions.

13 (P)artition Tolerance

Tradeoffs • A and C means you will not have P

• C and P means you will not have A

• A and P means you will not have C

14

(P)artition Tolerance

A+C

A+P C+P

Pick Any Two

Big Data at Loyalty

• We have systems all around the CAP triangle for different purposes.

15

Riak Mongo

Data warehouse

Pick Any Two

(P)artition Tolerance

Postgres

Pick

Any

Two

Implications for Analytics

• Some data and models are same old style: static, historical looking. • One model or segmentation applied across all customers

• Still very important to have these.

• More and more models using data based on current activity.

Examples

• Real time decisions are rule- based and trigger off of observed activity

• Recommendation engines use a model for every individual • Recommendation based on individual product preferences

16

Models

• Preferences and associations to be updated as data comes in.

• Model structure needs to be able to handle being updated as data is updated.

• Models need to be able to deal with eventual consistency.

• Information may not be 100% complete.

• E.g., Poisson Models coefficients to be updated additively.

• Good for billions of record that need to be processed online, like movie recommendations. 1

7

Recommendation Engines

• Two main types of recommendation systems:

• Content-based

• Collaborative filtering,

• Batch vs. real time algorithms

• Push and pull through all our channels

18

Design of data

• Combine big historical string with new information as it comes in….

• Final Prepare has helped us ‘prepare’ for the future,

• Unfortunately it’s not final.

19

Slow moving models and characteristics

(FINAL PREPARE)

Fast moving new data

Going from Unstructured to Structured data

• Modeling and analysis still requires that data has a structure.

• Find and define your associations. Applies to: • Text mining

• Web logs activity

• Smart phone behaviour

• Consumer behaviour.

Two words: Map and Reduce

• Software functions to go from unstructured to structured

• Tell your IT guys to code for the established mappings.

20

Example - Hypothetical

• Data on a customer and his history of buying patterns – historical data on CA server • Discover that he is online and looking to find out about specials – data feed to AP system. • RFI indicates he is entering the store X – another data feed but to CP server. • CP server triggers for an algorithm to calculate the best offer for that person.

• Proactive message sent by CP server to offer a discount on complementary product. • Capture record of actual spend at checkout – another data feed to CA server • Feedback survey send after leave store – capture on AP server. • Capture record of spend and feedback to perform post campaign analysis and improve models on CA

system. • Report campaign results online to retailer on CA system • Better shopping experience for customer, greater commitment, spreads the word. • Better models, more accurate offers next time. Continual improvement. • Applications are limitless.

21

SAS Products on the Big Data Front

• Real Time Decision Manager

• Social Network Analytics

• Text Miner

• Visual Analytics

22

The Future of the Analytics Industry and of LNZ

• More applications, more digital presence

• Analytics are being led by technology and data

• Implications is that models are changing • Hybrid models: slow models + real time data

• One model per customer

• Analysts will be shifting their focus • Build new types of models and real time solutions

• Unstructured to structured

• Tracking of models will change.

• Lab 360 will be there to provide technology and services for both internal partners and external clients.

23