Data Science Popup Austin: Predicting Customer Behavior & Enhancing Customer Experience

DATA SCIENCEPOP UP

AUSTIN

Predicting Customer Behavior & Enhancing Customer Experience

Vinny SenguttuvanSr. Data Scientist, Metis

DATA SCIENCEPOP UP

AUSTIN

#datapopupaustin

April 13, 2016Galvanize, Austin Campus

http://www.dominodatalab.com

PREDICTING CUSTOMERS

Vinny SenguttuvanSenior Data ScientistMETIS

5PREDICTING CUSTOMERS

What is the company/product?High 5 Games

A facebook and mobile app

A suite of about 100 games

Over 10 million users


What to Predict?User-Product Interaction (A/B testing)

Popularity

Survival Analysis

Lifetime Value

Recommendation systems


The main challenge:Most users play for free

2% of the users contribute to all revenue

2% of those 2% are responsible for over half the entire revenue

Double singularity

A/B TESTING

9A/B TESTING

Classic Hypothesis Testing

There is a current product design/feature: A

There is a suggested improvement or variation: B

We split the users into two groups and offer them A or B

10A/B TESTING

Classic Hypothesis TestingAn Example hypothesis:

Moving the “buy credits” button from top-right to a more prominent position in the mid-right of the screen and making it bigger and brighter will lead to at least 20% more clicks.

11A/B TESTING

Classic Hypothesis TestingAn Example hypothesis:

Moving the “buy credits” button from top-right to a more prominent position in the mid-right of the screen and making it bigger and brighter will lead to at least 20% more clicks.

It is important to have a minimum improvement requirement, since there is a cost involved in making the change.

12A/B TESTING

Classic Hypothesis TestingNeed enough data to:• separate the probability density functions• get the required sample size• achieve statistical significance

13A/B TESTING

Classic Hypothesis TestingNeed enough data to:• separate the probability density functions• get the required sample size• achieve statistical significance1. A long list of changes means not enough time per change

14A/B TESTING

Classic Hypothesis TestingNeed enough data to:• separate the probability density functions• get the required sample size• achieve statistical significance1. A long list of changes means not enough time per change

2. Often the high monetizers are the primary test subjects, so the double singularity is a big issue

15A/B TESTING

Work aroundWhen we don’t have enough samples, we could:• we could bring down the confidence value• we could bias on the side of one hypothesis• we could assume independence between various A/B and run

them simultaneously or at least overlap them

16A/B TESTING

Work aroundWhen we don’t have enough samples, we could:• we could bring down the confidence value• we could bias on the side of one hypothesis• we could assume independence between various A/B and run

them simultaneously or at least overlap them

We do all three.

17A/B TESTING

Work aroundWhen we don’t have enough samples:• we bring down the confidence value• we bias towards hypothesis A, since B has implementation cost• we order the various A/B tests such that the dependences

between adjacent ones are minimal

POPULARITY METRICS

19POPULARITY METRIC

Rank or classify products or games• New products added continuously.• Some locked and slowly unlocked.• Others fully unlocked to all users.

20POPULARITY METRIC

Rank or classify products or games• New products added continuously.• Some locked and slowly unlocked.• Others fully unlocked to all users.• They are released at vastly different conditions:

• one is game #15 when there are 2 million users• another is game #77 when there are 10 million users

21POPULARITY METRIC

Rank or classify products or gamesOpen problem

Also faced with music and other products

22POPULARITY METRIC

Rank or classify products or gamesOpen problem

Also faced with music and other products

We whiteboarded various approaches including N dimensional product spaces and harmonic distributions.

Either they were too complex to build or the data was too noisy.

23POPULARITY METRIC

Rank or classify products or gamesWent with two simple metrics:

1. An average-weighted-spin: Spins per game, over total spins, times number of games unlocked, time log number of days since the unlock of the game.

2. A Bayesian method (my colleague implemented this)

24POPULARITY METRIC

Rank or classify products or gamesWent with two simple metrics:

1. An average-weighted-spin: Spins per game, over total spins, times number of games unlocked, time log number of days since the unlock of the game.

2. A Bayesian method (my colleague implemented this)

Both performed well but had flaws. Used an intersection of the two.

SURVIVAL ANALYSIS

26SURVIVAL ANALYSIS

Survival of usersWanted to know:

1. How many users are still active after a given time since the creation of their account

2. When is a user at risk of discontinuing to use the product

27SURVIVAL ANALYSIS

Survival of usersWanted to know:

1. How many users are still active after a given time since the creation of their account

2. When is a user at risk of discontinuing to use the product

This problem had its beginnings in the medical industry but is widely among internet services and products

28SURVIVAL ANALYSIS

Survival of usersA lot can be learned by grouping users by cohorts and plotting survival rates over time

We can also observe return rates of users after each additional day of absence.

Cox survival model (my colleague worked on this)

29SURVIVAL ANALYSIS

Survival of usersOur observations:

The survival rate of new users were significantly lesser than earlier users and continues to drop

There was a clear day span, if a user was absent by more than that, the chance of their return dropped significantly

30SURVIVAL ANALYSIS

Survival of usersOur actions:

Factor in the decline in quality of the users while estimating revenue. Also focus on acquisition channels that provided better players

Reach out to users at risk of discontinuation with notifications and offers before they hit the absence threshold

PREDICTIVE LIFETIME VALUE

32PREDICTIVE LIFETIME VALUE

Lifetime ValueThe total revenue from a specific user over entire time with product

Often defined as the revenue during the first 365 days of use

Very significant because of costs:• 10% of the revenue spent on acquisition ads• 30% revenue goes to app hosts (facebook/apple)


Lifetime ValueThe cost of ads are per new player acquired

While the revenue comes from a small subset of those players


Lifetime ValueThe cost of ads are per new player acquired

While the revenue comes from a small subset of those players

So we wanted to build an aggregated LTV for various acquisition sources


Lifetime ValueIt’s a time series

Modeling that way would use all available information, but is difficult


Lifetime ValueIt’s a time series

Modeling that way would use all available information, but is difficult

Flipped it into a regression problem


Regression ProblemPredict 365 day revenue based on user’s first month data.



SpendPurchase countDays of playPlayer level achieved



SpendPurchase countDays of playPlayer level achievedDate joined (because of the decline in player quality)


Regression ProblemIncluding the date joined as a feature makes it a sort of time series

Had to be careful in the setting up of the train and test data


Regression ProblemIncluding the date joined as a feature makes it a sort of time series

Had to be careful in the setting up of the train and test data

Prediction rates were 80%, 90% and 95% after 1, 2 & 3 months

Surprising discoveries which made us change strategies

RECOMMENDATION SYSTEMS

44RECOMMENDATION SYSTEMS

How does Pandora do it?

Specialists listened to every song and gave each of them various attributes (about 30 to a hundred)

Two songs are similar if their distance in that n-dimensional space is minimal

This works with a binary scale or a point system


How does YouTube do it?

Algorithmic:• Find all users who listened to the song• Find the song that set of people had listened to the most

The actual solution is a little different but concept is the same


Pandora: • intrinsic recommendation system• internal elements (features) of the songs are used to

measure similarity

YouTube:• extrinsic recommendation system• recommendations are solely based on peer preferences• hence the term “collaborative filtering”


DIEHARD BRAVEHEART MULAN CASABLANCA HOME ALONE

5

3

1

1

3

4

5

1

2

3

3

4

2

5

5

4

3

2

1

2

TOM

SALLY

NIKIL

KIM


ACTION COMEDY HISTORY COMING-OF-AGE ROMANCE

5 3 1 1

5 5 1 4

1 3 5 4

1 2 1 4

1 5 1 3

TOM SALLYNIKIL

KIM

DIE HARD BRAVEHEART MULAN CASABLANCA HOME ALONE

5 2 1 2 1

5 1 5 2 4

4 2 4 4 4

2 2 4 2 5

3 4 1 4 1

ACTION COMEDY HISTORY

COMING-OF-AGE ROMANCE


svd: single value decomposition

You provide the MxN matrix and a K. And you get:

M x K (user-feature) matrix

K x N (feature-product) matrix


Collaborative FilteringUser & Product Matrix can be decomposed:

(User & Features) and (Features & Product)


Collaborative FilteringUser & Product Matrix can be decomposed:

(User & Features) and (Features & Product)

Then you can predict rating, find similar users or similar products.

AP News example: Articles and words


Collaborative Filtering

How to choose “K”?



How to choose “K”?

Could try various K, decompose, multiply and compare (RMSE) to original matrix (cross-validation)

Or look at Eigen Values



What if the ratings are not clearly defined?



What if the ratings are not clearly defined?

Could use click, percentage of view, number of views, like, share



For Missing values:Use averages. Or sparse matrix solvers.

Normalization is important.Outliers and noise get magnified.



What if there is a progression or order to the content?



What if there is a progression or order to the content?

Have to do some supervision.


Intrinsic recommendation systemsFeatures are well definedDon’t need user data to beginManual


Extrinsic recommendation systemsAutomaticFeatures are undefinedNeed a lot of user dataNot automatically adaptive to special cases (like order-specific entries)


Combined technique

Begin with the intrinsic knowledgeBetter understanding of the contentEffective initial recommendationOnce enough data points are accumulated, switch to the extrinsic model


Combined technique

Double the workMost effective, especially for timely contentThis is what Netflix does… likely the best recommendation system


What more, with recommender systems?


Plus recommender systems can be more than mere recommendation..

DATA SCIENCEPOP UP

AUSTIN

@datapopup #datapopupaustin

Data Science Popup Austin: Predicting Customer Behavior & Enhancing Customer Experience

Data & Analytics