More value from data using Data Mining Allan Mitchell SQL Server MVP.

More value from data using Data Mining

Allan MitchellSQL Server MVP

Who am I

• SQL Server MVP• SQL Server Consultant• Joint author on Wrox Professional SSIS book• Worked with SQL Server since version 6.5• www.SQLDTS.com and www.SQLIS.com• Partner of SQL Know How

Today’s Schedule

• what is data mining (Overview)• data mining terminology• myths around data mining• excel AddIn to Office2007

– Demo Setup– Demo Key Influencers– Demo Categories– Demo Make a Prediction– Demo “Other stuff” – if time

• Questions and answers

What is Data Mining

• The process of using statistical techniques to discover subtle relationships between data items, and the construction of predictive models based on them. The process is not the same as just using an OLAP tool to find exceptional items. Generally, data mining is a very different and more specialist application than OLAP, and uses different tools from different vendors. Normally the users are different, too. OLAP vendors have had little success with their data mining efforts.

OLAP REPORT

What does Data Mining Do?

Explores Your Data

Finds Patterns

Performs Predictions

Query, Reporting, Analysis Data Mining

What Why

How

Comparative BenefitsPredictive Projects versus Nonpredictive Projects

0%

10%

20%

30%

40%

50%

60%

70%

80%

Technology Productivity Business ProcessEnhancement

Predictive Nonpredictive

Source: IDC, 2003

Data Mining terminology

• mining structure• mining model• mining algorithm• training dataset• testing dataset

SQL Server 2005 Algorithms

Decision Trees Clustering Time Series

Sequence Clustering

Association Naïve Bayes

Neural NetPlus: Linear and Logistic Regression

Sequence Clustering

• Applied to– Click stream analysis– Customer segmentation with

sequence data– Sequence prediction

• Mix of clustering and sequence technologies

• Group individuals based on their profiles including sequence data

Time Series

• Applied to– Forecast sales– Web hits prediction– Stock value estimation

• Patented technique from Microsoft Research

• Uses regression tree technology to describe and predict series values

Clustering• Applied to

– Segmentation: Customer grouping, Mailing campaign

– Also support classification and regression

• Expectation Maximization– Probabilistic Clustering

• K-Means– Distance based

• Clusters both discrete and continuous values– Discrete values are “binarized”

• Anomaly detection• Check variable independence

– “Predict Only” attributes not used for clustering

ClusteringDiscrete

Male Female

Son

Daughter

Parent

Age

ClusteringAnomaly Detection

Male Female

Son

Daughter

Parent

Age

dm data flow

Cube

HistoricalDataset

NewDataset

Data Transform (SSIS)Reporting

Mining Models

ModelBrowsing

Prediction

LOBApplication

Cube

the steps to a successful model

MS BOL

DMX CREATE MINING MODEL CreditRisk

(CustID LONG KEY,

Gender TEXT DISCRETE,

Income LONG CONTINUOUS,

Profession TEXT DISCRETE,

Risk TEXT DISCRETE PREDICT)

USING Microsoft_Decision_Trees

CREATE MINING MODEL CreditRisk

(CustID LONG KEY,

Gender TEXT DISCRETE,

Income LONG CONTINUOUS,

Profession TEXT DISCRETE,

Risk TEXT DISCRETE PREDICT)

USING Microsoft_Decision_Trees

INSERT INTO CreditRisk

(CustId, Gender, Income, Profession, Risk)

Select

CustomerID, Gender, Income, Profession,Risk

From Customers

INSERT INTO CreditRisk

(CustId, Gender, Income, Profession, Risk)

Select

CustomerID, Gender, Income, Profession,Risk

From Customers

Select NewCustomers.CustomerID, CreditRisk.Risk, PredictProbability(CreditRisk)

FROM CreditRisk PREDICTION JOIN NewCustomers

ON CreditRisk.Gender=NewCustomer.Gender

AND CreditRisk.Income=NewCustomer.Income

AND CreditRisk.Profession=NewCustomer.Profession

Select NewCustomers.CustomerID, CreditRisk.Risk, PredictProbability(CreditRisk)

FROM CreditRisk PREDICTION JOIN NewCustomers

ON CreditRisk.Gender=NewCustomer.Gender

AND CreditRisk.Income=NewCustomer.Income

AND CreditRisk.Profession=NewCustomer.Profession

Myths around data mining

• You have to be a propeller head

• It’s a new concept.• Only works with SSAS cubes

Excel 2007 DMAddin

• DM visualisation• table analysis• Create session models/permanent models• Connect to ssas for full blown models• intuitive interface

Demos

• setup• key Influencers• categories• Make a prediction• other sexy stuff

Resources

• Loads to be honest (DMX, API to name two things)

• Big Subject but very sexy

Contact Details

[email protected]