Probability & Statistical Inference Lecture 1

MSc in Computing (Data Analytics)

Probability & Statistical Inference Lecture 1

Lecture Outline Introduction

General Info Questionnaire

Introduction to Statistics Statistics at work The Analytics Process Descriptive Statistics & Distributions Graphs and Visualisation

Introduction Name : Aoife D’Arcy Email: [email protected] Bio: Managing Director and Chief Consultant at the Analytics

Store, has degrees in statistics, computer science, and financial & industrial mathematics. With over 12 years of experience in analytics consultancy with major national and international companies in banking, finance, insurance, manufacturing and gaming; I have developed particular expertise in risk analytics, fraud analytics, and customer insight analytics.

Lecture Notes: Will be available online on

www.comp.dit.ie/bmacnamee and later on webcourses

Course OutlineWeek Topic

1 Introduction to Statistics2 & 3 Probability Theory4 Introduction to SAS Enterprise Guide5 Probability Distributions6 Confidence Intervals7 & 8 Hypothesis testing9 Assignment10 - 12 Regression Analysis13 Revision

Exam & AssignmentExam The end of term exam accounts for 60% of

the overall mark

Assignment The assignment is worth 40% of the overall

mark. The assignment will be handed out in week 5 Week 9’s class will be dedicated to working

on the assignment.

Software SAS Enterprise Guide will be the

software that will be used during the course.

Applied Statistics and Probability for EngineersJohn Wiley & SonsDouglas C. Montgomery

Probability and Statistics for Engineers and ScientistsPearson EducationR.E. Walpole, R.H. Myers, S.L. Myers, K. Ye

Modelling Binary DataChapman & HallDavid Collett

Probability and Random Processes Oxford University PressG. Grimmett & D. Stirzaker

Statistical InferenceBrooks/ColeGeorge Casella

Recommended Reading

Questionnaire

Section 1: Statistics are everywhere

We are bombarded with Statistics

http://www.irishtimes.com/newspaper/frontpage/2012/0918/1224324122326.html http://www.irishtimes.com/newspaper/world/2012/0914/1224324008884.html http://www.independent.ie/business/world/survey-names-oslo-the-worlds-priciest-city-ireland-ranks-

27th-3229426.html

The internet is full of interesting statistics

http://www.usatoday.com/news/politics/twitter-election-meter

Statistics can be misleading An ad claimed:

“9 Out of 10 Dentists prefer Colgate” What is wrong with this statement?

Consider these complaints about airlines published in US News and World Report on February 5, 2001

Can we conclude the United airlines has the worst customer service?

Statistics in Everyday Life With the increase in the amount

of data available and advancement`s in the power of computers, statistics are being used more and more frequently

Question: Is it good that statistics are used so much

and what happens when statistics are misused?

Statistics can be misleading

Misinterpreted Statistics can be Devastating

In 1999 Sally Clarke was wrongly convicted of the murder of two of her sons. The case was widely criticised because of the way statistical evidence was misrepresented in the original trial, particularly by paediatrician Professor Sir Roy Meadow.

He claimed that, for an affluent non-smoking family like the Clarks, the probability of a single cot death was 1 in 8,543, so the probability of two cot deaths in the same family was around "1 in 73 million" (8543 × 8543).

What is wrong with this assumption?

Video http://www.youtube.com/watch?v=4TKbIi

dbyhk&feature=fvwrel

Challenges As an Analytics practitioner you will

face a number of challenges:

Create insight from all available data (and there is lots of it)

Interpret statistic correctly Communicate statistically driven

insight in a way that is clearly understood

Objective of this course

Give you a set of statistical skills to allow you, as an analytics practitioner, turn data into insight!!

The Analytics Process & Statistics

Section Overview Statistics and Analytics Introduction to CRISP

Data Analytics Is Multidisciplinary

Databases

StatisticsPatternRecognition

KDD

MachineLearning AI

Neurocomputing

Predictive Analytics

Data Warehousing

Analytics Process

Data Insight Business Decision

Analytics Is A Lot Of ThingsWhat’s the best that can happen?

What will happen next?

What if these trends continue?

Why is this happening?

What actions are needed?Where exactly is the problem?How many, how often, where?What happened?

Optimization

Predictive modellingForecasting/extrapolation

Statistical analysisAlerts

Query/drill down

Ad hoc reports

Standard reports

Com

petit

ive

adva

ntag

e

Degree of intelligence

Pred

ictiv

e An

alyt

icsAc

cess

&

repo

rting

For this course we will concentrate on Statistical Analysis

What’s the best that can happen?

What will happen next?

What if these trends continue?

Why is this happening?

What actions are needed?Where exactly is the problem?How many, how often, where?What happened?

Optimization

Predictive modellingForecasting/extrapolation

Statistical analysisAlerts

Query/drill down

Ad hoc reports

Standard reports

Com

petit

ive

adva

ntag

e

Degree of intelligence

Pred

ictiv

e An

alyt

icsAc

cess

&

repo

rting

CRISP-DM Evolution Over 200 members of the CRISP-DM SIG

worldwide DM Vendors: SPSS, NCR, IBM, SAS, SGI,

Data Distilleries, Syllogic, etc System Suppliers/Consultants: Cap Gemini,

ICL Retail, Deloitte & Touche, etc End Users: BT, ABB, Lloyds Bank, AirTouch,

Experian, etc Crisp-DM 2.0 is due…

Complete information on CRISP-DM is available at: http://www.crisp-dm.org/

CRISP-DM Features of CRISP-DM:

Non-proprietary Application/Industry neutral Tool neutral Focus on business issues

As well as technical analysis Framework for guidance Experience base

Templates for Analysis

Data

Business Understandin

g

Data Understandin

g

Data Preparation

Modelling

Evaluation

Deployment

Phases & Generic TasksBusiness

Understanding Data

Understanding Data

Preparation Modeling Deployment Evaluation

Determine Business Objectives

AssessSituation

DetermineData Mining

Goals

ProduceProject Plan

Business Understanding

This initial phase focuses on understanding the project objectives and requirements from a business perspective, then converting this knowledge into a data mining problem definition and a preliminary plan designed to achieve the objectives

Phases & Generic TasksBusiness

Understanding Data

Understanding Data

Preparation Modeling Deployment Evaluation

CollectInitialData

DescribeData

ExploreData

VerifyData

Quality

Data Understanding

The data understanding phase starts with an initial data collection and proceeds with activities in order to get familiar with the data, to identify data quality problems, to discover first insights into the data or to detect interesting subsets to form hypotheses for hidden information.

Phases & Generic TasksBusiness

Understanding Data

Understanding Data

Preparation Modeling Deployment Evaluation

SelectData

CleanData

ConstructData

IntegrateData

FormatData

Data Preparation

The data preparation phase covers all activities to construct the data that will be fed into the modelling tools from the initial raw data. Data preparation tasks are likely to be performed multiple times and not in any prescribed order. Tasks include table, record and attribute selection as well as transformation and cleaning of data for modelling tools.

Phases & Generic TasksBusiness

Understanding Data

Understanding Data

Preparation Modeling Deployment Evaluation

SelectModelingTechnique

GenerateTest Design

BuildModel

AssessModel

Modelling

In this phase, various modelling techniques are selected and applied and their parameters are calibrated to optimal values. Typically, there are several techniques for the same data mining problem type. Some techniques have specific requirements on the form of data. Therefore, stepping back to the data preparation phase is often necessary.

Phases & Generic TasksBusiness

Understanding Data

Understanding Data

Preparation Modeling Deployment Evaluation

EvaluateResults

ReviewProcess

DetermineNext Steps

Evaluation

Before proceeding to final deployment of a model, it is important to thoroughly evaluate it and review the steps executed to construct it to be certain it properly achieves the business objectives. A key objective is to determine if there is some important business issue that has not been sufficiently considered. At the end of this phase, a decision on the use of the data mining results should be reached.

Phases & Generic TasksBusiness

Understanding Data

Understanding Data

Preparation Modeling Deployment Evaluation

Plan Deployment

Plan Monitoring &

Maintenance

ProduceFinal

Report

ReviewProject

Deployment

Creation of a model is generally not the end of the project. Even if the purpose of the model is to increase knowledge of the data, the knowledge gained will need to be organized and presented in a way that the customer can use it. Depending on the requirements, the deployment phase can be as simple as generating a report or as complex as implementing a repeatable data mining process across the enterprise.

Crisp - DM Business Understanding

Data Understanding

Data Preparation

Modelling

Evaluation

Deployment.

Crisp – DM – Areas covered in this course

Business Understanding

Data Understanding

Data Preparation

Modelling

Evaluation

Deployment

Section 2: Descriptive Statistics & Distributions

Topics1. Introduction to Statistics2. The Basics 3. Measures of location: Mean, Median &

Mode.4. Measures of location & Skew.5. Measures of dispersion: range, standard

deviation (variance) & interquartile range.

Introduction to Statistics According to The Random House College

Dictionary, statistics is “the science that deals with the collection, classification, analysis and interpretation of numerical facts or data.” In short, statistics is the science of data.

There are two main branches of Statistics: The branch of statistics devoted to the

organisation, summarization and the description of data sets is called Descriptive Statistics.

The branch of statistics concerned with using sample data to make an inference about a large set of data is called Inferential Statistics.

Process of Data AnalysisPopulation

Representative

Sample

Sample Statistic

A Statistical population is a data set that is our target of interest.

A sample is a subset of data selected from the target population.

If your sample is not representative then it is referred to as being bias

Describe

Make Inferenc

e

Types of Data: Numeric Data Numeric data can be of two types:

Continuous Data: Data is continuous if it has an interval of real numbers for its range The number of centimetres of rain that fell in

March Discrete Data: Data is defined as discrete

if it has a finite range The number of correct answers in a 10 question

quiz

Types of Data: Categorical Data Data that is broken into discrete categories is

referred to as categorical data Categorical data has two main types:

Nominal: A nominal variable has a discrete number of categories or levels with no logical order Gender: Male, Female Working Status: Employed, Unemployed, Home-maker,

Student, Retired Ordinal: An ordinal variable has a discrete number

of categories or levels with a logical order Income Level: Low, Medium, High Places in a race: 1st, 2nd, 3rd, 4th, 5th, 6th

Class Task Task: Classify the type of each of the

data the following examples: The profit margin made from customers of

an online clothing company The type of interest rate you can be

charged on a mortgage i.e. Fixed rate, Adjustable rate

Number of dependents a associated with a loan applicant

Let’s Start at the Very Beginning When learning to read and write we start

with A-B-C, when starting to count we start with 1-2-3 and of course The Von Trappe family singers started with Do-Re-Me!

When learning statistics you start with the arithmetic mean or a simple average

The Arithmetic Mean

Year Canada China Germany* Russia** United Kingdom

United States

  Total Gold Total Gold Total Gold Total Gold Total Gold Total Gold1992 18 7 54 16 82 33 112 45 20 5 108 371996 22 3 50 16 65 20 63 26 15 1 101 442000 14 3 59 28 56 13 88 32 28 11 92 372004 12 3 63 32 49 13 92 27 30 9 103 362008 18 3 100 51 41 16 72 23 47 19 110 36Mean 17 4 65 28 58 19 85 30 28 9 102 38

The table below shows the total medals won and gold medals won by each country in the last 5 Olympic games

• * Germany combines East and West Germany prior to reunification ** Russia or The Soviet Union• Data source http://www.databaseolympics.com/index.htm

Arithmetic Mean – The Formula The formula for calculating the sample

arithmetic mean of n data points x1, x2 ..... xn:

x x i

1

n

n

:x is referred to as x-bar

Attributes of the Arithmetic Mean It is straight-forward to calculate It is easy to interpret the mean It gives us a good estimate of where

a set of numbers is centred This is referred to as the central tendency of a sample

It is sensitive to outliers

Other Measures of Central Tendency

Median: The middle value of an ordered set of values, i.e. 50% higher and 50% lower

Mode: The most commonly occurring value in a distribution

Calculating the MedianYear Medals1964 901968 1071972 941976 941980 01984 1741988 941992 1081996 1012000 922004 1032008 110

Medals (Sorted)

17411010810710310194949492900

Sort the data Median =

97.5

Calculating the Mode

Medals Count174 1110 1108 1107 1103 1101 194 392 10 1

Mode = 94

Year Medals1964 901968 1071972 941976 941980 01984 1741988 941992 1081996 1012000 922004 1032008 110

Count frequenci

es

When to Use Each Central Tendency Value?

Question: When and why would you use the median over the mean?

Let’s Look at the Variation in our Data

01 - 2

424 - 4

848 - 7

474 - 9

797 - 1

21

121 - 146

146 - 170

0

2

4

6

8

10

12

14

16

18

20

Distribution of the Total Olympic Medals won by any Country from 1964 - 2008

Coun

t

01 - 2

424 - 4

848 - 7

474 - 9

797 - 1

21

121 - 146

146 - 170

0

2

4

6

8

10

12

14

16

18

20

Distribution of the Total Olympic Medals won by any Country from 1964 - 2008

Coun

t

Let’s Look at the Variation in our Data

Central Tendency / Location

Spread/Variation

Measures of Spread or VariationRangeVarianceStandard DeviationInter-quartile Range

Calculating the Range The Range in calculated by

subtracting the minimum value in a data set from the maximum value

The main advantage to using the range is the ease with which it is calculated

The major disadvantage of the range is that it is highly sensitive to outliers

Calculating the Variance As an example of Variance consider the

following data:OBS Data

1 32 43 8

Sum 15Mea

n5

Calculating the Variance As an example of Variance consider the

following data:OBS Data Mean Deviatio

n1 3 5 -22 4 5 -13 8 5 3

Sum 15 15 0Mea

n5 15 0

Calculating the Variance As an example of Variance consider the

following data:OBS Data Mean Deviatio

n(Deviation

)2

1 3 5 -2 42 4 5 -1 13 8 5 3 9

Sum 15 15 0 14Mea

n5 15 0 4.67

Variance – The Formula Square the deviations around the mean

before summing. For n data points x1, x2 ..... xn:

Divide by n-1 (?) to get the average of squared deviations:

x i xn 2i1

n

s2 x i xn 2

i1

n

n 1

Standard Deviation – The Formula Take the square root of the variance.

The value is in the original unit

s x i xn 2

i1

n

n 1

Standard Deviation

Question: Why might it be useful to have the value is in the original unit?

Percentiles The nth percentile is a value that has a

proportion of the sample taking values at or lower than it, and taking values larger than it

Example: if your grade in an industrial engineering class was located at the 84th percentile, then 84% of the grades were equal to or lower than your grade and 16% were higher

n100

100 n100

Inter-quartile Range The median is the 50th percentile The 25th percentile and the 75th

percentile are called the lower quartile and upper quartile respectively (or 1st and 3rd)

The difference between the lower and upper quartile is called the inter-quartile range

Quartiles ExampleMedals (Sorted)

17411010810710310194949492900

Sort the data

25th Percentile= 1st Quartile = 93

50th Percentile= Median= 97.5

75th Percentile = 3rd Quartile = 107.5

Inter-quartile Range 107.5 – 93 = 14.5

Year Medals1964 901968 1071972 941976 941980 01984 1741988 941992 1081996 1012000 922004 1032008 110

Proportions The proportion, p, of items in a population

that belong to a certain class, for example: The proportion of your customers that are

female The proportion of voters that will vote for

Labour in the next election A proportion is calculated as:

where C is the number of items in a population of size N that belong to the class of interest

p CN

Skew – The Shape of a DistributionThere are a number of ways of describing the shape of a distribution.

We will consider only one – skew.

Skew is a measure of how asymmetric a distribution is.

Symmetric Distributions  = skew is zero

There are few very large data points which create a 'tail' going to the right (i.e. up the number line)

Note: No axis of symmetry here - skew > 0 (i.e. it is positive)

Example: Lifetime of people, house prices

Positive Skew

There are few very small data points which create a 'tail' going to the left (i.e. down the number line)

Note: No axis of symmetry here - skew < 0 (i.e. it is negative)

Examples: Examination Scores, reaction times for drivers

Negative Skew

Mean, Median & Mode are the same and are found in the middle

66

5 6 74 5 6 7 8

3 4 5 6 7 8 9

Mean = 102/17 = 6Median = 6Mode = 6

Skew & Measures of Location - Symmetry

ModeMedianMean

66

5 6 75 6 7 8 95 6 7 8 9 10 11

Mean = 121/17 = 7.12Median = 7Mode = 6

In general: Mode < Median < Mean

Positive Skew

ModeMedianMean

Mean = 83/17 = 4.89Median = 5Mode = 6

In general: Mode > Median > Mean

66

5 6 73 4 5 6 7

1 2 3 4 5 6 7

Negative Skew

Section 3: Graphs and Visualisation

Graphical Displays A way of letting people get a 'picture' of

relationships in the data set.

The simpler the better should be a rule in graphical display.

People can remember pictures better.

A good graph should show something that is not easy to ‘see’ using tables.

Bar Charts Used to display categorical data or

discrete data with a modest number of values.

A Bar is drawn to represent each category. The Bar height represents the frequency

or % in each category. Allows for visual comparison of relative

frequencies. Need to draw up a frequency distribution

table first.

Core Statistical Plots

4265.

375 88.75

112.12

5135

.5

158.87

5182

.25

205.62

5More

0

5

10

15

20

25

Points Scored by any Team in Six Nations Champi-

onship 2000 - 2011

Core Statistical Plots Comparisons Column Charts

Box Plots

Core Statistical Plots Correlations Scatter Plots

Trends(time)

Line Charts

Core Statistical Plots Proportions Pie Chart

Column Chart

Some Hans Inspiration to Finish UP http://www.youtube.com/watch?v=fTznEI

ZRkLg