Transcript
CE 459 Statistics
Assistant Prof. Muhammet Vefa AKPINAR
ExpectedNormal
VAR1
Upper Boundaries (x <= boundary)
No
of
ob
s
0
2
4
6
8
10
12
14
16
50 55 60 65 70 75 80 85 90 95 100
08.10.2011 2
Lecture Notes
What is Statistics
Frequency Distribution
Descriptive Statistics
Normal Probability Distribution
Sampling Distribution of the Mean
Simple Linear Regression & Correlation
Multiple Regression & Correlation
08.10.2011 3
INTRODUCTION
Criticism
There is a general perception that statistical knowledge is all-too-frequently intentionally misused, by finding ways to interpret the data that are favorable to the presenter.
(A famous quote, variously attributed, but thought to be from Benjamin Disraeli is: "There are three types of lies - lies, damn lies, and statistics.") Indeed, the well-known book How to Lie with Statistics by Darrell Huff discusses many cases of deceptive uses of statistics, focusing on misleading graphs. By choosing (or rejecting, or modifying) a certain sample, results can be manipulated; throwing out outliers is one means of doing so. This may be the result of outright fraud or of subtle and unintentional bias on the part of the researcher.
WHAT IS STATISTICS?
Definition
Statistics is a group of methods used to collect, analyze, present, and interpret data and to make decisions.
08.10.2011 5
What is Statistics ?
American Heritage Dictionary defines statistics as: "The mathematics of the collection, organization, and interpretation of numerical data, especially the analysis of population characteristics by inference from sampling."
The Merriam-Webster‟s Collegiate Dictionary definition is: "A branch of mathematics dealing with the collection, analysis, interpretation, and presentation of masses of numerical data."
The word statistics is also the plural of statistic (singular), which refers to the result of applying a statistical algorithm to a set of data, as in employment statistics, accident statistics, etc.
08.10.2011 6
In applying statistics to a scientific, industrial, or societal problem, one begins with a process or population to be studied. This might be a population of people in a country, of crystal grains in a rock, or of goods manufactured by a particular factory during a given period.
For practical reasons, rather than compiling data about an entire population, one usually instead studies a chosen subset of the population, called a sample.
Data are collected about the sample in an observational or experimental setting. The data are then subjected to statistical analysis, which serves two related purposes: description and inference.
08.10.2011 7
Descriptive statistics and Inferential statistics.
Statistical data analysis can be subdivided into Descriptive statistics and Inferential statistics.
Descriptive statistics is concerned with exploring, visualising, and summarizing data but without fitting the data to any models. This kind of analysis is used to explore the data in the initial stages of data analysis. Since no models are involved, it can not be used to test hypotheses or to make testable predictions. Nevertheless, it is a very important part of analysis that can reveal many interesting features in the data.
Descriptive statistics can be used to summarize the data, either numerically or graphically, to describe the sample. Basic examples of numerical descriptors include the mean and standard deviation. Graphical summarizations include various kinds of charts and graphs.
08.10.2011 8
Inferential statistics is the next stage in data analysis and involves the identification of a suitable model. The data is then fit to the model to obtain an optimal estimation of the model's parameters. The model then undergoes validation by testing either predictions or hypotheses of the model. Models based on a unique sample of data can be used to infer generalities about features of the whole population.
Inferential statistics is used to model patterns in the data, accounting for randomness and drawing inferences about the larger population. These inferences may take the form of answers to yes/no questions (hypothesis testing), estimates of numerical characteristics (estimation), forecasting of future observations, descriptions of association (correlation), or modeling of relationships (regression).
Other modeling techniques include ANOVA, time series, and data mining.
Population and sample.
Population
Sample
A portion of the population selected for study is referred to as a sample.
A population consists of all elements – individuals, items, or objects – whose characteristics are being studied. The population that is being studied is also called the target population.
Measures of Central Tendency
Mean
Sum of all measurements divided by the number of measurements.
Median:
A number such that at most half of the measurements are below it and at most half of the measurements are above it.
Mode:
The most frequent measurement in the data.
The central tendency of a dataset, i.e. the centre of a frequency distribution, is most commonly measured by the 3 Ms:
= arithmetic mean =average
Mean The Sample Mean ( ) is the arithmetic average of a data set.
It is used to estimate the population mean, ( .
Calculated by taking the sum of the observed values (yi) divided by the number of observations (n).
System $K
1 22.2
2 17.3
3 11.8
4 9.6
5 8.8
6 7.6
7 6.8
8 3.2
9 1.7
10 1.6
n
yyy
n
yn
n
i
i 211y
K06.9$10
6.13.172.22y
yi
yy - yi
Residual
= 9.06
y
Historical Transmogrifier
Average Unit Production Costs
08.10.2011 12
The Mode
The mode, symbolized by Mo, is the most frequently occurring score value. If the scores for a given sample distribution are:
32 32 35 36 37 38 38 39 39 39 40 40 42 45
then the mode would be 39 because a score of 39 occurs 3 times, more than any other score.
08.10.2011 13
A distribution may have more than one mode if the two most frequently occurring scores occur the same number of times. For example, if the earlier score distribution were modified as follows:
32 32 32 36 37 38 38 39 39 39 40 40 42 45
then there would be two modes, 32 and 39. Such distributions are called bimodal. The frequency polygon of a bimodal distribution is presented below.
Example of Mode
Measurements
x
3
5
1
1
4
7
3
8
3
Mode: 3
Notice that it is possible for a data not to have any mode.
Mode
The Mode is the value of the data set that occurs most frequently
Example:
1, 2, 4, 5, 5, 6, 8
Here the Mode is 5, since 5 occurred twice and no other value occurred more than once
Data sets can have more than one mode, while the mean and median have one unique value
Data sets can also have NO mode, for example:
1, 3, 5, 6, 7, 8, 9
Here, no value occurs more frequently than any other, therefore no mode exists
You could also argue that this data set contains 7 modes since each value occurs as frequently as every other
Example of Mode
Measurements
x
3
5
5
1
7
2
6
7
0
4
In this case the data have tow modes:
5 and 7
Both measurements are repeated twice
08.10.2011 17
Median
Computation of Median When there is an odd number of numbers, the median is simply the middle number. For example, the median of 2, 4, and 7 is 4. When there is an even number of numbers, the median is the mean of the two middle numbers. Thus, the median of the numbers 2, 4, 7, 12 is (4+7)/2 = 5.5.
Example of Median
Median: (4+5)/2 = 4.5
Notice that only the two central values are used in the computation.
The median is not sensible to extreme values
Measurements Measurements
Ranked
x x
3 0
5 1
5 2
1 3
7 4
2 5
6 5
7 6
0 7
4 7
40 40
median rim diameter (cm)
unit 1 unit 2
9.7 9.0
11.5 11.2
11.6 11.3
12.1 11.7
12.4 12.2
12.6 12.5
12.9 <-- 13.2 13.2
13.1 13.8
13.5 14.0
13.6 15.5
14.8 15.6
16.3 16.2
26.9 16.4
Median
The Median is the middle observation of an ordered (from low to high) data set
Examples:
1, 2, 4, 5, 5, 6, 8
Here, the middle observation is 5, so the median is 5
1, 3, 4, 4, 5, 7, 8, 8
Here, there is no “middle” observation so we take the average of the two observations at the center
5.42
54Median
Mode Median
Mean
Mode = Median = Mean
Dispersion Statistics
The Mean, Median and Mode by themselves are not sufficient descriptors of a data set
Example:
Data Set 1: 48, 49, 50, 51, 52
Data Set 2: 5, 15, 50, 80, 100
Note that the Mean and Median for both data sets are identical, but the data sets are glaringly different!
The difference is in the dispersion of the data points
Dispersion Statistics we will discuss are:
Range
Variance
Standard Deviation
Range
The Range is simply the difference between the smallest and largest observation in a data set
Example
Data Set 1: 48, 49, 50, 51, 52
Data Set 2: 5, 15, 50, 80, 100
The Range of data set 1 is 52 - 48 = 4
The Range of data set 2 is 100 - 5 = 95
So, while both data sets have the same mean and median, the dispersion of the data, as depicted by the range, is much smaller in data set 1
08.10.2011 24
deviation score
A deviation score is a measure of by how much each point in a frequency distribution lies above or below the mean for the entire dataset:
where: X = raw score X= the mean
Note that if you add all the deviation scores for a dataset together, you automatically get the mean for that dataset.
Variance
The Variance, s2, represents the amount of variability of the data relative to their mean
As shown below, the variance is the “average” of the squared deviations of the observations about their mean
1
)( 2
2
n
yys
i
The Variance, s2, is the sample variance, and is used to estimate the actual population variance, 2
N
yi
2
2)(
Standard Deviation
The Variance is not a “common sense” statistic because it describes the data in terms of squared units
The Standard Deviation, s, is simply the square root of the variance
1
)( 2
n
yys
i
The Standard Deviation, s, is the sample standard deviation, and is used to estimate the actual population standard deviation,
N
yi
2)(
Standard Deviation
The sample standard deviation, s, is measured in the same units as the data from which the standard deviation is being calculated
)($4.449
8.399
110
7.559.677.172
1
)(
2
2
2
K
n
yys
i
System FY97$K
1 22.2 13.1 172.7
2 17.3 8.2 67.9
3 11.8 2.7 7.5
4 9.6 0.5 0.3
5 8.8 -0.3 0.1
6 7.6 -1.5 2.1
7 6.8 -2.3 5.1
8 3.2 -5.9 34.3
9 1.7 -7.4 54.2
10 1.6 -7.5 55.7
Average 9.06
yyi
2)y(yi
)($67.6
)($4.44 22
K
Kss
This number, $6.67K, represents the average estimating error for predicting subsequent observations
In other words: On average, when estimating the cost of transmogrifiers that belongs to the same population as the ten systems above, we would expect to be off by $6.67K
08.10.2011 28
Variance and the closely-related standard deviation
The variance and the closely-related standard deviation are measures of how spread out a distribution is. In other words, they are measures of variability.
In order to define the amount of deviation of a dataset from the mean, calculate the mean of all the deviation scores, i.e. the variance.
The variance is computed as the average squared deviation of each number from its mean.
For example, for the numbers 1, 2, and 3, the mean is 2 and the variance is: .
08.10.2011 29
variance in a population is:
variance in a sample is:
where; μ is the mean and N is the number of scores.
08.10.2011 30
The standard deviation is the square root of the variance.
08.10.2011 31
Variance and Standar Deviation
Example of Mean
Measurements Deviation
x x - mean
3 -1
5 1
5 1
1 -3
7 3
2 -2
6 2
7 3
0 -4
4 0
40 0
MEAN = 40/10 = 4
Notice that the sum of the “deviations” is 0.
Notice that every single observation intervenes in the computation of the mean.
Example of Variance
Measurements Deviations Square of
deviations
x x - mean
3 -1 1
5 1 1
5 1 1
1 -3 9
7 3 9
2 -2 4
6 2 4
7 3 9
0 -4 16
4 0 0
40 0 54
Variance = 54/9 = 6
It is a measure of “spread”.
Notice that the larger the deviations (positive or negative) the larger the variance
The standard deviation
It is defines as the square root of the variance
In the previous example
Variance = 6
Standard deviation = Square root of the variance = Square root of 6 = 2.45
08.10.2011 35
Observed Vehicle velocity
velocity km/saat
67 73 81 72 76 75 85 77 68 84
76 93 73 79 88 73 60 93 71 59
74 62 95 78 63 72 66 78 82 75
96 70 89 61 75 95 66 79 83 71
76 65 71 75 65 80 73 57 88 78
08.10.2011 36
Mean, Median, Standard Deviation
Valid
Numbers Range Mean Median Minimum Maximum Variance Standard.Dev.
50 39 75,62 75 57 96 96,362 9,816458
08.10.2011 37
Frequency Table
Number class class frequency relative freq. Cumulative freq.
Relative cumulative
freq.
of Class (intervals) intervals midpoints % %
1 50,000 < x <= 55,000 52,5 0 0 0 0
2 55,000 < x <= 60,000 57,5 3 6 3 6
3 60,000 < x <= 65,000 62,5 5 10 8 16
4 65,000 < x <= 70,000 67,5 5 10 13 26
5 70,000 < x <= 75,000 72,5 14 28 27 54
6 75,000 < x <= 80,000 77,5 10 20 37 74
7 80,000 < x <= 85,000 82,5 5 10 42 84
8 85,000 < x <= 90,000 87,5 3 6 45 90
9 90,000 < x <= 95,000 92,5 4 8 49 98
10 95,000 < x <= 100,00 97,5 1 2 50 100
08.10.2011 38
Frequency Table
A cumulative frequency distribution is a plot of the number of observations falling in or below an interval. The graph shown here is a cumulative frequency distribution of the scores on a statistics test.
A frequency table is constructed by dividing the scores into intervals and counting the number of scores in each interval. The actual number of scores as well as the percentage of scores in each interval are displayed. Cumulative frequencies are also usually displayed.
The X-axis shows various intervals of vehicle speed.
08.10.2011 39
Selecting the Interval Size
In order to find a starting interval size the first step is to find the range of the data by subtracting the smallest score from the largest. In the case of the example data, the range was 96-57 = 39. The range is then divided by the number of desired intervals, with a suggested starting number of intervals being ten (10). In the example, the result would be 50/10 = 5. The nearest odd integer value is used as the starting point for the selection of the interval size.
08.10.2011 40
Histogram
A histogram is constructed from a frequency table. The intervals are shown on
the X-axis and the number of scores in each interval is represented by the
height of a rectangle located above the interval. A histogram of the vehicle
speed from the dataset is shown below. The shapes of histograms will vary
depending on the choice of the size of the intervals.
ExpectedNormal
VAR1
Upper Boundaries (x <= boundary)
No
of
ob
s
0
2
4
6
8
10
12
14
16
50 55 60 65 70 75 80 85 90 95 100
08.10.2011 41
There are many different-shaped frequency distributions:
08.10.2011 42
A frequency polygon is a graphical display of a frequency table. The intervals are shown on the X-axis and the number of scores in each interval is represented by the height of a point located above the middle of the interval. The points are connected so that together with the X-axis they form a polygon.
08.10.2011 43
Spread, Dispersion, Variability
A variable's spread is the degree to which scores on the variable differ from each other. If every score on the variable were about equal, the variable would have very little spread. There are many measures of spread. The distributions shown below have the same mean but differ in spread: The distribution on the bottom is more spread out. Variability and dispersion are synonyms for spread.
08.10.2011 44
Skew
Further Notes
When the Mean is greater than the Median the data distribution is skewed to the Right.
When the Median is greater than the Mean the data distribution is skewed to the Left.
When Mean and Median are very close to each other the data distribution is approximately symmetric.
08.10.2011 46
The distribution shown below has a positive skew. The mean is larger than the median.
test was very difficult and almost everyone in the class did very poorly on it,
the resulting distribution would most likely be positively skewed.
The Effect of Skew on the Mean and Median
08.10.2011 47
The distribution shown below has a negative skew. The mean is smaller than the median.
08.10.2011 48
Probability
Likelihood or chance of occurrence. The probability of an event is the theoretical relative frequency of the event in a model of the population.
08.10.2011 49
Normal Distribution or Normal Curve
Normal distribution is probably one of the most important and
widely used continuous distribution. It is known as a normal random variable, and its probability distribution is called a normal distribution.
The normal distribution is a theoretical function commonly used in inferential statistics as an approximation to sampling distributions. In general, the normal distribution provides a good model for a random variable.
08.10.2011 50
In a normal distribution:
68% of samples fall between ±1 SD
95% of samples fall between ±2 SD (actually + 1.96 SD)
99.7% of samples fall between ±3 SD
08.10.2011 51
The normal distribution function
The normal distribution function is determined by the following formula:
Where;
: mean : standard deviation e: Euler's constant (2.71...) : constant Pi (3.14...)
08.10.2011 52
Characteristics of the Normal Distribution:
It is bell shaped and is symmetrical about its mean. It is asymptotic to the axis, i.e., it extends indefinitely in either direction from the mean.
They are symmetric with scores more concentrated in the middle than in the tails.
It is a family of curves, i.e., every unique pair of mean and standard deviation defines a different normal distribution. Thus, the normal distribution is completely described by two parameters: mean and standard deviation.
There is a strong tendency for the variable to take a central value. It is unimodal, i.e., values mound up only in the center of the curve.
The frequency of deviations falls off rapidly as the deviations become larger.
08.10.2011 53
Total area under the curve sums to 1, the area of the distribution on each side of the mean is 0.5.
The Area Under the Curve Between any Two Scores is a PROBABILITY The probability that a random variable will have a value between any
two points is equal to the area under the curve between those points. Positive and negative deviations from this central value are equally likely
08.10.2011 54
Examples of normal distributions
Notice that they differ in how spread out they are. The area under each curve is the same. The height of a normal distribution can be specified mathematically in terms
of two parameters: the mean (μ) and the standard deviation (σ). The two
parameters, and , each change the shape of the distribution in a different
manner.
08.10.2011 55
Changes in without changes in
Changes in , without changes in , result in moving the distribution to the right or left, depending upon whether the new value of was larger or smaller than the previous value, but does not change the shape of the distribution.
08.10.2011 56
Changes in the value of
Changes in the value of , change the shape of the distribution without affecting the midpoint, because d affects the spread or the dispersion of scores. The larger the value of , the more dispersed the scores; the smaller the value, the less dispersed. The distribution below demonstrates the effect of increasing the value of :
08.10.2011 57
THE STANDARD NORMAL CURVE
The standard normal curve is a member of the family of normal curves with = 0.0 and = 1.0.
Note that the integral calculus is used to find the area under the normal distribution curve. However, this can be avoided by transforming all normal distribution to fit the standard normal distribution. This conversion is done by rescalling the normal distribution axis from its true units (time, weight, dollars, and...) to a standard measure called Z score or Z value.
08.10.2011 58
Standard Scores (z Scores)
A Z score is the number of standard deviations that a value, X, is away from the mean.
Standard scores are therefore useful for comparing datapoints in different distributions.
If the value of X is greater than the mean, the Z score is positive; if the value of X is less than the mean, the Z score is negative. The Z score or equation is as follows:
where z is the z-score for the value of X
08.10.2011 59
Table of the Standard Normal (z) Distribution
z 0.00 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09
0.0 0.0000 0.0040 0.0080 0.0120 0.0160 0.0190 0.0239 0.0279 0.0319 0.0359
0.1 0.0398 0.0438 0.0478 0.0517 0.0557 0.0596 0.0636 0.0675 0.0714 0.0753
0.2 0.0793 0.0832 0.0871 0.0910 0.0948 0.0987 0.1026 0.1064 0.1103 0.1141
0.3 0.1179 0.1217 0.1255 0.1293 0.1331 0.1368 0.1406 0.1443 0.1480 0.1517
0.4 0.1554 0.1591 0.1628 0.1664 0.1700 0.1736 0.1772 0.1808 0.1844 0.1879
0.5 0.1915 0.1950 0.1985 0.2019 0.2054 0.2088 0.2123 0.2157 0.2190 0.2224
0.6 0.2257 0.2291 0.2324 0.2357 0.2389 0.2422 0.2454 0.2486 0.2517 0.2549
0.7 0.2580 0.2611 0.2642 0.2673 0.2704 0.2734 0.2764 0.2794 0.2823 0.2852
0.8 0.2881 0.2910 0.2939 0.2969 0.2995 0.3023 0.3051 0.3078 0.3106 0.3133
0.9 0.3159 0.3186 0.3212 0.3238 0.3264 0.3289 0.3315 0.3340 0.3365 0.3389
1.0 0.3413 0.3438 0.3461 0.3485 0.3508 0.3513 0.3554 0.3577 0.3529 0.3621
1.1 0.3643 0.3665 0.3686 0.3708 0.3729 0.3749 0.3770 0.3790 0.3810 0.3830
1.2 0.3849 0.3869 0.3888 0.3907 0.3925 0.3944 0.3962 0.3980 0.3997 0.4015
1.3 0.4032 0.4049 0.4066 0.4082 0.4099 0.4115 0.4131 0.4147 0.4162 0.4177
1.4 0.4192 0.4207 0.4222 0.4236 0.4251 0.4265 0.4279 0.4292 0.4306 0.4319
1.5 0.4332 0.4345 0.4357 0.4370 0.4382 0.4394 0.4406 0.4418 0.4429 0.4441
1.6 0.4452 0.4463 0.4474 0.4484 0.4495 0.4505 0.4515 0.4525 0.4535 0.4545
1.7 0.4554 0.4564 0.4573 0.4582 0.4591 0.4599 0.4608 0.4616 0.4625 0.4633
1.8 0.4641 0.4649 0.4656 0.4664 0.4671 0.4678 0.4686 0.4693 0.4699 0.4706
1.9 0.4713 0.4719 0.4726 0.4732 0.4738 0.4744 0.4750 0.4756 0.4761 0.4767
2.0 0.4772 0.4778 0.4783 0.4788 0.4793 0.4798 0.4803 0.4808 0.4812 0.4817
2.1 0.4821 0.4826 0.4830 0.4834 0.4838 0.4842 0.4846 0.4850 0.4854 0.4857
2.2 0.4861 0.4864 0.4868 0.4871 0.4875 0.4878 0.4881 0.4884 0.4887 0.4890
08.10.2011 60
Three areas on a standard normal curve
08.10.2011 61
Z-1.5
Total - infinity to Z-
1.5
Z
Total Z to +
infinity
Z
Total - infinity to
Z
+Z
- Z
Area from -Z to +Z
+Z
- Z
-infinity to -Z
plus
+Z to +
infinity
+Z
- Z
-infinity to -Z
plus
+Z to +
infinity
Z Area Under Curve
from negative infinity
to Z
Area Under Curve
from Z to positive
infinity
Area Under Curve
from -Z to +Z
Area Under Curve
(negative infinity to -
Z) PLUS
(+Z to positive infinity)
Convert
(negative infinity to -Z
) PLUS
(+Z to positive infinity)
into PPM
Area Under Curve
negative infinity to Z-
1.5
0,000 0,50000000000000 0,50000000000000 0,00000000000000 1,00000000000000 1.000.000,00000000 0,06680720126886
0,100 0,53982783727702 0,46017216272298 0,07965567455403 0,92034432544597 920.344,32544597 0,08075665923377
0,200 0,57925970943909 0,42074029056091 0,15851941887818 0,84148058112182 841.480,58112182 0,09680048458561
0,300 0,61791142218894 0,38208857781106 0,23582284437788 0,76417715562212 764.177,15562212 0,11506967022170
0,400 0,65542174161031 0,34457825838969 0,31084348322063 0,68915651677937 689.156,51677937 0,13566606094638
0,500 0,69146246127400 0,30853753872600 0,38292492254801 0,61707507745200 617.075,07745200 0,15865525393145
0,600 0,72574688224992 0,27425311775008 0,45149376449983 0,54850623550017 548.506,23550017 0,18406012534675
0,700 0,75803634777692 0,24196365222308 0,51607269555384 0,48392730444617 483.927,30444617 0,21185539858339
0,800 0,78814460141659 0,21185539858341 0,57628920283319 0,42371079716681 423.710,79716681 0,24196365222306
0,900 0,81593987465323 0,18406012534677 0,63187974930647 0,36812025069354 368.120,25069354 0,27425311775006
1,000 0,84134474606854 0,15865525393146 0,68268949213707 0,31731050786293 317.310,50786293 0,30853753872598
1,100 0,86433393905361 0,13566606094639 0,72866787810722 0,27133212189278 271.332,12189278 0,34457825838967
1,200 0,88493032977829 0,11506967022171 0,76986065955657 0,23013934044343 230.139,34044343 0,38208857781104
1,300 0,90319951541439 0,09680048458562 0,80639903082877 0,19360096917123 193.600,96917123 0,42074029056089
1,400 0,91924334076622 0,08075665923378 0,83848668153245 0,16151331846755 161.513,31846755 0,46017216272296
08.10.2011 62
The area between Z-scores of -1.00 and +1.00. It is .68 or 68%.
The area between Z-scores of -2.00 and +2.00 and is .95 or 95%.
08.10.2011 63
Exercise 1
An industrial sewing machine uses ball bearings that are targeted to have a diameter of 0.75 inch. The specification limits under which the ball bearing can operate are 0.74 inch (lower) and 0.76 inch (upper). Past experience has indicated that the actual diameter of the ball bearings is approximately normally distributed with a mean of 0.753 inch and a standard deviation of 0.004 inch.
For this problem, note that "Target" = .75, and "Actual mean" = .753.
08.10.2011 64
What is the probability that a ball bearing will be between the target and the actual mean?
P(-0.75 < Z < 0) = .2734
08.10.2011 65
What is the probability that a ball bearing will be between the lower specification limit and the target?
P(-3.25 < Z < -0.75) = .49942 - .2734 = .22602
08.10.2011 66
What is the probability that a ball bearing will be above the upper specification limit?
P(Z > 1.75) = .5 - .4599 = .0401
08.10.2011 67
What is the probability that a ball bearing will be below the lower specification limit?
P (Z < -3.25) = .5 - .49942 = .00058
08.10.2011 68
Above which value in diameter will 93% of the ball bearings be?
The value asked for here will be the 7th percentile, since 93% of the ball bearings will have diameters above that. So we will look up .4300 in the Z-table in a "backwards“ manner. The closest area to this is .4306, which corresponds to a Z-value of 1.48.
-0.00592 = X - 0.753 X = 0.74708
So 0.74708 in. is the value that 93% of the diameters are above.
08.10.2011 69
Exercise 2
Graduate Management Aptitude Test (GMAT) scores are widely used by graduate schools of business as an entrance requirement. Suppose that in one particular year, the mean score for the GMAT was 476, with a standard deviation of 107. Assuming that the GMAT scores are normally distributed, answer the following questions:
08.10.2011 70
Question 1
What is the probability that a randomly selected score from this GMAT falls between 476 and 650 (476 <= x <= 650) the following figure shows a graphic representation of this problem.
Answer: Z = (650 - 476)/107 = 1.62. The Z value of 1.62 indicates that the GMAT score of 650 is 1.62 standard
deviation above the mean. The standard normal table gives the probability of value falling between 650 and the mean. The whole number and tenths place portion of the Z score appear in the first column of the table. Across the top of the table are the values of the hundredths place portion of the Z score. Thus the answer is that 0.4474 or 44.74% of the scores on the GMAT fall between a score of 650 and 476.
08.10.2011 71
Question 2. What is the probability of receiving a score greater than 750 on a GMAT test that
has a mean of 476 and a standard deviation of 107 i.e., P(X >= 750) = ?. Answer This problem is asking for determining the area of the upper tail of the distribution. The Z score is: Z = ( 750 - 476)/107 = 2.56- Table- P(Z=2.56) = 0.4948.
This is the probability of a GMAT with a score between 476 and 750. 0.5 - 0.4948 = 0.0052 or 0.52%. Note that P(X >= 750) is the same as P(X >750), because, in continuous
distribution, the area under an exact number such as X=750 is zero.
08.10.2011 72
What is the probability of receiving a score of 540 or less on a GMAT test that has a mean of 476 and a standard deviation of 107 i.e., P(X <= 540)= ?
we are asked to determine the area under the curve for all values less than or equal to 540.
z score (540-476)/107=0.6 -Table- P (z= 0.2257) which is the probability of getting a score between the mean 476 and 540.
The answer to this problem is: 0.5 + 0.2257 = 0.73 or 73%. Graphic representation of this problem.
08.10.2011 73
Question 4 What is the probability of receiving a score between 440 and 330 on a GMAT test that
has a mean of 476 and a standard deviation of 107. i.e., P(330 < 440)="?."
The two values fall on the same side of the mean.
The Z scores are: Z1 = (330 - 476)/107 = -1.36, and Z2 = (440 - 476)/107 = -0.34. The probability associated with Z = -1.36 is 0.4131,
The probability associated with Z = -0.34 is 0.1331.
Thee answer to this problem is: 0.4131 - 0.1331 = 0.28 or 28%.
08.10.2011 74
Standard Error (SE)
Any statistic can have a standard error. Each sampling distribution has a standard error.
Standard errors are important because they reflect how much sampling fluctuation a statistic will show, i.e. how good an estimate of the population the sample statistic is
How good an estimate is the mean of a population? One way to determine this is to repeat the experiment many times and to determine the mean of the means. However, this is tedious and frequently impossible.
SE refers to the variability of the sample statistic, a measure of spread for random variables
The inferential statistics involved in the construction of confidence intervals (CI) and significance testing are based on standard errors.
08.10.2011 75
Standard Error of the Mean, SEM, σM
The standard deviation of the sampling distribution of the mean is called the standard error of the mean.
The size of the standard error of the mean is inversely proportional to the square root of the sample size.
Not:
08.10.2011 76
The standard error of any statistic depends on the sample size - in general, the larger the sample size the smaller the standard error.
Note that the spread of the sampling distribution of the mean decreases as the sample size increases.
Notice that the mean of the distribution is not affected by sample size.
08.10.2011 77
Comparing the Averages of Two Independent Samples
Is there "grade inflation" in KTU? How does the average GPA of KTU students today compare with, say 10, years ago?
Suppose a random sample of 100 student records from 10 years ago yields a sample average GPA of 2.90 with a standard deviation of .40.
A random sample of 100 current students today yields a sample average of 2.98 with a standard deviation of .45.
The difference between the two sample means is 2.98-2.90 = .08. Is this proof that GPA's are higher today than 10 years ago?
08.10.2011 78
First we need to account for the fact that 2.98 and 2.90 are not the true averages, but are computed from random samples. Therefore, .08 is not the true difference, but simply an estimate of the true difference.
Can this estimate miss by much? Fortunately, statistics has a way of measuring the expected size of the ``miss'' (or error of estimation) . For our example, it is .06 (we show how to calculate this later). Therefore, we can state the bottom line of the study as follows: "The average GPA of KTU students today is .08 higher than 10 years ago, give or take .06 or so."
08.10.2011 79
Overview of Confidence Intervals
Once the population is specified, the next step is to take a random sample from it. In this example, let's say that a sample of 10 students were drawn and each student's memory tested. The way to estimate the mean of all high school students would be to compute the mean of the 10 students in the sample. Indeed, the sample mean is an unbiased estimate of μ, the population mean.
Clearly, if you already knew the population mean, there would be no need for a confidence interval.
08.10.2011 80
We are interested in the mean weight of 10-year old kids living in Turkey. Since it would have been impractical to weigh all the 10-year old kids in Turkey, you took a sample of 16 and found that the mean weight was 90 pounds. This sample mean of 90 is a point estimate of the population mean.
A point estimate by itself is of limited usefulness because it does not reveal the uncertainty associated with the estimate; you do not have a good sense of how far this sample mean may be from the population mean. For example, can you be confident that the population mean is within 5 pounds of 90? You simply do not know.
08.10.2011 81
Confidence intervals provide more information than point estimates.
An example of a 95% confidence interval is shown below:
72.85 < μ < 107.15
There is good reason to believe that the population mean lies between these two bounds of 72.85 and 107.15 since 95% of the time confidence intervals contain the true mean.
If repeated samples were taken and the 95% confidence interval computed for each sample, 95% of the intervals would contain the population mean. Naturally, 5% of the intervals would not contain the population mean.
08.10.2011 82
It is natural to interpret a 95% confidence interval as an interval with a 0.95 probability of containing the population mean
The wider the interval, the more confident you are that it contains the parameter. The 99% confidence interval is therefore wider than the 95% confidence interval and extends from 4.19 to 7.61.
08.10.2011 83
Example
Assume that the weights of 10-year old children are normally distributed with a mean of 90 and a standard deviation of 36. What is the sampling distribution of the mean for a sample size of 9?
standard deviation of 36/3 = 12. Note that the standard deviation of a sampling distribution is its standard error.
90 - (1.96)(12) = 66.48
90 + (1.96)(12) = 113.52
The value of 1.96 is based on the fact that 95% of the area of a normal distribution is within 1.96 standard deviations of the mean; 12 is the standard error of the mean.
08.10.2011 84
Figure shows that 95% of the means are no more than 23.52 units
(1.96x12) from the mean of 90.
Now consider the probability that a sample mean computed in a
random sample is within 23.52 units of the population mean of 90.
Since 95% of the distribution is within 23.52 of 90, the probability that
the mean from any given sample will be within 23.52 of 90 is 0.95.
This means that if we repeatedly compute the mean (M) from a
sample, and create an interval ranging from M - 23.52 to M +
23.52, this interval will contain the population mean 95% of the
time.
08.10.2011 85
notice that you need to know the standard deviation (σ) in order
to estimate the mean. This may sound unrealistic, and it is.
However, computing a confidence interval when σ is known is
easier than when σ has to be estimated, and serves a
pedagogical purpose.
Suppose the following five were sampled from a normal
distribution with a standard deviation of 2.5: 2, 3, 5, 6, and 9. To
compute the 95% confidence interval, start by computing the
mean and standard error:
M = (2 + 3 + 5 + 6 + 9)/5 = 5. σm = = 1.118.
08.10.2011 86
Z.95 --the value is 1.96.
08.10.2011 87
If you had wanted to compute the 99% confidence interval, you would have set the shaded area to 0.99 and the result would have been 2.58.
The confidence interval can then be computed as follows:
Lower limit = 5 - (1.96)(1.118)= 2.81
Upper limit = 5 + (1.96)(1.118)= 7.19
08.10.2011 88
Estimating the Population Mean Using Intervals
Estimate the average GPA of the population of approximately 23000 KTU undergraduates.n=25 randomly selected students, sample average= 3.05.
Consider estimating the population average
Now chances are the true average is not equal to 3.05.
True KTU average GPA is between 1.00 and 4.00, and with high confidence between (2.50, 3.50); but what level of confidence do we have that it is between say, (2.75, 3.25) or (2.95, 3.15)?
Even better, can we find an interval (a, b) which will contain with 95%
certainty?
08.10.2011 89
Example:
Given the following GPA for 6 students: 2.80, 3.20, 3.75, 3.10, 2.95, 3.40
Calculate a 95% confidence interval for the population mean GPA.
08.10.2011 90
Determining Sample Size for Estimating the Mean
want to estimate the average GPA of KTU undergraduates this school year. Historically, the SD of student GPA is known to be .
If a random sample of size n=25 yields a sample mean of , then the population mean is estimated as lying within the interval
with 95% confidence. The plus-or-minus quantity .12 is called the margin of error of the sample mean associated with a 95% confidence level. It is also correct to say ``we are 95% confident that is within .12 of the sample mean 3.05''.
Confidence Interval for μ, Standard Deviation Estimated
It is very rare for a researcher wishing to estimate the mean of a population to already know its standard deviation. Therefore, the construction of a confidence interval almost always involves the estimation of both μ and σ. When σ is known -> M - zσM ≤ μ ≤ M + zσM is used for a confidence interval.
When σ is not known, Whenever the standard deviation is estimated (NOT KNOWN), the t rather than the normal (z) distribution should be used. for μ when σ is estimated is: M - t sM ≤ μ ≤ M + t sM where M is the sample mean, sM is an estimate of σM (standard error), and t depends on the degrees of freedom and the level of confidence.
confidence interval on the mean:
More generally, the formula for the 95% confidence interval on the mean is:
Lower limit = M - (t)(sm) Upper limit = M + (t)(sm)
where;
M is the sample mean, t is the t for the confidence level desired (0.95 in the above example), and sm is the estimated standard error of the mean.
A comparison of the t and normal distribution
A comparison of the t distribution with 4 df
(in blue) and the standard normal
distribution (in red).
Finding t-values
Find the t-value such that the area under the t distribution to the right of the t-value is 0.2 assuming 10 degrees of freedom. That is, find t0.20 with 10 degrees of freedom.
Upper tail probability p (area under the right side)
Example:
P[t(2) > 2.92] = 0.05
P[-2.92 < t(2) < 2.92] = 0.9
50% 60% 70% 80% 90% 95% 96% 98% 99% 99.5% 99.8% 99.9%
0.25 0.2 0.15 0.1 0.05 0.025 0.02 0.01 0.005 0.0025 0.001 0.0005
df
1 1.000 1.376 1.963 3.078 6.314 12.706 15.895 31.821 63.657 127.32 318.30 636.61
2 0.817 1.061 1.386 1.886 2.920 4.303 4.849 6.965 9.925 14.089 22.327 31.599
3 0.765 0.979 1.250 1.638 2.353 3.182 3.482 4.541 5.841 7.453 10.215 12.924
4 0.741 0.941 1.190 1.533 2.132 2.776 2.999 3.747 4.604 5.598 7.173 8.610
5 0.727 0.920 1.156 1.476 2.015 2.571 2.757 3.365 4.032 4.773 5.893 6.869
6 0.718 0.906 1.134 1.440 1.943 2.447 2.612 3.143 3.707 4.317 5.208 5.959
7 0.711 0.896 1.119 1.415 1.895 2.365 2.517 2.998 3.499 4.029 4.785 5.408
8 0.706 0.889 1.108 1.397 1.860 2.306 2.449 2.896 3.355 3.833 4.501 5.041
9 0.703 0.883 1.100 1.383 1.833 2.262 2.398 2.821 3.250 3.690 4.297 4.781
10 0.700 0.879 1.093 1.372 1.812 2.228 2.359 2.764 3.169 3.581 4.144 4.587
11 0.697 0.876 1.088 1.363 1.796 2.201 2.328 2.718 3.106 3.497 4.025 4.437
12 0.696 0.873 1.083 1.356 1.782 2.179 2.303 2.681 3.055 3.428 3.930 4.318
13 0.694 0.870 1.079 1.350 1.771 2.160 2.282 2.650 3.012 3.372 3.852 4.221
14 0.692 0.868 1.076 1.345 1.761 2.145 2.264 2.624 2.977 3.326 3.787 4.140
15 0.691 0.866 1.074 1.341 1.753 2.131 2.249 2.602 2.947 3.286 3.733 4.073
Abbreviated t table
df
0.95
0.99
2 4.303 9.925
3 3.182 5.841
4 2.776 4.604
5 2.571 4.032
8 2.306 3.355
10 2.228 3.169
20 2.086 2.845
50 2.009 2.678
100 1.984 2.626
Example
Assume that the following five numbers are sampled from a normal distribution: 2, 3, 5, 6, and 9 and that the standard deviation is not known. The first steps are to compute the sample mean and variance: M = 5 sm = 7.5 Standard error (sm)= 1.225
df = N - 1 = 4
t t tablethe value for the 95% interval for is
2.776.
Lower limit = 5 - (2.776)(1.225)= 1.60 Upper limit = 5 + (2.776)(1.225)= 8.40
Example
Suppose a researcher were interested in estimating the mean reading speed (number of words per minute) of high-school graduates and computing the 95% confidence interval. A sample of 6 graduates was taken and the reading speeds were: 200, 240, 300, 410, 450, and 600. For these data, M = 366.6667 sM= 60.9736 df = 6-1 = 5 t = 2.571
lower limit is: M - (t) (sM) = 209.904
upper limit is: M + (t) (sM) = 523.430,
95% confidence interval is: 209.904 ≤ μ ≤ 523.430
Thus, the researcher can conclude based on the rounded off 95% confidence interval that the mean reading speed of high-school graduates is between 210 and 523.
Homework 1
The mean time difference for all 47 subjects is 16.362 seconds and the standard deviation is 7.470 seconds. The standard error of the mean is 1.090.
A t table shows the critical value of t for 47 - 1 = 46 degrees of freedom is 2.013 (for a 95% confidence interval). The confidence interval is computed as follows:
Lower limit = 16.362 - (2.013)(1.090)= 14.17 Upper limit = 16.362 + (2.013)(1.090)= 18.56
Therefore, the interference effect (difference) for the whole population is likely to be between 14.17 and 18.56 seconds.
Homework 2
The pasteurization process reduces the amount of bacteria found in dairy products, such as milk. The following data represent the counts of bacteria in pasteurized milk (in CFU/mL) for a random sample of 12 pasteurized glasses of milk.
Construct a 95% confidence interval for the bacteria count.
NOTE: Each observation is in tens of thousand. So, 9.06 represents 9.06 x 104.
Prediction with Regression Analysis
The relationship(s) between values of the response variable and corresponding values of the predictor variable(s) is (are) not deterministic.
Thus the value of y is estimated given the value of x. The estimated value of the dependent variable is denoted y, and the population slope and intercept are usually denoted β1 and β0.
Linear Regression
The idea is to fit a straight line through data points
Linear Regression - Indicates that the relationship(s) between the dependent variable and the independent variable(s).
Can extend to multiple dimensions
correlation analysis is applied to independent factors: if X increases, what will Y do (increase, decrease, or perhaps not change at all)?
In regression analysis a unilateral response is assumed: changes in X result in changes in Y, but changes in Y do not result in changes in X.
0.1 0.0-0.1-0.2
0.4
0.3
0.2
0.1
0.0
-0.1
-0.2
-0.3
vwmkt
m1
S = 0.0590370 R-Sq = 31.3 % R-Sq(adj) = 30.8 %
m1 = 0.0095937 + 0.880436 vwmkt
Regression Plot
Linear regression means a regression that is linear in the parameters
A linear regression can be non-linear in the variables
Example: Y = β0 + β1X2
Some non-linear regression models can be transformed
into a linear regression model
(e.g., Y=aXbZc can be transformed into
lnY = ln a + b*ln X + c*ln Z)
Example
Given one variable
Goal: Predict Y
Example: Given Years of
Experience
Predict Salary
Questions: When X=10, what is Y?
When X=25, what is Y?
This is known as regression
X
(years)
Y (salary, $1,000)
3 30
8 57
9 64
13 72
3 36
6 43
11 59
21 90
1 20
For the example data
xy 5.32.23
5.3
,2.23
x=10 years prediction of y (salary) is:
23.2+35=58.2 K dollars/year.
Linear Regression Example Linear Regression: Y=3.5*X+23.2
0
20
40
60
80
100
120
0 5 10 15 20 25
Years
Sal
ary
XY
xy
xx
yyxx
i
i
i
ii
2)(
))((
Regression Error
We can also write a regression equation slightly differently:
Also called the residual, this is the difference between our estimate of the value of
the dependent variable y and the actual value of the dependent variable y.
Unless we have perfect prediction, many of the y values will fall off of the line. The added e in the equation refers to this fact. It would be incorrect to write the equation without the e, because it would suggest that the y scores are completely accounted for by just knowing the slope, x values, and the intercept. Almost always, that is not true. There is some error in prediction, so we need to add an e for error variation into the equation.
The actual values of y can be accounted for by the regression line equation (y=a+bx) plus some degree of error in our prediction (the e's).
r correlation coefficient
The correlation between X and Y is expressed by the correlation coefficient r :
xi = data X, ¯x = mean of data X yi = data Y, ¯y = mean of data Y
1 >r > -1
r = 1 perfect positive linear correlation between two variables
r = 0 no linear correlation (maybe other correlation) r = -1 perfect negative linear correlation
Notice that for the perfect correlation, there is a perfect line of points. They do not deviate from that line.
least squares
The principle is to establish a statistical linear relationship between two sets of corresponding data by fitting the data to a straight line by means of the "least squares" technique.
The resulting line takes the general form: y = bx + a
a = intercept of the line with the y-axis
b = slope (tangent)
a = 0, b= 1 perfect positive correlation without bias a= 0 systematic discrepancy (bias, error) between X and Y; b = 1 proportional response or difference between X and Y.
Example
Each point represents one student with a certain score for time on the exam, x, and grade, y. The scatter plot reveals that, in general, longer times on the exam tend to be associated with higher grades.
0.64
ID Grade on
Exam (x)
Time on
Exam (y)
X-X avr Y-Yavr (X-Xavr)*(Y-Yavr) (X-Xavr)2
1 88 60 8.6 18.55 159.53 73.96
2 96 53 16.6 11.55 191.73 275.56
3 72 22 -7.4 -19.45 143.93 54.76
4 78 44 -1.4 2.55 -3.57 1.96
5 65 34 -14.4 -7.45 107.28 207.36
6 80 47 0.6 5.55 3.33 0.36
7 77 38 -2.4 -3.45 8.28 5.76
8 83 50 3.6 8.55 30.78 12.96
9 79 51 -0.4 9.55 -3.82 0.16
10 68 35 -11.4 -6.45 73.53 129.96
11 84 46 4.6 4.55 20.93 21.16
12 76 36 -3.4 -5.45 18.53 11.56
13 92 48 12.6 6.55 82.53 158.76
r correlation
The Pearson r can be positive or negative, ranging from -1.0 to
1.0. If the correlation is 1.0, the longer the amount of time spent on
the exam, the higher the grade will be--without any exceptions. An r value of -1.0 indicates a perfect negative correlation--
without an exception, the longer one spends on the exam, the poorer the grade.
If r=0, there is absolutely no relationship between the two variables. When r=0, on average, longer time spent on the exam does not result in any higher or lower grade. Most often r is somewhere in between -1.0 and +1.0.
ID Grade on Exam (x) x2 Time on Exam (y) y2 xy
1 88 7744 60 3600 5280
2 96 9216 53 2809 5088
3 72 5184 22 484 1584
4 78 6084 44 1936 3432
5 65 4225 34 1156 2210
6 80 6400 47 2209 3760
7 77 5929 38 1444 2926
8 83 6889 50 2500 4150
9 79 6241 51 2601 4029
10 68 4624 35 1225 2380
11 84 7056 46 2116 3864
12 76 5776 36 1296 2736
13 92 8464 48 2304 4416
14 80 6400 43 1849 3440
15 67 4489 40 1600 2680
16 78 6084 32 1024 2496
17 74 5476 27 729 1998
18 73 5329 41 1681 2993
19 88 7744 39 1521 3432
20 90 8100 43 1849 3870
S 1588 127454 829 35933 66764
ID Grade on
Exam (x)
Time on
Exam (y)
X-X ort Y-Yort (X-Xort)*(Y-Yort) (X-Xort)2 (Y-Yort)
2
1 88 60 8,6 18,55 159,53 73,96 344,1025
2 96 53 16,6 11,55 191,73 275,56 133,4025
3 72 22 -7,4 -19,45 143,93 54,76 378,3025
4 78 44 -1,4 2,55 -3,57 1,96 6,5025
5 65 34 -14,4 -7,45 107,28 207,36 55,5025
6 80 47 0,6 5,55 3,33 0,36 30,8025
7 77 38 -2,4 -3,45 8,28 5,76 11,9025
8 83 50 3,6 8,55 30,78 12,96 73,1025
9 79 51 -0,4 9,55 -3,82 0,16 91,2025
10 68 35 -11,4 -6,45 73,53 129,96 41,6025
11 84 46 4,6 4,55 20,93 21,16 20,7025
12 76 36 -3,4 -5,45 18,53 11,56 29,7025
13 92 48 12,6 6,55 82,53 158,76 42,9025
14 80 43 0,6 1,55 0,93 0,36 2,4025
15 67 40 -12,4 -1,45 17,98 153,76 2,1025
16 78 32 -1,4 -9,45 13,23 1,96 89,3025
17 74 27 -5,4 -14,45 78,03 29,16 208,8025
18 73 41 -6,4 -0,45 2,88 40,96 0,2025
19 88 39 8,6 -2,45 -21,07 73,96 6,0025
20 90 43 10,6 1,55 16,43 112,36 2,4025
Total 1588 829 941,4 1366,8 1570,95
Average 79,4 41,45
r = 0.6424
r2 square of the correlation coefficient
r² is the proportion of the sum of squares explained in one-variable regression,
r² is the proportion of the sum of squares explained in multiple regression.
Is an R-Square < 1.00 Good or bad?
This is both a statistical and a philosophical question; It is quite rare, especially in the social sciences, to get an R-square that is really high (e.g., 98%).
The goal is NOT to get the highest R-square per se. Instead, the goal is to develop a model that is both statistically and theoretically sound, creating the best fit with existing data.
Do you want just the best fit, or a model that theoretically/conceptually makes sense? Yes, you might get a good fit with nonsensical explanatory variables. But, this opens you to spurious/intervening relationships. THEREFORE: hard to use model for explanation.
Why might an R-Square be less than 1.00?
underdetermined model (need more variables) nonlinear relationships measurement error sampling error not fully predictable/explainable even with all data
available; there is a certain amount of unexplainable chaos/static/randomness in the universe (which may be reassuring)
the unit of analysis is too aggregated (e.g., you are predicting mean housing values for a city -- you might get better results with predicting individual housing prices, or neighborhood housing prices).
Adjusted R2 (R-square)
What is an "Adjusted" R-Square? The Adjusted R-Square takes into account not only how much of the variation is explained, but also the impact of the degrees of freedom. It "adjusts" for the number of variables use. That is, look at the adjusted R- Square to see how adding another variable to the model both increases the explained variance but also lowers the degrees of freedom. Adjusted R2 = 1- (1 - R2 )((n - 1)/(n - k - 1)). As the number of variables in the model increases, the gap between the R-square and the adjusted R-square will increase. This serves as a disincentive to simply throwing in a huge number of variables into the model to increase the R-square.
This adjusted value for R-square will be equal or smaller than the regular R-square. The adjusted R-square adjusts for a bias in R-square. R-square tends to over estimate the variance accounted for compared to an estimate that would be obtained from the population. There are two reasons for the overestimate, a large number of predictors and a small sample size.
So, with a small sample and with few predictors, adjusted R-square should be very similar to the R-square value. Researchers and statisticians differ on whether to use the adjusted R-square. It is probably a good idea to look at it to see how much your R-square might be inflated, especially with a small sample and many predictors.
Example
Suppose we have collected the following sample of 6 observations on age and income:
Find the estimated regression line for the sample of six observations we have collected on age and income:
Which is the independent variable and which is the dependent variable for this problem?
Cautions About Simple Linear Regression
Correlation and regression describe only linear relations Correlation and least-squares regression line are not resistant to
outliers Predictions outside the range of observed data are often
inaccurate Correlation and regression are powerful tools for describing
relationship between two variables, but be aware of their limitations
Multiple Prediction
Regression analysis allows us to use more than one independent variable to predict values of y. Take the fat intake and blood cholesterol level study as an example. If we want to predict cholesterol as accurately as possible, we need to know more about diet than just how much fat intake there is.
On the island of Crete, they consume a lot of olive oil, so there fat intake is high. This, however, seems to have no dramatic affect on cholesterol (at least the bad cholesterol, the LDLs). They also consume very little cholesterol in their diet, which consists more of fish than high cholesterol foods like cheese and beef (hopefully this won't be considered libelous in Texas). So, to improve our prediction of blood cholesterol levels, it would be helpful to add another predictor, dietary cholesterol.
From Bivariate to Multiple regression: what changes?
potentially more explanatory power with more variables.
the ability to control for other variables: and one sees the interaction of the various explanatory variables. partial correlations and multicollinearity.
harder to visualize drawing a line through three+ n-dimensional space.
the R is no longer simply the square of the correlation statistic r.
From Two to Three Dimensions With simple regression (one predictor) we had only the x-axis and the y-axis. Now we need an axis for x1, x2, and y.
where Y' is the predicted score, X1 is the score on the first predictor variable, X2 is the score on the second, etc. The Y intercept is A. The regression coefficients (b1, b2, etc.) are analogous to the slope in simple regression.
If we want to predict these points, we now need a regression plane rather than just a regression line. That looks something like this:
More than one prediction attribute
X1, X2
For example,
X1=„years of experience‟
X2=„age‟
Y=„salary‟
2211 xxY
x1
x2
y
0=10
0
(xi1, xi2)
E(yi)
yi
i
Response Surface
The parameters β0, β1, β2,… , βk are called partial regression coefficients.
β1 represents the change in y corresponding to a unit increase in x1, holding all the other predictors constant.
A similar interpretation can be made for β2, β3, ……, βk
Regression Statistics
Multiple R 0,995
R Square 0,990
Adjusted R Square 0,989
Standard Error 0,008
Observations 30
ANOVA
df SS MS F
Significa
nce F
Regression 4 0,164 0,041 628,372 0,000
Residual 25 0,002 0,000
Total 29 0,165
Coefficie
nts
Standard
Error t Stat P-value
Intercept 0,500 0,008 60,294 0,000
Percent of Gross Hhd Income Spent on rent -0,399 0,016 -24,610 0,000
percent 2-parent families -0,288 0,015 -19,422 0,000
Police Anti-Drug Program? -0,004 0,004 -1,238 0,227
Active Tenants Group? (1 = yes; 0 = no) -0,102 0,004 -28,827 0,000
Controlling also for this new variable, the police anti-drug program is no
longer statistically significant, an instead the presence of the active
tenants group makes the dramatic difference. (and look at that great R
square!). However, we are no quite done…
SUMMARY OUTPUT
Regression Statistics
Multiple R 0.928
R Square 0.861
Adjusted R Square 0.850
Standard Error 0.030
Observations 30
ANOVA
df SS MS F Significance F
Regression 2 0.149 0.074 83.484 0.000
Residual 27 0.024 0.001
Total 29 0.173
Coeffici
ents
Standard
Error t Stat P-value BETA
Intercept 0.36582 0.017 20.908 0.000
percent 2-parent
families -0.2565 0.051 -5.017 0.000 -0.362
Active Tenants Group?
(1 = yes; 0 = no) -0.1246 0.011 -11.347 0.000 -0.821
Since the police variable now has a statistically insignificant t-score, we remove it
from the model. (We also remove the income variable, since it also becomes
insignificant after we remove the police variable.) We are left with two independent
variables: percent of 2-parent families and active tenants group.
Stepwise Regression Algorithms
• Backward Elimination
• Forward Selection
• Stepwise Selection
Backward Elimination
1. Fit the model containing all (remaining)
predictors.
2. Test each predictor variable, one at a
time, for a significant relationship with y.
3. Identify the variable with the largest pvalue.
If p > α, remove this variable from
the model, and return to (1.).
4. Otherwise, stop and use the existing
model.
Forward Selection
1. Fit all models with one (more) predictor.
2. Test each of these predictor variables,
for a significant relationship with y.
3. Identify the variable with the smallest pvalue.
If p < α, add this variable to the
model, and return to (1.).
4. Otherwise, stop and use the existing
model.
Stepwise Selection
• The Stepwise Selection method is
basically Forward Selection with Backward
Elimination added in at every step.
Stepwise Selection 1. Fit all models with one (more) predictor. 2. Test each of these predictor variables, for a significant relationship with y. 3. Identify the variable with the smallest p-value. If p < α, add this variable to the model, and return to (1.). 4. Now, for the model being considered, test each predictor variable, one at a time, for a significant relationship with y. 5. Identify the variable with the largest p-value. If p > α, remove this variable from the model, and return to (1.). 6. Otherwise, stop and use the existing model.
Linear regression
Review
Multiple Regression Models
Chapter Topics
The Multiple Regression Model
Contribution of Individual Independent Variables
Coefficient of Determination
Categorical Explanatory Variables
Transformation of Variables
Violations of Assumptions
Qualitative Dependent Variables
Multiple Regression Models
MultipleRegression
Models
LinearDummy
Variable
LinearNon-
Linear
Inter-action
Poly-
Nomial
SquareRoot
Log Reciprocal Exponential
Linear Multiple Regression Model
Additional Assumption for Multiple Regression
No exact linear relation exists between any subset of explanatory variables (perfect
"multicollinearity")
The Multiple Regression Model
ipipiii XXXY 22110
Relationship between 1 dependent & 2 or more independent variables is a linear
function Population
Y-intercept Population slopes
Dependent (Response)
variable for sample
Independent (Explanatory)
variables for sample model
Random
Error
ipipiii eXbXbXbbY 22110
Population Multiple Regression Model
X2
Y
X1
YX =
0 +
1X
1i +
2X
2i
0
Yi =
0 +
1X
1i +
2X
2i +
i
Response
Plane
(X1i
,X2i
)
(Observed Y)
i
Bivariate model
Sample Multiple Regression Model
X2
Y
X1
b0
Yi = b
0 + b
1X
1i + b
2X
2i + e
i
Response
Plane
(X1i
,X2i
)
(Observed Y)
^
ei
Yi = b
0 + b
1X
1i + b
2X
2i
Bivariate model
Parameter Estimation
Linear Multiple Regression Model
O il (G a l) T e m p In su la tio n
275.30 40 3
363.80 27 3
164.30 40 10
40.80 73 6
94.30 64 6
230.90 34 6
366.70 9 6
300.60 8 10
237.80 23 10
121.40 63 3
31.40 65 10
203.50 41 6
441.10 21 3
323.00 38 3
52.50 58 10
Multiple Regression Model: Example
(0F)
Develop a model for estimating
heating oil used for a single
family home in the month of
January based on average
temperature and amount of
insulation in inches.
Interpretation of Estimated Coefficients
Slope (bP)
Estimated Y changes by bP for each 1 unit increase in XP holding all other variables constant (ceterus paribus) Example: If b1 = -2, then fuel oil usage (Y) is
expected to decrease by 2 gallons for each 1 degree increase in temperature (X1) given the inches of insulation (X2)
Y-Intercept (b0) Average value of Y when all XP = 0
Sample Regression Model: Example
C o e ffic ie n ts
I n te r c e p t 5 6 2 . 1 5 1 0 0 9 2
X V a r i a b l e 1 -5 . 4 3 6 5 8 0 5 8 8
X V a r i a b l e 2 -2 0 . 0 1 2 3 2 0 6 7
iii X.X..Y 21 012204375151562
For each degree increase in
temperature, the average amount of
heating oil used is decreased by 5.437
gallons, holding insulation constant.
For each increase in one inch of
insulation, the use of heating oil is
decreased by 20.012 gallons,
holding temperature constant.
Evaluating the Model
Evaluating Multiple Regression Model Steps
Examine variation measures
Test parameter significance
Overall model
Portions of model
Individual coefficients
Variation Measures
Coefficient of Multiple Determination
r2Y.12..P = Explained variation = SSR
Total variation SST
r2=0 all the variables taken together do
not explain variation in Y
NOT proportion of variation in Y „explained‟ by all X variables taken together
Reflects
Sample size
Number of independent variables
Smaller than r2Y.12..P
Sometimes used to compare models
Adjusted Coefficient of Multiple Determination
Simple and Multiple Regression Compared:Example
Two simple regressions:
ABSENCES= + 1AUTONOMY
ABSENCES= + 2SKILLVARIETY
Multiple Regression:
ABSENCES= + 1AUTONOMY+
2SKILLVARIETY
Overlap in Explanation
SIMPLE REGRESSION: AUTONOMY MULTIPLE REGRESSION
Multiple R 0,169171 Multiple R 0,231298
R Square 0,028619 R Square 0,053499
Adjusted R Square0,027709 Adjusted R Square0,051723
Standard Error 12,443 Standard Error12,28837
Observations 1069 Observations 1069
ANOVA ANOVA
df SS MS F Significance F df SS MS F
Regression 1 4867,198 4867,198 31,43612 2,62392E-08 Regression 2 9098,483 4549,242 30,1266
Residual 1067 165201,7 154,8282 Residual 1066 160970,4 151,0041
Total 1068 170068,9 Total 1068 170068,9
SIMPLE REGRESSION: SKILL VARIETY
Multiple R 0,193838 0,06619206 SUM OF SIMPLE R2
R Square 0,037573 0,05349881 MULTIPLE R2
Adjusted R Square0,036671 0,01269325 OVERLAP ATTRIBUTED TO BOTH
Standard Error 12,38552
Observations 1069
11257,2098 SUM OF REGRESSION SUM OF SQUARES
ANOVA 9098,4831 REGRESSION SUM OF SQUARES
df SS MS F Significance F 2158,72671 OVERLAP
Regression 1 6390,011 6390,011 41,6556 1,64882E-10
Residual 1067 163678,9 153,401
Total 1068 170068,9
Testing Parameters
F 0 3.89
H0: 1 = 2 = … = p = 0
H1: At least one I 0
= .05
df = 2 and 12
Critical Value(s):
Test Statistic:
Decision:
Conclusion:
Reject at = 0.05
There is evidence that at
least one independent
variable affects Y
= 0.05
F
Test for Overall Significance Example Solution
168.47
Test for Significance: Individual Variables
•Shows if there is a linear relationship between the
variable Xi and Y
•Use t test Statistic
•Hypotheses:
H0: i = 0 (No linear relationship)
H1: i 0 (Linear relationship between Xi and Y)
C o e ffic ie n ts S ta n d a rd E rro r t S ta t
I n te r c e p t 5 6 2 . 1 5 1 0 0 9 2 1 . 0 9 3 1 0 4 3 3 2 6 . 6 5 0 9 4
X V a r i a b l e 1 -5 . 4 3 6 5 8 0 6 0 . 3 3 6 2 1 6 1 6 7 -1 6 . 1 6 9 9
X V a r i a b l e 2 -2 0 . 0 1 2 3 2 1 2 . 3 4 2 5 0 5 2 2 7 -8 . 5 4 3 1 3
t Test Statistic Excel Output: Example
t Test Statistic for X1
(Temperature)
t Test Statistic for X2
(Insulation) Seb
bt
H0: 1 = 0
H1: 1 0
df = 12
Critical Value(s):
Test Statistic:
Decision:
Conclusion:
Reject H0 at = 0.05
There is evidence of a
significant effect of
temperature on oil
consumption. Z 0 2.1788 -2.1788
.025
Reject H 0 Reject H 0
.025
Does temperature have a significant effect on monthly
consumption of heating oil? Test at = 0.05.
t Test : Example Solution
t Test Statistic = -16.1699
Example: Analysis of job earnings
What is the impact of employer tenure
(ERTEN), unemployment (UNEM) and
education (EDU) on job earnings (JEARN)?
Example: Analysis of job earnings
Correlations
Results: Anova
Results
Examines the contribution of a set of X variables to the relationship with Y
Null hypothesis:
Variables in set do not improve significantly the model when all other variables are included
Alternative hypothesis:
At least one variable is significant
Testing Model Portions
Only one-tail test
Requires comparison of two regressions
One regression includes everything
One regression includes everything except the portion to be tested.
Testing Model Portions
Testing Model Portions Test Statistic
)X, ,
))/k-)X, ,
3
3
21
321
(
(((
XXMSE
XSSRXXSSRF
From ANOVA section
of regression for
iiii XbXbXbbY 3322110ˆ
ii XbbY 330ˆ
From ANOVA section
of regression for
Test H0: 1= 2 = 0 in a 3 variable model
Testing Portions of Model: SSR
Contribution of X1 and X2 given X3 has been
included:
SSR(X1and X2 X3) = SSR(X1,X2 and X3) -
SSR(X3)
From ANOVA section of
regression for
iiii XbXbXbbY 3322110ˆ
From ANOVA section of
regression for
ii XbbY 320ˆ
Partial F Test For Contribution of Set of X variables
Hypotheses:
H0 : Variables Xi... do not significantly improve
the model given all others included
H1 : Variables Xi... significantly improve the
model given all others included
Test Statistic:
F = MSE
kothersallXSSR i /)....(
With df = k and (n - p -1)
k=# of variables
tested
Testing Portions of Model: Example
Test at the = .05 level
to determine if the
variable of average
temperature
significantly improves
the model given that
insulation is included.
Testing Portions of Model: Example
H0: X1 does not improve
model (X2 included)
H1: X1 does improve model
= .05, df = 1 and 12
Critical Value = 4.75
A N O V A
S S
R e g r e s s i o n 5 1 0 7 6 . 4 7
R e s i d u a l 1 8 5 0 5 8 . 8
T o t a l 2 3 6 1 3 5 . 2
717,676
076,51015,228)( 21
MSE
XXSSRF
A N O V A
S S M S
R e g re ssio n 2 2 8 0 1 4 .6 2 6 3 1 1 4 0 0 7 .3 1 3
R e sid u a l 8 1 2 0 .6 0 3 0 1 6 6 7 6 .7 1 6 9 1 8
T o ta l 2 3 6 1 3 5 .2 2 9 3
(For X1 and X2) (For X2)
= 261.47
Conclusion: Reject H0. X1 does improve model
Do I need to do this for one variable?
•The F test for the inclusion of a single variable
after all other variables are included in the
model is IDENTICAL to the t test of the slope
for that variable
•The only reason to do an F test is to test several
variables together.
Example: Collinear Variables
20,000 Execs in 439 Corps: Dependent Variable=base pay+bonus
Individual Simple Regression Multiple Regression
R2 Contribution to R2
Company Dummies .33 .08
Occupational Dummies .52 .022
Position in hierarchy .69 .104
Human Capital Vars .28 .032
Shared .632
TOTAL .87
Yedek
Multiple Regression
The value of outcome variable depends on several explanatory variables.
F-test. To judge whether the explanatory variables in the model adequately describe the outcome variable.
t-test. Applies to each individual explanatory variable. Significant t indicates whether the explanatory variable has an effect on outcome variable while controlling for other X‟s.
T-ratio. To judge the relative importance of the explanatory variable.
Problem of Multicollinearity
When explanatory variables are correlated there is difficulty in interpreting the effect of explanatory variables on the outcome.
Check by:
Correlation coefficient matrix (see next slide).
F-test significant with insignificant t.
Large changes occur in the regression coefficients when variables are added or deleted. (Variance Inflation). Vi > 4 or 5 means there is multicollinearity.
Example of a Matrix Plot
This matrix plot comprises several scatter plots to provide visual information as to whether variables are correlated
The arrow points at a scatter plot where two explanatory variables are strongly correlated
Selecting the most Economic Model
The purpose is to find the smallest number of explanatory variables which make the maximum contribution to the outcome.
After excluding variables that may be causing multicollinearity, examine the table of t-ratios in the full model. Those variables with a significant t are included in the sub-set.
In the Analysis of Variance table examine the column headed SEQ SS. Check that the candidate variables are indeed making a sizable contribution to the Regression Sum of Squares
Stepwise Regression Analysis
Stepwise finds the explanatory variable with the highest R2 to start with. It then checks each of the remaining variables until two variables with highest R2 are found. It then repeats the process until three variables with highest R2 are found, and so on.
The overall R2 gets larger as more variables are added.
Stepwise may be useful in the early exploratory stage of data analysis, but not to be relied upon for the confirmatory stage.
Is the Model Adequate?
Judged by the following:
R2 value. Increase in R2 on adding another variable gives a useful hint
Adjusted R2 is a more sensitive measure.
Smallest value of s (standard deviation).
C-p statistic. A model with the smallest C-p is used such that Cp value is closest to p (the number of parameters in the
Confidence Interval Estimate For The Slope
Provide the 95% confidence interval for the population
slope 1 (the effect of temperature on oil consumption).
111 bpn Stb
Coefficients Lower 95% Upper 95%
Intercept 562,151009 516,1930837 608,108935
X Variable 1 -5,4365806 -6,169132673 -4,7040285
X Variable 2 -20,012321 -25,11620102 -14,90844
-6.169 1 -4.704
The average consumption of oil is reduced by between
4.7 gallons to 6.17 gallons per each increase of 10 F in
houses with the same insulation.
top related