Properties of Estimators

1/27

EC114 Introduction to Quantitative Economics14. Properties of Estimators

Marcus Chambers

Department of EconomicsUniversity of Essex

07/09 February 2012

EC114 Introduction to Quantitative Economics 14. Properties of Estimators

2/27

Outline

1 Introduction

2 Unbiased Estimators

3 Efficient Estimators

4 Linear Estimators and Mean Square Error

5 An Example

Reference: R. L. Thomas, Using Statistics in Economics,McGraw-Hill, 2005, sections 11.1 and 11.2.


Introduction 3/27

To provide some background, consider a population ofvalues for a random variable X.The variable X will have a probability distribution, whichmay be known or unkown (typically the latter).Suppose this distribution can be characterised by anunknown parameter θ.The parameter θ could represent the mean (µ) or variance(σ2) of the distribution, or it could represent the regressionslope parameter (β).Whatever it represents, the parameter θ needs to beestimated using sample information.


Introduction 4/27

Suppose we have a random sample of n observationstaken from the population of X:

X1,X2, . . . ,Xn,

where Xi denotes the i’th observation.Let the estimator of θ be denoted Q, which will be somefunction of the observations i.e.

Q = Q(X1,X2, . . . ,Xn).

For example, if θ were the population mean µ, then ourestimator would be the sample mean

X̄ =X1 + X2 + . . .+ Xn

n.


Introduction 5/27

Whatever the estimator Q, it will have a samplingdistribution.This is because the value of Q changes with each differentpossible sample, and the sampling distribution representsthe distribution of Q across all possible samples.We can therefore talk about quantities such as the meanand variance of the estimator Q i.e.

E(Q) and E[Q− E(Q)]2.

In the regression model we have two estimators, a and b,of α and β, the intercept and slope parameters.Both a and b will have their own sampling distributions, aswell as a joint sampling distribution.


Introduction 6/27

The small sample properties of estimators are determinedby the sampling distribution for estimators obtained using agiven sample size n.Such properties hold even when n may be small.This contrasts with large sample, or asymptotic, propertieswhich are obtained as the sample size n gets larger andlarger i.e. as n→∞.You have already seen the Central Limit Theorem, which isan example of an asymptotic property for the samplemean.It is often written in the form

Zn =

√n(X̄ − µ)

σ

d→ N(0, 1) as n→∞.

We shall not be concerned with asymptotic propertieshere, only small sample properties.


Unbiased Estimators 7/27

A desirable small sample property for an estimator topossess is unbiasedness:

DefinitionAn estimator Q is said to be an unbiased estimator of θ if, andonly if, E(Q) = θ.

Put another way, the mean of the sampling distribution ofthe estimator Q is equal to the true population parameter θ.Or, if we were to take many samples, the average of all Q’sobtained would be equal to θ.This is illustrated on the next slide:



The distribution of Q is centred at θ, meaning that there isno systematic tendency to either over-estimate orunder-estimate θ.This does not mean that in any given sample the estimatorwill equal θ (it almost certainly won’t)!



We have already seen that the sample mean is anunbiased estimator of the population mean: E(X̄) = µ.Note that the property of unbiasedness does not dependon the sample size; it is therefore a small sample property.If an estimator does not satisfy the unbiasedness propertyit is said to be biased and so there is a systematictendency to error in estimating θ.The bias of an estimator Q is defined as

bias(Q) = E(Q)− θ.

If Q tends to over-estimate θ then E(Q) > θ andbias(Q) > 0.Alternatively, if Q tends to under-estimate θ then E(Q) < θand bias(Q) < 0.



An example of a biased estimator is

v2 =

∑(Xi − X̄)2

n,

which is an estimator of σ2, the population variance.It can be shown that

E(v2) =

(n− 1

n

)σ2 < σ2,

and hence bias(v2) < 0.This motivates the unbiased estimator, s2, of σ2, in which nin the denominator is replaced by n− 1:

s2 =

∑(Xi − X̄)2

n− 1;

this results in E(s2) = σ2.



The negative bias of v2 is depicted above.But is unbiasedness all that we require of an estimator?


Efficient Estimators 12/27

In practice we are faced with using a single sample todetermine our estimator Q.Although unbiasedness is a good property for Q topossess, we need to consider other aspects of thesampling distribution as well, such as the variance.Consider two unbiased estimators, Q1 and Q2, so thatE(Q1) = θ and E(Q2) = θ.Suppose, however, that the variance of Q1 is larger thanthe variance of Q2, so that V(Q1) > V(Q2).Which estimator would we prefer?The diagram on the next slide will help answer thisquestion. . .



Although both estimators are unbiased, there is a higherprobability of being far away from θ using the estimator Q1than with Q2 i.e.

Pr(Q1 > θ∗) > Pr(Q2 > θ∗).



We would therefore prefer to use Q2 whose distribution ismore condensed around θ than the distribution of Q1.Another desirable property for an estimator to possess isefficiency:

DefinitionAn estimator Q is said to be an efficient estimator of θ if, andonly if:

(i) it is unbiased, so that E(Q) = θ; and(ii) no other unbiased estimator of θ has a smaller variance.

The two important properties for efficiency, therefore, areunbiasedness and smallest (or minimum) variance.Efficient estimators are sometimes also called bestunbiased estimators.


Linear Estimators and Mean Square Error 15/27

It can be difficult, in practice, to show that an unbiasedestimator has the smallest variance among all estimators.It is often easier to restrict attention to linear estimators,that is, ones which are a linear combination of the sampleobservations.A linear estimator is therefore of the form

Q = a1X1 + a2X2 + . . .+ anXn,

where a1, . . . , an are a set of constants (or weights).An example of a linear estimator is the sample mean:

X̄ =X1 + . . .+ Xn

n=

1n

X1 + . . .+1n

Xn,

so the weights are a1 = 1/n, . . . , an = 1/n.



DefinitionAn estimator, Q, is said to be a best linear unbiased estimator(BLUE) of θ if, and only if:

(i) it is a linear estimator, that is, Q =∑

i aiXi, where the ai areconstants;

(ii) it is unbiased, so that E(Q) = θ; and(iii) no other linear unbiased estimator has a smaller variance.

Note that a BLUE estimator may not be the best possible,because there may exist an efficient nonlinear estimatorwith a smaller variance.In terms of estimating the population mean µ, it turns outthat the sample mean X̄ is both BLUE and efficient.



The concept of efficiency is useful when comparingunbiased estimators, because we always prefer the onewith the smaller variance.But suppose we are faced with the following problem.There are two estimators, Q1 and Q2, of a parameter θ.Q2 is unbiased and efficient while Q1 has a small (positive)bias but also a smaller variance than Q2.We have E(Q1) > θ, E(Q2) = θ, V(Q1) < V(Q2).Can we compare these estimators in a meaningful way?Consider the diagram on the next slide:



The distribution of Q2 is centred at θ but has largedispersion while the distribution of Q1 lies to the right of θbut with a smaller variance.There is a trade-off here between bias and variance.Can we combine both in a single measure?



We shall consider the error of the estimator, given by Q− θ.The mean square error (MSE) of Q is given by

MSE(Q) = E(Q− θ)2.

It can be shown that

MSE(Q) = V(Q) + [bias(Q)]2,

i.e. the MSE of Q is equal to the variance plus the squareof the bias.One way of choosing the preferred estimator would be tochoose the one that has the smallest MSE, which placesequal weight on variance and squared bias.



A more general method uses weights to reflect the relativeimportance placed on variance and squared bias.We could then choose the estimator as the one whichminimises

M(Q) = φV(Q) + (1− φ)[bias(Q)]2,

where 0 < φ < 1 denotes the weight.The larger is φ, the more importance is placed on variance,while the smaller is φ, the more importance is placed on(squared) bias.



It is usually a good idea, when estimating a parameter, touse all the sample information that is available.In general, the more information used, the more efficientthe estimator will be.For example, it wouldn’t make sense to ignore 50% of asample when estimating the population mean – the moreinformation (observations) the better the estimator.An estimator that uses all the sample information is said tobe sufficient.However, if an estimator uses all the observationsinappropriately, it will not be efficient despite beingsufficient.The point is that an estimator cannot be efficient unless itis sufficient.


An Example 22/27

Example (Thomas, p.327). A variable X is normallydistributed with mean µ and variance σ2. Three estimatorsof µ are proposed:

m̂ = X̄ − 10, m̃ = X̄ +5n, m∗ =

(n− 1n− 2

)X̄,

where X̄ is the sample mean and n the sample size.(a) Explain why all three estimators will have sampling

distributions that are normal in shape.(b) Recalling that E(X̄) = µ, use Theorem 1.1 to show that

all the proposed estimators are biased and hencedetermine the bias in each case. If n = 10 and µ = 8,which estimator has the smallest absolute bias?

(c) Recalling that V(X̄) = σ2/n, use Theorem 1.1 to findthe variance of the sampling distribution for eachestimator. Hence determine which estimator has thelargest variance.


An Example 23/27

Solution (a). First, note that m̂, m̃ and m∗ are all linearfunctions of X̄.From the central limit theorem, we know that X̄ has anormal distribution.As all linear functions of normally distributed variables arethemselves normally distributed, it follows that thesampling distributions of m̂, m̃ and m∗ must all also benormal.


An Example 24/27

Solution (b). Recall, from Theorem 1.1, that

E(a + bX) = a + bE(X),

where a and b are constants.We begin by finding the expected values of each estimator:

E(m̂) = E(X̄ − 10) = E(X̄)− 10 = µ− 10;

E(m̃) = E(

X̄ +5n

)= E(X̄) +

5n

= µ+5n

;

E(m∗) = E[(

n− 1n− 2

)X̄]

=

(n− 1n− 2

)E(X̄) =

(n− 1n− 2

)µ.

Thus all the estimators are biased.


An Example 25/27

The biases are

bias(m̂) = E(m̂)− µ = µ− 10− µ = −10;

bias(m̃) = E(m̃)− µ = µ+5n− µ =

5n

;

bias(m∗) = E(m∗)− µ =

(n− 1n− 2

)µ− µ =

(1

n− 2

)µ.

When n = 10 and µ = 8,

bias(m̂) = −10;

bias(m̃) =510

= 0.5;

bias(m∗) =

(1

10− 2

)8 = 1.

Hence m̃ has the smallest bias in absolute terms.


An Example 26/27

Solution (c). Recall, from Theorem 1.1, that

V(a + bX) = b2V(X),

where a and b are constants.The variances are

V(m̂) = V(X̄) =σ2

n;

V(m̃) = V(X̄) =σ2

n;

V(m∗) =

(n− 1n− 2

)2

V(X̄) =

(n− 1n− 2

)2 σ2

n.

Thus, since (n− 1)/(n− 2) > 1, it follows that m∗ has thelargest variance.


Summary 27/27

Summary

Small sample properties of estimators.

Next week:the Classical two-variable regression model.


Properties of Estimators

Documents