Chapter 7 Point Estimation - University of Arizonajwatkins/G1_pointestimation.pdfChapter 7 Point Estimation Method of Moments 1/23. IntroductionMethod of MomentsProcedureMark and RecaptureMonsoon

Introduction Method of Moments Procedure Mark and Recapture Monsoon Rains

Chapter 7Point EstimationMethod of Moments

1 / 23


Outline

IntroductionClassical Statistics

Method of Moments

Procedure

Mark and Recapture

Monsoon Rains

2 / 23


Parameter Estimation

For parameter estimation, we consider X = (X1, . . . ,Xn), independent randomvariables chosen according to one of a family of probabilities Pθ where θ is elementfrom the parameter space Θ. Based on our analysis, we choose an estimator θ̂(X ). Ifthe data x takes on the values x1, x2, . . . , xn, then

θ̂(x1, x2, . . . , xn)

is called the estimate of θ. Thus we have three closely related objects.

1. θ - the parameter, an element of the parameter space, is a number or a vector.

2. θ̂(x1, x2, . . . , xn) - the estimate, is a number or a vector obtained by evaluating theestimator on the data x = (x1, x2, . . . , xn).

3. θ̂(X1, . . . ,Xn) - the estimator, is a random variable. We will analyze thedistribution of this random variable to decide how well it performs in estimating θ.

3 / 23


Classical Statistics

In classical statistics, the state of nature is assumed to be fixed, but unknown to us.Thus, one goal of estimation is to determine which of the Pθ is the source of the data.The estimate is a statistic

θ̂ : data→ Θ.

For estimation procedures, the classical approach to statistics is based on twofundamental questions:

• How do we determine estimators?

• How do we evaluate estimators?• Does this estimator in any way systematically under or over estimate the parameter?• Does it has large or small variance?• How does it compare to a notion of best possible estimator?• How easy is it to determine and to compute?• How does the procedure improve with increased sample size?

4 / 23


Method of Moments

Method of moments estimation is based solely on the law of large numbers,

Let M1,M2, . . . be independent random variables having a common distributionpossessing a mean µM . Then the sample means converge to the distributional mean asthe number of observations increase.

M̄n =1

n

n∑i=1

Mi → µM as n→∞

almost surely and in mean.

In addition, if the random variables in this sequence fail to have a mean, then the limitwill fail to exist.

5 / 23


Procedure

• Step 1. If the model is based on a parametric family of densities fX (x |θ) with ad-dimensional parameter space (θ1, θ2 . . . , θd), we compute

µm = EXm = km(θ) =

∫SxmfX (x |θ) ν(dx), m = 1, . . . , d

the first d moments,

µ1 = k1(θ1, θ2 . . . , θd), µ2 = k2(θ1, θ2 . . . , θd), . . . , µd = kd(θ1, θ2 . . . , θd),

obtaining d equations in d unknowns.

6 / 23


ExampleLet X1,X2, . . . ,Xn be a simple random sample of Pareto random variables with density

fX (x |β) =β

xβ+1, x > 1, β > 0.

The cumulative distribution function is

FX (x) = 1− x−β, x > 1.

The mean and the variance are, respectively,

µ =β

β − 1, σ2 =

β

(β − 1)2(β − 2).

In this situation, we have one parameter, namely β. Thus, in step 1, we will only needto determine the first moment

µ1 = µ = k1(β) =β

β − 1

to find the method of moments estimator β̂ for β.7 / 23


Procedure

• Step 2. We then solve for the d parameters as a function of the moments.

θ1 = g1(µ1, µ2, · · · , µd), θ2 = g2(µ1, µ2, · · · , µd),

. . . , θd = gd(µ1, µ2, · · · , µd).

• Step 3. Now, based on the data x = (x1, x2, . . . , xn), we compute the first dsample moments,

x =1

n

n∑i=1

xi , x2 =1

n

n∑i=1

x2i , . . . , xd =1

n

n∑i=1

xdi .

8 / 23


ExampleExercise. If

µ =β

β − 1, show that β =

µ

µ− 1.

For step 2, we solve for β as a function of the mean µ.

β = g1(µ) =µ

µ− 1.

For step 3, we compute the sample mean

x =1

n

n∑i=1

xi .

9 / 23


Procedure

• Step 4. Using the law of large numbers, we have, for each moment, m = 1, . . . , d ,that

µm ≈ xm.

For the equations derived in step 2, we replace the distributional moments µm bythe sample moments xm to give us formulas for the method of moment estimates

(θ̂1, θ̂2, . . . , θ̂d).

For the data x, these estimates are

θ̂1(x) = g1(x̄ , x2, · · · , xd), θ̂2(x) = g2(x̄ , x2, · · · , xd),

. . . , θ̂d(x) = gd(x̄ , x2, · · · , xd).

10 / 23


ExampleConsequently, a method of moments estimate for β is obtained by replacing thedistributional mean µ by the sample mean x̄ in the equation for g1(µ). Thus,

β̂ =x̄

x̄ − 1.

A good estimator should have a small variance. To use the delta method to estimatethe variance of β̂,

σ2β̂≈ g ′1(µ)2

σ2

n.

We compute

g1(µ) =µ

µ− 1and so g ′1(µ) = − 1

(µ− 1)2

giving

g ′1

(β

β − 1

)= − 1

( ββ−1 − 1)2

= − (β − 1)2

(β − (β − 1))2= −(β − 1)2.

11 / 23


ExampleWe find that β̂ has mean approximately equal to β and variance

σ2β̂≈ g ′1(µ)2

σ2

n= (β − 1)4

β

n(β − 1)2(β − 2)=β(β − 1)2

n(β − 2).

Let’s consider the case with β = 4 and n = 225. Then,

σ2β̂≈ 4 · 32

225 · 2=

36

450=

2

25, σβ̂ ≈

√2

5= 0.283.

To simulate, we use the probability transform

u = FX (x) = 1− x−β, then x = (1− u)−1/β = 1/ β√

(1− u).

Note that if Ui are U(0, 1) random variables, then 1/ β√

(1− U1), 1/ β√

(1− U2), · · ·have the appropriate Pareto distribution.

12 / 23


Example> paretobar<-rep(0,1000)

> for (i in 1:1000){u<-runif(225);

pareto<-1/(1-u)^(1/4);

paretobar[i]<-mean(pareto)}

> betahat<-paretobar/(paretobar-1)

> mean(betahat)

[1] 4.03508

> sd(betahat)

[1] 0.2833142

Note that the mean is above 4, but thestandard deviation is very close to thevalue given by the delta method.

Exercise. Reproduce the simulationabove and compare. Simulate using adifferent value for β.

Histogram of betahat

betahat

Density

0 5 10 15

0.00

0.05

0.10

0.15

Figure: 1000 simulations for the method ofmoments estimate for the case β = 4.

13 / 23


Mark and Recapture

The Lincoln-Peterson method of mark and recapture

• The size of an animal population in a habitat of interest is an important questionin conservation biology.

• In many case, individuals are often too difficult to find and a census is not feasible.

• One estimation technique is to capture some of the animals, mark them andrelease them back into the wild to mix randomly with the population.

• Some time later, a second capture from the population is made.

14 / 23


Mark and Recapture

Some of the animals were not in the first capture and some, which are tagged, arerecaptured. Let

• t be the number captured and tagged,

• k be the number in the second capture,

• r be the number in the second capture that are tagged, and let

• N be the total population size.

Thus, t and k is under the control of the experimenter. The value of r is random andthe populations size N is the parameter to be estimated.

15 / 23


Mark and Recapture

We can guess the the estimate of N by considering two proportions.

the proportion of the tagged fish ≈ the proportion of tagged fish

in the second capture in the population

r

k≈ t

N

This can be solved for N to find

N ≈ kt

r.

The advantage of obtaining this as a method of moments estimator is that we evaluatethe precision of this estimator by determining, for example, its variance.

16 / 23


Mark and RecaptureTo begin, let

Xi =

{1 if the i-th individual in the second capture has a tag.0 if the i-th individual in the second capture does not have a tag.

The Xi are Bernoulli random variables with success probability P{Xi = 1} = t/N. Thenumber of tagged individuals is X = X1 + X2 + · · ·+ Xk and the expected number oftagged individuals is

µ = EX = EX1 + EX2 + · · ·+ EXk =t

N+

t

N+ · · ·+ t

N=

kt

N.

The proportion of tagged individuals, X̄ = (X1 + · · ·+ Xk)/k, has expected value

EX̄ =µ

k=

t

N. Thus, N =

kt

µ.

17 / 23


Mark and RecaptureNow in this case, we are estimating µ, the mean number recaptured with r , the actualnumber recaptured. So, to obtain the estimate N̂. we replace µ with the previousequation by r .

N̂ =kt

r.

We simulate the process in a lake having 4500 fish.

> N<-4500;t<-400;k<-500 #population 4500, 400 tagged, recapture 500

> r<-rep(0,2000) #set a vector of zeros for 2000 simulations

> fish<-c(rep(1,t),rep(0,N-t)) #tag t fish

> for (j in 1:2000){r[j]<-sum(sample(fish,k))}

> Nhat<-k*t/r #compute estimate of population

> mean(Nhat);sd(Nhat)

[1] 4606.933

[1] 666.1918

18 / 23


Mark and Recapture

Histogram of r

r

Frequency

20 30 40 50 60

0100200300400500600

Histogram of Nhat

Nhat

Frequency

3000 5000 7000

0100200300400500600

Exercise. Describe the histograms above. Comment of the mean and standarddeviation of the estimate N̂.

19 / 23


Monsoon RainsMonsoon is used to describe the rainy phase for seasonal changes in atmosphericcirculation and precipitation associated with the asymmetric heating of land and sea.Our model for this distribution will be the gamma family of random variables.

A Γ(α, β) random variable has mean α/β and variance α/β2. Because we have twoparameters, the method of moments methodology requires us, in step 1, to determinethe first two moments.

µ1 = E(α,β)X1 =α

β

µ2 = E(α,β)X21 = Var(α,β)(X1) + (E(α,β)X1)2

=α

β2+

(α

β

)2

=α

β2+α2

β2=α(1 + α)

β2.

NB. Var(Y ) = EY 2 − (EY )2. So, EY 2 = Var(Y ) + (EY )2.20 / 23


Monsoon RainsThe first two moments

µ1 =α

βand µ2 =

α

β2+α2

β2=α(1 + α)

β2

For step 2, we solve for α and β. Note that

µ2 − µ21 =α

β2,

µ1µ2 − µ21

=α/β

α/β2= β,

and

µ1 ·µ1

µ2 − µ21=α

β· β = α, or α =

µ21µ2 − µ21

.

21 / 23


Monsoon Rains

β =µ1

µ2 − µ21=µ1σ2

and α = βµ1 =µ21

µ2 − µ21=µ21σ2.

For step 3, set the first two sample moments

x̄ =1

n

n∑i=1

xi and x2 =1

n

n∑i=1

x2i

to obtain estimates

β̂ =x̄

x2 − (x̄)2and α̂ = β̂x̄ =

(x̄)2

x2 − (x̄)2

as required in step 4.

22 / 23


Monsoon Rains

In 2017, Tucson, Arizona had 17summer monsoon rainstorms. Thedata are the rainfall in millimeters.

x<-c(3,15,1,37,5,1,8,11,6,9,12,

35,22,3,38,1,2)

> (xbar<-mean(x));(s2<-var(x))

[1] 12.29412

[1] 167.0956

> (betahat<-xbar/s2);

(alphahat<-betahat*xbar)

[1] 0.07357536

[1] 0.9045441

0 10 20 30 40

0.0

0.2

0.4

0.6

0.8

1.0

ecdf(x)

rain in millimeters

probability

0 10 20 30 40

0.0

0.2

0.4

0.6

0.8

1.0

rain in millimeters

probability

Figure: Comparison of the empirical distribution functionfor the monsoon rainfall data and the plot of theestimated gamma distribution function.

23 / 23

Chapter 7 Point Estimation - University of Arizonajwatkins/G1_pointestimation.pdfChapter 7 Point Estimation Method of Moments 1/23. IntroductionMethod of MomentsProcedureMark and RecaptureMonsoon

Documents