COMPARISON OF VALIDATION PROCEDURES TO DETECT MEASUREMENT ERRORS IN AN AREA FRAME SAMPLE SURVEY

11 july 2008 European Conference on Quality

COMPARISON OF VALIDATION PROCEDURES TO COMPARISON OF VALIDATION PROCEDURES TO DETECT MEASUREMENT ERRORS IN AN AREA FRAME DETECT MEASUREMENT ERRORS IN AN AREA FRAME

SAMPLE SURVEYSAMPLE SURVEYLaura Martino, Marco Fritz, Marjo Kasanko & Javier Gallego

European Conference on quality in official statistics, Rome 8-11th July 2008

11 july 2008 European Conference on Quality

Contents

Scope of the work

Methods for measurement errors detection

The case study: LUCAS survey

Conclusions

11 july 2008 European Conference on Quality

Scope of the work

The scope of the work is comparing the efficacy of three selected procedures to detect measurement errors in an area frame sample survey that uses a combination of orthophoto interpretation and ground survey

11 july 2008 European Conference on Quality

Methods for detection of measurement errors

Three methods have been selected to be compared:

double-blind survey;

check of ground data with ausiliary data sources;

cross-check of orto-photo interpretation and ground-survey outcomes to evaluate the accuracy of stratification

11 july 2008 European Conference on Quality

Area frame sample survey carried out by Eurostat since 2001.Main objectives are providing:

a) coherent and harmonised statistics on land use and land cover;

b) a common sampling base (frame, nomenclature, data treatment) to be used for further scopes;

c) ground evidence for calibration of satellite images;

d) information on aspects relating to the agro- environment

Case study: Land Use and Coverage Area frame sample Survey (LUCAS)

11 july 2008 European Conference on Quality

Second phase sample:Ground survey

Record land cover and land use according to the full nomenclature

East

During the ground survey, surveyors: Reach the point using GPS and

cartographic material

Take pictures (Point, Cov,N,E,S,W,irr)

Walk the transect (250 m. East)

11 july 2008 European Conference on Quality

1° procedure: Double blind survey

A second survey was organised in 2006 in parallel to the main LUCAS 2006 survey.

It was conducted by an independent company not in charge of the LUCAS survey in the countries involved.

No information were provided on the results of the main survey.

The double blind-survey sample size represented 5% of the total survey sample and counted almost 8200 points.

11 july 2008 European Conference on Quality

Outcomes of double-blind survey

The possible outcomes of the double-blind survey were:

Land Cover/Land Use being the same between the two surveys (correct points observed from the same location);

Land Cover/Land Use being different between the two surveys but both correct (different rules of observation - look north, east, etc. - or change in the land cover between the two visits (crops harvested, sown, building built, etc...);

Land Cover/Land Use being different between the two surveys and lack of sufficient information to say which survey is correct;

Land Cover/Land Use being different between the two surveys and the double-blind survey being correct;

Land Cover/Land Use being different between the two surveys and the main survey being correct.

11 july 2008 European Conference on Quality

CountryRough data

comparison

BE 69.86%

CZ 80.07%

DE 77.49%

ES 63.22%

FR 73.09%

HU 64.89%

IT 67.12%

PL 65.03%

SK 81.25%M

ain su

rvey 169,000 pts

D-B

su

rvey

8,

200

pts

Double blind survey

D-B survey introduced as

many errors as the main survey

11 july 2008 European Conference on Quality

2° procedure: Use of ausiliary data to correct results

CountryRough data

comparison

Post-processed data

comparison

BE 69.86% 96.58%

CZ 80.07% 95.73%

DE 77.49% 98.71%

ES 63.22% 95.03%

FR 73.09% 96.36%

HU 64.89% 98.55%

IT 67.12% 95.08%

PL 65.03% 90.62%

SK 81.25% 97.50%

Large improvement in

data quality through the use

of pictures

IACS data

Pictures

Other georeferenced information

11 july 2008 European Conference on Quality

3° procedure: comparison of orthophoto interpretation and ground survey

A comparison of the classification of the points according to the ground observation and the photointerpretation has been conducted in a comparable nomenclature of 7 classes.

A proportion of agreement has been computed

Since points belonging to different strata have been subsampled with probability that could be 5 times larger/lower, a weighted proportion of agreement is computed in addition to the unweighted one.

11 july 2008 European Conference on Quality

Weighted confusion matrix of point photo-interpretation Ground obs

Strata A

rabl

e

Per

m.

Cro

ps

Per

m.

Gra

ss

Woo

d,

shru

b

Bar

e

Art

ific

ial

Wat

er

Tot

al

Arable land 61165 1894 18728 2139 5846 1611 255 91638 Permanent Crops 410 9536 601 630 217 172 6 11572 Permanent Grassland 4210 682 29068 4340 1187 1425 421 41333 Wooded areas, shrubland 1220 930 9725 84715 1245 2225 805

100865

Bare land or low vegetation 75 65 505 530 725 395 195 2490 Artificial land 365 170 2255 520 245 9280 90 12925 Water, wetland 15 10 100 70 45 20 2765 3025

Total 67460 13287 60982 92944 9510 15128 4537 26384

8

11 july 2008 European Conference on Quality

Measurements of agreement

Frequency: The unweighted proportion of agreement is 70.8%, while the weighted agreement is 74.8%.

Kappa index (Bishop et al., 1975)

where are the proportions of each cell of the table

The unweighted Kappa is 0.58 (0.66 if we take into account the strata weights).

ii

i

ii

ii

ii

pp

pppK

1

N

np ij

ij

11 july 2008 European Conference on Quality

Rate of agreement

Unweighted weighted

Cty agreement KAPPA agreement KAPPA

BE 78.8 0.671 82.0 0.759

CZ 79.6 0.669 82.3 0.743

DE 71.1 0.537 73.4 0.632

ES 64.0 0.536 70.0 0.598

FR 78.0 0.667 79.9 0.723

HU 73.1 0.531 74.7 0.638

IT 65.2 0.531 71.2 0.619

LU 78.2 83.1

NL 74.5 0.574 74.9 0.660

PL 67.5 0.473 73.3 0.615

SK 83.1 0.737 87.3 0.806

11 july 2008 European Conference on Quality

Mainly confused items

Some points that are agricultural land are photo-interpreted as non-agricultural for several reasons: location inaccuracy,

land cover change or simply photo-interpretation errors

The main confusions occured between:

permanent grass and bare land are often photo-interpreted as arable.

Woodland or shrubland interpreted as grass. This may be partly due to problems to apply in practice the definition of forest or woodland

11 july 2008 European Conference on Quality

Conclusions

The pre-eminent result of the double blind survey was that it introduced as much errors as the main survey for many reasons

The analysis of pictures and the use of auxiliary information appeared to be fundamental to detect measurement errors and improve data quality

Orthophoto interpretation produce some inefficiency (stratification accuracy) -> poststratification???

11 july 2008 European Conference on Quality

THANK YOU FOR YOUR ATTENTION!!!