11 july 2008 European Conference on Quality
COMPARISON OF VALIDATION PROCEDURES TO COMPARISON OF VALIDATION PROCEDURES TO DETECT MEASUREMENT ERRORS IN AN AREA FRAME DETECT MEASUREMENT ERRORS IN AN AREA FRAME
SAMPLE SURVEYSAMPLE SURVEYLaura Martino, Marco Fritz, Marjo Kasanko & Javier Gallego
European Conference on quality in official statistics, Rome 8-11th July 2008
11 july 2008 European Conference on Quality
Contents
Scope of the work
Methods for measurement errors detection
The case study: LUCAS survey
Conclusions
11 july 2008 European Conference on Quality
Scope of the work
The scope of the work is comparing the efficacy of three selected procedures to detect measurement errors in an area frame sample survey that uses a combination of orthophoto interpretation and ground survey
11 july 2008 European Conference on Quality
Methods for detection of measurement errors
Three methods have been selected to be compared:
double-blind survey;
check of ground data with ausiliary data sources;
cross-check of orto-photo interpretation and ground-survey outcomes to evaluate the accuracy of stratification
11 july 2008 European Conference on Quality
Area frame sample survey carried out by Eurostat since 2001.Main objectives are providing:
a) coherent and harmonised statistics on land use and land cover;
b) a common sampling base (frame, nomenclature, data treatment) to be used for further scopes;
c) ground evidence for calibration of satellite images;
d) information on aspects relating to the agro- environment
Case study: Land Use and Coverage Area frame sample Survey (LUCAS)
11 july 2008 European Conference on Quality
Second phase sample:Ground survey
Record land cover and land use according to the full nomenclature
East
During the ground survey, surveyors: Reach the point using GPS and
cartographic material
Take pictures (Point, Cov,N,E,S,W,irr)
Walk the transect (250 m. East)
11 july 2008 European Conference on Quality
1° procedure: Double blind survey
A second survey was organised in 2006 in parallel to the main LUCAS 2006 survey.
It was conducted by an independent company not in charge of the LUCAS survey in the countries involved.
No information were provided on the results of the main survey.
The double blind-survey sample size represented 5% of the total survey sample and counted almost 8200 points.
11 july 2008 European Conference on Quality
Outcomes of double-blind survey
The possible outcomes of the double-blind survey were:
Land Cover/Land Use being the same between the two surveys (correct points observed from the same location);
Land Cover/Land Use being different between the two surveys but both correct (different rules of observation - look north, east, etc. - or change in the land cover between the two visits (crops harvested, sown, building built, etc...);
Land Cover/Land Use being different between the two surveys and lack of sufficient information to say which survey is correct;
Land Cover/Land Use being different between the two surveys and the double-blind survey being correct;
Land Cover/Land Use being different between the two surveys and the main survey being correct.
11 july 2008 European Conference on Quality
CountryRough data
comparison
BE 69.86%
CZ 80.07%
DE 77.49%
ES 63.22%
FR 73.09%
HU 64.89%
IT 67.12%
PL 65.03%
SK 81.25%M
ain su
rvey 169,000 pts
D-B
su
rvey
8,
200
pts
Double blind survey
D-B survey introduced as
many errors as the main survey
11 july 2008 European Conference on Quality
2° procedure: Use of ausiliary data to correct results
CountryRough data
comparison
Post-processed data
comparison
BE 69.86% 96.58%
CZ 80.07% 95.73%
DE 77.49% 98.71%
ES 63.22% 95.03%
FR 73.09% 96.36%
HU 64.89% 98.55%
IT 67.12% 95.08%
PL 65.03% 90.62%
SK 81.25% 97.50%
Large improvement in
data quality through the use
of pictures
IACS data
Pictures
Other georeferenced information
11 july 2008 European Conference on Quality
3° procedure: comparison of orthophoto interpretation and ground survey
A comparison of the classification of the points according to the ground observation and the photointerpretation has been conducted in a comparable nomenclature of 7 classes.
A proportion of agreement has been computed
Since points belonging to different strata have been subsampled with probability that could be 5 times larger/lower, a weighted proportion of agreement is computed in addition to the unweighted one.
11 july 2008 European Conference on Quality
Weighted confusion matrix of point photo-interpretation Ground obs
Strata A
rabl
e
Per
m.
Cro
ps
Per
m.
Gra
ss
Woo
d,
shru
b
Bar
e
Art
ific
ial
Wat
er
Tot
al
Arable land 61165 1894 18728 2139 5846 1611 255 91638 Permanent Crops 410 9536 601 630 217 172 6 11572 Permanent Grassland 4210 682 29068 4340 1187 1425 421 41333 Wooded areas, shrubland 1220 930 9725 84715 1245 2225 805
100865
Bare land or low vegetation 75 65 505 530 725 395 195 2490 Artificial land 365 170 2255 520 245 9280 90 12925 Water, wetland 15 10 100 70 45 20 2765 3025
Total 67460 13287 60982 92944 9510 15128 4537 26384
8
11 july 2008 European Conference on Quality
Measurements of agreement
Frequency: The unweighted proportion of agreement is 70.8%, while the weighted agreement is 74.8%.
Kappa index (Bishop et al., 1975)
where are the proportions of each cell of the table
The unweighted Kappa is 0.58 (0.66 if we take into account the strata weights).
ii
i
ii
ii
ii
pp
pppK
1
N
np ij
ij
11 july 2008 European Conference on Quality
Rate of agreement
Unweighted weighted
Cty agreement KAPPA agreement KAPPA
BE 78.8 0.671 82.0 0.759
CZ 79.6 0.669 82.3 0.743
DE 71.1 0.537 73.4 0.632
ES 64.0 0.536 70.0 0.598
FR 78.0 0.667 79.9 0.723
HU 73.1 0.531 74.7 0.638
IT 65.2 0.531 71.2 0.619
LU 78.2 83.1
NL 74.5 0.574 74.9 0.660
PL 67.5 0.473 73.3 0.615
SK 83.1 0.737 87.3 0.806
11 july 2008 European Conference on Quality
Mainly confused items
Some points that are agricultural land are photo-interpreted as non-agricultural for several reasons: location inaccuracy,
land cover change or simply photo-interpretation errors
The main confusions occured between:
permanent grass and bare land are often photo-interpreted as arable.
Woodland or shrubland interpreted as grass. This may be partly due to problems to apply in practice the definition of forest or woodland
11 july 2008 European Conference on Quality
Conclusions
The pre-eminent result of the double blind survey was that it introduced as much errors as the main survey for many reasons
The analysis of pictures and the use of auxiliary information appeared to be fundamental to detect measurement errors and improve data quality
Orthophoto interpretation produce some inefficiency (stratification accuracy) -> poststratification???
11 july 2008 European Conference on Quality
THANK YOU FOR YOUR ATTENTION!!!