IMPLEMENTING UNIT VALUE INDICES IN THE ANNUAL OECD INTERNATIONAL TRADE IN COMMODITY STATISTICS (ITCS) DATABASE Handling missing values, outliers in unit.

IMPLEMENTING UNIT VALUE INDICES IN THE ANNUAL OECD INTERNATIONAL TRADE IN COMMODITY STATISTICS (ITCS) DATABASE

Handling missing values, outliers in unit value variations, representativity of the indices, data conversion across classifications

Working Party on Trade in Goods and Services

7-9 November 2011OECD Statistics Directorate

Outline of presentation

• Background• Data universe and formulae• Supplementary units or net weights?• Conversion issues• Estimation of missing values• Dealing with outliers• Specific products

Background 1/2

Advantages of UVIs and Quantities Indices: • proxies for X & M price indices and for volume measures• available at product detail level and in a timely manner• international comparability• analysis of :– terms of trade– price and non-price competitiveness in X &M– quantity effect from the transmission of inflation via foreign trade

• Context• follow-up of the WPTGS 2010• recommandation of the IMTS 2010• results of the short Survey WPTGS 2010

Background 2/2

• Stocktaking of the implementation:– on tracks with PWB– process with SAS– chained Laspeyres Paasche Fisher indices– on 34 countries– HS88 at a total level / partner World

• Issues that are being looking at : – missing values (of quantities and trade values) – outliers– conversion across HS classifications

Data Universe 1/3

ITCS database :– UNSD / OECD joint data processing

system– Contains Information on

– Trade values in US dollars– Quantities (liters, m²…) – Net Weight (kg)

– Data Availability– HS (1988,1996,2002 & 2007) – SITC (rev.2, rev.3)– ISIC rev.3

Data Universe 2/3

• CIF / FOB Valuation

• Calculation based on 6 digits of HS1988

• Estimation of missing quantity/weight by the UNSD method :

1) use of the available info (i.e. net weight proxied from quantity)

2) use of a Standard Unit Value

• Exclusion of 6 digits commodities without information on weights (whole chapter 99 )

Data Universe 3/3

•UV = Value in US dollars Quantity

•Unit Values Indices are sensitive to exchange rates fluctuations; not Quantity Indices …

(Pt)

Computation of Quantity indices

LASPEYRES

PAASCHE

FISHER

111

11

11

11/ *

*

*

*

*)(

t

t

tt

tt

tt

tttt Q

Q

QP

QP

QP

QPQL

t

t

tt

tt

tt

tttt Q

Q

QP

QP

QP

QPQP 1

11/ *

*

*/1

*

*)(

)(*)()( 1/1/1/ QLQPQF tttttt

Quantity ratio form

Weighing system

Computation of Unit Value indices

LASPEYRES

PAASCHE

FISHER

111

11

11

11/ *

*

*

*

*)(

t

t

tt

tt

tt

tttt P

P

QP

QP

QP

QPPL

t

t

tt

tt

tt

tttt P

P

QP

QP

QP

QPPP 1

11/ *

*

*/1

*

*)(

)(*)()( 1/1/1/ PLPPPF tttttt

Quantity ratio form

Weighing system

Defining the denominator of the UV

Supplementary Units or Net Weights ?

Should we use supplementary units?

• Supplementary units more accurate than Net weights for some commodities BUT

How to handle changes of quantity units if for instance the quantity unit is one year, ‘’number of items (5)’’ and

the next year ‘’ thousands of items (9)’’

On long series, net weights are more reported UNSD estimated that 75% of supplementary units is KG

=> OECD choice : net weight in kg

Representativity of the sample:Issue with historical series

• 2009 :– <50% Israel X (80% for Israel M)– >75% for All others OECD countries (M and X)

• 1999 : more problematic – <25% Canada(M and X) , Australia M and USA M – <60% + New Zealand (M), USA (X), Norway (X),

Australia (X)– <75% + Japan (X), Norway (M), Mexico (M), New

Zealand (X) and Netherlands (M & X)

• 1989 / 21 countries– <75% Canada (M & X), USA (M&X), New Zealand (M)

and Australia (M & X), Japan (X) , Norway (M & X)

=> Thresholds on UVI for coverage ?

Conversion issues

use of HS 1988conversion of

HS 1996, HS 2002 HS 2007

Conversion of HS

Conversion issues: looking at HS1988

Handling gaps in series

Estimating Missing Values and Quantities:

Unconvincing tests of 2 methods

Estimation of Trade Values cmd 010111

Estimation of Quantities based on Standard Unit Values (SUV)

• SUV– Compiled by UNSD– median unit value of – each 6-digit commodity/year/flow – after elimination of outliers– of a sample of Unit Value of– available data of the latest reporting year– that respect a certain number of criteria

(box 1)

Missing values estimation conclusions

• Thresholds for estimation for trade values: 1% of values estimated for the whole chapter

• Calculated indices present dubious movements :• Large fluctuations for chained type

Indexes at 6 digits level=> Issues for more disaggregated indices (2&4

digits)

Outliers detection

Outlier detection

• “an unrealistic price growth in the product specific distribution of unit value ratios”

CEPII indices :“ Unit values ratios are compared with the

product specific median change in unit values computed over the whole period”

• .

Outlier detection • “Value that lie far from the

middle of the distribution in either direction”

• Mexico and Italy :>100 obs : Asymmetric Fence

Method<100 obs : Mean Absolute Deviation• UNSD : Tukey Method

AFM and MAD Formulae

• Assymetric Fence Method

• Mean Absolute Deviation

UV =logarithm of the unit valueQ1, Q2; Q3 1st 2nd 3rd quartiles of log unit values of trade distributionc, k, A parameters

|)|,max(

|)|,max(

22338

21281

qcqqkquv

qcqqkuvq

AFMihs

AFMihs

||*||

*||

228

828

qAquv

MADkquv

ihs

hsMADihs )( 288 quvmedianMAD i

hshs

Specific products

chapters

% of outliers within a chapter using symetric yearly variation detection method

% of outliers within a chapter using AFM MAD detection method

84: Nuclear reactors, boilers, machinery, etc 15% 15%

85: Electrical, electronic equipment 3% 5%

29: Organic chemicals 1% 2%

Comparing OECD total level indexes with those available from other international organisations and other frameworks (SNA)

Percentage Change of Quantity Indices for Iceland Imports

Percentage Change of Quantity Indices for Turkey Imports

Percentage Change of Unit Values Indices for Japan Import

Percentage Change of Unit Values Indices for Italy Export

Next Steps

Following the Program of Work Begining of 2012 : • 34 countries at a 2 digits level applying

methodologies for outliers and missing values• Finding some specific treatments for specific

chapters (including those that lose 30 % of their trade just by changing classification (chapters 84 -85 )

Summer 2012 :• Matrix of exports and imports unit value and

quantity indices available online for comments at the 2012 WPTGS

Questions to delegates

• Thresholds : on customs transactions ?• Measure of Quantity: choice of supplementary

quantity or net weight values? • HS Conversion issues : How to deal with cmd of HS

change ? • Estimation of Missing Values : What kind of

methodology do you recommend? • More disaggregated indices (2 digits indices or

more detailed) do you have special warnings or experiences to share ?