Centraal Bureau voor de Statistiek Marcel Pommée National Accounts Department Statistics Netherlands (CBS) Semi-automatic integration in Semi-automatic.

Centraal Bureau voor de Statistiek

Marcel PomméeMarcel PomméeNational Accounts DepartmentNational Accounts Department

Statistics Netherlands (CBS)Statistics Netherlands (CBS)

Semi-automatic integration inSemi-automatic integration in

Dutch Supply and Use Tables – with special Dutch Supply and Use Tables – with special reference to time-seriesreference to time-series

Centraal Bureau voor de Statistiek

OutlineOutline

• SUTs, characteristicsSUTs, characteristics• Reasons for automationReasons for automation• Automation with ‘machines’Automation with ‘machines’• Balancing machineBalancing machine• Quarterly machineQuarterly machine• Time-series machineTime-series machine• Time-series projectTime-series project• SummarySummary

Centraal Bureau voor de Statistiek

SUTs, characteristicsSUTs, characteristics

• Annually, quarterly (t+45 and t+30 ?)Annually, quarterly (t+45 and t+30 ?)

• Detail: industries (120), commodities (630), expenditure categories (50)Detail: industries (120), commodities (630), expenditure categories (50)

• Focus on year on year changes, no seasonal adjustmentFocus on year on year changes, no seasonal adjustment

• Simultaneous balancing of current and constant pricesSimultaneous balancing of current and constant prices

• Data gaps filled with assumptions and extrapolationsData gaps filled with assumptions and extrapolations

• Manual balancingManual balancing

Centraal Bureau voor de Statistiek

Reasons for automationReasons for automation

• Efficiency gains: budget reductions up to 34% expected in 2018 Efficiency gains: budget reductions up to 34% expected in 2018

• Part of redesign of chain of economic statistics:Part of redesign of chain of economic statistics:• More structured processMore structured process• Top-down approachTop-down approach• Focus on major problemsFocus on major problems

• QualityQuality• Visibility various adjustment steps (transparancy)Visibility various adjustment steps (transparancy)• Consistency over timeConsistency over time• Reproduce results (consistency) Reproduce results (consistency) • First gdp estimates quickly available (analysis)First gdp estimates quickly available (analysis)

Centraal Bureau voor de Statistiek

Automation with ‘machines’Automation with ‘machines’

• Quadratic optimization modelQuadratic optimization model• Minimizing the adjustment needed to the growth rates of quarterly seriesMinimizing the adjustment needed to the growth rates of quarterly series• T-1 and unbalanced data in current and constant prices availableT-1 and unbalanced data in current and constant prices available

• Only semi-automatic integrationOnly semi-automatic integration• Major problems tackled manuallyMajor problems tackled manually• Small problems resolved through automationSmall problems resolved through automation

• Machines for different purposes:Machines for different purposes:• Balancing machine: balancing single SUTBalancing machine: balancing single SUT• Quarterly machine: rebasing years and aligning quartersQuarterly machine: rebasing years and aligning quarters• Time-series machine: rebasing time-seriesTime-series machine: rebasing time-series

Centraal Bureau voor de Statistiek

Balancing machineBalancing machine

• Balancing single SUT: major problems solved manuallyBalancing single SUT: major problems solved manually

• Hard and soft constraintsHard and soft constraints• Suppy is equal to use by commoditySuppy is equal to use by commodity• Preserve price indices by commodityPreserve price indices by commodity• Preserve i/o-ratios by branches of industryPreserve i/o-ratios by branches of industry• Compute trade and transport marginsCompute trade and transport margins• Compute taxes and subsidies on productsCompute taxes and subsidies on products• Upper and lower bounds for individual variablesUpper and lower bounds for individual variables• Fixation of variablesFixation of variables• Weighting of variables based on quality of datasourceWeighting of variables based on quality of datasource• Specific relations (import and re-exports, building materials and construction)Specific relations (import and re-exports, building materials and construction)

Centraal Bureau voor de Statistiek

Quarterly machineQuarterly machine

• Compilation cycle: final (F), preliminary (P), very preliminary (V)Compilation cycle: final (F), preliminary (P), very preliminary (V)

• Quarterly machineQuarterly machine• Input: F-year and 12 quarters of previous cycleInput: F-year and 12 quarters of previous cycle• Output: rebased P- and V- year and 12 aligned quartersOutput: rebased P- and V- year and 12 aligned quarters

• Updating of P- and V-yearUpdating of P- and V-year• Selected information addedSelected information added• With balancing machineWith balancing machine

Centraal Bureau voor de Statistiek

Time-series machineTime-series machine

• Time-series (1990-2009) based on benchmark revision 2010Time-series (1990-2009) based on benchmark revision 2010

• Reconstruction of complete SU and IO tablesReconstruction of complete SU and IO tables

• EarlierEarlier• Year by year compilation => very time-consumingYear by year compilation => very time-consuming• Difficult to preserve price and volume indices of original seriesDifficult to preserve price and volume indices of original series

• Time-series machineTime-series machine• New levels given by benchmark year and reference yearsNew levels given by benchmark year and reference years• Reference years, e.g. 1987, 1995, 2001 (previous revision years)Reference years, e.g. 1987, 1995, 2001 (previous revision years)• Preservation of price and volume indices of original seriesPreservation of price and volume indices of original series• Iterative process, manual interventionIterative process, manual intervention• Result: fully consistent time-series in current and constant pricesResult: fully consistent time-series in current and constant prices

Centraal Bureau voor de Statistiek

Time-series project (1)Time-series project (1)

• Year 2010 ESA 2010 conceptual revision and benchmark (statistical) revision Year 2010 ESA 2010 conceptual revision and benchmark (statistical) revision Revised GDP 7,6% higher (concepts 3% and benchmark 4,6%)Revised GDP 7,6% higher (concepts 3% and benchmark 4,6%)

• Covers Fully consistent ANA, QNA, ASA, QSA, LA,Covers Fully consistent ANA, QNA, ASA, QSA, LA,SUTs, IO-tables, and regional accountsSUTs, IO-tables, and regional accounts

• Planning Benchmark year 2010Planning Benchmark year 2010 publication 6th March 2014publication 6th March 2014 2001-2009, up to 20132001-2009, up to 2013 publication 20th June publication 20th June 20142014 1995-20001995-2000 publication 24th September 2014 publication 24th September 2014

• Extremely tight scheduleExtremely tight schedule

Centraal Bureau voor de Statistiek

Time-series project (2)Time-series project (2)

• Time-series 2001-2009: series of problemsTime-series 2001-2009: series of problems

• Start-up problems time series machine (coding errors, retrieving data, capacity Start-up problems time series machine (coding errors, retrieving data, capacity limitations, processing time)limitations, processing time)

• Takes a lot of time to specify constraints (fixation of variables, notably government Takes a lot of time to specify constraints (fixation of variables, notably government data, weighting of variables, notably prices)data, weighting of variables, notably prices)

• Many interdependencies with other NA-modules (LA, ASA, government data and Many interdependencies with other NA-modules (LA, ASA, government data and financial institutions, fisim) financial institutions, fisim)

• Time-series machine had to make quite large adjustments in the original series due Time-series machine had to make quite large adjustments in the original series due to substantial level shifts in revision yearto substantial level shifts in revision year

Centraal Bureau voor de Statistiek

Time-series project (3)Time-series project (3)

• Time-series 2001-2009: consequencesTime-series 2001-2009: consequences

• Difficult to understand what the machine is doing (black box)Difficult to understand what the machine is doing (black box)

• Planning deadlines were not metPlanning deadlines were not met

• Hardly any documentationHardly any documentation

• Data results machine less than optimal: extensive manual interventions Data results machine less than optimal: extensive manual interventions => publication is on provisional basis=> publication is on provisional basis

• Highly motivated team but tension and frustration due to adversitiesHighly motivated team but tension and frustration due to adversities

Centraal Bureau voor de Statistiek

Time-series project (4)Time-series project (4)

• Time series 1995-2000: gaining experienceTime series 1995-2000: gaining experience

Compilation process split into parts withCompilation process split into parts with::•Firstly a basic run without some constraints (no commodity balancing) => less Firstly a basic run without some constraints (no commodity balancing) => less adjustments by machineadjustments by machine•Secondly manual intervention to solve major problems (imbalances)Secondly manual intervention to solve major problems (imbalances)•Final run to solve minor inconsistencies and restore all constraintsFinal run to solve minor inconsistencies and restore all constraints

Results were much betterResults were much better•Better understanding of the machine outputBetter understanding of the machine output•Less or almost no manual intervention neededLess or almost no manual intervention needed

Centraal Bureau voor de Statistiek

Time-series project (5)Time-series project (5)

• Time series 1995-2000: some lessons learnedTime series 1995-2000: some lessons learned

• Takes a lot of time to smoothly run the time-series machineTakes a lot of time to smoothly run the time-series machine

• Manual adjustments are mostly complex as they often affect large parts or the whole Manual adjustments are mostly complex as they often affect large parts or the whole time-seriestime-series

• Due to all interdependencies it is important to stick to the planningDue to all interdependencies it is important to stick to the planning

• Complexities warrant an experienced team of compilersComplexities warrant an experienced team of compilers

Centraal Bureau voor de Statistiek

SummarySummary

• Almost 3 years experience with ‘machines’Almost 3 years experience with ‘machines’

• ProsPros• Efficiency gains: in terms of less fte’sEfficiency gains: in terms of less fte’s• More robust statistical processMore robust statistical process• Improved qualityImproved quality

• ConsCons• Investment to develop and to build Investment to develop and to build • Programming errorsProgramming errors• More complex: takes time to understand what the machine is doingMore complex: takes time to understand what the machine is doing