7/29/2019 Data Transfer Mechanisation_DST001CEn
1/22
UNSD - OECD Joint System
Data Transfer Mechanism - Technical SpecificationsUN Comtrade Version
Ref : 135664293.doc
7/29/2019 Data Transfer Mechanisation_DST001CEn
2/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page ii
1 Feb 2006
INFOOECD Ref: 135664293.doc
7/29/2019 Data Transfer Mechanisation_DST001CEn
3/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page iii
UPDATE FORM
Name and Versionnumber
Sent By Signed Approved By Signed
DST001C 1 Feb 06 OECD/STD
DISTRIBUTIONAndreas Lindner OECDTrevor Fletcher OECDGrgory Legoff OECD
Lynda Hawe OECDRonald Jansen UN
Markie Muryawan UNHenri Kayali Info
VERSION DATE OPERATION SPECIFICS
A 19 Oct05
Creation of the document
B 12 Dec05
Add-ons: Comtrade Distribution details (operating mode)
C 1 Feb 06 Cover page and Introduction edited
INFOOECD Ref: 135664293.doc
7/29/2019 Data Transfer Mechanisation_DST001CEn
4/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page iv
TABLE OF CONTENTS
1. INTRODUCTION .................................................................................................................................................... ......1
1.1. WHAT IS MEANTBY JOINT SYSTEM? .......................................................... ............. ............. ............. ............. ..........11.2. ABOUTTHIS DOCUMENT............................................................................................................................................. 21.3. ASSOCIATED DOCUMENTS........................................................................................................................................... 2
2. OVERVIEW ...................................................................................................................................................... ......... ....3
3. TECHNICAL SPECIFICATIONS ............................................................................................................................ ...4
3.1. REQUIREMENTS......................................................................................................................................................... 43.2. DATA STORAGE FORMATS........................................................................................................................................... 4
3.2.1. Data Capture ........................................................................................................................................ ......... .......43.2.2. Data Processing and Dissemination ............................................................................................................. .......4
3.3. DATA FLOW FORMATS............................................................................................................................................... 53.4. DATA TRANSFER PROTOCOLS...................................................................................................................................... 63.5. SECURITY................................................................................................................................................................ 73.6. SYNCHRONIZATION ................................................................................................................................................... 7
4. OPERATING MODE .......................................................................................................................................... ......... .8
4.1. DOWNLOAD COMTRADETOOLS SETUP FILES................................................................................................................... 84.2. INSTALL COMTRADETOOLS......................................................................................................................................... 94.3. TEST WEB SERVICES............................................................................................................................................... 114.4. DOWNLOAD DATATOA CSV FILE............................................................................................................................. 124.5. USINGTHE CONFIGURATION FILE................................................................................................................................ 14
4.5.1. Change Column Mapping ...................................................................................................................................154.5.2. Use ProjectID ...................................................................................................................................... ......... ......15
4.6. IMPORT DATAINTOA SQL SERVER DATABASE............................................................................................................ 164.6.1. Check SQL Server Updates .......................................................................................................................... ......164.6.2. Create the Destination Table .................................................................................................................... ........ .16
4.6.3. Define the Connection String ........................................................................................................................... ..164.6.4. Define the Column Mapping Between CSV File and Database Table ......................................................... .....174.6.5. Import a Dataset into the Table Debug Mode ......................................................................................... .......174.6.6. Import Datasets ...................................................................................................................................................174.6.7. Use the New Resume Parameter ................................................................................................................... ...17
FIGURES ................................................................................. ............. ............. ............. ............. ............ ............. ...18
INFOOECD Ref: 135664293.doc
7/29/2019 Data Transfer Mechanisation_DST001CEn
5/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page 1
1. Introduction
1.1. What Is Meant by Joint System?
Since 2000, UNSD1 and OECD2 have joined forces to reduce the response burden for national data
providers in the field of merchandise trade statistics and to agree on one unique and transparent set ofdata instead of having parallel (and often not harmonized) data collections. This considerable amount ofvery detailed investigation has led to the signing of an UNSD/OECD Memorandum of Understanding inJanuary 2001 andfurther extensions, where UNSD and OECD jointly developed
(i) an agreement on data collection and data sharing,(ii) common trade data processing standards, and(iii) computer applications for use in processing of trade data.
This set of agreements, standards and applications is referenced by both organizations as:
UNOECD Joint Trade Data Collection and Processing System (Joint System)
Both organizations agreed to use the above name in any documents describing their cooperative
activities. It is also to be used when referring to the respective databases.
The Joint System is the foundation of the trade databases maintained by both organizations. Bothorganizations continue to keep separate (institutionally and legally) trade databases to ensure rights andability of each organization to pursue their own policies with regard to
(i) storing various data in addition to those agreed in Joint System [e,g., historical data,memorandum items, derived data etc],
(ii) developing additional analytical and presentational functionalities and controlling user access tothem, and
(iii) dissemination practices including pricing of user access to the database services, entering intocontracts with users, etc.
Each organization maintains its own brand name for use in data dissemination activities. The names ofthose databases are:
For UNSD: UN Comtrade3
For OECD: OECD ITCS4 Database
To ensure user awareness of the joint efforts of both organizations in the area of international merchandisetrade statistics, both organizations agreed to make data responsibility very clear.
Consequently, the UN Comtrade disclaimer contains the following statement:
Beginning 2005, trade data of OECD member countries are compiled by OECD in accordance withUN-OECD Common Trade Data Processing Standards and are available in UN Comtrade inaccordance with UN/OECD Data Sharing Agreement. Any deviation from this practice is reflected
in the country/period metadata.
Similarly, the OECD trade database disclaimer contains the following statement:
The trade data for the selected non-OECD member countries are compiled by UNSD inaccordance with UN/OECD Common Trade Data Processing Standards and are available in OECDITCS Database in accordance with UN-OECD Data Sharing Agreement. Any deviation from thispractice is reflected in the country/period
1 United Nations Statistics Division
2 Organization for Economic Co-operation and Development3 Commodity Trade Statistics Database4 International Trade by Commodity Statistics
INFO/OECD Ref: 135664293.doc
7/29/2019 Data Transfer Mechanisation_DST001CEn
6/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page 2
1.2. About this Document
This document makes up the technical specifications of the Joint System data transfer mechanism(DTM) between UNSD and OECD databases. DTM is introduced in Section 2. below.
Data transfers between UNSD and OECD are the responsibility of specialized OECD statistics database
Administrators.
Word links are available throughout this document like for instance, at the beginning of the section.
Pictures include detailed comments in callouts or in their captions. These comments contribute totrimming down the text and making it clearer. This layout is generally appreciated by non-expert readers.
1.3. Associated Documents
The UNSD OECD Joint System package also includes the following documents:
Data Capture / Data Processing Manual Web Browser Client User Manual Database Administrator Client Manual Database Dissemination Model Guide
INFO/OECD Ref: 135664293.doc
7/29/2019 Data Transfer Mechanisation_DST001CEn
7/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page 3
2. Overview
As mentioned in the previous section, UNSD and OECD will use a unique logical storage to make theirstatistical data available through the Web.
In fact, the Joint System will still use two physical SQL Serverdatabases, one for each organization, tostore the collected and processed commodity trade statistical data.
These two databases will be continuously updated and synchronized through a data exchange processexplained later in this document. Thus, the databases will stay identical regarding their logical contents. Aspecific flavor will also be added to each database contents to reflect OECD and UNSD standards,particular additions and habits.
This process is illustrated below:
ITS Meeting Sept 20055
Common
Processing
Module
Common
Processing
Module
Common
Processing
Module
Common
Processing
Module
Working
Database
Working
Database
OECD UNSD
Working
Database
Working
Database
Data capture Data capture
Process Process
Disseminate Disseminate
Dissemination
Database
Dissemination
DatabaseDissemination
Database
Dissemination
Database
Data Exchange &
Synchronisation
Data Access
Figure 1: statistical data flow for UNSD and OECD in Joint System
Data transfers and conversions are physically carried out by an UNSD software package, namedComtradeTools, which makes use of Web Services. ComtradeTools functions, Configuration file andcommands are discussed in theOperating Mode below.
The purpose of this document is to describe, step by step, the technical specifications of data transfer asrepresented above:
Transfer protocols Data flow formats Data storage formats Security OECD UNSD synchronization Operating mode
Some parts of this document are closely related to the Database Administrator ClientManual.
INFO/OECD Ref: 135664293.doc
7/29/2019 Data Transfer Mechanisation_DST001CEn
8/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page 4
3. Technical Specifications
3.1. Requirements
The OECD Client who wishes to import updated data from the UNSD database should be equipped withthe following software components:
Microsoft Windows 2000 or a higher version; Microsoft .NET Framework version 1.1; Microsoft SQL Server 2000; WinZip program (note: some versions of Windows have a built-in unzip capability)
In addition, the OECD Client should have a basic knowledge of XML files and SQL Server QueryAnalyzer.
3.2. Data Storage Formats
3.2.1. Data Capture
Contributing countries generally send their annual statistical data as plain text, Excel or .DBF (FoxPro)files. In fact, any format is acceptable.
MS Access is used as an intermediate storage for all these files before loading them up to the SQL Serverdatabase. Importation into Access and SQL Server database are carried out through the Get External Data(Access) and Import Data (SQL Server) functions.
Note: data in Access format is kept for archival purposes.
3.2.2. Data Processing and Dissemination
As mentioned before, SQL Server is used to store the statistical data to be processed (WorkingDatabases) and distributed (Dissemination Databases)
Data processing (importation, checking, conversion, aggregation) is described in the Data Capture /ProcessingManual.
INFO/OECD Ref: 135664293.doc
7/29/2019 Data Transfer Mechanisation_DST001CEn
9/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page 5
3.3. Data Flow Formats
The OECD and UNSD databases are kept synchronized through bi-directional data transfers. Thesynchronization process is explained in this document and in the Database Administrator ClientManual.During these transfers, the XML5format is used, with SDMX6 cross-sectional schema. The incoming datais finally converted into CSV7 format and loaded into the destination database.
The following insertion is an example of SDMX cross-sectional schema:
-
-
UN626573874
truefalse
2005-11-10T04:34:38
-
United Nations
UN_COMTRADE_BE
01
Update
2005-11-10T04:34:38
-
--
Figure 2: example of SDMX cross-sectional schema
Link tohttp://www.sdmx.org
for more detail about SDMX.
The same data format is used for transfers between working and dissemination databases (see fig. 1).
The SDMX data is compressed by 70% to 80% before the data transfer.
5 Extensible Markup Language
6 Statistical Data and Metadata Exchange7 Comma Separated Values; e.g., the contents of a plain text file. Such a file can be used by Notepad orExcel
INFO/OECD Ref: 135664293.doc
http://c/Documents%20and%20Settings/simon/Local%20Settings/Temporary%20Internet%20Files/Content.IE5/HXMZPR5L/comtrade_trade_data[1].sdmx##http://c/Documents%20and%20Settings/simon/Local%20Settings/Temporary%20Internet%20Files/Content.IE5/HXMZPR5L/comtrade_trade_data[1].sdmx##http://c/Documents%20and%20Settings/simon/Local%20Settings/Temporary%20Internet%20Files/Content.IE5/HXMZPR5L/comtrade_trade_data[1].sdmx##http://c/Documents%20and%20Settings/simon/Local%20Settings/Temporary%20Internet%20Files/Content.IE5/HXMZPR5L/comtrade_trade_data[1].sdmx##http://c/Documents%20and%20Settings/simon/Local%20Settings/Temporary%20Internet%20Files/Content.IE5/HXMZPR5L/comtrade_trade_data[1].sdmx##http://c/Documents%20and%20Settings/simon/Local%20Settings/Temporary%20Internet%20Files/Content.IE5/HXMZPR5L/comtrade_trade_data[1].sdmx##http://www.sdmx.org/http://www.sdmx.org/http://c/Documents%20and%20Settings/simon/Local%20Settings/Temporary%20Internet%20Files/Content.IE5/HXMZPR5L/comtrade_trade_data[1].sdmx##http://c/Documents%20and%20Settings/simon/Local%20Settings/Temporary%20Internet%20Files/Content.IE5/HXMZPR5L/comtrade_trade_data[1].sdmx##http://c/Documents%20and%20Settings/simon/Local%20Settings/Temporary%20Internet%20Files/Content.IE5/HXMZPR5L/comtrade_trade_data[1].sdmx##http://c/Documents%20and%20Settings/simon/Local%20Settings/Temporary%20Internet%20Files/Content.IE5/HXMZPR5L/comtrade_trade_data[1].sdmx##http://c/Documents%20and%20Settings/simon/Local%20Settings/Temporary%20Internet%20Files/Content.IE5/HXMZPR5L/comtrade_trade_data[1].sdmx##http://c/Documents%20and%20Settings/simon/Local%20Settings/Temporary%20Internet%20Files/Content.IE5/HXMZPR5L/comtrade_trade_data[1].sdmx##http://www.sdmx.org/7/29/2019 Data Transfer Mechanisation_DST001CEn
10/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page 6
3.4. Data Transfer Protocols
The common HTTP8is used all over the Internet via Web Services. Web Services is a remote library (orClass, from a language point of view), published on the Web via XML standards, with properties andmethods, some of which devoted to Web data transfers. In our case, due to data size, REST Web serviceis selected because of its ability to stream and use on-the-fly compression.
The picture below illustrates the Joint System data transfer mechanism between UNSD and OECD:
Figure 3: transfer mechanism between UNSD and OECD
Transfers are Pulled by OECD: data is sent on OECD Users requests to ComtradeTools see OperatingMode below.
8 Hypertext Transfer Protocol
INFO/OECD Ref: 135664293.doc
7/29/2019 Data Transfer Mechanisation_DST001CEn
11/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page 7
3.5. Security
It is based on IP Addresses of Users who wish to connect to the Web Server and on passwords to thedissemination database. The IP address seen by ComtradeTools is the Web access providers, not thelocal network address of the connecting station.
An example ofdenied access is given in section 4. Web Services is also used via the Access Control Listfunction. In addition, the User ID / password combination used when logging on to Windows is used toaccess the CoprAapplication (see Data Capture / Data ProcessingManual).
The secured (encrypted) HTTP, HTTPS, could possibly be used.
3.6. Synchronization
The OECD and UNSD databases are synchronized through a time stamp mechanism insideComtradeTools: each time a data set is imported from reporting countries to the UNSD database, its timestamp is updated to the current date, and a comparison with the previous time stamp tells whether thisdata item must be downloaded to the OECD database to update it.
The format of a time stamp is YYYY-MM-DDTHH:MM:SS (labels in italics areplaceholders)
From a general point of view, the updating data can be filtered by country, year or commodity, in additionto being selected by the time stamp. Examples of such selection parameters are given in the next section.
Selection criteria can be defined directly in the command line or by Keys in the ComtradeToolsConfiguration file. The example below is the definition of such a key:
7/29/2019 Data Transfer Mechanisation_DST001CEn
12/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page 8
4. Operating Mode
This section is an illustration of the operating procedure used to import UN data into the OECD database.The conversion and synchronization phases are also reproduced. An example of column mapping and
SQL Server data creation is provided. In fact, all the useful operations are discussed. Depending on thestatus of the operating workstation and the data to be transferred, some of the operations described herewont be necessary to carry out an import.
As mentioned above, a software package, ComtradeTools, is used to initiate the various transfer /conversion functions. ComtradeTools is a command line program: it is launched from the Run window,like, for instance, the Registry Editor.
The first thing to do is to download the ComtradeTools package, if not already present on the Usersworkstation.
4.1. Download ComtradeTools Setup Files
Get ComtradeTools installation files by linking tohttp://unstats.un.org/unsd/comtradedist/comtradeToolsSetup.zip
The usual Open/Save interface is displayed. The easiest way to proceed is to choose Open; WinZip will beactivated and will display the three expanded installation files (.exe, .ini, .msi):
Figure 5: the three ComtradeTools setup files unpacked by WinZip
Transfer the installation files to a proper local directory(suggested: :\Program Files\UNSD\ComtradeTools; System Drive is usually C).
INFO/OECD Ref: 135664293.doc
Installation package (compressed file)
http://unstats.un.org/unsd/comtradedist/comtradeToolsSetup.ziphttp://unstats.un.org/unsd/comtradedist/comtradeToolsSetup.zip7/29/2019 Data Transfer Mechanisation_DST001CEn
13/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page 9
4.2. Install ComtradeTools
Double-click9the .exe file. A wizard will help you install the Application; in fact, you just have to choose theApplication folder (the installation folder defined during the previous step is the default). The installationprocess takes a few seconds. The following files are created:
Figure 6: the ComtradeTools files
(See next page)
9 Depending on the System options, a single click may be enough.
INFO/OECD Ref: 135664293.doc
ComtradeTools task
Configuration file
Installation / Application folder
7/29/2019 Data Transfer Mechanisation_DST001CEn
14/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page 10
The presence of the Application can also be checked through the usual Add/Remove Programs interface:
Figure 7: Add/Remove Programs interface
This utility should be used to uninstall ComtradeTools, as it is for any other application.
INFO/OECD Ref: 135664293.doc
7/29/2019 Data Transfer Mechanisation_DST001CEn
15/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page 11
4.3. Test Web Services
To make sure that everything is in order especially regarding access rights, a request to ComtradeToolscan be sent through the Web Browser. For instance:
http://unstats.un.org/unsd/comtradedist/getSdmxV1.aspx?px=H1&r=381&y=2003,2002&cc=TOTAL&p=0&comp=false
This request asks for the transfer of a selected data set from the UN database to the OECD Clientworkstation. The selection parameters are identified by keywords like px or r. The meanings of thesekeywords are explained below, in section 4.4. If the transfer operation works fine, the destination is a XMLfile with contents like:
Figure 8: contents of the destination XML file after the transfer (partial view, may need enlargement)
The access control concerns the sender IP address as seen by the UNSD Server, as mentioned in the
Securitysection. If access is denied, the following message is sent:
Figure 9: access to ComtradeTools function denied
INFO/OECD Ref: 135664293.doc
http://unstats.un.org/unsd/comtradedist/getSdmxV1.aspx?px=H1&r=381&y=2003,2002&cc=TOTAL&p=0&comp=falsehttp://unstats.un.org/unsd/comtradedist/getSdmxV1.aspx?px=H1&r=381&y=2003,2002&cc=TOTAL&p=0&comp=false7/29/2019 Data Transfer Mechanisation_DST001CEn
16/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page 12
4.4. Download Data to a CSV File
The next requests will be sent from the Command Line (aka DOS) window. This window pops up whenselecting Start > Run. The following command is an example of download request:
ComtradeTools /r:360 /y:2003 /px:H1 /action:DownloadAndConvertToCSV
The program name, ComtradeTools, is followed by a list of slash-separated parameters. Theseparameters are:
r: reporting country code. Here, 360 stands for Indonesiay: reporting yearpx: classification used. H1 means HS1996action: here, a CSV file will be created with the converted imported data.
The Command Line window with the command (leftmost part) in it is reproduced below:
Figure 10: command line window
Clicking OK submits the transfer request to ComtradeTools. The resulting process can be followed on aDOS pane that pops up:
Figure 11: command line execution error(s) occurred
We see that the transfer failed. Submitting the URL that appears in this DOS pane would return the accessdenied message in fig. 9.
(See next page)
INFO/OECD Ref: 135664293.doc
7/29/2019 Data Transfer Mechanisation_DST001CEn
17/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page 13
Had the access been granted, the transfer would have been successful, with the following messages onthe DOS pane:
Figure 12: command line execution successful transfer
The code of the reporting country is now 8 (Albania): Indonesia would have returned too large a data set.
We notice an extra parameter in the command line: outputDirectorywith C:\Temp as value. This is thelocal directory the CSV file is created in. The name of this file is Temph120018_CSV.txt, with theselected dataset values converted to CSV in it. The destination file name is built from the selectioncriteria: classification, year and country, prefixed by the destination folder.
The picture below shows the first records of the destination CSV file:
Figure 13: dataset downloaded into a CSV file first records
The first line is made of column headers, with, for instance, 8 as reporting country. The next records are
the extracted data values.
INFO/OECD Ref: 135664293.doc
Columnheaders
Another
command line
7/29/2019 Data Transfer Mechanisation_DST001CEn
18/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page 14
4.5. Using the Configuration File
This file has the XML format and contains the definition of Keys that are used by ComtradeTools asparameters. It is part of the ComtradeTools package (seefig. 6)
Its original contents (for the example we use) are displayed below:
Figure 14: the original ComtradeTools Configuration file notice the comments (HTML format)
The keys in this file are defined by their names and values within an add key tag. The 2000Datakey hasbeen introduced in the Synchronization section. In the rest of this section, well give examples of how theConfiguration file can be used.
INFO/OECD Ref: 135664293.doc
7/29/2019 Data Transfer Mechanisation_DST001CEn
19/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page 15
4.5.1. Change Column Mapping
When the original UNSD data (SDMX format) is transferred to a CSV file, and later when the CSV data isimported into the OECD SQL Server database, a column mapping must take place between the sourceand the destination. The correspondence is defined by the two following keys in the Configuration file:
AttributeMapping: SDMXto CSVSQLColumnMapping: CSV to SQL Server table
In the Configuration file, well first change the AttributeMapping entry:
to:
We notice that:1. two correspondences have been removed: RPT=reporter; time=year.
2. two other correspondences have been added:REPORTED_CURRENCY=cur; CONVERSION_FACTOR=cf.
We can then run the transfer request again:
ComtradeTools /r:8 /y:2001 /px:H1 /action:DownloadAndConvertToCSV OutputDirectory:C:\temp
And compare the new CSV file we get with the previous one (fig. 13).
Finally, we restore the two removed correspondences (reporter and year) in the AttributeMapping key.
SQLColumnMapping will be used later.
4.5.2. Use ProjectID
Keys can be added to the Configuration file to replace a parameter string in the ComtradeTools command.These keys (and therefore the parameter string) can be submitted via the ProjectID label in the commandline. For instance, we can insert the following definition in the Configuration file:
The name of the key is IndonesiaH1 and its contents make a selection by reporting country(360=Indonesia) and classification (H1=HS1996). LastSynch is reset and TIMESTAMP is used assynchronization option. The command line can now be:
ComtradeTools /y:2003 /action:DownloadAndConvertToCSV /outputDirectory:c:\temp\sdmx\/ProjectID:IndonesiaH1
The transfer and conversion to CSV will be carried out, according to the parameters in IndonesianH1 andin the command line. When the process is completed, the timestamp will be updated to keep thesynchronization process in order.
INFO/OECD Ref: 135664293.doc
7/29/2019 Data Transfer Mechanisation_DST001CEn
20/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page 16
4.6. Import Data into a SQL Server Database
4.6.1. Check SQL Server Updates
The User should first make sure that the sql2ksp3.exe Client component is installed on their workstation bylinking to:
http://www.microsoft.com/sql/downloads/2000/sp3.asp file
If this component is missing, the following error will occur:
"Unhandled Exception: System.InvalidCastException: QueryInterface for interface DTS.CustomTask failed."
Of course, this operation should be carried out only once for a given system.
4.6.2. Create the Destination Table
The following commands and source data should be submitted to SQL Server Query Analyzer, either fromthe keyboard or from a script file, to create the destination table in a database (for this example, anyexisting database managed by SQL Server will do):
if exists (select * from dbo.sysobjects where id =object_id(N'[dbo].[tempLoad]') and OBJECTPROPERTY(id, N'IsUserTable') = 1)drop table [dbo].[tempLoad]GO
CREATE TABLE [dbo].[tempLoad] ([pfCode] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,[yr] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,[rgCode] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,[rtCode] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,[ptCode] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,[cmdCode] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,[cmdID] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,[qtCode] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
[TradeQuantity] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,[NetWeight] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,[TradeValue] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,[source] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,[currency] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,[conversion_factor] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL) ON [PRIMARY]
GO
Figure 15: SQL Query Analyzer input to create the destination table in a database
This SQL sequence first checks for table tempLoad existence; if the table exists, it is removed from thedatabase. tempLoad is then (re-)created and its columns are defined with their names, type and length.Here, type=Character and length=50 or 255.
4.6.3. Define the Connection String
The following key should be added to the Configuration fi le:
This is the path to the SQL Server database and the target table.
In this example, Northwind, the standard Windows demonstration database, is used. Optional valueparameters make it possible to customize the destination table according to Year, Reporting Country orClassification.
INFO/OECD Ref: 135664293.doc
7/29/2019 Data Transfer Mechanisation_DST001CEn
21/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page 17
4.6.4. Define the Column Mapping Between CSV File and Database Table
Key SQLColumnMapping in the Configuration file is used. Its value should be:
Compared to the original value in the Configuration file, the new columns currency andconversion_factor have been added.
Constant values (e.g., labels) can also be mapped through the SQLConstValueMapping key. In theoriginal Configuration File, we have:
4.6.5. Import a Dataset into the Table Debug Mode
The ComtradeTools command line is used again to transfer data from the UN database and import theresulting CSV file into tempLoad table in Northwind database, as defined in the Configuration File:
ComtradeTools /r:8 /y:2001 /px:H1 /action:DownloadAndImportToSQLDB
/removeTemp:false /DTSDebug:true
With the DownloadAndImportToSQLDB action, the intermediate step that results in the creation of the CSVfile is transparent to the User its name is still built from the selection parameters. This action is carriedout through SQL Server DTS (Data Transformation Services). The DTSDebug option makes it possible tocreate and examine the DTS package; this is useful for troubleshooting.
Refer to SQL Server documentation for details on DTS.
Once the action is completed the User should check the contents of table tempLoad in the database.
4.6.6. Import Datasets
The IndonesianH1 key has already been appended to the Configuration file:
This key includes the reset of LastSynch (see section 4.5.2)
We can now re-run the direct import with the synchronization option:
ComtradeTools /action:DownloadAndImportToSQLDB /outputDirectory:c:\temp\sdmx\/projectID:IndonesiaH1 /synchOption:Timestamp
4.6.7. Use the New Resume Parameter
As from 15 Aug 2005, a new parameter in the command line makes it possible to resume the selectedComtradeTools action up to n times if errors are encountered. This parameter is used in the command
below:ComtradeTools /action:DownloadAndImportToSQLDB /outputDirectory:c:\temp\sdmx\/projectID:IndonesiaH1 /synchOption:Timestamp /resume:3
The operation will be resumed up to three times until no error occurs.
INFO/OECD Ref: 135664293.doc
7/29/2019 Data Transfer Mechanisation_DST001CEn
22/22
UNSD - OECD Joint SystemData Transfer Mechanism - Technical Specifications UN Comtrade Version
Page 18
FIGURES
FIGURE 1: STATISTICAL DATA FLOW FOR UNSD AND OECD IN JOINT SYSTEM....................................3
FIGURE 2: EXAMPLE OF SDMX CROSS-SECTIONAL SCHEMA........................................................................5
FIGURE 3: TRANSFER MECHANISM BETWEEN UNSD AND OECD.................................................................6
FIGURE 4: THE IMPORT - SYNCHRONIZATION PROCESS................................................................................7
FIGURE 5: THE THREE COMTRADETOOLS SETUP FILES UNPACKED BY WINZIP..............................8
FIGURE 6: THE COMTRADETOOLS FILES.............................................................................................................9
FIGURE 7: ADD/REMOVE PROGRAMS INTERFACE..........................................................................................10
FIGURE 8: CONTENTS OF THE DESTINATION XML FILE AFTER THE TRANSFER (PARTIAL VIEW,MAY NEED ENLARGEMENT)....................................................................................................................................11
FIGURE 9: ACCESS TO COMTRADETOOLS FUNCTION DENIED...................................................................11
FIGURE 10: COMMAND LINE WINDOW................................................................................................................12
FIGURE 11: COMMAND LINE EXECUTION ERROR(S) OCCURRED...........................................................12
FIGURE 12: COMMAND LINE EXECUTION SUCCESSFUL TRANSFER......................................................13
FIGURE 13: DATASET DOWNLOADED INTO A CSV FILE FIRST RECORDS............................................13
FIGURE 14: THE ORIGINAL COMTRADETOOLS CONFIGURATION FILE NOTICE THE
COMMENTS (HTML FORMAT).................................................................................................................................14
FIGURE 15: SQL QUERY ANALYZER INPUT TO CREATE THE DESTINATION TABLE IN ADATABASE......................................................................................................................................................................16