Report from A+M Data Centre, RRC “Kurchatov Institute” Yu.V.Martynenko 19th Meeting of the Atomic and Molecular Data Centers and ALADDIN Network Vienna,

Report from A+M Data Centre,

RRC “Kurchatov Institute”

Yu.V.Martynenko

19th Meeting of the Atomic and Molecular

Data Centers and ALADDIN Network

Vienna, 03-05 October 2007 

The main activities on A+M/PSI DATA:

1. The new A+M Data generation. (Experiment, theory, codes).

2.  Computer code for tokamak plasma processes

3. Data Acquisition System + (DAS+). This software is developed for operation with experimental

data of various devices of controlled nuclear fusion (storage, transmission, processing and results representation). The software allows to create Data warehouse for data and gives possibility for multi-devices fusion researches.

.

3. Data base.

3

DATA GENERATIONA+M

1. Atomic particles cross section measurement (split beam methods).

2. Theory of atomic particles interaction.3. Analytic models and codes for atomic

kinetic and radiations.

PSI1. Surface erosion and modification• ion beams, plasma exposition• theory2. H retention and permeability

4

COMPUTER CODES FOR TOKAMAK

PLASMA PROCESSES

The integration of already existing codes

for comprehensive SOL description:

- Plasma dynamic, convection transport (Shurygin / Pastukhov)

- Universal Numerical Code for

atomic kinetics and radiation (V. Lisitsa)

- Material response on plasma affect (Martynenko, Moskovkin)

(DB sputtering yields and reflection coefficients)

5

THE UNIFICATION OF ACCESS TO EXPERIMENTAL DATA OF FUSION DEVICES: Т-10, GTB, PN-3, S300, L-2M, Tuman, Globus, PS-3, Castor NFI RRC “Kurchatov Institute”. http://t10.fusion.ru, [email protected], M. Sokolov.

This software is developed for operation with experimental data of various devices (storage, transport, processing and results presentation). It allows to create the Data warehouse and to carry multi devices researches.

The software includes:• DB DASSQL for storage of the diagnostic and descriptive information of various devices.

• Local archives DASFile for transport of data and off-line work with fragments of DB DASSQL.

• Network service DASWeb for remote access to DB DASSQL from the Internet browser.

• Network service DASTCP for remote access to DB DASSQL from applications.

• Cross-platform library DAS Access for creation of user's applications in the environment of MS Windows and Linux, having remote access through the Internet to DB DASSQL, and also to local archives DASFile.

•The set of Mex-functions of data reading from DB DASSQL and DASFile in MATLAB.

• Utilities DASText, DASLoad and DASDec for data exchange as the text and binary files between DB DASSQL, DASFile and user's applications.

• Multifunctional graphics shell DASTools with a package of mathematical processing DAS Processing for operation with experimental data of plasma devices.

Data Acquisition System +

6

Data files

Application server

High perf. PC

Runs servicesand data handlingroutines.Acts as a server for remote terminals.

Data math.processing

DB loading/extracting

Data files reading

Service1 for client programs (TCP)

Service2 forremote

terminals

.

System modules and data flows.

T-10

_||_… _||_

Experimental shots

Data Readers

Industrial PCs with ADCs installed.Read multi-channelraw data.Compress data.Create data files.

Data Center

High perf. PC or blade-servers unit.Integrates and saves many data files into one, loads data into database.

Database Server

High perf.PC.SQL-databasestores raw and processed data of experiments, incl.data from other experim. devices.Currently 200 Gb +

Web server

High perf. PC

Runs Website with static descriptive information and facility of interactive access to the experimental data and to do more calculations.

Service3 for Web clients to access

data via browsers

Service4 for moremathematic calculations

to be executed

(http)

Data files

Data convertersImport data of other experimental devices. Export data into other formats (Text, XML, XLS, GIF,…)

Compressed data

Researchers

LAN / Internet

Researchers are provided with full-featured program utilityfor selecting, viewing and analysing the data.

IntegratedData file of a shot300 Mb +

Researchers can integrate provided library modulesinto their specific applications, and thus gain access to the data.

Efficientbinary format

…

Other experimental devices

7

Program facilities for data access and analysis.Researchers can access the experimental data in several ways:

Internet browser with HTML-input forms as interfaces to access the experimental data.The data are selected and rendered on the server and the results are displayed in the user’s browser,in binary, text, graphic, image and media presentation.

Internet browser (SigmaWeb) with input forms allowing users to run certain mathematic routines on the server with user-defined initial data.The calculations are carried out on the server and the resultsare displayed in the browser.

Other user applications for experimental data processing and presentationcan integrate library modules which implement connection to Application Server and thus the applications gain access to the experimental data.The user applications list includes Matlab, MDS, any Pascal,C++,Python, …- programs.

Full-featured program utility for data selecting, viewing and analysing,with a graphic user interface and rich variety of data processing and presentation ways (tables, 2D-, 3D-graphics, scaling, profiles).Can run on users’ PCs (usually in LAN of the experimental installation)or on Application Server while in remote terminal mode*. Can work with local copies of the experimental data (data files).Provides a unified user interface for researchers (remote included).

to Service1

to Service2

to Service3

to Service4

to Service1

*remote terminal mode is an advantageous way of using full-featured program utility to access the data via the Internet, as it results in minimal data traffic over the network.In this mode a user is provided with the program interface, while fetching the data, calculations and rendering are carried out on the server.

8

Flow-chart of the main components

Database

DASSQL

Internet service

SigmaWeb

Internet service

DASTCP

Remote access Data sources

Graphics shell DAS Tools

Internet browser

Program library DAS Access

Utilities: DASТехт, DASLoad, DASDec

Applications and codes. Mathematical packages like MATLAB MDS Plus, IDL

HTTP

TCP

DASFile

Data acquisition systems: T-10 (DASHost) CASTOR TUMAN …

Codes

Decoders

Local archives DASFile

Files: ASCII, media, image, binary

Internet service

DASWeb

9

Unification of data access for various devices

Common DB DASSQL data of device-1 … data of device-N

DASTCP DASWeb

Shared data access

Data warehouse REMOTE USERS The common tools for the analysis of experimental data:

Internet browser DASTools DASAccess-Applications DASMex-MATLAB …

MDSPlus

DASFile

DB of device-1 DASTCP / DASWeb Decoder

DB of device-N DASTCP / DASWeb Decoder

…

Unified databases

DASSQL of device-1 DASTCP / DASWeb

DASSQL of device-N DASTCP / DASWeb

…

TCP/HTTP

This software forms the common information system, which is built over the existing systems of data acquisition and over the existing data bases of experimental devices.

10

NETWORK RESOURCES OF T-10

Network services T-10:DASTCP (for DASTools, DASAccess) 195.9.0.8:8888DASWeb http://cpunfi.fusion.ru/dassql/dasweb2.dll/showgl SigmaWeb http://cpunfi.fusion.ru:8088/SIGMAFRAME.htmlWebsite T-10:http://T10.fusion.ru (also, useful links at the page “Database”)http://T10.fusion.ru/JE page of Joint experimenthttp://T10.fusion.ru/Files page of shared documents (from directory JE T10.WWW)http://T10.fusion.ru/DAST10 page of software DAS T-10Network T-10 disks:Netware network, server Pls:\\pls\nw1 DAS T-10 software and documentation\\pls\nw2 T-10 users archives\\pls\jet10 common directory of JE T10 participantsT-10 disks also can be accessed as:MS Windows network, workgroup Soflab, server NS:\\Softlab\\NS\DASPLS DAS T-10 software and documentation\\Softlab\\NS\T10Common T-10 users archives\\Softlab\\NS\JE T-10 Common directory of JE T-10 participantsT-10 disks also can be accessed via FTP.FUSION.RU (secure SSL is preferred, for example WS_FTP client): /!JE T-10 Common directory of JE T-10 participantsThe Common directory of JE T-10 participants contains two folders:JECommon folder for archives and files exchangeJEFile.WWW files from this folder are published on website http://T10.fusion.ru/Files

11

Structure of data records

Data record

Header Passport of a variable length (up to 2Gb)

Data of a variable length (up to 2Gb)

Name of device

Shot number Name of record Hyperlink Date-time Record type Comment The auxiliary fields

Metadata (XML or the Train of descriptive parameters):

Date-time Text (string) Number Structures (binary

string) Data records

DB DASSQL

The quick reference on shots

(Indexed parameters)

The quick reference on devices,

diagnostics, channels and indexed parameters The main types:

Variable arrays Tables of variable

dimension Media (video) Images ASCII Documents

The fields Passport and Data are not mandatory; they have the variable size and may be compressed.At storage and data transfer via a network, the data compression by a combination of delta-method and open algorithm Zlib is carried out.

12

Tree of records in DASSQL Device 1 (Globus)

… Device N (T-10) Shot 1

… Shot M Indexed parameters for the quick reference on shots

Additional shot’s description (record “0.inf”) Diagnostics Diagnostic 1

… Diagnostic K (@Ece) Record 1

… Record J (Ece.xxx)

SQL search:•Device, Shot, •Date Time •Diagnostics, Name•Commentary•Indexing parameters (B, Ipl,U, …)

Data ownership (Signals and parameters):•Group of names - Ch001..Ch999•Domain – Ch.*

Key: Device-Shot-ParameterName. Values: Double / Datetime / Text

The dynamic table of Indexing parameters:

13

Various ways of data access and viewing.

DAS Tools

MATLAB

DASWeb - Интернет браузер

MDS Plus

14

DASSQL data acquisition system, storage, access and device search

• Choice of data by criteria using the resources of SQL• For pulse experiments• Used in several Russian facilities• Rich set of interfaces• Unified format and the common methods of operation with

various data. • Data compression in 3-6 times at storage and transmission via

Internet.• Remote multi-user operation with data of many devices.• Specialized processing and the unified representation of

results.• Possibility of the subsequent integration with the Grid

infrastructure.

15

DASSQL/DASFile -- MATLAB

Applications/server MATLAB

Mex-functions DASMex

Library DAS Access

Internet service DASTCP

Database DASSQL

Local archive DASFile

Reading from the DASSQL/DASFile » DASMexDevices - get list of devices » DASMexShots - get list of shots » DASMexShotInf - get description of shot » DASMexNames - get list of names » DASMexData - get data » DASMexParamByName - get parameter

ADO/TCP

TCP/IP

Examples:[Names, ErrorMessage] =DASMexNames (‘dassql.ini’, ‘T10’, 37856);[Data, Passport, ErrorMessage] =DASMexData (‘dassql.ini’, ‘T10’, 37856, ‘Bol.d-01’, 1, 500, 1e5, 7e5);

16

DASWEB

• DB DASSQL are automatically published in the Internet.

• The Web-interface provides formation of inquiries through the Internet-browser.

• Dynamic creation HTML and images, compression transmitted by Deflate.

• Data processing are made on the side of a server, results in the form of graphic (GIF), tables (ASCII) or binary (DAS File) and transmitted to the user.

17

SigmaWeb: Run codes Start of a code is carried out on a following chain:{The client, the Internet a browser} - Internet HTTP/Post- {Application server, SigmaWeb} - Files: Input, OutPut, Errors, Status - {the Code}At start on platform SigmaWeb the code representing the simple console application working with files, becomes multi-user and receives the graphic interface in the form of the Internet of a browser. The code can represent group of the executed modules incorporated in a command file. Performance of a code in the multi-user mode is provided with creation of a temporary context for each started copy of a code. All network connections of clients with SigmaWeb are started in separate streams.

SigmaWeb console on Applications server.

18

SigmaWeb: Run math code AT-Light on the Server (Service4)

Задание параметров для кода

Результаты выполнения кода

Setting the initial data -> Viewing in browser the results of the

calculations carried out on the server.Sending request ->

19

DAS TOOLS• Access to a DB via Internet.

• Work with local archives, text, binary and media files.

• Effective navigation and work with data sets.

• Fast mathematical processing - DAS Processing.

• Integrated with the special codes.

• Wide set of means of data presentation.

• Work with big multichannel data and media formats.

20

DAS PROCESSING

1. Methods of the moving average (MA, WMAH, WMAT, Average).2. Median filter.3. Subtraction of shelf (Zero line).4. Transformation of the AXIS «Y» (dimension).5. Transformation of the AXIS «X» (dimension).6. Suppression of casual emissions.7. Inverting of the signal.8. Allocation of positive values of the signal.9. Detailed elaboration.10. Extracting.11. Normalizing the signal on the maximum.12. Calculation of the chord.13. Interpolation given by multinominal of Lagrange.14. Derivatives and integrals.15. Data presentation of multichannel diagnostics in the form of structures and

surfaces.16. Addition, multiplication and division for groups of signals.17. Fourier analysis.18. Use of graphics of harmonious fluctuations.19. Fitting of function approximation of many variables to experimental data.

Methods DAS Processing are accessible through network services DASWeb and DASTCP, library DAS Access and task interface DASText.

21

Nuclear Fusioninter-facilities collaboration

A lot of standards applied

Collaboration:No common interoperation

standardAsdex

Mast

JET

TCVTore Supra

W-7X

L-2M

GOL-3

T-11M

T-10

Globus

FTU

Textor

22

EGEE

infrastructuremiddleware &

Asdex

Mast

JET

TCVTore Supra

W-7X

L-2M

GOL-3

T-11M

T-10

Globus

FTU

Textor

Unified interoperation needed

Nuclear Fusioninter-facilities collaboration

23

PROSPECTIVE

The DAS is going to be modernized to accommodate increasing amounts of experimental data.Modern electronic equipment and computational concepts are going to be used,including blade-servers and parallel multi-channel data processing.

The System future development will keep up a cooperation with other researchers for working out common standards for better interoperability of different data-systems in nuclear fusion experimental field.This cooperation includes issues of integration into scientific structures based on Grid-technologies and other prospective formations.

The System as the whole or its parts can be used in different applications: - development of new data-systems for any devices;- integration its data viewing/processing/analysis utilities into existing data-systems;- storing large amounts of data from other data sources in the System database with ability to manipulate this data with the DAS utilities.

24

CONCLUSION

This software forms the information system, which is built over the existing systems of data acquisition and over the existing data bases of experimental devices.

At present, the records from devices: T-10, GTB, PN-3, PS-1, S-300 (RRC KI), CASTOR (host lab experiment, IPP, Praguesep.2005) and also separate shots from devices: TUMAN-3M, GLOBUS, L-2M are presented in the DB DASSQL.

The following possibilities of operations with experimental data have been realized:

• Choice of data by criteria using the resources of SQL• Rich set of interfaces• Unified format and the common methods of operation with various data. • Data compression in 3-6 times at storage and transmission via Internet.• Remote multi-user operation with data of many devices.• Specialized processing and the unified representation of results.• Possibility of the subsequent integration with the Grid infrastructure.