Integration of the ICAT API at Institut Laue-Langevin (ILL) (Grenoble, France) Tuesday, August 25, 2009 Jean-François Perrin ICAT developper Workshop (25/26 August 2009)
Dec 29, 2015
Integration of the ICAT APIat
Institut Laue-Langevin (ILL)(Grenoble, France)
Tuesday, August 25, 2009 Jean-François Perrin
ICAT developper Workshop (25/26 August 2009)
Agenda
• Overview (ILL - Project)
• Data(file) import (ILL data import into ICAT)
• User Interface (web-based user Frontend)
• Conclusions (Problems, Questions)
IT Service ILL
• Besides two other IT services at ILL: – SCI (Instrument Control) and CS (Computing for science)
• Responsibilities (groups): – Infrastructure (network and phone)– DB – Usersupport– Developpement – Administration
• ICAT Team (Developpement group): – Holger Gebhard– Najor Cruz-Cruz (since July)
ILL-ICAT-Project
• Current state at ILL (IDA application): – text and script based system, – user must know how to retrieve date (not a search)
• 3 steps to integrate ICAT at ILL– replace existing (IDA) functionalities– public, more comfortable access for a larger group of users– Extend meta-data base and search functionalities
• Practical point of viewInstallation and exploration data-import GUI (Web App)
Data Import 1• General:
– Since 1973 all ILL data are available online– about 40 active instruments (60 overall) – Filesystem organised by cycle – instrument
stored on NFS (sometimes CIFS)– Total number of files: 29.145.595 (148 cycles)– Average size: 229 kb max ~ 40 mb (compressed)
• ILL formats: – Historical ILL Formats (ascii) – Nexus (2 Instruments)– edf (proprietary binary format)
• Impacts for ICAT integration at ILL relevant parameters are specific to (groups) of instrument(s). Need for expert advice
Data Import 2
Example: Import first cycle of 2006 for instrument d22 No relation to proposals IDA replacement !
Analyse directory levelAnalyse file level
Map files to dataset
Integrate found•Data files•Data sets •Investigationsinto Database
…/061/d22/
015673.Z
…
021354.Z
…/061/d22/
015673.Z…
015713.Z
….
020967.Z… 021354.Z
Read parameters and metadata
One specific parser class per (group) of instrument(s)
Data Import 3
• DATASET
• DATAFILE• DATAFILE_PARAMETER
• INVESTIGATION
• ICAT_AUTHORIZATION: – (user_id = any, role=downloader)
• PARAMETER– (ex. wavelength, pressure, voltage)
•DATAFILE_FORMAT•ILL Nexus, ILL ASCII
•INSTRUMENT (# 110, 61 activ)
•FACILITY_CYCLE (#142)
•INVESTIGATION_TYPE
•DATASET_STATUS (complete)•DATASET_TYPE(experiment_raw)
•USER_ROLES (some testusers)
Used (filled) tables:
ILL User Interface
Available in a stand-alone application and/or integrated into intra-/internet
As described in documentation: Compile WSDL togenerate remote proxy and classes for business delegates
Call remote proxy methods to interact with ICAT API
Feedback
• Tests and developments done so far worked fine for us
• ICAT Installation: straightforward – Database Scripts, ICAT API– Test with a dummy prototype
• Certificate handling and user authentication during installation
• No performance critical tests
ILL’s open issues
ICAT integration: • Mapping between data files and proposal data
no proper notion of "ICAT investigation" at ILL • Specimen data (and others) are stored with proposal• Control of data format and parameter variety
General• ILL Data Policy
– data ownership – delay until public availability
• Unification of data file formats– Competence and responsibility– Complexity
Some open issues
• Search by parameter in ICAT
• Future prospect: Opening ICAT’s Authentication mechanism to ILL’s SSO/LDAP
• Future prospect: Search between ICAT facilities repositories
Technical envorinment
• Hardware and Database– Virtual Server (Suse SLES 10.2) for ICAT– Oracle 10g (Cluster RAC) as DB
• Development Environment:– Java 1.6, – Glassfish 1.4, – Ant 1.7.1
• ICAT version: 3.3.3
Question details searchable parameters
Parameter table:‘Wavelength’ , ‘Å’, ‘Amstrong’ , Y, Y, N, Y, Y, ‘Desc Wavelenght’, Y, 999, GEBHARD, 31/07/09 02:20, 31/07/09 02:20,
‘NAJOR’, Y, N
Also: Voltage and Pressure
Join to tables: data_file_parameter and data_set_parameter
Known searches: • SearchByAdvanced• SearchByKeywords• SearchByRunNumber• SearchByUserId• SearchByUserSurName• SearchDatasetsBySample
Advanced Search Fields private java.lang.String datafileName; private java.util.Calendar dateRangeEnd; private java.util.Calendar dateRangeStart; private java.lang.String experimentNumber; private java.lang.Long grantId; private java.lang.String[] instruments; private java.lang.String investigationAbstract; private uk.icat3.client.InvestigationInclude investigationInclude; private java.lang.String investigationName; private java.lang.String investigationType; private java.lang.String[] investigators; private java.lang.String[] keywords; private java.lang.Double runEnd; private java.lang.Double runStart; private java.lang.String sampleName; private java.lang.String visitId;
ILL-ICAT-Project 2
From the practical point of view• ICAT installation and exploration
• Data Import (File metadata to ICAT, ~ 10 of 40 instruments)
• Webbased User Interface (Simple version, planned for end of the year)
• Extension of metadata and userinterfaceAutomated, continious data import (all intruments, all cycles)
Data File exampleRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR 068762 0 4 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 80 1 Inst User L.C. Date Time D19 tf sam 14-May-09 10:16:11 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 80 1 Title Scantypebackgr om-mono IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 31 4 nvers ntype kctrl manip nbang nkmes npdone jcode ipara ianal imode itgv iregul ivolt naxe npstart ilast1 isa flgkif ih ik nbsqs nb_det nbdata icdesc1 icdesc2 icdesc3 icdesc4 icdesc5 icdesc6
4 2 4 8 1 21 21 1 1 0 0 0 1 0 1 0 0 0 0 0 0 0 1 163840 8 0 0 0 0 0FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF 50 10 H (Hmin) K (Kmin) L (Lmin) phi chi omega 2theta (gamma) psi ub(1,1) ub(1,2) ub(1,3) ub(2,1) ub(2,2) ub(2,3) ub(3,1) ub(3,2) ub(3,3) wavelength dmonochromator danalyser energy Hmax Kmax Lmax DeltaH-0.10000000E+01 0.50000000E+00 0.50000000E+00 -0.49880001E+02 0.15721400E+03 0.23701000E+02 0.64004000E+02 0.00000000E+00 0.12238280E+00 0.53997743E-01 -0.64825200E-02 -0.53893240E-01 0.12252440E+00 0.37250000E-02 0.69433160E-02 -0.74291900E-03 0.14317410E+00 0.24220001E+01 0.10000000E+01 0.10000000E+01 0.00000000E+00 0.00000000E+00 0.00000000E+00 0.00000000E+00 0.00000000E+00