25th & 26th August 2009ICAT developer workshop 1.
Post on 03-Jan-2016
213 Views
Preview:
Transcript
125th & 26th August 2009
ICAT developer workshop
NXextract
Extracting too l for NeXus filesS téphane P o ir ie r – S ynch ro tron S o le il
225th & 26th August 2009
ICAT developer workshop
Overview
Introduction
Writing templates
Real example
Main usages
Future
325th & 26th August 2009
ICAT developer workshop
Introduction
Why does we need a extracting tool ? Because the analysis software the users use to process data are not yet
able to read NeXus files
What were the Requirements A tool able to extract any kind of data from NeXus files and write it a
arbitrary file format (in ASCII or binary mode) Output description as simpliest as possible Efficiency
2 possibilities Extending a scripting language like Python Developing a specific tool that process template files describing the
output
Why choosing the second way ? The templates have to be as short as possible while keeping a good
readability: they must be understood even by a scientist ! Fast to execute Easy to deploy (very few dependencies, apart from NeXus and HDF5)
425th & 26th August 2009
ICAT developer workshop
Introduction
CapabilitiesExtracting any kind of data
Writing one or more files (ASCII or binary, it's not a matter) from a single NeXus file
Directly writing 2D data as jpeg or bmp files (png format will be added soon)
525th & 26th August 2009
ICAT developer workshop
Writing a template
In a template you describe the extraction in a “WYSIWYM”-like way (What You See Is What You Mean)
Just write that you want to get
The result is written on the console
Defining a output file
No keywords, only symbols !
Actually, it's not a language just a way to describe a output format
"Some text\n""Value = %?\n" nxs:/path/to/a/dataset
Some textValue = 1
> "out.txt"
625th & 26th August 2009
ICAT developer workshop
Writing a template
Looping over a collectionThe NXentry groups
Testing values, attributes or dataset presence
Defining data blockNot directly written on
output
Defining and using variablesRecursive evaluation of
content
@( entry = [nxs:/<NXentry>/]...@)
?( nxs:/path/to/dataset = 2...?)
( binarydata # output into this bloc as 2 bytes unsigned integers 'ui2' nxs:/path/to/a/group<NXdata>/data)
% image_grp = /path/to/a/group"value: %g\n" nxs:$(image_grp)/a_dataset
725th & 26th August 2009
ICAT developer workshop
Writing a template
Looping over values index is in range [0,
array_size[
Inserting sub-templateThe sub-template inherit
variables and data blocks of its parent
@( i = 0, nxs:/path/to/a/group/dataset._size_(1) "%g\n" nxs:/path/to/a/group/dataset[i]@)
@( entry = [nxs:/<NXentry>/] + 'operation.nxe'@)
825th & 26th August 2009
ICAT developer workshop
Real example
A script used on the SWING beamline used to extract images data from Nexus file and output them as EDF (ESRF Data format) files.
Characteristics of the output formatThe file is made of 2 parts: a ASCII header and a binary image
data
ASCII Header length must be a power of 2
a lot of cabbalistics fields must be filled
bina ry data must be 2 bytes integer
Input NeXus files contains several image, each image in a dedicated NXdata group and named image#i with i in range [0, image_count[ (I known, a image stack would be a better way for storing data in that case)
925th & 26th August 2009
ICAT developer workshop
Main usages at Soleil
During the data acquisition process a extraction is (most of the time) applied automatically for each NXentry group or for the whole NeXus file (on 7 out of 11 beamlines)
Our data portal application allows users to download NeXus files as well as on the fly extractions using nxextract on server sideTemplates depends of the beamline
1025th & 26th August 2009
ICAT developer workshop
Future
Adding PNG format support
Defining standard template(s) for exporting data in CIF or imgCif files ? (but no interest for Soleil up to now)
Embeding the Python engine ?Allowing to define data block filled by Python script, like this:
( integer_array 'i2' nxs:/path/to/a/dataset)py( toto import os toto = integer_array[0]py)'b' toto
top related