NeXus and Collaboration DANSE kick-off meeting

Australian Nuclear Science & Technology Organisation

NeXus and CollaborationNeXus and CollaborationDANSE kick-off meetingDANSE kick-off meeting

Tony Lam, Darren Kelly &

Nick Hauser

Bragg Institute

August 14-15, 2006

Congratulations from ‘Down Under’Congratulations from ‘Down Under’

Thanks to Brent Fultz driving DANSE to a successful NSF proposal

Thanks to Mike McKerns and Taura Scott for organising the meeting.

1. NeXus “The State Of NeXus” Physica B. In press (October 2006) Downstream developments

2. Collaboration Minimizing effort and maximising impact

3. Components Using your own toothbrush in someone else’s bathroom.

NIAC Meeting - October 2006NIAC Meeting - October 2006

The next meeting of the NeXus International Advisory Committee will be held at the Lawrence Berkeley National Laboratory near San Francisco, California, USA. The meeting will be held from October 5 to 6, 2006, immediately following NOBUGS 2006.

If you are interested in attending or proposing agenda item contact Andrew Götz.

Minutes of previous NIAC meetings available on the mediawiki

COME ALONG

NXroot The root level of a NeXus file.

NXentry All the data, including instrument and sample descriptions, which logically make up a single scan or measurement. At many facilities, this corresponds to the entity that is defined by a single run number, which could be used to name the NXentry group. There can be many NXentry groups in each NeXus file.

NXinstrument The information needed to describe the instrument. This group contains other groups that describe instrument components e.g. choppers, collimators, detectors.

NXsource

NXmoderator

NXinsertion_device

NXbending_magnet

NXcrystal

NXdisk_chopper

NXfermi_chopper

NXvelocity_selector

NXguide

NXcollimator

NXaperture

NXfilter

NXattenuator

NXpolarizer

NXflipper

NXmirror

NXdetector

NXbeam_stop

NXpositioner

NXsample The information needed to define the physical state of the sample during the scan e.g. temperature, magnetic field, crystal mosaic.

NXmonitor Monitor data, i.e., counts, integrals, etc.

NXdata The data to be plotted i.e. a single data set comprising the measurements along with the data errors, and the default axis scales and labels required to plot the data. There can be more than one NXentry, e.g., if there are several detector banks producing plottable data.

NXevent_data Event-based data, i.e. a data set in which each count is recorded as a separate data event. This form might be chosen when the data are too sparse to be stored efficiently in histograms. Normally, such data will have to be converted to a regular NXdata group before it can be analyzed.

NXuser Details of a user, i.e., name, affiliation, email address, etc.

NXprocess Group used to store details of how the data have been processed.

NXcharacterizations Information required for the analysis of this NXentry, e.g. identification of empty can runs, vanadium runs, etc.

In addition to these classes, which appear in the locations shown above, the following groups can be added to any group in a NeXus file.

NXlog This group contains any logged information, i.e., information monitored during the run. It comprises the logged values and the times at which they were measured as elapsed time since a specified starting time.

NXnote This is a generic group designed to hold annotations or images stored in order to describe the experiment.

NXbeam This group records the state of the neutron or X-ray beam at any location. It could be referenced by instrument components within the NXinstrument group or by the NXsample group. In storing the results of instrument simulations in which it is useful to specify the beam profile, time distribution etc., at various stages down the beamline. Otherwise, its most likely use is in the NXsample group in which it defines the results of the neutron scattering by the sample, e.g., energy transfer, polarizations.

NXgeometry This group contains the geometrical information required to define the position, shape, and orientation of a NeXus object. This is especially important if the NeXus file contains the results of a comprehensive instrument simulation.

NXtranslation This group defines the position of the object in either absolute coordinates or relative to another object.

NXshape This group defines the shape of an object.

NXorientation This group defines the orientation of the object. The NXgeometry group defines the shape of an object in terms of a conventional set of coordinate axes, but this group allows it to be rotated into an arbitrary orientation.

NXenvironment This group contains details of the environment of a beamline component.

NXsensor This group defines an environment sensor.

<?xml version="1.0" encoding="UTF-8"?> <!-- URL: http://www.nexus.anl.gov/classes/xml/NXdata.xml Editor: NIAC $Id: NXdata.xml,v 1.2 2005/07/19 04:10:26 rio Exp $ Template of plottable data and their dimension scales. It is mandatory that there is at least one group of this type in each NXentry group. Note that "variable" and "data" can be defined with different names. The "signal" and "axes" attribute of the "data" item define which items are plottable data and which are dimension scales. --> <NXdata name="{Name of data block}"> <variable type="NX_FLOAT[:]|NX_INT[:]" long_name="{Axis label}" distribution="0|1" first_good="{Index of first good value}" last_good="{Index of last good value}"> {Dimension scale defining an axis of the data}? </variable> <variable_errors type="NX_FLOAT[:]|NX_INT[:]"> {Errors associated with axis "variable"}? </variable_errors> <data type="NX_FLOAT[:...]|NX_INT[:...]" signal="1" axes="[...]" long_name="{Title of data}"> {Data values}? </data> <errors type="NX_FLOAT[:...]"> {Standard deviations of data values - the data array is identified by the attribute signal="1". This array must have the same dimensions as the data}? </errors> </NXdata>

NeXus meta-DTDNeXus meta-DTDhttp://www.nexus.anl.gov/mediawiki/index.php?title=Main_Pagehttp://www.nexus.anl.gov/mediawiki/index.php?title=Main_Page

NeXusBeanNeXusBeanhttp://www.webel.com.au/nexmlhttp://www.webel.com.au/nexml

BeansBeanshttp://java.sun.com/products/javabeans/http://java.sun.com/products/javabeans/

CloningCloning

from http://www.abc.net.au/science/features/wollemi/default.htm

Wollemi nobilis.

Discovered in 1994, 200km west of Sydney

Claim to fame: one of the oldest and rarest trees

Reproduces by cloning.

Population <100. DNA of entire population is identical

Dinosaurs may have eaten its leaves

And this is what a neutron beam instrument And this is what a neutron beam instrument clone looks likeclone looks like

dke@gumtree:~/software/au.gov.ansto.bragg/release/1.0>

-rw-r--r-- 1 dke users 544 2006-08-08 12:58 ExampleLocalEchidna.java

-rw-r--r-- 1 dke users 4443808 2006-08-08 12:58 bragg-base-1.0.jar

-rw-r--r-- 1 dke users 9345780 2006-08-08 12:58 bragg-el-1.0.jar

-rw-r--r-- 1 dke users 1531683 2006-08-08 12:58 bragg-models-1.0.jar

-rw-r--r-- 1 dke users 1757674 2006-08-08 12:58 bragg-nexus-1.0.jar

-rw-r--r-- 1 dke users 5509976 2006-08-08 12:58 bragg-sics-1.0.jar

-rw-r--r-- 1 dke users 313898 2006-08-08 13:02 dom4j-1.6.1.jar

-rw-r--r-- 1 dke users 365 2006-08-08 13:02 j2ee.jar

-rw-r--r-- 1 dke users 358180 2006-08-08 13:01 log4j-1.2.13.jar

20

©1998-2003, Michael Aivazis

Example dataflow diagram

NeXusReaderNeXusReader SelectorSelector

BckgrndBckgrnd

SelectorSelector

SelectorSelector

EnergyEnergy NeXusWriterNeXusWritertimestimes

instrument infoinstrument info

raw countsraw counts

filenamefilename

time intervaltime interval

energy binsenergy bins filenamefilename

A picture that’s worth a thousand questions…

NeXML and DANSENeXML and DANSE

DANSE is welcome to review and use: UML and XSD representations of NeXus NeXusBeans and XStream inside or outside as a NeXus

server (using web services) or pyre component NeXusReader and NeXusWriter

Java code. How to reuse Java developments in DANSE. Jpipe or ORB? See Gumtree poster

We’d like to see the UML description of DANSE

Data processing inside GumtreeData processing inside Gumtree

Gumtree’s data processor Uses SDO data representation http://www-128.ibm.com/developerworks/java/library/j-sdo/ Use of NXprocess for provenance Extend NXprocess to provide unique ID’s for each process Scope of processor background subtraction, geometric

corrections and data reduction algorithms Can be replaced by DANSE components

Pyre in GumtreePyre in Gumtree

““A scientist would rather use someone A scientist would rather use someone else’s toothbrush, than use their code…” else’s toothbrush, than use their code…”

ACA July 2006ACA July 2006

25

©1998-2003, Michael Aivazis

ComponentComponent

Component schematic

input portsinput ports output portsoutput ports

propertiesproperties

component corecomponent core namename

controlcontrol

26

©1998-2003, Michael Aivazis

ComponentComponent

Component schematic

input portsinput ports output portsoutput ports

propertiesproperties

component corecomponent core namename

controlcontrol

Why does anyone collaborate?Why does anyone collaborate?

Benefits of collaboration “You scratch my back…”. We do bigger things and/or

things that we otherwise couldn’t do as an individual organisation (unless your organisation has unlimited resources).

Higher quality and functionality. More ideas, more testing. For science and for the community. Altruism and

interoperability. We provide interoperability for the community (if it were profit driven, we’d do it to gain a share of the market – could you sell a printer without USB?)

New generation of collaborationNew generation of collaboration

OPAL market survey, 2002 Collaboration was beginning to appear in neutron

scattering data acquisition or data analysis. IPNS and Los Alamos had a mature hardware and software collaboration.

EPICS and TANGO continue to be successful collaborations

Lower levels of functionality and quality than expected

Why does anyone collaborate?Why does anyone collaborate?

Tips to good collaboration Communication. Face to face preferred. Take

opportunities for satellite meetings Respect Separate framework development from domain specific

content. Decoupling software engineering from scientific content.

Provide good tools for collaboration e.g. wiki, content management system, subversion, bug and issue tracking

Collaborations require a manageable structure and clearly defined rules of engagement e.g. how are features of the framework edited?

Have adequate resources to enable collaboration. Create an economy to maximize outcomes e.g. deliverables of equal value

Why does anyone collaborate?Why does anyone collaborate?

Tips to good collaboration Clearly defined priorities within development team, visible

to the community Clearly defined milestones (dates and deliverables).

Allowing stakeholders to make plans based on your plans.

NeXus collaborationNeXus collaboration

Suggested collaborations Suggest the NeXus community has a full-time technical

secretary for next 12-24 months, jointly sponsored by each stakeholder

Institutes that write NeXus files are data providers. NeXus provides an interface between data providers and data requirers

Suggest DANSE has representation on the NIAC as a stakeholder that requires NeXus

Suggest that DANSE and NIAC look at the X-ray community and imgCIF. NSF proposal submitted to integrate imgCIF and NeXus

Suggest that the data requirers (e.g. DANSE and the Gumtree processor) collaborate on defining NXprocess for data provenance

There’s more to DANSE and GRID than There’s more to DANSE and GRID than large datasets and HPClarge datasets and HPC

DANSE and GRID have advantages for reactor sources too Deployment. Users don’t have to try to load codes on their

machines Code reuse, testing, verification and validation. Versioning of codes Provenance and curation Data access, virtual organisations Data availability and curation Bragg has an inventory of code, both in-house and

external, that we would like to integrate into DANSE. Looking to the DANSE team to provide coordination of legacy code integration

End presentationEnd presentation