Www.lightsource.ca Science Studio and ANISE (Jan 2010 update) Elder Matias Canadian Light Source.

www.lightsource.ca

Science Studio and ANISE(Jan 2010 update)

Elder Matias

Canadian Light Source

www.lightsource.ca

• Overview• Beamline Control Traditional Technology• Beamline Control – ScienceStudio • Nan-Fabrication – ScienceStudio• User Office• Grid Computing• User Configurable Light Paths

www.lightsource.ca

Where is Saskatoon?

www.lightsource.ca

Scientific American May 2008

Science 2.0 – The Risk and Reward of Web-Based Research

---------------------------------

“Our real mission isn’t to publish journals but to facilitate scientific communication” Timo Hannay – Head of Web Publishing at Nature Publishing Group

www.lightsource.ca

What Is the Web 2.0?

• In plain English ….– Automating tedious tasks using web technology– Tools to help people and software collaborate

www.lightsource.ca

Project Team Members

Dionisio Medrano

Dylan Maxwell

Dong Liu

Elder Matias

Chris Armstrong

Khalid Subratee

Jinhui Qin

Yuzhen Xie

Mike Bauer

Stewart McIntyre

Marina Suominen Fuller

Ludeng (Eric) Zhao

Yuhong Yan

www.lightsource.ca

Goals of the Two Projects• Science Studio

– User Office Automation– Integrated Experiment Management– Remote Access to VESPERS (XRD and XRF)– Access Scanning Electron Microscope (UWO)– Grid Computing (Cell Processor)

• ANISE– Integrated Stream Computing to providing

near real-time data processing

www.lightsource.ca

Beamline ControlTraditional Technology

www.lightsource.ca

EPICS

DetectorsCameras etc.

ModbusTCP/IP

GPIB

RS-232

Channel A

ccess Protocol

IOC

IOC

IOC

IOC

State MachineEngine

CA

CA

CA

CA

CA

Single BoardComputer

CA

Operator WorkstationUser Applications

TouchPanels

CA

CA

Telemecanique Momentum PLC

VME

IOC

CA

www.lightsource.ca

Beamline Controls Interfaces

• EPICS Tools • Configuration Tool• User GUI & ROOT

www.lightsource.ca

Beamline Visualization

As Map

- Raster scan with fluorescence spectroscopy of a pine needle contaminated with arsenic. The technique allows multiple elements to be detected simultaneously at each point of the raster. These images show distributions of arsenic, iron, and manganese, respectively, near the tip. The highest intensity displays in red, lowest in blue.- Custom on-line plotting application scripted in CERN Root, with data stream from the CLS data acquisition application. (G. Wright, R. Igarashi, K. Chang-Yong, N. Chen)

Fe Map

www.lightsource.ca

Traditional Beamline Visualization

Cross section views of the beam spot intensity distribution for varying degrees of monochromator detuning (50-80%). (Only one image shown).

(G. Wright, R. Igarashi, K. Chang-Yong, N. Chen)

www.lightsource.ca

Beamline ControlScienceStudio Technology

www.lightsource.ca

Requirements

• New User Office Functionality– Proposal submission– Peer review– User Feedback Tracking– Experiment Management– User Training/ Safety Testing

• Remote Beamline Access• Integration with grid data-storage• Grid computing

www.lightsource.ca

Science Studio and ANISE

ProcessingService

DataServiceData

ServiceProcessing

Service

ANISE

Business ModelLayer

PersistenceLayer

DeviceProxies

Client ServicesLayer

BrowserBrowser

LaboratoryControl Module

ServiceProxies

Science Studio

LabatoryControl Module

IOCs

Beamline

IOCs

Beamline

General, commonComponent

XRD Processing

XRF Processing

Science Studio specificComponent

www.lightsource.ca

System Architecture

WebApplication

BeamlineControlModule

DBSANSAN

JMS CA

VESPERS

HTTP

1. VESPERS Beamline2. EPICS control system3. Beamline Control Module (BCM)4. Web Application5. Database6. File Storage7. Web Interface

www.lightsource.ca

VESPERS Beamline

• VESPERS — Very Sensitive Elemental and Structural Probe Employing Radiation from a Synchrotron

• A bending magnet beamline on sector 6 at the Canadian Light Source synchrotron in Saskatoon, Saskatchewan.

• A hard x-ray microprobe with an energy range of 6 to 30keV.

• Techniques: X-Ray Fluorescence (XRF) & X-Ray Diffraction (XRD)

WebApplication

BeamlineControlModule

DBSANSAN

JMS CA

VESPERS

HTTP

www.lightsource.ca

EPICS Low-level Control System

• EPICS — Experimental Physics and Industrial Control System

• The standard control system at the CLS.

• EPICS consists of a network of Input-Output Controls (IOCs) which are connected to directly to devices.

• An IOC provides many Process Variables (PVs) which relate to either an input or output from a device and have a unique name.

• Channel Access (CA) is used to read or write to any PV without knowing which IOC provides the PV.

• More than 50,000 PVs in the CLS control system.

WebApplication

BeamlineControlModule

DBSANSAN

JMS CA

VESPERS

HTTP

www.lightsource.ca

Beamline Control Module (BCM)

• The BCM provides a high-level interface to the low-level control system (EPICS).

• Logical and physical separation of business logic and control logic.

• Virtual device abstraction that provides independence from low-level control system.

• Virtual devices can be logically organized into a device hierarchy.

• Basic devices can be combined to build more functional devices.

• Communication with external applications using two message queues (ActiveMQ).

WebApplication

BeamlineControlModule

DBSANSAN

JMS CA

VESPERS

HTTP

www.lightsource.ca

Web Application

• A J2EE Servlet application that provides a web-based interface Science Studio.

• Tools: Spring (MVC), iBATIS (ORM), JSecurity (Apache Ki), Apache Tomcat

• Divided into two parts: the Core application and the VESPERS beamline application.

• Core application is responsible for providing access to the business objects.

• VESPERS application is responsible for remote control of the VESPERS beamline.

WebApplication

BeamlineControlModule

DBSANSAN

JMS CA

VESPERS

HTTP

www.lightsource.ca

Database

• Metadata associated with the operation of a remote controlled beamline and the organization of experimental data collected on that beamline.

• A project is the top level organizational unit and is associated with a project team.

• A session defines a period of time allocated to a project team to conduct experiments.

• An experiment relates a sample and the technique being applied to that sample.

• A scan records the location of the acquired experimental data.

WebApplication

BeamlineControlModule

DBSANSAN

JMS CA

VESPERS

HTTP

www.lightsource.ca

Experimental Data Storage

• Experimental data is stored at the CLS.

• Common directory structure shared with other beamlines.

• A large data storage facility is now operational at the University of

Saskatchewan as part of WestGrid.

WebApplication

BeamlineControlModule

DBSANSAN

JMS CA

VESPERS

HTTP

www.lightsource.ca

VESPERS Web Interface

• Rich web interface to Science Studio and the VESPERS beamline.

• Designed to be used over commodity broadband internet.

• Developed for the Firefox web browser without any additional plugins or extensions.

• Known to work with other browsers, but requires the Canvas HTML tag.

• AJAX is used for the VESPERS interface to provide device values in pseudo real time.

• ExtJS, a JavaScript framework, provides many advanced GUI elements.

WebApplication

BeamlineControlModule

DBSANSAN

JMS CA

VESPERS

HTTP

www.lightsource.ca

www.lightsource.ca

Beamline Setup

www.lightsource.ca

Experiment Setup

www.lightsource.ca

XRF (X-Ray Fluorescence)

www.lightsource.ca

Beamline Hutch Cameras

www.lightsource.ca

Experimental Data Viewer

www.lightsource.ca

www.lightsource.ca

ScienceStudio

www.lightsource.ca

Scanning

www.lightsource.ca

XRD Screen

www.lightsource.ca

(a) X ray Fluorescence spectra and mapping

(b) Laue X ray Diffraction and mapping Orientation Strain

www.lightsource.ca

VM strain

XRF Image XRD: orientation XRD:strain

Optical image

www.lightsource.ca

University of Western OntarioNan-Fabrication Facility

Using ScienceStudio

www.lightsource.ca

Nanofabrication Facility

The Western Nanofabrication Facility is an open user facility at the University of Western Ontario for the fabrication of micro and nano structures.

LEO 1540XB Scanning Electron Microscope (SEM) with an integrated Oxford x-ray system.

Oxford x-ray system is controlled by INCA.

INCA provides limited control of the SEM, and spectral analysis and quantification.

www.lightsource.ca

Nanofab Integration Design

www.lightsource.ca

Nanofab Implementation

www.lightsource.ca

Initial Test: INCA Remotely

www.lightsource.ca

User Office

www.lightsource.ca

www.lightsource.ca

User Office Modules• User Profile/Authentication

– User Profile Deployed– User Roles Deployed– Authentication (LDAP/AD) Deployed

• Information Model– Beamlines Done– End-stations Done– Techniques Done

• Notification Module Deployed

www.lightsource.ca

User Office Modules

• Proposal Submission In Progress• User Feedback Survey Prototype• Scheduling Module Prototype• Workflow Engine Evaluation• Sample Management

Requirements• Publication Tracking Requirements

www.lightsource.ca

User Office - Training Module• First User Office Module Deployed• User Profile

• 1,400 + user records/accounts created

• Training Module– 53 Training Modules Setup and past user

records manually transferred

www.lightsource.ca

Calendar view in JQuery

www.lightsource.ca

User Office Workflow Example

Prototype Implementation1. CLS issues a call for proposals and gives deadline2. Beamline users submit proposals3. User Office administrator ends registration or extends deadline4. User Office administrator assigns proposals to user office reviewers5. Reviewers look at proposals and rank them6. User Office looks at ranking and chooses the proposals to accept7. Accepted proposals contact persons are notified8. Beamline User completes training (web service)9. After training is completed (simulated by a delay) the CLS is notified

www.lightsource.ca

User Login

www.lightsource.ca

Admin – Add Proposal

www.lightsource.ca

Admin - Setup for Cycle

www.lightsource.ca

Admin – Creating Roles

www.lightsource.ca

Grid Computing

www.lightsource.ca

HPC Computing in Canada

• Compute Canada– Responsible for High Performance Computing in Canada– Each regional grid is a member of Compute Canada

• ACEnet – Atlantic Canada• CLUMEQ - Quebec• SCINET - UofT• HPCVL – Queens, Royal Military Collage

St. Lawrence, Carlson, Ottawa, …• RQCHP - Quebec• SHARCNET - Ontario• WESTGRID – Western Canada

CLUMEQ Facility

www.lightsource.ca

Grid Data Storage?

• UofS is the host for the new WestGrid data storage facility

• Cost: $3.2 M• Includes on-line and

archival storage• Two sites on campus

Photo: tape backup unit holding 6,000 tape (each @1TB)

www.lightsource.ca

IBM Cell Processor (3.2 GHz)

www.lightsource.ca

www.lightsource.ca

User Configurable Light Paths

www.lightsource.ca

Lightpath Accelerator controls a software virtual cross-connect that commands UCLP.

Integrating LPA with User Configurable Light Paths

In effect, CA*Net4 is treated as a single lightpath cross-connect

real device real devicevirtual device

www.lightsource.ca

File Transfer Demo