MACS Architecture MACS Week June 24th, 2010 Johannes Gutleber

J. Gutleber1

MACS ArchitectureMACS Week

June 24th, 2010Johannes Gutleber

PR-100616-b-JGU, June 24th, 2010

R. Gutleber

J. Gutleber2

Functional Modeling Concept (FMC)

• Notation used to describe architecture

PR-100616-b-JGU, June 24th, 2010

J. Gutleber3

Scope

PR-100616-b-JGU, June 24th, 2010

MedicalSoftwareSystems

Access ControlSystem

BuildingAutomation

System

AcceleratorFrontendSystems

Beam deliveryFrontendSystems

BeamInterlockSystem

Authentication&

AuthorizationSupervisory Control System

AcceleratorControl System

Beam DeliveryControlSystem

Scope of Work Package

SafetyManagement

System

RadiationMonitoring

System

Scope of MACS

J. Gutleber4

3 + 1 Tier Architecture

PR-100616-b-JGU, June 24th, 2010

Control System Work Package

Presentation tier (1)

Processing tier (2)

Equipment tier (3)

Subsystem Work Package

Frontend tier (4)

Common work

Protocols and data format adapters, fan-out and data concentration, timing, SADS

Configuration for devicesSupervisory state machinesData logging, SCADA

Graphical user interfacesRemote operation

Control logic and closedReal-time control loops

Front-end adaptation and lean framework layer

J. Gutleber5

Repository Management System

PR-100616-b-JGU, June 24th, 2010

Graphical User Interfaces & Data Import/Export Interfaces

Data Model Views

Data Model Access Controller

Data Model

ComponentSetpoints

Waveforms

Alarm Thresholds

Alarm Conditions

AcceleratorComponentDefinitions

ElectronicsComponentDefinitions

SoftwareComponentDefinitions

ComponentConnectionProtocols

Relations amongComponents

ManufacturingData

Virtual AcceleratorDefinitions

Data ArchivingSpecifications

Associations Components and User Interfaces

Geographical Component

Layouts

Accelerator Component

Optics

Main TimingSequences

Accelerator Cycle Definitions

Rooms

Working Set Definitions

Asset TrackingReports

Definition of Device

Permissions

DBMS

ComponentCommands

ComponentEvents

ComponentMonitorables

J. Gutleber6

Component Model

• Physical accelerator element or assemblies of physical elements are represented in software, called component

• Each component has• Unique identifier• Name• Properties• Ownership at runtime• State machine• May react to timing events (cycle dependent operation)

PR-100616-b-JGU, June 24th, 2010

J. Gutleber7

Uniform State Machine

• Every device in 1 of 4 modes• Service, Physics, QA, Clinical

• Every device implements all states and transitions

• Additional, device specific commands possible in “Op” state

• State machine implemented in “software representation” of device• Front End Controller (FEC)

PR-100616-b-JGU, June 24th, 2010

Init

Reset

Hold Ready Op

FaultFail

Reset Reset

EnableConfigure

Clear or Modify QuiesceClear or Modify

Clear

Reset

Initialize

J. Gutleber8

Working Sets• Working Set is a collection of defined components• Can be controlled as a “single virtual device”• Can be assigned a “single set of configuration parameters”

PR-100616-b-JGU, June 24th, 2010

Working Set

DeviceSoftware

Component

Device

Limited to 1 layerImplement the sameState-machine

J. Gutleber9

Errors and Alarms• Devices (hardware and software) limit themselves to provide

information that can be processed by SCS to generate alarms• Alarms are defined in PVSS based on numeric quantities and

Boolean expressions• “Failed” and “Fault” states for each device

• Device must stop working (fail-silent behaviour)• Possibility to recover from “Failed” via operator intervention• Alarm raised by SCS in either case that must be acknowledged

• Error during processing a command is notified to SCS• Not a failure! Alarm can be defined by operator at SCS level is needed

• QoS indicator for each “software” service

PR-100616-b-JGU, June 24th, 2010

J. Gutleber10

Logging (Execution)• Process of tracking changes in execution of a program• Rely on standard protocol, data format and tools

• Log4Net tool, Log4J protocol, all logs end up in PVSS

PR-100616-b-JGU, June 24th, 2010

Presentation TierProcessing TierEquipment Tier

Log Files

FECOSComponent

ProShellComponent

Other Component

LogCollector

SCADATool Visualization

J. Gutleber11

Data Logging• Automated process of continuously collecting and archiving

process measurements on a timely basis. • O(3000) values collected at average frequency of O(1) Hz. • Carried out by PVSS SCADA tool• For each data item conditions can be specified that determine

when the value is collected or archived. • Limited transformations (e.g. unit conversion) may be applied

to the gathered values in order to record a set of derived data values.

• Archiving can be configured for each data item independent of the time when it is collected• tradeoff between storage space efficiency and measurement accuracy

PR-100616-b-JGU, June 24th, 2010

J. Gutleber12

Data Logging Architecture

PR-100616-b-JGU, June 24th, 2010

Processing TierEquipment Tier

FECOSComponent

ProShellComponent

OtherComponent

SCADATool

R

Configuration

RepositoryManagement

System

Archive

Alarms

Field data

Oracle

PVSS

Oracle

OPC

NI LV-RT

C#

PVSS Internal

J. Gutleber13

Beam Generation Process

PR-100616-b-JGU, June 24th, 2010

Generate beams

J. Gutleber14

Operation Principle

• Accelerator is split into logical machine subsystems (LMS)• Sources• LEBT• Linac• MEBT• Main Ring• Extraction Line• Individual irradiation lines

• Virtual Accelerator (Vacc) comprises at least 1 LMS working set

• VAcc works on a “cycle” basis

PR-100616-b-JGU, June 24th, 2010

J. Gutleber15

Cycle Definition

• Accelerator cycle describes beam characteristics to be produced during a period of time by specifying a set of configuration parameters.• What (parameters)• When (timing)

• Generation of cycle causes activation of pre-defined actions at pre-defined, cycle-dependent times in front-end software components that control the physical elements that constitute the accelerator.

• At each point in time, a single cycle is active per VAcc• A defined cycle can be played over a “VAcc”.

PR-100616-b-JGU, June 24th, 2010

J. Gutleber16

Cycle Definition Contents

• Ion type• Source type (optional, allows binding of cycle to source)• Target room of beam (optional, for clinical mode)• Energy• Beam dimension• Intensity• Spill length

PR-100616-b-JGU, June 24th, 2010

J. Gutleber17

Cycle Code64 bit wide identifier to which configuration parameters can be associated for each front end controller

PR-100616-b-JGU, June 24th, 2010

MSB LSBWord

31 20 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

Beam line Spill length Degrader Source Particle Energy 0

Major Version Minor Ver. Fmt Mode Y size X size 4

Bit Size Name Description0-15 16 Energy n x 0,1 MeV from 0 to 800 MeV (8000 steps)

16-18 3 Particle species

0 unused, p, C and space for other light ions. 7 in total

19-21 3 Source number 0 means unbound, 7 sources if indication needed22-24 3 Degrader 0: 0%, 1: 10%, 2: 20%, 3: 50% (3 values)25-28 4 Spill length 0,1; 0,2; 0,5; 1, 2, 3, 4, 5, 6, 7, 10 sec (11 values)29-31 3 Beam line 0 means unbound, up to 7 if indication needed32-35 4 Beam x size Size in mm from 1 to 1036-39 4 Beam y size Size in mm from 1 to 1040-41 2 Mode 0: service, 1: physics, 2: QA, 3: clinical42-43 2 Code format 0: current version, others for future format changes44-47 4 Minor version Intermediary versions for MD, modes 0 (S) and 1 (P).48-63 16 Major version Version indicates which settings to use for this cycle

J. Gutleber18

Beam Stamp• A record of information updated throughout the entire

lifetime of the facility• Unambiguously associates in time any piece of information

generated during operation to a cycle • e.g. occurrence of a command, acquired data, alarms and errors

• Consists of a run number and a cycle number

PR-100616-b-JGU, June 24th, 2010

Start Cycle Stop CycleStart Cycle Stop CycleStart Run

action action action action

time

UTC timestamp

Run number,Cycle number,UTC timestamp

UTC timestamp

Run number,Cycle number,UTC timestamp

Cycle n Cycle n+1

J. Gutleber19

Virtual Accelerators

• Virtual accelerator (VAcc) is a collection of working sets (LMS)• Used for operation purposes (beam generation)• Freely definable using RMS• Non-overlapping VAccs can be active concurrently• If VAccs overlap, only 1 VAcc active at that timePR-100616-b-JGU, June 24th, 2010

VAcc 1

VAcc 2

J. Gutleber20

VAcc State Machine

• BDCS allocates a Vacc from Virtual Accelerator Allocator (VAA)• If VAcc overlaps becomes “pending”• If “allocated”, resources are owned by requestor• If “active” VAcc is ready for beam generation

• Beam interception devices and safety elements are “armed”• If VAcc is released/deactivated, “pending” gets allocated

PR-100616-b-JGU, June 24th, 2010

allocate activate

deactivate

Inactive Pending Allocated Active

Finished

release

J. Gutleber21

Beam Allocation

PR-100616-b-JGU, June 24th, 2010

BDCS

BeamGeneratorProcedure

MACSR

PRVSR

R

BeamGeneratorProcedure

SoftBDCS

R

VAA

R

BDCSBDCS VAA

SCADA Tool

MTS

RBDCS

R

R

J. Gutleber22

Blueprint ofSupervisory andAcceleratorControl SystemComponents

PR-100616-b-JGU, June 24th, 2010

J. Gutleber23

Network Organization

• SCS/ACS devices in physically separate technical network (TN) • Applies also to control room equipment

• General purpose network for office IT and private computers

PR-100616-b-JGU, June 24th, 2010

GPN

Switch

TN

Appliance

J. Gutleber24

Status Displays

• Current status of operation and near real-time update of cycle on various displays in the facility

• Available in techn. network and general purpose network

PR-100616-b-JGU, June 24th, 2010

SCADATool

StatusApplication withUser interface

PublisherSubscriber

B-TrainApplication

StatusApplication withUser interface

Push statusand field data

Push statusand

field data Push status

Push field data

StatusMessageGenerator

Server

Server

Client

General purpose network (GPN) Technical Network (TN)

J. Gutleber25

SCADA Tool

• PVSS to be configured from RMS database• Keeps state of entire system• Single point of entry to interact with all subsystems

PR-100616-b-JGU, June 24th, 2010

SCADA Tool

User InterfaceManager

ControlManager

ProShellManager

EventManager

DatabaseManager

ConfigurationManager

ArchiveManager

Driver

ArchiveDatabase

OnlineDatabase

RepositoryManagement

System

RedundancyManager

J. Gutleber26

Datapoint Concept• Data structures that abstract from physical devices (DP)• PVSS interacts with devices by reading and writing to

elements (DPE) of data structures that represent those devices.

• Data point type (DPT) defines the structure of DP and contains a set of hierarchically ordered DPEs. • Can be of a simple type (e.g. string or integer) or a complex type (e.g.

static or dynamic lists of integers). • Every subsystem or component is represented as a datapoint

in the system• All interaction with components/subsystems goes through

the datapoints to maintain consistency and operation safety

PR-100616-b-JGU, June 24th, 2010

J. Gutleber27

Accessing Device Components• Supervisory panels

to control entire subsystems

• Virtual Instrument panels• Communication via

PVSS datapoints• Communication via

NI SV

PR-100616-b-JGU, June 24th, 2010

28

SADS

• Connect analogue signal sources to devices acting as sampling scopes display it on operator screens.

• Select video sources and tell on which screen to show them.• Acts as MD, analysis tool not as primary operation display• Requirements to be captured by October 2010

PR-100616-b-JGU, June 24th, 2010 J. Gutleber

SADS

Signalsource

Videosource

Signal anddestination

selector

Oscilloscopedisplay

Video sourceand destination

selector

Videodisplay

R

R

R

R

Signalsource

J. Gutleber29

SADS Architecture

PR-100616-b-JGU, June 24th, 2010

SADSRF

SwitchingMatrix

SignalSource 1

SADSProcessing

Element

SignalSource 100

MTS

Timing event

SwitchControllerGUI 2

GUI 1

R

Disk

J. Gutleber30

Procedure Framework (ProShell)

• Functionality to automate repetitive supervisory tasks• Implement beam diagnostics procedures• Framework written in-house in C#• Procedures to be written by “experts” in C#

• Training and documentation to be provided by WP CO.

PR-100616-b-JGU, June 24th, 2010

J. Gutleber31

ProShell Architecture

PR-100616-b-JGU, June 24th, 2010

ProShell

ProshellManagement

GUI

ProcedureSpecific

GUIProcedure

R

Load, run, stopprocedure

Procedure Storage

Load procedures

SCADAEvent

Manager

Publisher/Subscriber

ConfigurationRepository

ProcedureLoader and

Registry

SCADAProxy

Dispatcher

ComponentFacade

R

Commands

Publisher/Subscriber

Proxy

R

Commands

Events

RepositoryProxy

R

Events

J. Gutleber32

T3 and T4 Architecture

• WP CO provides LV-RT programming framework (FECOS)• Uniform state management, configuration, communication• FEC software components provided by WP CO for individual

work packages on case-by-case basis (by default in WP)• FEC can be operated standalone or via Working Set/VAcc

PR-100616-b-JGU, June 24th, 2010

FEC

OS

FEC FED Device

Analogueand

DigitalIO

Industrialfieldbus

TechnicalNetwork

Slow Control

Technical Network

Fast Control

Tier 2 Tier 3 Tier 4

J. Gutleber33

FEC Communication• FEC is building

block at T3• Defined

interfaces to T2• State machine

and slow control via OPC

• Direct data access via NI Shared Variable engine and DIM

• BLOBs via HTTP

PR-100616-b-JGU, June 24th, 2010

FEC

ProShell

SCADA Tool

NIMeasurement

Studio

SharedVariableEngine

OPCServer

SV

OPC

SV

DIM

HTTPWeb

Server

Tier 2 Tier 3

J. Gutleber34

Main Timing System• Coordinates devices in each subsystem (ES-090512-a-JGU)

• Precision of ordinary timing events O (1 μsec)• Synchronization of 2 receivers O(50 nsec), local solution• Non real-time events distributed via DIM (Ethernet, TN)• Reference clock O(10 MHz) foreseen• Provision O(10 Hz) stability pulse that can be picked on demand

• Broadcasts events• Occurrences in time that are assigned a logical name• E.g. start injection, start acceleration, start extraction, etc.• Sequences are statically defined for individual cycles

• Cycle definition comprises• Record of settings (WHAT)• Sequence of timing events (WHEN)

PR-100616-b-JGU, June 24th, 2010

J. Gutleber35

Typical Timing Events

PR-100616-b-JGU, June 24th, 2010

J. Gutleber36

Definition of Timing Sequence

PR-100616-b-JGU, June 24th, 2010

Generation of events is dynamicsince the BDCS decides, whichcycle to generate next.

J. Gutleber37

Main Timing System Concept

PR-100616-b-JGU, June 24th, 2010

MTG

Fanout

MTR

SRC RFQ/IH RFC Kickers Dipoles Fast Deflector BDIQuad.-

polesBetatron

CoreSext.-poles

OrbitCorrectors

MTR MTR MTR MTR MTR MTR MTR

Reference clock

J. Gutleber38

MTS Architecture

• Event broadcast sequencer (Main Timing Generator)• Real-time and non real-time event broadcast• Timing Box for scope trigger signal reception

PR-100616-b-JGU, June 24th, 2010

Main Timing System

Subsystem (RT) or TiBx

Beam DeliveryControl System

ProShellFramework

TechnicalNetwork

Main TimingGenerator

R

R

ReferenceClock

Main TimingReceiver

Physicaldeviceor TiBx

software

Timing Broadcast Network

Main TimingTransmitters

Gateway toTechnicalNetwork

Subsystem (Non-RT)

Gateway toTechnicalNetwork

TechnicalNetwork

Technical Network

Physicaldevice

R

J. Gutleber39

High-Precision Operation

• Precision of MTS events 1 µsec• For higher-precision, local solution based on programming the

timing receiver board (MTR) possible• Dynamic reconfiguration not foreseen• Output can be simple optical or electrical signals

• Precision of O(nsecs) can be achieved

PR-100616-b-JGU, June 24th, 2010

Main Timing Receiver

Delay 1

Delay 2

EventReceiver

TimingBroadcastNetwork

InjectionBumper

PCO

FastDeflector

PCO

Delay in nsec range

Event

Optical or electrical trigger link

J. Gutleber40

Power Converter Controller Architecture

PR-100616-b-JGU, June 24th, 2010

PCO FEC

PCO FEC

PCCComponent

PCOFED

PCCComponent

PCOFED

PCOFED

PCOFED

PCO

PCO

PCO

PCO

PCCComponent

PCOFED PCO

CPU

PCCComponentCPU

100 MBit/s (optical)1-to-1

MTS

Setpointsequence

driven(MR, BDS)

SetpointDriven(LEBT, MEBT, HEBT)

Ethernet (non-realtime)and

Optical (realtime)

Responsibility of WP:CO Responsibility of WP:PCO

MTR

Slow control

Fast control

PX

I bac

kpla

ne

MTR

PX

I bac

kpla

ne

Slow control

Optical trigger LEMO trigger

Optical trigger LEMO trigger

J. Gutleber41

PCC-FED-PCO Chain

PR-100616-b-JGU, June 24th, 2010

FED

AnalogueRegulation

BoardMemorybus

PCOV

V

I

Slow control, serial

FECCustomOptical

link

FED PCODigital values

Digital values

Slow control, serial

FECCustomOptical

link

Responsibility of WP:PCOResponsibility of WP:CO

J. Gutleber42

Conventional Magnets Slow Control

• Only Boolean values• Will be implemented reading out Beam Interlock System (BIS)• Saving on cabling via Profinet instead of point-to-point

PR-100616-b-JGU, June 24th, 2010

Magnet

TemperatureSwitch

FlowIndicator

PLCSCADAR

TechnicalNetwork

Tier 2 Tier 3 Tier 4

PLCIO

ProcessorFieldbus

Magnet

TemperatureSwitch

FlowIndicator

PLCIO

ProcessorFieldbus

J. Gutleber43

Special Magnet Slow Control

• Flow and Temperature via BIS• Cooling and positionning subject to analysis (summer student)PR-100616-b-JGU, June 24th, 2010

Magnet

TemperatureSensor

FlowMeter

PLC

SCADA

R

TechnicalNetwork

PLCIO

ProcessorFieldbus

FrontendPositioning

SystemPositionDetector

MotorR

TechnicalNetwork

MagnetTemperatureSensor

FlowMeter

MagnetCoolingSystem

Tier 4

Valve

R

TechnicalNetwork

Tier 2 Tier 3 Tier 4

J. Gutleber44

Vacuum Control Architecture

PR-100616-b-JGU, June 24th, 2010

VAC FEC

VACController

VACFED Gauge

VACController

Gauge

CPU

PX

I bac

kpla

ne

PumpVACFED

Valve

Pump

Fieldbus

Digital IO

Tier 3 Tier 4

FECOS and CrateProvided by WP CO,Mounted in same rackAs FEDs

Component programmedBy WP VAC

FED provided by WP VAC,serial RS422/RS485 interfaces,digital IO interfaces

J. Gutleber45

Sources Slow Control• Existing LV software

from supplier to be re-engineered

• CWO to Cosylab for provision of FECOS based component

• Beam diagnostics integrated into overall BDI system by BDI programmer

• Provision of MTS events foreseen• BDI, pulsed operation

PR-100616-b-JGU, June 24th, 2010

SRC BDI FEC

SRC FEC

Gas Injection FED Valves

SRC BDIComponent

Gauge

CPU

PX

I bac

kpla

ne

BDIDevices

Fieldbus

Analogue and Digital IO

Tier 3 Tier 4

MTR

Pump

FED Microwaveamplifier

Microwavetuner

Microwavegenerator

PlasmaGeneration

HV

Chiller

FED Valves

Gauge

Fieldbus

Pump

VacuumController

CoolingController

ExtractionController

PX

I bac

kpla

ne CPU

MTR

J. Gutleber46

Beam Diagnostics & Instrumentation

• WP BDI programmer works with FECOS provided by WP CO• Standalone operation possible for each FEC• GUI: ProShell and PVSS with Measurement Studio plugins

PR-100616-b-JGU, June 24th, 2010

SCADAEvent

Manager

Publisher/Subscriber

TN

BDI FEC

BDIMonitor

Controller

BDIMonitor

CPU

PX

I bac

kpla

ne

MTR

FEC

OS

LocalStorage

R

Local debug GUI

BDI FEC

TN

J. Gutleber47

Technologies Used• Primary means of communication with front-end:

• Transport: Ethernet + TCP/IP, RS 232/422/485, digital I/O for interlocks• Protocols: OPC (+ NI SV) + DIM (+ HTTP)• Data contents: Definition starts now based on CNAO XML schema• PLCs: Siemens, S7 protocol over Ethernet preferred

• SCADA + GUIs: PVSS II + Measurement Studio• Platform for Frontend:

• PXI crates and crate controllers, FlexRIO FPGAs• LV-RT operating system

• Platform for T2,T3:• Rack-mounted server PCs• Virtualized MS Windows XP (upgrade to 7 for deployment)

PR-100616-b-JGU, June 24th, 2010

J. Gutleber48

Technology Overview

PR-100616-b-JGU, June 24th, 2010

Presentation tier

Processing tier

Equipment tier

Frontend tier

Ethernet, TCP/IP

Ethernet, TCP/IP

Ethernet, TCP/IP

Recommendations forRS 232/422/485Digital IO

OPC, DIM, MRF for timing8B10B custom for setpoint sequences

PXI

Virtual Machines

Rack mountedPC (PCIe)

PLC

OPC, DIM, HTTP

PVSS, SRDP

MeasurementStudio

Inhouse C#

Ethernet,NI SV