Data Acquisition for the 12 GeV Upgrade CODA 3. The good news… There is a group dedicated to development and support of data acquisition at Jefferson.

Data Acquisition for the 12 GeV UpgradeData Acquisition for the 12 GeV Upgrade

CODA 3CODA 3

The good news…The good news…

There is a group dedicated to development There is a group dedicated to development and support of data acquisition at Jefferson and support of data acquisition at Jefferson Lab. This includes Hall D.Lab. This includes Hall D.

Much of what Hall D needs is generally Much of what Hall D needs is generally useful to the whole JLAB experimental useful to the whole JLAB experimental program.program.

We are not waiting on the 12GeV upgrade.We are not waiting on the 12GeV upgrade.

In the short term…In the short term…

Hall D Requirements drive development Hall D Requirements drive development Replace aging technologiesReplace aging technologies

Run ControlRun Control Tcl-Based DAQ componentsTcl-Based DAQ components mSQLmSQL

Maintain cross-platform compatibilityMaintain cross-platform compatibility Linux, Solaris, OS X, vxWorksLinux, Solaris, OS X, vxWorks

Support new commercial hardware Support new commercial hardware advancesadvances

HALL DHALL D

ExistingHalls

GlueX - RequirementsGlueX - Requirements Pipelined Electronics (ADC, TDC)Pipelined Electronics (ADC, TDC)

Dead-timeless systemDead-timeless system 200 KHz L1 Trigger200 KHz L1 Trigger

Parallel/Staged Event BuildingParallel/Staged Event Building Up to 100 Front-end CratesUp to 100 Front-end Crates 1 GByte/s aggregate data throughput1 GByte/s aggregate data throughput

L3 Online FarmL3 Online Farm 200+ nodes200+ nodes x10 reduction in data to diskx10 reduction in data to disk

Storage Management Storage Management Ordering/sorting of built events (at 15-20 kHz)Ordering/sorting of built events (at 15-20 kHz) 100 MB/s --> 8 TB/day --> 1 PB/year100 MB/s --> 8 TB/day --> 1 PB/year

Front-End IssuesFront-End Issues

Trigger rate - 200 KHzTrigger rate - 200 KHz Block up Events (200 event block -> 2kHz)Block up Events (200 event block -> 2kHz) Move some ROL runtime code to modules (FPGAs)Move some ROL runtime code to modules (FPGAs) ADCs provide L1 trigger data ( hence we need a ADCs provide L1 trigger data ( hence we need a

distributed high speed clock - 62.5 MHz ??)distributed high speed clock - 62.5 MHz ??)

New Trigger SupervisorNew Trigger Supervisor Perhaps 100+ cratesPerhaps 100+ crates Support pipeline, event blockingSupport pipeline, event blocking Manage flow control into DAQ system backendManage flow control into DAQ system backend

Level 1 TriggerLevel 1 Trigger

• Distributed high speed clockDistributed high speed clock• Subset of ROCs collect sampled ADC dataSubset of ROCs collect sampled ADC data and send it to L1 Trigger in syncand send it to L1 Trigger in sync• 12 bit sums/crate x 250MHz --> 3 Gbit/s links12 bit sums/crate x 250MHz --> 3 Gbit/s links• Trigger decision goes to Trigger SupervisorTrigger decision goes to Trigger Supervisor

Front-End Issues cont…Front-End Issues cont… Form-Factor for electronicsForm-Factor for electronics

VME64XVME64X New commercial bridge (TSi148) supports 300MB/s on New commercial bridge (TSi148) supports 300MB/s on

existing VME backplanesexisting VME backplanes Support other Hall DAQ applicationsSupport other Hall DAQ applications

High speed switched serial interconnect (4Gbit/s links) High speed switched serial interconnect (4Gbit/s links) needed for GlueX L1 triggerneeded for GlueX L1 trigger Commercial solutions - VXS, ATCA ?Commercial solutions - VXS, ATCA ?

DAQ - Trigger - Modules : All must be designed to work DAQ - Trigger - Modules : All must be designed to work together. together.

Staged/Parallel Event BuildingStaged/Parallel Event Building

• EMU built around the ETEMU built around the ET system for customizablesystem for customizable processing/distribution ofprocessing/distribution of event streams.event streams.

Examples:Examples:• Data Concentrator for ROCsData Concentrator for ROCs• Sub-Event builderSub-Event builder• Farm distribution pointFarm distribution point• Event RecorderEvent Recorder• User Processes can attach toUser Processes can attach to any EMU in the systemany EMU in the system

L3 FarmL3 Farm

Can be used for analysis Can be used for analysis or filteringor filtering

Support 100s of nodesSupport 100s of nodes Nodes can come and go Nodes can come and go

during event takingduring event taking

-------------- Do other experimental Do other experimental

halls need this (Hall B)?halls need this (Hall B)? Do filtered events need Do filtered events need

to be time ordered? to be time ordered?

1 GB/s

100 MB/s

RunControl / Monitoring / Slow ControlsRunControl / Monitoring / Slow Controls First generation Java Agent RunControl is hereFirst generation Java Agent RunControl is here

Robust fault toleranceRobust fault tolerance Process abstraction through COOL languageProcess abstraction through COOL language

Integration of foreign processesIntegration of foreign processes DP, vxServer, shellsDP, vxServer, shells EPICS, CAEN OPC comingEPICS, CAEN OPC coming Move toward full integration of Slow ControlsMove toward full integration of Slow Controls

Web Interface for remote monitoring.Web Interface for remote monitoring. Extended and customizable graphing and DAQ Extended and customizable graphing and DAQ

system monitoring capabilitiessystem monitoring capabilities Basis for Cmsg - CODA messaging system Basis for Cmsg - CODA messaging system

currently under developmentcurrently under development

Other IssuesOther Issues Integrate existing Hall requirements into a single Integrate existing Hall requirements into a single

supportable distribution.supportable distribution. Transition toward Hall D requirements. Transition toward Hall D requirements.

Maintain cross platform compatibilityMaintain cross platform compatibility SUN, LINUX, VxWorksSUN, LINUX, VxWorks 64bit Arch - Opteron, G5 (Mac X)64bit Arch - Opteron, G5 (Mac X) Embedded Linux (on Single Board Computers)Embedded Linux (on Single Board Computers)

Move to database independenceMove to database independence Proxy Server (JDBC) to support User’s database choiceProxy Server (JDBC) to support User’s database choice

User Hooks into the DAQ systemUser Hooks into the DAQ system JAVAJAVA Updated Tcl supportUpdated Tcl support Others…??Others…??

SummarySummary

CODA version 3 is now being molded - nothing is CODA version 3 is now being molded - nothing is irreversible. irreversible.

Our plan is to phase in new tools to provide a Our plan is to phase in new tools to provide a smooth transition from CODA2 --> CODA 3smooth transition from CODA2 --> CODA 3

Much software support for Hall D requirements are Much software support for Hall D requirements are on a short term timeline (2-3 years).on a short term timeline (2-3 years).

Front-End (hardware) support is longer term and Front-End (hardware) support is longer term and may go through a “revision 1” iteration for use in may go through a “revision 1” iteration for use in existing experiments. existing experiments.

Extra slidesExtra slides

Pipelines (Dead-timeless DAQ)Pipelines (Dead-timeless DAQ)

250MHz

10µs “snapshot” can be stored in memory (5KB/FADC)

A Trigger generates a lookback and extraction of the sampled ADC data

VME64X - VXS InterconnectVME64X - VXS Interconnect

J0 - 200 total pins 45 differential pairs 6 GHz Bandwidth

18 VME Payload Slots

2 Switching slots

What is CODAWhat is CODACODA is a software toolkit from which data acquisition systems with varying degrees of complexity can be built.

A typical system might look

Single Board Computer

FASTBUSVMECAMAC

ROC rcServer

EB

ROC

Network: Ethernet FDDI ATM

UNIX/LINUX

DISK/TAPEET

User Proc.

RunControlGUI

ER

cdev

dp_tcl

mSQL Database

msqld cmlogServer

cmlogDatabase