The CERN LHC central timing A vertical slice Pablo Alvarez Jean-Claude Bau Stephane Deghaye Ioan Kozsar Julian Lewis Javier Serrano.

The CERN LHC central timing

A vertical slicePablo Alvarez

Jean-Claude BauStephane Deghaye

Ioan KozsarJulian Lewis

Javier Serrano

Talk layout

• LHC Injector Chain Timing• Timing Distribution Overview• Central Timing Overview• Reflective Memory• UTC Time and GPS• The Multitask Timing Generator• The Controls Timing Receiver Module• Safe Machine Parameters Hardware

Checking• LHC Software Architecture and Event Tables

R1LHC

CNGS

Linac PSB CPS SPS

D3Dump

TI8Dump

TI2DumpSPS

Dump

R2LHC

TCLP

The LHC Proton Injector Chain

Strongly time coupled

CERN accelerator networksequenced by central timing

generator

….…. ….….

PSB

CPS

SPS

LHC

Experimentalarea

Experimentalarea

ExperimentalArea

CBCM Sequence Manager

PSB1 PSB1 PSB2PSB2 PSB3PSB3 PSB4 PSB4

CPS Batch 1 CPS Batch 2 CPS Batch 3 CPS Batch 4

SPS Cycle for the LHC

SPS injection plateaux

The LHC Beam

LSA Beam request:RF bucketRingCPS batches

ExtractionForewarning

LHC Injection plateaux

Injection Injection

ExtractionExtraction

The LHC timing is onlycoupled by extraction

start-rampevent

ExtractionForewarning

Timing Distribution Overview

• RS485 drop nets distribute timing around the CERN accelerator complex

• Long distance transmission over optical fibers

• One timing network for each accelerator• Hundreds/Thousands of timing receiver

modules distributed around the complex• One timing generator drives one timing

network

Timing Frames

• Timing frames are Manchester encoded at 500kbit using a 1MHz clock.

• Each frame carries 32 data bits, parity, start and stop bits.• One frame transmitted each 125us, 8 per millisecond.• Frames are broken into bit fields

– 4 Bits Accelerator [A]– 4 Bits frame Type [T]– 8 Bits Code [CODE]– 16 Bits Payload [PAYLOAD]

• Some frame are recognized by the hardware and cause special treatment– Two UTC frames carry the time of day in their payload– Millisecond frames are always sent in phase with the PPS– Telegram frames are stored in double buffers– Event frames cause counters to be loaded and triggered and

may produce bus interrupts

MillisecondMillisecond

A T CODE PAYLOAD

7 Other frames

LHCCentral Timing

CBCM Controlling

Injector Chain

CS-CCR-CTMGWFESA API

LSASequencer

“LSA”

SPS destination request R1,R2

CPS Batch Request 1,2,3,4

LIC Timing

HIX.FW1K

LHC TimingInhibitsRequestsInterlocks

SIS TI8/TI2& SPS DumpInhibits

LHC Fill Requests:BucketRingBatches

MasterON/OFF

CTR

LHC Fill Requests:BucketRingBatches

ReflectiveMemoryLink

CBCM SequenceManager

LIC Sequence

TI8/TI2Dump

RequestTI8/TI2Dump

LHCUser

RequestLHC User

NormalSpare

LSA changes Allowed

LSA Master

SEX.FW1KSPS.COMLN.LSA_ALW

Central TimingOverview

LHC GatewayImplements FESA API

2Gbit/SToken ringVMIACC-5595

Single ModeHub

64MbyteVMIPMC-5565Reflective memory

LHC Timing generator A LHC Timing generator B

Reflective Memory

Reflective memory:

A and B must always bein the same state.If no restrictions forswitch over

Protects token ring

IdenticalexceptID event

GPSOne pulse per Second

GPS

SymmetricomXLI

PLLOne pulse

per Second

Phase locked10MHz

Basic Period1200/900/600 ms

Advanced (100us)One pulseper Second

Synchronized 1KHz(slow timing clock)

Phase locked10MHz

Phase looked40 MHz Eventencoding clock

40MHz PLL

Synchronizationmodule in eachtiming generatorcrate

RS485Timing

MTTMultitaskTimingGeneratorMTT

UTC time (NTP or GPS)

Eventtables

External events

UTC Time and GPS

CERN UTC Time

Set once on startup & on Leap Seconds

RS485Timing

CTR

PPS

10 MHz

1 KHz

40MHz

Delay

Control SystemCERN UTC Time

25ns steps

Timing receiver

SymmetricomCS4000portable

Atomic Clock

Multitask Timing Generator MTT

• Hardware multitasking for 16 tasks – 32 local registers per task– 218 Global registers– 6 Memory mapped IO registers

• Timing frame out register• VME P2 in register …

• Host processor access to all registers

• Implements general purpose CPU– Op-codes are triadic: AND SrcREG,SrcREG,DstREG – AND 0x7,VMEP2,TMP

• Arithmetic and logical• Move indexed, literal, register• Wait value, relative• Conditional branch • Interrupt host

• Tasks defined from host via Task Control Block– PC– PC Offset– Processor Status Word

• Command and status registers allow host access to running tasks

MTT hardware module

FPGA (Spartan2 600E)

CPU

VME

Interface

UTC SERIALIZER

40MH

z

PP

S

SY

NC

Timing Signal

VME

P2

External Events

VME BUS

See: The LHC central timing hardware implementationP. Alvarez, J. Lewis, J. Serrano CERN, Geneva, Switzerland This conference

strp: % Start program and interrupt survey taskmovv TskStsRUNNING LRegTASK_STATUS % Say we are runningmovv ConsNOT_SET LRegParRUN_COUNT % Run foreverint 2 % Notify survey we are running

cont: % Wait for the VME P2 bits and send out events accordinglyworv ConsVMEP2_BITS VMEP2 % Wait for VME P2 bitsmovr VMEP2 RegVmeP2 % Copy reg and clear bits

tdmp1: % Test for dump ring 1andv ConsHX_DMPD1_BIT RegVmeP2 LRegTEMP % Test dump 1 bitbeq tdmp2 % Go check for dump 2 bitmovv ConsHX_DMPD1 EVOUT % Send dump 1movv ConsHX_ENBPM1 EVOUT % Re-enable after a dump

tdmp2: % Test for dump ring 2andv ConsHX_DMPD2_BIT RegVmeP2 LRegTEMP % Test dump 2 bitbeq tinj % Go injection warning bitmovv ConsHX_DMPD2 EVOUT % Send dump 2movv ConsHX_ENBPM2 EVOUT % Re-enable after a dump

tinj: % Test for LHC injectionandv ConsHIX_FW_BIT RegVmeP2 LRegTEMP % Test inject bitbeq tpm1 % Go check for PM ring 1movv ConsHIX_FW EVOUT % Send injection forewarning

tpm1: % Test for post mortem bit 1andv ConsHX_PM1_BIT RegVmeP2 LRegTEMP % Test PM ring 1 bitbeq tpm2 % Go check for PM ring 2movv ConsHX_PM1 EVOUT % Send PM-1 trigger

tpm2: % Test for post mortem bit 2andv ConsHX_PM2_BIT RegVmeP2 LRegTEMP % Test PM ring 2 bitbeq cont % Go check for PM ring 2movv ConsHX_PM1 EVOUT % Send PM-1 trigger (not PM-2)

jmp cont % Go wait for next P2 interrupt

MTT External Events Task

Controls Timing Receiver CTR

HybridPLL

Delay

FrameDecoder

TGM

TGMUTC

Content AddressedMemory - Triggering

CounterConfigurationsLoader

22Bit 50MHzCounters

BusInterrupt

Output

Timing

40MHz

LookupLoad

Trigger

•Start•Previous•External•Event

•Clock•Previous•40MHz harmonic•External

•Modes•Once•Multi-pulse•Burst

•Trigger•Event frame•Wildcard•Telegram

•Action•Output•Bus interrupt

•Output•TTL/TTL bar•Pulse width

CTRV – CTRI – CTRP formats

Reads thresholds

@1kHz-24Bit 108

Safe Machine Parameters Controller

for LHC

Energy A

I_beam1 & 2

Reads status

Energy B

BEM

BCT “A”

BEM

ManagementCritical

Settings

(CTRV)

CTRV

CTRx

LSA

BCT “B”

EXPLine

driver

LHCTiming

Generator

SMP @ 10Hz 16Bit 1010

(Flags, E & Int.)

BLMCTRVCTRV BLM BLM

CTRVCTRV

BIS BIS

BIS

CTRVCTRVCTRV

Kickers Kickers

Timing Network

Events, UTC, & Telegrams (including SMP)

Safe Machine Parameters Distribution

BIS

CTRxCTRxCTRx

EXP

Flags TTLHw Output

If length > 5m

LSA and FESA

• The FESA API is implemented on the LHC timing gateway

• Accesses timing generators across reflective memory

• Implements – Load or Unload event table– Get running tables list– Set event table run count and synchronization

event– Stop or Abort event table– Set telegram parameters– Send event– Read status of tasks and MTT module

Event tables

• Event tables are ASCII strings sent from LSA to the LHC timing gateway

• Each entry in the table contains– The name of the event to send “HX.START-RAMP”– The relative time to wait in seconds before sending

the event “1.01”– The payload the event will carry “0x0005”– A carriage-return <CRTN> terminates the entry

• The table run count and synchronization event will be filled in later when the table is to be executed

Garbage CollectLoad objectInitialize TCBRun task

Assembletask

Translateand Merge

HX.Start-Ramp 1.01 0x5HX.Start-Freq 0 0x05

waitr MSFR,1001movv 0x14020005 EVOUTmovv 0x14030005 EVOUT

Event Table Processing

FESA API

LSAEventTable

Table Compiler TranslationEvent Table

template

Table Task

Assembler

Object CodeReflectiveMemory

TaskLoader

MTTProgramMemory

Hot Standby

Data

Process

Conclusion

– The LHC machine processes such as squeezing, physics… may occur at any time and run independently of each other, they share no common basic time period. This has led to a new approach in which machine processes are modeled as independent timing event tasks that can be started, stopped at any time.

• Reuse of existing timing boards was facilitated by using FPGAs and writing new VHDL

– The LHC timing distributes machine safety critical data around the control system. This has led to the development of hardware to monitor its performance and dump the beam when necessary.

– Reflective memory has greatly simplified the implementation of the hot-standby and the timing generators themselves which are no longer required communicate directly with the rest of the control system. This has increased their reliability considerably.