Physics 335 project update Anton Kapliy April 2007.

Physics 335 project update

Anton Kapliy

April 2007

Experiment - JPARCProcess: FCNC

sd → dd

Final state:

Two photons

Major BG:

Four photons, two missed

Critical to correctly detect all photons that hit CsI, and nail down the timing

Data flow

PMT – 1 channel(total = hundreds)

14-bit DIGI125 MHz

FPGATrigger

Data packingPipelining

Interface to PC

Amp/shaper

VME, GLink

Sum of 16

Trigger board…

x16

Global trigger

My work

Why need a trigger?

• >300 RO channels, digitized at 125 MhZ• 300*(14 bits)*(125,000,000) = 60 TB/sec• No computer can keep up!

• BUT: mostly junk• Physics simulations predict event rate <100KHz

for entire detector!• (14 bit)*100k = 2 GB/sec• Can save and apply HLT on PC

Note: in reality, the rates must be multiplied by 30

Data gets into FPGA

2 µs data pipelinedata

dataLocal trig

(threshold)

yes/no2 µs trigger pipeline

Data waits here for global trigger decision

data x 16 Sum of 16

These are slightly out of time

To trigger boardUsed to make global trigger decision

Described on next page

ADC data

ANDGlobal trigger

Local trigger Trigger pipeline

To global trigger boardBoard sum

Readout initiator

Raw pipeline data

data Data packer

Temp buffer 1Temp buffer 2Temp buffer 3Temp buffer 4

prio1prio2prio3prio4

Channel hypervisor MUXData (ch 1)

(to board hypervisor)

Board hypervisor

Data (ch 1)Data (ch 2)

Data (ch 16)

.

.

.

Per-channel logic:

Per-board logic (serves 16 channels):

VME buffer for writing

VME buffer for reading To VME crate

Local Triggering

Subtract pedestal Average over 4 cyclesdata Trigger

We massage the data before applying a trigger threshold:

Pedestal:•2 seconds idle•1 second collisions•When idle: collect 16 clocks of noise, save the average (“pedestal”)•Subtract pedestal from real data in the subsequent 1-sec interval

Average over 4 cycles:•Smoothes out small spikes

Local triggering: illustration

Threshold

Subtract pedestal

Avg over 4clocks

Note: input signal has correct shape, but baseline and noise have been over-emphasized to better illustrate the algorithm.

Global Trigger comes

2 µs data pipeline

2 µs trigger pipeline

Global triggerAND

•Coincidence between LOCAL and GLOBAL triggers•Start readout of the data from the pipeline

Start readout

Local trigger

Data packer

2 µs data pipeline

read out 32 clocks around trigger point

Data packer: timing and IDs

0010 1110 00 …

timin

g timin

g ch/b id

Data (29 words)16 bit

00

32 words

•Timing: an integer 0 - 1.25M identifies an event within a 1/100 sec spill•Id: channel ID (0-15) within a board, and board ID (1 to ~400)•Data: remaining 29 words

DFFtrig offset + pulse offset (~19)

Can we now save it to a computer?

NOT YET!

0010 1110 00 …

timin

g timin

g ch/b id

Data (29 words)16 bit

0032 words

Control bits

PC readout basics

PC – VME masterBuffer for WRITING

Buffer for READINGVME-addressable

Readout board serves 16 channels. VME slave

Saves ~100KhZ of data for the duration of spill cycle

New data from any of 16 channelsCan process 1at a time

Switchesin 1/100 s

Intermediate buffers

32x16 bit packed data prio1

32x16 bit packed data prio2

32x16 bit packed data prio3

32x16 bit packed data prio4

2 µs data pipeline

For *each* channel

Uses priorities to choosenext available buffer

Channel hypervisor

SUM reports channel priorityto board hypervisor

To VME slave buffer

Buffer for Writing

Chooses highest-priority buffer to read out

Board hypervisor

One per *board*

VME/GLinkBuffer for Reading

Switch every 1/100 s

EARLIERLATER

|

……01231 30 29

0|

Already saved!

16

16

|

Readout!

21 16

… …012

Overlap

0

Trigger sampling time

Data time

No downtime, little overlap

1/100 sec switching caveat

PC – VME masterBuffer for WRITING

Buffer for READINGVME-addressable

New data from any of 16 channelsCan process 1at a time

•Every 1/100 sec write and read buffers switch roles.

•New spill is started, and timing stamp is reset.

What if there are old intermediate buffers that are not read-out yet?

Solution: delay the read-write switch until all those are saved.

Project status

• Implemented entire trigger/readout logic• Tested portions, trying to get the whole system up• Most work done in dedicated Altera simulation• Programmed actual board (16 channels)• Observed scope traces of the signals• Altera Stratix II appears to meet timing requirements

Plans and issues

• Very technical stuff, hard to present

• Currently devising more visual tests

• Still unable to run the whole system

• Must be ready for FNAL test in December

• But I am confident I’ll get it to work!

SUPPLEMENTAL SLIDES

Intermediate buffer data flow

Detailed description

1

0

0

0

Raw pipeline

1

2

1

0

0

Raw pipeline

3

3

2

1

0

Raw pipeline

6

0

2

1

0

Raw pipeline

3

0

7

6

0

Raw pipeline

13

1

7

6

0

Raw pipeline

14

(a)

(d)

(b)

(e)

(c)

(f)

1

0

0

0

Raw data pipeline

1

Suppose a trigger comes (i.e. coincidence between local and global).

Choose 1st buffer with zero priority. Start writing and increment prio to 1.

address 0 address 31

0

0

0

0

Raw data pipeline

0

While we are writing, suppose a board hypervisor selects this channel to read.

We can start reading this buffer, even though not all 32 words are written yet!

to VME RAM

read out

NOTE: even though prio1=0,the channel hypervisor knowsnot to write to this buffer!

0

0

0

0

Raw data pipeline

0

Buffer 1 if completely read out. The channel is now empty.

1

0

0

0

Raw data pipeline

1

A new trigger comes. It is routed to buffer 1, which is now empty.

It’s priority is immediately increased.

address 0 address 31

2

1

0

0

Raw data pipeline

3

Another trigger comes when 1st one is more than 16 clocks done.Start writing (initially with the same data) into next 0-prio buffer.

Increment all non-zero priorities by 1 - i.e. those that are older

Note: we require that 1st buffer is >16 cycles.See next slide!

|

……01231 30 29

0

|

Already saved!

15

15

21 15

…

…

012

Overlap

Trigger sampling

Second data readout

(A)

(B)

First data readout

1 32

1 32

16

1 32

Overlap

1 32

(A) (B)

No trigger Can trigger

1221

2nd readout1st readout

trigger

readout

trigger

readout

2

1

0

0

Raw data pipeline

3

Another trigger comes when the two are still writing.

Since we are already saving everything, ignore it.

3

2

1

0

Raw data pipeline

6

Another trigger comes after 1st buffer is fully uploaded.

Start its write (again, duplicating some of the data from buffer 2).

Increase prio of old buffers, because we want to read them ASAP.Note: maximum priority value is 4!

0

2

1

0

Raw data pipeline

3

Now, the channel has a high enough prio that the board hypervisor selects it.

Then the channel hypervisor selects the highest-prio buffer *within* the channel.

In our case, we read out buffer 1. It’s prio is immediately nulled

to VME RAM

1

3

2

0

Raw data pipeline

6

Suppose at this moment another trigger comes.

The data is routed to buffer 1 – it’s already available! Its prio is increased.

Note that we again increase the priorities of old buffers!

to VME RAM

new event old event

1

0

2

0

Raw data pipeline

6

After this read is finished, another read comes.Buffer 2 – having the highest priority – is selected.

to VME RAM

1

0

0

0

Raw data pipeline

6

After the two reads are finished, another read comes. Buffer 3 is selected.

to VME RAM

6

0

0

0

Raw data pipeline

6

Suppose at this moment a VME spill comes (every 1/100 s).We’d like to read out the large VME-addressable buffer, but we cannot:The 32-deep buffers contain data from a previous spill!

Solution: promote up to two priorities to “critical”: prio+5.Don’t let VME read the data until all critical buffers are emptied!

to VME RAM

CRITReported to board hypervisor

0

1

0

0

Raw data pipeline

1

After 3rd buffer is read out, 1st one is selected.VME will be able to read out when: none critical, none reading.

If another trigger comes at this moment (belonging to a new spill),it will be saved as it normally would. No downtime!

to VME RAM

Last critical buffer!

0

1

0

0

Raw data pipeline

1

When the last critical buffer is read-out, the VME addressable buffers switch.VME is informed that it can start the readout from the current buffer.Data from the new spill can now be saved to a fresh VME buffer!

VME can RO!

VME-addressable pipe8K x 32 bit

VME-addressable pipe8K x 32 bit

Data from buffers

reports “vme ready”

Must be finished within 1/100 sec!

VME readout

reports max addressBoard hypervisor

VME block

MU

XVME can RO!