Discussion of Design Parameters for the ProtoDUNE-SP ... · -SP requires 9:1 for a MIP at the cathode in all three views-SP induction-plane s:n is lower than collection-plane s:n-s/n

Discussion of Design Parameters for the ProtoDUNE-SP Online ComputingTom JunkDUNE Software and Computing General MeetingJune 6, 2016

Requirements• Still being debated what the definition of "success" is

• Minimal success from the DUNE Strategy Document DocDB 1050 (M. Thomson)- Strategic goal of commissioning and operation of two ProtoDUNE detectors

prior to the reviews of the DUNE TDRs to demonstrate the validity of the designs using full-scale components and to mitigate risks associated with the far detector construction;

- Strategic goal of recording charged-particle test-beam data in both ProtoDUNE detectors prior to LS2 of the LHC;

- Use the ProtoDUNE-SP TDR requested by the SPSC as a major milestone to define the single-phase far detector design;

- Other goals related to collaboration responsibilities and working groups

• Hallway discussions: Commissioning with cosmics can satisfy the CD-2 review needs. But is it enough to build the required team to make it real?

June 6, 2016 Tom Junk DUNE S&C ProtoDUNE-SP Online2

ProtoDUNE-SP Requirements• Take instead the ProtoDUNE-SP proposal, DocDB 186 (T.

Kutter et. al.)


5MTotalTriggers(analyzable)

Unknownfractionof"good"triggers.

Beamcompositionwilldifferfromourneeds –needtriggering

ProtoDUNE-SP Measurement Program

• From the Proposal, DocDB 186 (T. Kutter et al) -- more or less sorted by importance, weighted by feasibility:- Shower calibration

- Cross section measurements

- Angular dependence of dE/dx

- Reconstruction Performance

- e/gamma separation

- Michel Electrons

- Charge sign determination

- Nucleon Decay sensitivity and background samples

- Antiproton annihilation


Requirements of the Trigger/DAQ System• Beam trigger with PID:- Precise timing (requirement -- ±1𝛍s sounds reasonable, much better is

doubtless possible)

- PID performance numbers If we don't ask, we don't get. – The more imperfect these are, the more data we need.

- Triggers must be prompt. Lack of buffering in the DAQ means we cannot tolerate all of the beam instrumentation information given to us in a big buffer slowly read out between spills. Need information to make prompt trigger decisions.

- Upstream momentum measurement. Performance numbers to be specified

- No beam trigger at all (!) means a much larger sample of data is required to fish out the particles we need.


Requirements of the Trigger/DAQ System• Events must be self-contained. No 35-ton-like sharing of

the data on a trigger from one event to another.• Corrupt data do not count towards our total data request.

Examples of corrupt data:- mismatched timestamps between detector components

- missing detector components' data when expected

- de-synchronized ADC bits

- Events of the wrong length

- Neighboring ADC samples not from neighboring times (skipping ticks)

- corrupted headers (wrong channel ID's)


High-rate scenario• Goldilocks gets us, with 10 Hz triggering, 5M triggers in 20.3 days:

Brett Viren, DocDB 1086

• Running efficiency may be low- beam availability – maybe 1 day/wk retuning. Sharing beam with other

experiments? Running together with ProtoDUNE-DP or serially?

• Fraction of good triggers may be low. Depends on the quality of the upstream instrumentation, readiness of the detector.

• Ask for Factor of 10 over 5M to estimate the fraction of useful triggers.

• Easiest to make up in the trigger rate. Puts us in the High-rate scenario, but for longer time. 20.3 days at 100Hz gets us 50M triggers. == High-rate with ½ of detector read out. Total data volume ~1 PB.


artdaq Topologies


KurtBiery,Nov.2015

artdaq Discussions• At the CM, Kurt floated the idea of removing the Aggregator step

and write straight from the EB to disk. He is now thinking of using the aggregators again.

• Need to put the online monitoring tasks in the EB's.• Fragments go from BoardReaders to EB's in a round-robin

schedule• We lack the buffering memory in the RCE's to store a whole 4.5-

second-long spill untriggered. Need to write disk during the spill (?) unless memory is spec'd for the artdaq nodes and artdaqcan accumulate events in the pipeline before they go to aggregators or disk. Disk cache?


artdaq discussions• In the high-rate scenario, the instantaneous data rate (triggered,

uncompressed) is 4.6 GBytes/sec = 37 GBits/sec. Includes factor of 5 compression!

• If we have 1-GBit network (or GBit disk-writing speed, which is coincidentally similar, seen in 35-ton), we'd need ~40 output streams.

• If boardreaders are not on the same computers as the disk writing processes, we need >40 network connections from them to the EB's. - Easier to put memory buffers in the BoardReader computers? 4.6 GBytes/sec * 4.5

seconds = 21 GBytes, spread across computers running the BoardReaders.

- Average rate 1.3 GBytes/sec doable in just a few streams (12?)

- Sounds like we need 1 server running BoardReaders per 10 RCE's, if we are limited to 1 GBit/sec internal network, just to have enough cables between the BR's and the EB's. Or more NICs per server.

• Higher-rate internal network speeds. 10 GBit possible? Cost?


Disk Buffer and Analysis• 3-day requirement – 341 TB from Brett's high-rate scenario is not crazy.

• Need to investigate/evaluate file systems. In the worst case, we can always alternate between disks ourselves so we are not writing many files to the same disk at the same time. Reads are faster. Disk caching can help with reading a recently written file.

• Can arrange to write 12 streams while reading N streams out and funnel them through a switch capable of driving the 20 GBit network out of EHN1.

• Even with slow hardware, and slow software (LArSoft), 35-ton was able to analyze a small fraction of events online. - Event display does not need to see a large fraction of events. Maybe 1 per spill is

okay?

- Online analysis much simpler for SP. But need to see that the beam particles are as expected.

• Bernd Panzer Steindel notes we haven't requested the 20 GBit network yet. Will cost money. Digging a trench for the fibers.


Differences between SP and DP• Very useful discussion with Dario May 28.

• Elisabetta's slides of Dec. 15 provide the answers too:http://indico.fnal.gov/getFile.py/access?contribId=4&resId=0&materialId=slides&confId=11030

• But some things I was slow to realize why the WA105 computing farm was sized the way it was.

• Extra requirement: DP needs online measurement of gain and electron lifetime to monitor data quality. Hard to disentangle these two parameters without reconstructing events.

• See André Rubbia's talk at the Detector R&D Workshop in January of this year: DP gain is not constant. Charging-up effect:

http://indico.fnal.gov/getFile.py/access?contribId=26&sessionId=6&resId=0&materialId=slides&confId=10548

• SP: gain is assumed to be constant.


Differences between SP and DP• signal-to-noise ratio is bigger in DP.- SP requires 9:1 for a MIP at the cathode in all three views

- SP induction-plane s:n is lower than collection-plane s:n

- s/n for dual phase is at ~100:1 (Elisabetta's talk) à factor of 10 compression expected for DP.

- s/n for induction-plane wires in 35-ton was ~2:1. à factor of 2 compression obtained. Can always dial the electronics gain down to have less noise and thus more compressible data, but also lose signal strength this way.


Differences between SP and DP• SP is running on a tight schedule. Likely not much time to take data if

components arrive and are commissioned late.

• We should plan for success. But the chances something is "not quite right" are high.

• Lots of talk about a post-LS2 SP run to get more precise calibrations of physics deliverables.

• ProtoDUNE-DP not (currently) very interested in a post-LS2 run.

• SP Online farm may then be upgradeable if the goals of the first run are modest.

• Ambitious plans however attract more collaborators. Modest plans mean students and postdocs will work on other experiments.


ProtoDUNE Beam Requests• NP02 has requested 120 days of beam time from the SPSC.

No objections, not (much) competition for beam time.• Dario noted that the NP04 beam request has not yet been

made.


Discussion of Design Parameters for the ProtoDUNE-SP ... · -SP requires 9:1 for a MIP at the cathode in all three views-SP induction-plane s:n is lower than collection-plane s:n-s/n

Documents