Data Filtering and Software

Erik BlaufussUniversity of Maryland

Data Filtering and Software

IceCube Collaboration Meeting

Monday, March 21, 2005

March 21, 2005Data Filtering and Software – E. Blaufuss

Overview• Data Filtering and Software includes:

– Processing and Filtering (PnF) System at Pole– Production Processing software environment

• Offline software suite• IceTray software framework• Dataclasses event containers• Build system and tools

– Database design and implementation• Database server• Access tools

– Data visualization• Event Viewer• Root-based analysis tools

March 21, 2005Data Filtering and Software – E. Blaufuss

PnF System Overview

DAQ

Dispatch

P&F Server

Recon WorkerX 50-60?

4-15MB/s1500-2000Hz

Tape

Transient Buffer

DAQ Land

Raw Data

Online Analysis

Online Analysis

Online Analysis

Recon Monitor

Data Handling

Rec/FiltData

ALL: RAW+FILT+REC

FILT:RAW+FILT+REC

ALL:FILT+REC

Expt. Control / Logging

Offline Software/IceTray

March 21, 2005Data Filtering and Software – E. Blaufuss

PnF system at Pole• V1.0 of PnF system was deployed at Pole

this past season.– Running IceTray in Online environment– Parses and decodes the incoming DAQ stream– Currently performs no reconstruction.

• Currently not in the Main data stream.– Lack of global run information at Pole– Lack of system-wide integration testing

• No PnF output sent via satellite– TestDAQ/Monolith output tarfile only output sent

• New version of Monolith soon (April)– Real testing of data flow to PnF (SPTS -> SPS)

March 21, 2005Data Filtering and Software – E. Blaufuss

PnF Plans for next year

• With the “Real DAQ” in place:– PnF output IS the output stream for RAW event

data from IceCube DAQ.– All other non-event data will be handled by

databases.

• After this next deployment season:– Filtering at South Pole will be REQUIRED.– Will require reconstruction/filtering modules be

ready and fully tested in IceTray offline!

March 21, 2005Data Filtering and Software – E. Blaufuss

Offline Software

• Several incremental releases this past year– In use by reconstruction and simulation groups.

• Current release (new!):– OFFLINE-SOFTWARE V01-05-00

• Full details on release, and HEAD at:– http://glacier.lbl.gov/

• Plans for more functionality and tools over the next year.

• More details regarding “readiness” of software later….

March 21, 2005Data Filtering and Software – E. Blaufuss

Databases (Mons group)

• DOM assembly and testing database setup and in use for FAT testing.

• Geometry database for String 21/IceTop up to date with Kurt’s latest numbers.– Includes “nominal” locations for other strings

• Preliminary version of DOM calibration DB• IceTray module for DB access in place• In the next few months:

– Run configuration DB for data from current year– Continued integration of database into IceTray

March 21, 2005Data Filtering and Software – E. Blaufuss

Visualization

• Event viewer in place– Animations– Fits, features,

readouts

• Root analysis Tree tools in place– Fit/reconstruction

level analyses– Waveforms, features,

hits level analyses

• Plans for more over the next year.

March 21, 2005Data Filtering and Software – E. Blaufuss

Current State of Offline SW(Is it “analysis ready”?)

• IceTray software Framework– Has been released for ~1 year.– Can handle single events in a linear chain of

modules• More advanced module topologies planned this year

• Dataclasses event storage classes– Have been released for >9 months.– Designed for storage of

• IceTray DOM data• Amanda data (treat as stand-alone/together)• Reconstruction results• Geometry, calibration, monitoring structures, more…

March 21, 2005Data Filtering and Software – E. Blaufuss

State of Software (cont’d)

– More extensions planned as need dictates.– Can hold TestDAQ data from this year

• I/O routines for IceCube data– Monolith-reader can decode data from Pole.– Triggered events from South Pole in dataclasses– Basic functionality is there.

• Includes domcal information (Thanks Dima…)• Still some work to do (trigger ID, run ID, etc)

• Run “meta-data” – Run configuration info.– Data exists (files), but documentation and wider

understanding is minimal.– Need docs, fill run configuration database to

make properly available for analysis in IceTray.

March 21, 2005Data Filtering and Software – E. Blaufuss

State of Software (cont’d)

• Cleaning, feature extraction– Basic functionality in place and working

• Hit selection/cleaning• FeatureExtractor (Dima)

– Advanced cleaning needs to be implemented• Amanda specific: Cross talk, flare checking, etc.• Plans for that in next few weeks with Peter Steffen and

others

• Reconstruction– Basic algorithms are done (and reviewed!)

• First guess (linefit, dipole, c-first, TOI, JAMS)• Standard Log-likelihood for muon tracks

– Advanced (exotic PDFs, iterative fits) needed

March 21, 2005Data Filtering and Software – E. Blaufuss

State of Software (cont’d)

– Cut parameters/observables• Work has just started here and much remains.

• Filtering capabilities– Currently possible to use linear chain of modules

to reduce data files.– Branching of flow in modules planned for this

year.• Allow muon and cascade recos in same chain.• Needs a “decision” module

• Amanda– Full support for calibrated F2K files

• Includes TWR in F2K files

March 21, 2005Data Filtering and Software – E. Blaufuss

State of Software (cont’d)

– TWR readouts (non-F2K) support this year– Working towards being ready to merge

AMANDA/IceCube data this summer.• IceCube and Amanda data are both handled naturally

together in IceTray/dataclasses.

• Simulation(Significant driving force in the design and

requirements for IceTray.)• Merging input frames to overlay events

– Currently developing simulation releases against OFFLINE-SOFTWARE releases.

March 21, 2005Data Filtering and Software – E. Blaufuss

String 21 Data – A few plots

March 21, 2005Data Filtering and Software – E. Blaufuss

String 21 Data – A few plots

More like this in Gary’s talk

March 21, 2005Data Filtering and Software – E. Blaufuss

Status SummaryFor example…

Is…. Work provided by…

Core Software IceTray,

Dataclasses

I/O

V1 of all ready and being used

UMD, LBNL, UDel

Core Utilities DB access,

File readers,

eventviewer

80% done, all in a usable state

UMD, Mons, LBNL, Dortmund

Application Software

Reconst.,

Filtering, Calib, Sim.

Work is underway(some useable)

Entire Collaboration

March 21, 2005Data Filtering and Software – E. Blaufuss

Status Summary (2)• IceCube Offline software effort needs more:

– Users– Developers of applications– Feedback, bugs found, documentation readers,

etc.

• We’re providing the framework and some tools. – Real work of providing reconstructions,

analyses, simulations will be the work of the entire collaboration.

• Start using it. It’s ready.– There are many ways to help out in software.