Implementation of the STAR Data Acquisition System using a Myrinet Network J.M. Landgraf, M.J. LeVine, A. Ljubicic, Jr., M.W. Schulz (Brookhaven National Laboratory) J.M. Nelson (University of Birmingham) C. Adler, J.S. Lange (University of Frankfurt)
17
Embed
Implementation of the STAR Data Acquisition System using a Myrinet Network
Implementation of the STAR Data Acquisition System using a Myrinet Network. J.M. Landgraf, M.J. LeVine, A. Ljubicic, Jr., M.W. Schulz (Brookhaven National Laboratory) J.M. Nelson (University of Birmingham) C. Adler, J.S. Lange (University of Frankfurt). First Collisions at RHIC!. - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Implementation of the STAR Data Acquisition System using a Myrinet
Network
J.M. Landgraf, M.J. LeVine,
A. Ljubicic, Jr., M.W. Schulz(Brookhaven National Laboratory)
J.M. Nelson(University of Birmingham)
C. Adler, J.S. Lange(University of Frankfurt)
First Collisions at RHIC!
Star Control Room June 12th, 2000 9:00pm
Outline
• The STAR DAQ System– Components– Event Building Network
• Introduction to Myrinet
• Myrinet Implementation– Myrinet Software (GM)– STAR DAQ Software– myriLib
L3– Linux Farm (Compaq Alpha workstations)– Physics based build decision
Event Building Network– Token Management– Event Building– Event Storage and Buffering
DAQ / L3 Event Building Network
Squares: MVME / VxWorksCircles: Alpha Workstations / Linux
Diamonds: Ultrasparc Workstations / Solaris
What is Myrinet?• Commercial Network From Myricom
(www.myri.com)
• Low cost (~$1K / Card, $4-6K / Switch) • PCI / PMC Network Interface Cards• High bandwidth (1.28 + 1.28 Gb/sec)• Low Latency (13 usec)• Scalable switched topology• Network control performed in software• Open-source MCP / Driver software
Myrinet Architecture
• Network Card Interface (PCI64B)
– Lanai processor controls network– Local memory buffer– Both network & PCI DMA engines
• Switches
– Cut-through wormhole routing– CRC is recalculated at each stage
Including header– Stop/Go flow control mediated with
Small slack buffer
Myrinet Throughput
We Tested: 32 / 64 bit Myrinet cards VxWorks MVME 2604, MVME 2306 Linux Compaq Alpha Linux Intell Solaris Ultrasparc
Myrinet Software Network mapping
• Each myrinet node maintains list of port offsets to each other node
• Dynamic and Static mapping supported• Alternate routes can be forced by user
Myrinet driver (GM)• Variable length Messages
–Sender / Receiver provide buffersin advance for each size
– Sender / Receiver notified and mustreturn buffer to gm
• Directed Sends– DMA directly to host memory– Receiver not notified• GM imposes structure on user program– Poll / Block on gm_receive()– GM is not thread-safe
DAQ SoftwareSoftware is Message Based
for(;;){ msgQReceive(&msg); switch(msg.cmd) { }}
Sending is routed to the proper network
Each network has an associated daemon
daqMsgSend(node, &msg)
node/task/domain Local QueueMyrinetEthernetVME
myriLibethComLibvmComLib
que[task]
ICCP Message Protocol• 120 byte messages• Standard header
myriLib
DAQ library which wraps gmmyriMsgSend()myriMemCpy()
What does it do?• Manages the DMA message buffers • Handles callback functions• Thread synchronization• Misc… (Byte order, 32 vs. 64 bit etc.)• Bypasses DMA limitations on Solaris
Several Flavors• Threaded vs Process• Buffered vs Unbuffered DMA copies