Sun Constellation Linux Cluster: Ranger System Name: Ranger Host Name: ranger.tacc.utexas.edu IP Address: 129.114.50.163 Operating System: Linux Number of Nodes: 3,936 Number of Processing Cores: 62,976 Total Memory: 123TB Peak Performance: 579.4TFlops Total Disk: 1.73PB (shared) 31.4TB (local)
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Sun Constellation Linux Cluster: Ranger
System Name: Ranger
Host Name: ranger.tacc.utexas.edu
IP Address: 129.114.50.163
Operating System: Linux
Number of Nodes: 3,936
Number of Processing Cores: 62,976
Total Memory: 123TB
Peak Performance: 579.4TFlops
Total Disk: 1.73PB (shared) 31.4TB (local)
System Overview
The Sun Constellation Linux Cluster, named Ranger, is one of the largest computational resources in the world. Ranger was made possible by a grant awarded by the National Science Foundation in September 2006 to TACC and its partners including Sun Microsystems, Arizona State University, and Cornell University. Ranger entered formal production on February 4, 2008 and supports high-end computational science for NSF TeraGrid researchers throughout the United States, academic institutions within Texas, and components of the University of Texas System.
The Ranger system is comprised of 3,936 16-way SMP compute nodes providing 15,744 AMD Opteron™ processors for a total of 62,976 compute cores, 123 TB of total memory and 1.7 PB of raw global disk space. It has a theoretical peak performance of 579 TFLOPS. All Ranger nodes are interconnected using InfiniBand technology in a full-CLOS topology providing a 1GB/sec point-to-point bandwidth. A 10 PB capacity archival system is available for long term storage and backups. Example pictures highlighting various components of the system are shown in Figures 1-3.
Figure 1. One of six Ranger rows: Management/IO Racks (front, black), Compute Rack (silver), and In-row Heat Exchanger (black).
Figure 3. InfiniBand Constellation Core Switch.
Architecture
The Ranger compute and login nodes run a Linux OS and are managed by the Rocks 4.1 cluster toolkit. Two 3456 port Constellation switches provide dual-plane access between NEMs (Network Element Modules) of each 12-blade chassis. Several global, parallel Lustre file systems have been configured to target different storage needs. The configuration and features for the compute nodes, interconnect and I/O systems are described below, and summarized in Tables 1.1 through 1.3.
Ranger is a blade-based system. Each node is a SunBlade x6420 blade running a 2.6.18.8 Linux kernel. Each node contains four AMD Opteron Quad-Core 64-bit processors (16 cores in all) on a single board, as an SMP unit. The core frequency is 2.3 GHz and supports 4 floating-point operations per clock period with a peak performance of 9.2 GFLOPS/core or 128 GFLOPS/node.
Each node contains 32 GB of memory. The memory subsystem has a 1.0 GHz HyperTransport system Bus, and 2 channels with 667 MHz DDR2 DIMMS. Each socket possesses an independent memory controller connected directly to an L3 cache.
The interconnect topology is a 7-stage, full-CLOS fat tree with two large Sun InfiniBand Datacenter switches at the core of the fabric (each switch can support up to a maximum of 3,456 SDR InfiniBand ports). Each of the 328 compute chassis is connected directly to the 2 core switches. Twelve additional frames are also connected directly to the core switches and provide files system, administration and login capabilities.
File systems: Ranger's file systems are hosted on 72 Sun x4500 disk servers, each containing 48 SATA drives, and six Sun x4600 metadata servers. From this aggregate space of 1.7PB, three global file systems are available for all users (see Table 1.5).
Table 1.1 System Configuration and PerformanceComponent Technology Performance/SizePeak FloatingPoint Operations