Minimalist’s Linux Cluster Changyoung Choi, Jeonghyun Kim, Seyong Kim Department of Physics Sejong University.

Minimalist’s Linux Cluster

Changyoung Choi, Jeonghyun Kim, Seyong Kim

Department of PhysicsSejong University

Reported in Lattice’99

1. Usual Linux cluster

• Easy to build, use and maintain• Off-The-Shelf everything!

– Ordinary PC’s– Ordinary network components– Free OS (Linux)– Free parallel programming environment

(MPI)

Room for improvement

• Occupies too much space• Consumes too much power• Not all the components in an ordinary

PC is necessary for computation– Possible further reduction in cost!– Possible reduction in power consumption

• Supporting full bisection bandwidth is costly!

Conditions

• No custom-made hardwares– If it takes too long to develop ->

no advantage– If there is, should be recyclable

• Should be efficient enough• Should be do-able with small manpower

2. Construction(hardware)

• Node – ATX size standard mother board– Intel Pentium IV-2.4 GHz– 512 Mbytes DDR SDRAM– 100 Mbps ethernet card (3+1)– 180W power

• Cage for ATX size mother board• 1 front end server

CPU Intel P-4,2.4GHz

\236,500(=$198)x36

motherboard

ASUS P4PE \203,500(=$170)x36

memory 512MB PC2700 DDR

\112,200(=$93.5)x36

NIC 3+1 realtek \53,000(=$44)x36

power Sun ATX 180W \25,300(=$21)x36

cable \10,000(=$8)x36

cabinet 2000x910x750

\880,000(=$733)

cabinet accessory \365,200(=$304)

Cost per node $534.5

Total cost $534.5x36+$1037 = $20,279

Price Includes Tax

Software

• WOW Linux 7.1 (equiv. RedHat 7.1)• Linux Terminal Server Project (LTSP)

– Remote booting (DHCP + TFTP)– Diskless node system

• 2-D mesh network configuration– Script developed in-house

• MPI programming environment– MPICH/LAM

Setup

• Front end wait• F-end send DHCP

info• send kernel using

TFTP service• send client file

system by NFS

• Node 1 power-up• NIC listens to DHCP

server• IP setup complete

and download kernel• setup file system on

memory and copy NFS mounted file system to memory

Setup(continued)

• Node 1 wait• Node 1 send DHCP

info• Node 1 send kernel

image• Node 1 act as a

pipe line between front end and node 2 for NFS

• Node 2 power-up• NIC listens to node 1

DHCP server• IP setup complete and

download kernel from node 1

• setup file system on memory and copy NFS mounted file system to memory

Parallel boot

3. Performance

• 1-D ring full QCD code– Not suitable for 2-D mesh structure

• 8^3 x 512 lattice

8^3 x 32 lattice testedon 1-node

1187 sec, CG = 181.2 -> 369 MFLOPS

8^3 x 512 lattice testedon 4-node

5522 sec, CG = 391.4 -> 1.24 GFLOPS

8^3 x 512 lattice testedon 8-node

3029 sec, CG = 390.9 -> 2.25 GFLOPS

4. Discussion

• recyclable– cabinet takes standard motherboard size

-> replacing node is enough for a upgrade

• occupies smaller foot-print, consumes less power

• need to fine-tune• need to modify the code for 1-D ring

to 2-D mesh