COMPUTE | STORE | ANALYZE Cray Cluster Solutions & Cray Aries Interconnect Mathias Pütz Pre-Sales Manager Cray Europe Thanks for contributions from Thomas Alrutz & Christian Simmendinger T-Systems – Solutions for Research May 13, 2014 – HPCN Workshop / Göttingen 1
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
C O M P U T E | S T O R E | A N A L Y Z E
Cray Cluster Solutions &
Cray Aries Interconnect
Mathias Pütz Pre-Sales Manager Cray Europe
Thanks for contributions from Thomas Alrutz & Christian Simmendinger
T-Systems – Solutions for Research
May 13, 2014 – HPCN Workshop / Göttingen
1
C O M P U T E | S T O R E | A N A L Y Z E
Cray Overview
Copyright 2014 Cray Inc. -
Confidential and
Proprietary
● Cray founded in 1972 ● ~1000 employees across 20 countries
● Headquarters in Seattle, WA
● Six Major Development Sites: ● Bristol, UK
● Chippewa Falls, WI
● Pleasanton, CA
● St. Paul, MN
● Seattle, WA
● Austin, TX
● UK based European Exascale Research Initiative
We build the world’s fastest solutions to help solve “Grand Challenges” in
science, engineering and analytics
5/13/2014
Manufacturing Earth
Sciences Energy Life Sciences Financial
Services
Government
and Defense
Higher
Education Cyber-
security
C O M P U T E | S T O R E | A N A L Y Z E
We Build Computational Tools That Help Change The World
Supercomputers
Flexible Clusters
Hybrid Architectures
Supercomputing Big Data
3
Compute
Integrated Storage
& Data Management
Tiered Storage Archive Store
Analyze Graph Analytics
Hadoop Solutions
C O M P U T E | S T O R E | A N A L Y Z E
4
Supercomputing Leadership
C O M P U T E | S T O R E | A N A L Y Z E
Why Cluster Supercomputers from Cray?
5
• End-to-End Cluster
Deployments
• Fast-to-Market Innovative
Technologies based on
Industry Standards
• Cluster Solution Expertise
• Experienced Building
Clusters Based on Workload
Requirements
• Air-Cooled or Direct
Liquid-Cooled Solutions
• Node and Network
Configuration Options
• Supported by Cray cluster
SW ecosystem
• Large Memory, Shared
Memory Configurations
• Problem/Solution
Customer Approach
• Fast Deployment Time
• Reliable and Scalable
Performance
• Outstanding Customer
Experience
Flexible Configurable Technologies to Solve Challenging Problems
Leadership Flexibility Results
C O M P U T E | S T O R E | A N A L Y Z E
Only from Cray
6
Unique offerings to Solve Challenging Problems
• Cray’s Scalable Cluster
Management Software
• Framework for Cray’s
Cluster HPC Software
Stack
• Deployed in some of the
largest production clusters
in the world
• Modular in design to
support 3rd party cluster
tools
Advanced Cluster Engine
• Cray’s Optimized C, C+ &
Fortran Compilers
• Cray’s Math & Scientific
Libraries
• LibSci
• LibSci-acc
Cray Compiler Environment
C O M P U T E | S T O R E | A N A L Y Z E
Cray Customers
7
C O M P U T E | S T O R E | A N A L Y Z E
“Catalyst”, Lawrence Livermore National Laboratory
Based on the Cray CS300-AC™, the Catalyst system is a
collaborative system between Lawrence Livermore,
Intel and Cray Inc. The cluster will be managed through
LLNL’s HPC Innovation Center and used to explore non-
volatile memory in HPC simulation and big data analytics
Research Science
Allows for the exploration of new approaches to
big data analytics and HPC simulations requiring
superb memory and I/O capabilities for :
• Bioinformatics
• Business analytics
• Machine learning
• Natural language processing
• Simulations
For more information: http://hpcinnovationcenter.llnl.gov/
https://asc.llnl.gov/
System Overview
• 150TF of performance with a total system cores of 7,776
• Intel Xeon ES-2695v2 processors (12 cores, 2.4Ghz)
• 42TB of dynamics random access memory (DRAM)
• 282TB of non-volatile memory (NVRAM)
• Over 600GB/s of I/O bandwidth, over 60 million IOPS
• 12 Lustre and 2 NFS router nodes (128GB DRAM)
• VRAM: Intel SSD 910 Series (800GB)
• Fabric: Intel True Scale Fabric (dual-rail QDR-80)
• Storage software: Intel Lustre, Livermore DIMMAP (Data
Immersion MMAP), and SCR (Scalable Checkpoint Restart)
• Delivered to LLNL in October 2013
Catalyst
8
C O M P U T E | S T O R E | A N A L Y Z E9
Cray CS300 Series
Cluster Supercomputers
Air-Cooled Liquid-Cooled
C O M P U T E | S T O R E | A N A L Y Z E
Cray CS300-AC™ Cluster Supercomputer Overview
Flexible and Scalable • Up to 80 nodes/rack configured with 2 CPUs/node
• Interconnect: 3D Torus/Fat Tree, single/dual rail,
QDR/FDR/EDR, IB/GigE, STL-1 and STL-2
• Supports a wide range of parallel Storage File
Systems
Manageable • Integrated with Cray’s HPC Cluster Software Stack
• Includes Cray’s Advanced Cluster Engine™
Management software
• Includes network, server, cluster and storage
management
Reliable • Node and System level redundancy
• Hardened software stack tested by CCS
Development team
End-To-End Supercomputer Solutions • From design, integration, delivery to on-site
installation and customized services
10
Performance and Flexibility
Air-Cooled Solution
C O M P U T E | S T O R E | A N A L Y Z E
2x Motherboards
Compute Blades
Service Blades
Hybrid Blades
Chassis
Cray CS300™ Cluster Supercomputer
Optimal Building Blocks for Cluster Supercomputer
11
C O M P U T E | S T O R E | A N A L Y Z E12
Compute Platform
• Design optimized for Compute platform
• Support for 2x Intel Ivy Bridge EP CPUs
• Support for 8x DDR3 1866MHz memory modules
• Access to up to 32 PCIe Gen3 lanes
— 1x PCIe Gen3 x16 expansion slots
— 2 x PCIe Gen3 x8 expansion slots
• Option for on-board PCIe Gen3 Mellanox Connext-X3
• Future generation supporting Haswell/Broadwell in plan
Hybrid Platform
• Design optimized for Hybrid platform
• Support for 2x Intel Haswell (Broadwell) EP CPUs
• Support for 16x DDR4 2133MHz memory modules
• Access to up to 48 PCIe Gen3 lanes
— 2x PCIe Gen3 x16 expansion slots
— 1 x PCIe Gen3 x8 expansion slots
• Option for on-board PCIe Gen3 Mellanox Connect-X3
• Future generation supporting Haswell/Broadwell in plan
CS300 Cluster Supercomputer Roadmap Romley System Boards
C O M P U T E | S T O R E | A N A L Y Z E
● 0.5RU effective density Compute Blades ● Supports Intel Xeon E5 Ivy Bridge
CPUs
● Supports up to 128GB DDR3 1866 MHz system memory
● Integrated PCIe Gen3 QDR/FDR IB (optional)
● Support for one internal 2.5” HDD
● 1RU effective density Hybrid
Blades ● Host + Expansion Blade
● Supports 2x nVIDIA Keplers
● Supports 2x Intel KNCs
Cray CS300-AC™ Cluster Supercomputer Configurable Compute Nodes
13
C O M P U T E | S T O R E | A N A L Y Z E
● SR5000 Chassis supports either: ● 10x Compute Blades
● 5x Hybrid Blades
● Compute Blades are 0.5RU 2P x86 servers
● Hybrid Blades are 1RU 2P x86 combined with either 2x nVIDIA Keplers or 2x Intel KNCs
● Three hot-swappable, redundant Cooling Fan Units
● Up to four hot-swappable, redundant 1630W PS
● Can support one or two redundant iSCB chassis managers
Cray CS300-AC™ Cluster Supercomputer Reliability & Serviceability – SR5000 Series
14
C O M P U T E | S T O R E | A N A L Y Z E15
Cray proprietary design ● Common interface to all blade servers
with iSCB: Chassis Management Appliance
ARM-based appliance running embedded Linux
● Node health monitoring
● Supports concurrent console sessions to each node (terminal concentrator)
● Reliable power control
● Active dynamic fan control
● Power monitoring
● Dedicated powerman/conman interface
● GPU/MIC power control
Can easily upgrade features
Cray CS300™ Cluster Supercomputer Reliable Management Designed for HPC
C O M P U T E | S T O R E | A N A L Y Z E
2RU effective density
● Service Node w/ expanded IO + 3x 3.5” HDDs ● Up to three 3.5” SATA/SAS HDDs
● Up to four PCIe expansion slots
● Ideal as a GW/Login Node
● Service Node w/ expanded IO + 6x 2.5” HDDs ● Up to six 2.5” SATA/SAS HDDs
● One DVD ROM Drive
● Up to four PCIe expansion slots
● Ideal as a Management/Boot Node
Cray CS300-AC™ Cluster Supercomputer Highly Configurable Service & Hybrid Nodes
16
Used for ACE Management Nodes, Login Nodes,
Data Transfer Nodes, and Ancillary Nodes
C O M P U T E | S T O R E | A N A L Y Z E
● 2RU effective density ● Hybrid Node w/ 4x Accelerators + 2x 2.5” HDDs
● Two (2) IVY BRIDGE Processors (to 115W)
● Washington pass motherboard (Intel PCSD Mfg) ● Will support Taylor Pass for Haswell (Broadwell)
● Four nVIDIA Keplers or Intel KNCs
Cray CS300-AC™ Cluster Supercomputer Highly Configurable Service & Hybrid Nodes
17
C O M P U T E | S T O R E | A N A L Y Z E
● SR8000 Chassis family supports either: ● 4x double-wide Service Blades
● Compute Blades are 0.5RU 2P x86 servers ● Hybrid Blades are 1RU 2P x86 combined with
either 2x nVIDIA Keplers or 2x Intel KNCs
● Double-wide Service Blades are 2RU 2P x86 servers
● Double-wide Hybrid Blades are 2RU 2P x86 servers with either 4x nVIDIA Keplers or 4x Intel KNCs
● Six hot-swappable, redundant Cooling Fan Units
● Up to six hot-swappable, redundant 1630W PS
● Can support one or two redundant iSCB chassis managers
Cray CS300-AC™ Cluster Supercomputer Reliability & Serviceability – SR8000 Series
18
C O M P U T E | S T O R E | A N A L Y Z E
Cray CS300-LC™ Cluster Supercomputer Overview
Energy-Efficient Architecture • Liquid-Cooled architecture
• Uses warm water instead of chilled water
• Directly cools the processor and memory
• Uses low-pressure isolated secondary loop
• Leak detection and prevention features
integrated with remote monitoring and
reporting
Flexible and Scalable • Up to 60 nodes/rack configured with 2
CPUs/node
• Interconnect: 3D Torus/Fat Tree, single/dual
rail, QDR/FDR, IB/GigE
• Supports a wide range of parallel Storage File
Systems
End-To-End Supercomputer Solutions • From design, integration, delivery to on-site
installation and customized services
Less Energy – Lower TCO – Faster ROI
Liquid-Cooled Solution
C O M P U T E | S T O R E | A N A L Y Z E20
Liquid-to-liquid “Heat Exchangers”
exchange heat between datacenter
liquid loop and server liquid loops.
Datacenter and server liquids are
kept separate and are never mixed
Warm water from datacenter
dry cooler or cooling tower
enters RackCDU and the
heated water returns
Tubes move cooling liquid to and
from RackCDU to blade servers
Pump/cold plate units atop CPUs
(or GPUs) circulate liquid through
server and RackCDU, collecting
heat and returning to RackCDU for
exchange with datacenter liquid
Cray CS300-LC™ Cluster Supercomputer How It Works
C O M P U T E | S T O R E | A N A L Y Z E21
Cray CS300-LC™ Cluster Supercomputer Pump Details
C O M P U T E | S T O R E | A N A L Y Z E
● Two expansion blade models – one
supporting 2x Intel MICs, the other
NVIDIA Keplers
● Remove 80-90% of heat generated
● Two compute blade models – one
based on CPU cooling only, the
other CPU+memory cooling
● Remove 65-75% of heat generated
22
Cray CS300-LC™ Cluster Supercomputer Configurable Compute Nodes
C O M P U T E | S T O R E | A N A L Y Z E23
Cray CS300-LC™ Cluster Supercomputer Highly Configurable RAS Chassis
● SR6000 Chassis supports either: ● 10x Liquid-cooled Compute Blades
● 5x Liquid-cooled Hybrid Blades
● Compute Blades are 0.6RU 2P x86 servers
● Hybrid Blades are 1,2RU 2P x86 combined with either 2x nVIDIA Keplers or 2x Intel KNCs
● Three hot-swappable, redundant Cooling Fan Units
● Up to four hot-swappable, redundant 1630W PS
● Can support one or two redundant iSCB chassis managers
● Enhanced monitoring – leak detection, water pressure, pump failure detection
C O M P U T E | S T O R E | A N A L Y Z E
InfiniBand Network
Compute Nodes
Management Group
Storage Network
Redundant
Management
Nodes
Core Operational
NW ( 10 GbE )
Operational
GigE Network
Login Nodes
External Network
( 10 GbE )
Core Management
Network (1GbE)
1 GbE (Operational)
10 GbE (Operational)
High Speed Interconnect
1 GbE (Management)
NFS Servers
Sub-Management Nodes
(Active/Active pairs)
Operational
GigE Network
Operational
GigE Network
Operational
GigE Network
CN CN
CN CN
CN
CN CN
CN CN
CN
CN CN
CN CN
CN
CN CN
CN CN CN
Global Storage
LSN LSN
MN MN
SMN NFS
SMN SMN SMN SMN SMN SMN SMN SMN
24
Cray CS300™ Cluster Supercomputer System Architecture
C O M P U T E | S T O R E | A N A L Y Z E25
Cray HPC Cluster Software
C O M P U T E | S T O R E | A N A L Y Z E26
Intel® Cluster Studio PGI (PGI CDK) GNU
MVAPICH2 OpenMPI Intel® MPI-
(Cluster Studio) Application Libraries
Development Tools
HPCC IOR netperf Performance Monitoring Perfctr PAPI/IPM
Cray® Compiler Environment (CCE)
Cray® LibSci
HPC Programming Tools
Essential Software and Management Tools Needed to Build a
Powerful, Flexible, and Highly Available Supercomputer.
Linux (Red Hat, CentOS, SUSE) Operating System Operating Systems
Operating Systems
Cray ACE (iSCB and OpenIPMI)
Cray ACE
Cluster Monitoring
Remote Power Mgmt
Remote Console Mgmt Cray ACE
Grid Engine Integrated
Resource Management / Job Scheduling
Altair PBS Pro
NFS File System Lustre Local FS
(ext3, ext4, XFS)
Cray® Advanced Cluster Engine (ACE) management software Provisioning
PanFS
SLURM Integrated
IBM Platform LSF
Torque/Maui
Middleware Applications and Management
MOAB
Cray HPC Cluster Software Stack Flexible SW Architecture for our flexible HW
C O M P U T E | S T O R E | A N A L Y Z E
Programming Environment from Cray
27
The Cray Programming Environment Suite for Clusters is a fully integrated programming environment with compilers, tools, and libraries designed to maximize programmer productivity, application scalability, and performance. It consists of ● The Cray Compiling Environment
● Compilers for Fortran, C and C++ supporting OpenMP 3.1 and OpenACC 2.0
● Fortran 2003 & 2008 compliance ● Significant performance gains over competitive compilers in particular for
well vectorizing applications ● Highest performance without sacrificing accuracy
● The Cray Scientific Libraries
● Autotuned BLAS library, LAPACK, ScaLAPACK ● Highly optimized non-square DGEMM/SGEMM/ZGEMM routines
● Iterative Refinement Toolkit (IRT) ● Cray LibSci_ACC providing accelerated BLAS and LAPACK routines for
GPUs
C O M P U T E | S T O R E | A N A L Y Z E
• Diskless/Stateless Operation
– Simplifies System Administration
– Improves Performance
– Multi-Level Cached Root File System
– Does not require a Light Weight Kernel
– Local Storage also Supported
– State maintained in two HA management servers
• “Instant” Provisioning
– Multiple Logical Clusters
– Multiple OS Configurations
– Provisioning Time = Reboot Time = Less than 10
minutes
• Configuration Management
– Multiple Revisions with Roll-Back Capability
– Rolling Upgrades on Individual servers between jobs
28
Cray HPC Cluster Software Stack ACE Features
C O M P U T E | S T O R E | A N A L Y Z E
Management
Compute
Servers
Storage
Networks
Clusters
ACE-Managed Objects
Root
File
System
Image
Hosts
prod-0001 prod-0002 prod-0003 prod-0004
...
Root
File
System
Image
Hosts test-0001 test-0002
test-0003 test-0004
...
iSCB
29
Cray HPC Cluster Software Stack ACE Managed Objects
C O M P U T E | S T O R E | A N A L Y Z E30
Cray Aries Interconnect
Scale out to 1000s of nodes
C O M P U T E | S T O R E | A N A L Y Z E31
0%
5%
10%
15%
20%
25%
30%
35%
40%
32 64 128 256 512 1024 2048
IB Cost % (Fat Tree)
IB Cost %
Node Count
Inflection Points
Infiniband scale-out cost Infiniband cost as a % of system cost
Compute Nodes
Service Nodes
InfiniBand
Ethernet
Software
Infrastructure
Services
32 Node
Compute Nodes
Service Nodes
InfiniBand
Ethernet
Software
Infrastructure
Services
1024 Node
Introduce
islands to keep
cost scaling flat
C O M P U T E | S T O R E | A N A L Y Z E
Aries connectivity to Compute Nodes
32
C O M P U T E | S T O R E | A N A L Y Z E
Aries Rank-1 network – adaptive routing
o Chassis with 16 compute blades
o Inter-Aries communication over
backplane
o Per-Packet adaptive Routing
o Proactive vs. Reactive
Copyright 2014 Cray Inc. -
Confidential and
Proprietary
C O M P U T E | S T O R E | A N A L Y Z E
16 Aries connected
by backplane
“Green Network”
Cray XC30 Rank-2 Copper Network
4 nodes
connect to a
single Aries
6 backplanes
connected with
copper cables in a 2-
cabinet group:
“Black Network”
Active optical
cables interconnect
groups
“Blue Network”
2 Cabinet
Group
768 Sockets
C O M P U T E | S T O R E | A N A L Y Z E
Configurable Global Bandwidth
● An all-to-all pattern is wired between the groups using optical cables
● Installed in integrated overhead trays
● Site-specific lengths are ordered to minimize excess
● The global bandwidth can be tuned by varying the number of optical cables in the group-to-group connections
C O M P U T E | S T O R E | A N A L Y Z E36
● Sandia miniApp, models performance of CTH
● Running on 2256 node CSCS system (¼ global bandwidth)
● Four concurrent 512 node parallel jobs, different layouts
● Runtime in seconds for 100 cycles
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 4 4 4 4 4 4 4 4 4 4 4 4 4 4 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 4 4 4 4 4 4 4 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 2 4 4 4 4 4 4 1 1 1 1 1 1 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 1 1 1 1 1 1 1 1 1 1 1 1 1
2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
3 1 3 2 1 3 2 3 3 1 1 4 4 1 3 3 4 2 1 2 3 2 2 2 3 2 3 1 4 2 4 3 4 1 3 1 4 2 1 1 4 1 3 2 3 3 2 3 4 4 3 2 1 3 3 3 1 1 2 4
1 4 3 3 3 3 4 2 3 3 4 4 3 4 4 2 3 2 2 3 2 3 1 1 1 1 1 2 2 2 2 2 1 4 1 4 2 4 3 1 2 2 3 2 4 2 3 4 4 3 3 1 4 2 2 3 3 3 1 2 4 3 2 1
3 1 2 1 3 2 3 4 4 4 2 1 3 4 1 1 1 4 1 2 1 4 2 1 3 2 4 1 2 1 2 4 4 3 2 3 1 2 1 1 2 1 1 1 1 4 1 4 3 1 3 2 1 3 2 2 4 1 4 1 4 1 3 4
3 3 4 2 4 4 4 3 3 4 3 4 4 4 3 1 1 2 3 1 4 2 1 2 2 1 2 4 2 1 4 4 2 1 2 1 4 1 4 3 4 1 2 4 2 2 4 4 4 1 4 2 3 1 4 2 3 1 2 2
4 3 2 4 3 4 3 1 2 3 2 4 1 2 4 2 2 3 3 2 4 1 1 3 4 4 2 1 1 2 3 1 2 1 4 4 2 1 2 3 4 3 1 2 1 4 2 2 4 4 1 2 4 4 4 2 1 3 3 3 1 4 3 3
1 1 2 4 1 3 4 4 4 2 2 1 1 1 1 4 4 4 1 4 2 2 3 3 1 1 4 4 2 2 1 1 4 1 1 1 2 2 2 3 1 2 4 1 1 1 3 4 2 2 4 1 3 4 3 4 4 4 4 1 1 4 4 3
3 3 4 2 1 4 2 3 3 1 3 3 4 3 1 1 3 4 1 3 2 2 4 3 2 3 4 1 2 3 2 1 2 1 4 4 3 2 1 3 2 1 4 2 3 4 3 4 2 4 4 4 1 3 2 2 3 2 3 4
4 3 1 2 1 1 4 2 4 3 3 4 3 4 3 3 2 3 4 1 2 4 2 4 4 3 1 2 4 1 1 3 4 3 3 4 4 2 4 4 4 1 4 2 4 4 2 4 4 1 2 3 1 3 2 1 1 3 1 2 3 4 3
1 2 3 2 1 4 3 1 2 2 4 4 4 3 3 3 3 2 2 4 4 2 4 4 1 2 3 2 1 2 4 1 2 1 1 3 2 4 2 2 4 3 1 2 4 3 3 2 4 4 1 1 2 2 1 1 4 4 2 3 2 4 3 1
2 2 2 2 2 3 2 4 3 2 3 4 4 4 2 2 1 1 2 3 2 3 1 1 4 2 1 2 2 1 2 1 2 1 4 3 2 3 3 2 2 1 3 4 1 2 3 3 4 2 2 4 4 3 4 2 3 1 3 1
1 1 1 1 1 1 3 1 3 4 4 4 2 1 1 4 2 1 1 2 4 3 4 1 3 4 2 2 4 1 1 2 2 1 4 4 3 3 2 2 3 4 3 3 1 1 4 2 1 2 2 2 2 3 1 3 1 4 2 1 3 1 2 3
3 3 4 2 4 3 2 1 2 4 3 2 1 3 4 3 4 3 1 3 4 3 1 4 4 2 2 2 1 2 3 1 2 1 3 4 1 4 2 1 2 1 4 2 2 4 2 3 4 2 2 2 4 3 3 1 1 3 1 1 1 3 3 4
4 2 2 1 1 3 3 4 1 2 2 2 2 3 2 2 1 2 1 2 1 2 4 1 3 1 3 3 3 2 3 4 1 2 2 1 1 3 1 4 4 3 1 3 3 1 3 4 3 3 1 4 4 1 3 2 4 1 4 3
3 1 3 1 4 1 1 1 3 2 3 2 3 4 1 4 4 2 2 1 1 2 4 3 4 3 1 3 2 4 4 2 2 4 1 3 2 3 4 4 1 1 1 3 2 1 2 1 1 2 2 1 4 1 1 4 1 2 1 4 2 1 3 3
3 1 4 2 3 1 4 3 2 3 4 1 4 2 4 3 3 4 4 4 4 2 4 2 3 4 3 4 4 4 1 2 2 1 2 1 3 2 3 1 3 2 4 1 2 4 3 1 1 4 3 1 1 3 1 1 3 1 1 1 3 1 3 4
4 3 3 2 2 1 3 3 3 3 4 2 4 2 3 3 3 4 4 2 3 2 4 2 4 2 1 1 1 4 3 4 4 1 1 3 3 4 2 4 2 2 4 2 2 4 1 2 2 2 2 1 1 2 1 4 2 3 3 1
4 2 2 3 2 4 1 2 4 2 2 4 4 2 1 4 3 2 3 2 4 3 2 3 4 3 4 3 1 4 4 1 1 1 4 4 2 4 1 1 3 4 1 4 2 3 3 3 1 4 2 1 4 2 3 1 4 2 2 4 2 4 3 1
3 3 1 4 1 4 1 1 4 4 1 3 3 3 1 1 2 2 2 1 3 4 2 1 1 1 1 3 4 4 1 1 4 3 2 1 2 1 4 2 1 4 1 2 3 4 4 4 3 1 4 2 3 1 3 4 2 2 2 2 3 3 2 1
2 1 1 2 1 2 3 4 4 2 4 4 2 3 3 2 3 3 2 3 2 4 1 2 4 4 1 4 4 2 1 1 2 1 3 4 3 4 1 3 2 4 4 1 1 3 3 1 4 2 1 4 3 2 4 2 3 3 2 2
2 1 2 4 4 4 1 1 3 3 3 3 3 1 4 2 3 1 3 2 4 1 2 2 3 3 1 2 3 3 2 2 1 3 2 4 4 4 1 1 1 3 4 3 3 2 4 2 4 1 4 1 4 4 3 4 4 2 4 1 3 4 3 4
1 1 1 4 1 3 4 3 1 3 1 1 1 1 4 2 1 3 1 2 4 4 2 3 1 4 4 2 1 3 2 2 3 3 2 3 4 4 3 3 4 4 1 3 3 3 3 1 1 2 3 1 3 3 1 2 2 4 2 3 3 2 4 3
2 1 2 2 1 3 4 2 1 1 1 3 3 1 3 3 4 2 2 2 2 3 2 1 4 4 1 2 2 3 3 4 2 2 4 1 4 4 1 3 4 3 1 3 1 2 3 1 2 1 4 4 2 2 3 1 2 2 2 4
2 1 4 3 4 2 4 1 4 3 1 2 1 2 4 4 3 2 3 2 4 1 4 1 4 2 2 1 4 2 4 1 2 4 4 4 3 3 4 4 3 1 3 4 1 1 2 2 3 1 3 3 1 3 4 3 3 2 1 2 2 1 4 2
3 4 1 2 3 3 1 2 3 1 1 4 3 1 4 1 2 3 3 3 1 2 1 4 2 2 2 3 1 3 1 1 1 2 1 1 4 2 2 1 4 2 1 3 4 2 2 1 4 1 3 2 1 3 2 2 4 3 4 4 4 4 4 2
4 4 2 1 3 2 1 4 4 2 1 4 2 3 3 2 1 2 3 1 4 2 3 2 2 3 2 2 3 3 4 3 4 4 2 4 3 1 2 3 2 4 4 4 3 1 4 4 1 1 3 3 2 2 1 4 4 4 2 4
3 1 2 3 4 1 4 3 4 3 2 1 1 1 2 2 2 4 4 3 3 4 3 1 3 4 4 4 1 1 3 3 3 1 1 1 1 3 3 4 3 2 4 4 2 3 2 2 4 4 3 2 4 2 4 2 4 4 3 2 3 2 1 2
3 1 2 3 3 2 1 2 3 4 1 2 4 4 2 1 4 4 3 3 4 4 4 3 4 1 3 2 4 3 3 1 3 3 1 3 2 2 1 2 4 1 1 4 1 2 1 2 2 4 1 4 2 4 4 2 3 4 3 4 1 4 3 1
4 3 1 3 1 4 2 3 2 3 3 2 3 3 3 2 2 1 4 2 4 3 4 2 1 4 4 3 1 2 4 2 1 2 2 1 2 2 2 1 2 4 3 1 3 2 4 4 1 3 1 1 4 2 1 4 3 3 1 1
2 3 2 2 2 4 1 2 3 2 4 1 3 4 2 3 1 4 4 4 1 2 3 1 3 4 1 3 4 3 3 4 2 1 1 3 3 4 2 2 1 3 4 3 1 4 4 4 1 3 3 3 1 1 4 2 4 1 1 4 1 4 2 2
2 3 1 1 1 4 4 1 3 1 2 1 4 4 4 2 2 4 1 4 2 1 4 4 1 1 2 1 4 3 4 1 2 1 2 3 3 1 4 3 3 4 2 1 1 4 3 4 2 4 2 4 2 2 4 3 4 1 2 1 3 4 4 2
2 4 4 1 2 2 1 4 3 1 3 3 3 3 4 1 3 2 2 2 4 3 3 3 2 3 2 1 1 3 4 1 1 2 3 4 4 1 4 3 4 2 3 2 3 2 4 3 1 2 1 2 4 1 2 3 3 1 3 4
3 3 4 2 4 4 3 3 3 3 4 2 1 1 1 2 3 4 2 4 2 4 3 1 3 4 1 1 3 2 3 3 4 1 1 1 3 3 4 2 2 3 2 3 4 1 1 4 1 3 3 2 4 4 4 2 3 3 3 3 1 2 4 1
4 4 3 2 2 2 3 2 2 2 1 4 3 2 4 3 4 4 2 3 4 3 4 2 3 3 1 3 2 1 1 2 3 2 1 4 4 1 2 3 2 4 2 3 3 4 4 4 1 2 4 4 2 3 4 4 4 3 2 3 2 3 1 3
2 4 4 1 3 4 2 3 2 4 4 1 4 3 2 2 2 3 1 1 2 2 2 4 4 4 2 4 1 4 2 3 4 1 3 3 1 4 3 4 2 3 4 2 2 4 2 1 1 4 3 2 4 3 3 3 2 2 3 2
3 1 2 2 1 3 3 4 3 3 2 2 3 1 2 2 4 2 4 2 2 4 1 3 3 2 3 4 2 1 2 1 3 1 3 1 1 3 3 2 4 3 1 4 4 1 1 1 4 4 3 1 1 3 3 2 1 2 1 2 2 3 1 2
1 1 3 2 2 3 3 3 1 2 1 2 2 1 2 3 1 3 3 3 1 2 3 3 1 1 3 3 3 1 3 1 3 3 3 1 3 3 1 3 3 1 3 1 1 1 3 3 1 1 3 1 1 3 1 1 1 1 1 1 1 1 1 1
Blocks of 512 nodes Blocks of 64 nodes Average of 32 nodes Random nodes
69.0 68.8 68.9 68.9 69.4 69.4 69.4 69.5 69.7 69.7 69.4 69.7 70.9 71.0 70.6 70.5
Application Explorations for Future Interconnects, R.F. Barrett, C.T. Vaughan, S.D. Hammond, and
D.Roweth, Workshop on Large-Scale Parallel Processing, at the International Parallel and Distributed
Processing Symposium: IPDPS 2013, Boston, MA, 2013.
Performance independent of placement
C O M P U T E | S T O R E | A N A L Y Z E
Real world-performance: TAU code single-grid
37
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
240 480 960 1920 3840 7680 15360
FDR efficiency (CASE-2)
Aries efficiency
F6-DPWII benchmark single-grid parallel scaling efficiency
normalized on 240 core performance
CASE-2: Intel XEON E5-2695v2 (12-core, 2.4 GHzwith turbo), Mellanox Connect-X3 FDR
XC30:Intel XEON E5-2697v2 (12-core, 2.5 GHz w/o turbo, Aries interconnect
C O M P U T E | S T O R E | A N A L Y Z E
Real world-performance: TAU code multi-grid
38
F6-DPWII benchmark 4w multi-grid parallel scaling efficiency
normalized on 240 core performance
CASE-2: Intel XEON E5-2695v2 (12-core, 2.4 GHz with turbo), Mellanox Connect-X3 FDR
XC30:Intel XEON E5-2697v2 (12-core, 2.5 GHz w/o turbo, Aries interconnect
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
240 480 960 1920 3840 7680 15360
FDR efficiency (CASE-2)
Aries efficiency
C O M P U T E | S T O R E | A N A L Y Z E
STAR-CCM+ Scaling: Cray XC30 compared to FDR InfiniBand Cluster
39
cores Cray XC30 FDR IB Cluster IB/XC30 32 34.08 33.34 0.98
64 16.53 17.11 1.04
128 7.75 8.99 1.19
256 3.98 4.85 1.22
512 2.11 2.72 1.29
1024 1.11 1.53 1.38
2048 0.59 0.96 1.62
3072 0.49 NA*
Automotive External Aerodynamics: 100M cells
STAR-CCM+ version: 7.04.011
Average Time per Iteration [s]
Intel “Sandy Bridge” processors on both systems, 2.6 GHz
XC30 results obtained in cluster ompatibility mode
* Only 2048 cores available in the CCS cluster
C O M P U T E | S T O R E | A N A L Y Z E
LS-DYNA Scaling: Cray XC30 compared to InfiniBand Cluster
40
cores Cray XC30 FDR IB Cluster IB/XC30 32 41,474
64 20,652 22,499 1.09
128 11,278 12,427 1.10
256 6,849 7,781 1.14
432 4,781 5,729 1.20
512 4,054
Automotive “crash”: 2.4M elements LS-DYNA, mpp971
Elapsed time (sec)
Intel “Sandy Bridge” processors on both systems, 2.6 GHz
Native LS-Dyna binary used on XC30
* Only 464 cores available in the CCS cluster
C O M P U T E | S T O R E | A N A L Y Z E
Legal Disclaimer
Information in this document is provided in connection with Cray Inc. products. No license, express or implied, to any intellectual property rights is granted by this document.
Cray Inc. may make changes to specifications and product descriptions at any time, without notice.
All products, dates and figures specified are preliminary based on current expectations, and are subject to change without notice.
Cray hardware and software products may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request.
Cray uses codenames internally to identify products that are in development and not yet publically announced for release. Customers and other third parties are not authorized by Cray Inc. to use codenames in advertising, promotion or marketing and any use of Cray Inc. internal codenames is at the sole risk of the user.
Performance tests and ratings are measured using specific systems and/or components and reflect the approximate performance of Cray Inc. products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance.
The following are trademarks of Cray Inc. and are registered in the United States and other countries: CRAY and design, SONEXION, URIKA, and YARCDATA. The following are trademarks of Cray Inc.: ACE, APPRENTICE2, CHAPEL, CLUSTER CONNECT, CRAYPAT, CRAYPORT, ECOPHLEX, LIBSCI, NODEKARE, THREADSTORM. The following system family marks, and associated model number marks, are trademarks of Cray Inc.: CS, CX, XC, XE, XK, XMT, and XT. The registered trademark LINUX is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a worldwide basis. Other trademarks used in this document are the property of their respective owners.
Copyright 2014 Cray Inc.
41
C O M P U T E | S T O R E | A N A L Y Z E42
Special thanks to
Thomas Alrutz & Christian Simmendinger
T-Systems – Solution for Research
For providing TAU benchmark results on CASE-2 system, native XC30 binaries and input data sets
C O M P U T E | S T O R E | A N A L Y Z E43
Thank you !
Questions ?
Related Documents
