Big Data Platform in High Performance Computing at NISCI Dr. Tran Minh Deputy Director of NISCI [email protected]MINISTRY OF INFORMATION AND COMMUNICATIONS (MIC) VIETNAM NATIONAL INSTITUTE OF SOFTWARE AND DIGITAL CONTENT INDUSTRY (NISCI) Workshop on “Big Data Analasys on Cloud and High Performance Computing Platform” Hanoi – 24 th July, 2014
20
Embed
Tran Minh: big data platform in high performance computing at NISCI
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
2 QLogic IB QDR 36 Port Switch. Switch 3800 for Storage connection Switch 2920 và 2620
HPC System Hardware
Big Data Hardware Infrastructure + Servers + Storage + Networks + HPC and Super Computers
Workshop on “Big Data Analasys on Cloud and High Performance Computing Platform” 5/18
Xeon Phi (Xeon Phi 5110p)
Multi core: # of Cores: 16 Clock Speed:1.053 GHz L2 Cache: 30MB Instruction Set:64-bit Max memory size : 8GB # of Memory Channels:16 Max Memory Bandwidth:320GB/s
InfiniBand
All ports on 4X QDR HCA card and switch are capable of supporting 40 Gbps signaling rate
Storage
50 Tb
HPC hardware
Proccessor
(1)
N of
proccessor
(2)
N of core
(3)
Speed Ghz
(4)
N of
operations/s
(5)
GFlops
(6) = 2*3*4*5
Xeon E5-2670 96 8 2.6 8 15.974
Xeon Phi 5110p 48 16 1.053 16 48.522
Total 64.496
Workshop on “Big Data Analasys on Cloud and High Performance Computing Platform” 6/18
Cluster LAN 1GbE
iLO LAN
IB QDR Network
Storage LAN 1GbE
2 x Master Node
StoreAll 40 TB
24 x Compute Node(s) –
GPU
External Users
Cluster System Admin
Public Network
HP 642 1200mm Rack
HP X9320 43TB
6 x SL6500 chassis Customer existing switch
HP 2920 48 port switch
HP 2620 48 port switch
HP 3800-24G-8XG Switch
Qlogic QDR IB switch
2 x DL380p
24x Compute Node(s)
6 x SL6500 chassis
HPC hardware architecture
Workshop on “Big Data Analasys on Cloud and High Performance Computing Platform” 7/18
HPC system in NISCI
Software and data processing library
Data source for Big Data
Development environment in HPC
Some proposal
Contents
Workshop on “Big Data Analasys on Cloud and High Performance Computing Platform” 8/18
Intel parallel Studio XE Intel® C, C++ and Fortran Compilers – Industry-Leading Compilers Intel® MKL and Intel® IPP – Performance Libraries Intel® Threading Building Blocks and Intel® Cilk™ Plus – Parallel Programming Models Intel® Advisor XE – Threading Assistant Intel® VTune™ Amplifier XE – Performance & Thread Profiler Intel® Inspector XE – Memory and Thread Checker Static Analysis – Locate Difficult to find Defects
Software development tools
HP Open MPI
Workshop on “Big Data Analasys on Cloud and High Performance Computing Platform” 9/18
Workshop on “Big Data Analasys on Cloud and High Performance Computing Platform” 10/18
HPC software platform
Big Data Software Platforms + SQL / Data Warehouse + NoSQL / Big Data Storage + Digital Content Repositories + Public and Private Clouds + Hadoop Distributed File System (HDFS) + Map / Reduce + Pig / Hive Query ...
Workshop on “Big Data Analasys on Cloud and High Performance Computing Platform” 11/18
HPC system in NISCI
Software and data processing library
Some data source for Big Data
Development environment in HPC
Some proposal
Contents
Workshop on “Big Data Analasys on Cloud and High Performance Computing Platform” 12/18
Workshop on “Big Data Analasys on Cloud and High Performance Computing Platform” 13/18
Sensor Network and Big Data
1. Sensor Network - Danang - Cantho - Hochiminh city 2. Document image processing (800 Milion pages/year) 3. Network administration data (Danang) 4. Satelite Image (VN RED-SAT, Ministry of Public
Security)
Workshop on “Big Data Analasys on Cloud and High Performance Computing Platform” 14/18
Sensor’s Network Big Data
Workshop on “Big Data Analasys on Cloud and High Performance Computing Platform” 15/18
- Monitor situation of wave, ships. - Monitor level of river - Monitor swirling water and dangerous area. - Auto send sms/email of warning water level to leaders. - Sharing data among gov. offices
Workshop on “Big Data Analasys on Cloud and High Performance Computing Platform” 16/18
Workshop on “Big Data Analasys on Cloud and High Performance Computing Platform” 17/18
Đà Nẵng, Cần Thơ,
Workshop on “Big Data Analasys on Cloud and High Performance Computing Platform” 18/18
Hospital
Station
Office
Show Room Store House
Public Home
Hotel
Digital Contents and Big Data
Workshop on “Big Data Analasys on Cloud and High Performance Computing Platform” 19/18
- Banks - Insurance - Securities - VNPT - Viettel
When Vietnam ?
Workshop on “Big Data Analasys on Cloud and High Performance Computing Platform” 20/18
Thanks for your attentions! Vietnam National Institute of Software and Digital Content Industry (NISCI)
The Authority of Radio and Frequency Management Building 8 floor, 115 Tran Duy Hung, Cau Giay District, Ha Noi.