© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HP Big DataReference Architecture
Philipp Koik, Strategic Presales
Jochen Mohr, Technology Services
5. Mai, 2015
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Time
Vo
lum
e o
f d
ata
Data
Technologygap
Human data
Machine data
Business data
Big DatashiftMobile apps
System logs
Data centers
Compliance archives
Internet of Things
Sensors
Social networking
Photo sharing
Wearable devices
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
IT infrastructures trendsHow they influence and are influenced by Big Data & Analytics transformation
Big Data
Server Shift from high-end to medium-/low-
end servers Massive Parallel Processing (MPP),
Scale-out computing architecture
Network Dramatic increase in fabric speeds and
bandwidth demand Access from anywhere at anytime LAN, WAN, Internet access SDN
Storage Shift from high-end centralized
storage to local DAS Shared-nothing Scale-out storage architecture
Software/Application Clustering NoSQL, NewSQL , Columnar DBs BI & BA working with structured &
unstructured data RT-Analytics Data life-cycle Management
Security Configuration Monitoring Management BuRA
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
IT infrastructures must evolve to handle Big Data demands
• Multiple silos with multiple copies of the same data
• Difficult to standardize on a consistent server architecture
• Less elastic than other virtualized or converged infrastructure
• Large scale makes density, cost and power problematic
Challenges
NoSQL Hadoop
Analytics
Data center
ServersStorage
NetworkingPower & Cooling
Management
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Reference Architecture
ConvergedSystem
HP Big Data – Hardware Solution PortfolioMarket-driven offerings and services
HP ConvergedSystem for Big Data
DL180 DL380 SL4540 HP Moonshot
HP Apollo
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7
HP ConvergedSystem 300 for Microsoft Analytics Platform
The only appliance with integrated in-memory performance, MPP DW and Hadoop
50%Lower cost per TB2
100XFaster query speed1
30%Better scan rate1
1 Than previous generations2 Than competitive offerings
• Next-generation data warehouse for mission-critical environments
• Factory built, appliance-based on HP Converged Infrastructure
• Pre-loaded with Microsoft software, integrated, tested, and tuned
• Architecture chosen for best data warehouse performance
• Single view of information across the enterprise
• New addition to the Converged Systems “Sharks” family
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
New approach to address Big Data demands
Current traditional Big Data approach
• Compute and storage are always collocated• All servers are identical• Data is partitioned across servers on direct-attached
storage (DAS)
New HP Big Data approach
• Separate compute and storage tiers connected by Ethernet networking
• Standard Hadoop installed asymmetrically with storage components on the storage servers and yarn applications on the compute servers
Two Socket, 2U Servers
YARN Applications, HDFS, ORC Files, Parquet, Hbase,
Cassandra
Compute Optimized Servers
Storage Optimized Servers
YARN Applications
HDFS, ORC Files, Parquet, Hbase,
Cassandra
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Benefits of HP Big Data Reference ArchitectureHP Moonshot and SL4540 addresses a variety of enterprise big data needs
Ethernet (RoCE)
Cluster consolidationMultiple big data environments can directly access a shared pool of data
Flexibility to scaleScale compute and storage independently
Maximum elasticityRapidly provision compute without affecting storage
Breakthrough economicsSignificantly better density, cost and power through workload optimized components
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Building blocks for the HP Big Data Reference Architecture
HP Moonshot System
HP ProLiant SL4540Scalable System
A complete server system engineered for specific workloads and delivered in a dense, energy-efficient package
A cost-effective industry standard storage server purpose built for big data with converged infrastructure that offers high density energy-efficient storage
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HOT COLD
Independent Scaling of Compute and Storage
HP Big Data Reference ArchitectureTraditional Architecture
4x compute60% of the storage capacity
72% of the Hadoop IO
1.7x compute1.5x the storage capacity
2.1x the Hadoop IO
60% of the compute2x the storage capacity
2.9x the Hadoop IO
Compared with traditional architecture, full rack
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Maximum Elasticity for Big Data workloadsHadoop Labels feature (jira YARN-796)
• HP contributed IP into the Hadoop trunk, working with Hortonworks
• Specifying labels on nodes allows for scheduling of YARN containers to specific pools of nodes
• Admins able to target workloads at optimized platforms
• Combined with the HP Big Data Reference Architecture, compute nodes can be dynamically assigned
• No data repartitioning
Hadoop Cluster 1 Vertica Analytics Spark
12am – 6am
6am – 12am
Hadoop Cluster 2
Hadoop Cluster 1 Hadoop Cluster 2
Storage Node Storage Node
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
No
de
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Evolve to support multiple compute and storage blocks
Minotaur CI for Big Data long term view
Low Cost Nodes
SSD Nodes Disk Nodes Archive Nodes
Multi-temperate Storage using HDFS Tiering, NoSQLs and Objectstores
GPU Nodes FPGA Nodes Big Memory Nodes
Workload Optimized compute nodes to accelerate various big data software
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HP Moonshot System + SL4540 for Big Data
14
HP Big Data Reference Architecture
Hardware
Hadoop Distribution
Operating System
Hortonworks Data Platform
Linux
Cloudera Enterprise 5
HP ProLiant m710 Server Cartridge HP ProLiant SL4540 Scalable System
Intel® Xeon® E3 Processor
480GB storage
32GB Memory
Dual port 10GbE
Co
mp
ute W
orker N
od
e Sto
rag
e W
ork
er N
od
e Intel® Xeon® E5 Processor
1PB storage
192GB Memory
Dual port 10GbE
Built on Standard Hadoop DistributionsNo proprietary software. Leverage the latest versions of Hadoop and consumer plug-ins
Optional HP Consulting ServicesExpedite the sizing and configuration of the infrastructure through the Hadoop Reference Architecture implementation service
Optional Factory BuildHardware racked, wired and tested, delivered to your data center
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Advise Transform Manage
HP BD Service – An exhaustive Big Data IT portfolio IT Consulting Services for Big Data Service Delivery Transformation
Big Data IT strategy and architecture services• Big Data Strategy Workshop
• Big Data Infrastructure Transformation Experience Workshop
• HP Enterprise Planning for HAVEn
• Big Data Protection and Compliance Analysis
• HP Vertica Deployment Roadmap
Big Data system infrastructure • HP professional services for HAVEn
solutions implementation
• Enterprise Design Service for Hadoop
• Reference Architecture Implementation Service for Hadoop, Microsoft PDW and SAP HANA
• HP Vertica Implementation Accelerator
Big Data protection• Data Loss Prevention
Big Data operationAchieve best-in-class operational efficiency of a client’s big data environment leveraging the unique knowledge of HP experts and our global infrastructure
Big Data educationTrain and certify a client’s IT staff and third-party partners to help them architect, integrate, and administer Big Data solutions. Assist Management of Change
Enabling provisioning of Big Data Services to your customers
Helping accelerate adoption and integration of Big Data technologies
Supporting IT transformation
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Big Data Advisory Services Positioning
Vertica deployment roadmap
Big Data Strategy Workshop
Big Data Infrastructure Transformation Workshop
De
plo
ym
en
t R
oa
dm
ap
Ma
turi
ty
Strategy & Planning Maturity
• Value of new technology to support business strategy
• Elements of Use Case(s)• Technology Impact • Initiative and Roadmap
• Overview of complete experience and impact of New Technologies introduction
• Initial Initiative Definition and Roadmap
Enterprise Planning for HAVEn
(Big Data Platform Architecture)
• Strategic Architecture Standards, Principles, Models & Measures
• Complete Technology Roadmap
Big Data Protection and Compliance Analysis
• Gap assessment, remediation plan, risk analysis and roadmap to improve readiness posture in protecting Big Data
• Elements of Business and Use Case analysis• Technology impact of Vertica• Initial Initiative & Arch. Definition and Roadmap
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
IT infrastructure modernization & consolidation
DWH
Modernisation
Big Data
On Demand Security
Analytics
Operations
Analytics
Data Center
Modernisation
Cloud
Data
Security
Big Data
Management
Rationales: Standardization, Costs, Control, Monitoring, Elasticity, Data Security, ….
Challenges: In-House-Dev, Performance, Security, Network Integration, Volume, Management Tools, Non-Standard HW, Risks, Interfaces, Backup- and Restore, ….
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Big Data Technology Consulting solution value
We provide leadership to IT to help achieve business objectives.
We minimize implementation and integration risk, improving time-to-value speed.
We facilitate IT to ramp up on skills to manage transition and transformation.
We will accelerate adoption and integration of Big Data technologies.
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.19
Rekordjäger Rainer Zietlow
THANK YOU
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
BACK-UP
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HP Moonshot 1500 Chassis front and rear view
45 hot-plug cartridges
Processor• x86 , ARM, or Accelerator
• Single-server = 45 servers per chassis• Quad-server =180 servers per chassis
Dual low-latency switches• HP Moonshot-45G Switch Module
(45 x1Gb downlinks)• HP Moonshot-180G Switch Module
(180 x1Gb downlinks)• Moonshot-45XGc Switch Module• (45 x10Gb downlinks)
Dual Network Uplinks• HP Moonshot-6SFP Uplink Module
(6 x10Gb Stackable Uplinks)• HP Moonshot-4 QSFP Uplink
Module (4 x 40Gb Stackable Uplinks)
5 hot-plug fan modules
HP Common-Slot Power Supplies
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HP ProLiant m710 Server Cartridge
CPU Intel Xeon E3-1284Lv3 with Iris Pro P5200 GPU
4 core / 1.8 GHz (3.4Ghz Turbo) / GPU + 128MB eDDR
MEMORY Total of 32GB of ECC protected memory, dual-memory channels(4) 8GB LV SO-DIMMs at 1600MHz with (8) embedded DRAM for ECCprotection.
NETWORK Integrated NIC: dual port 10GbE Mellanox CX3 PROSupported Switch(s): 45 port 10Gb Downlinks, (4) 40GbE QSFP uplinks
STORAGE Local SSD boot, 480GB m.2 (2280)
POWER Cartridge: <69W
OS Ubuntu 14.04 w/KVM, RHEL 6.5,7.0 w/KVM, SLES 11 SP3 w/KISO/KVM, Windows Server 2012 R2, CentOS 6.5, 7.0
Intel Media SDK (media libraries, OpenCL in beta), purchased direct from Intel
Big Data Compute Node
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HP ProLiant SL4500 Scalable System
CPU Up to two Intel® Xeon® E5-2400 or E5-2400v2 (4, 6, or 8 core) per node
MEMORY 12DIMMs – Up to 32GB’s, 384GB’s max1333/1066 MHz DDR3 RDIMM
NETWORK HP Ethernet 1Gb 2-port 361i OR HP Ethernet 1Gb 2-port 361i and HP Ethernet 10Gb 2P 544i Adapter; One 10GbE SFP+ connector; One 10GbE/40GbIB QSFP connector (option to be converted to Infiniband)
STORAGE 25x 3.5” SAS, SATA, or SATA SSD (hot-plug)2x 2.5”SATA for boot
POWER 4x 750W or 1200W hot plug, redundant optional, Platinum Plus supplies
OS MS Windows Server 2008 SP2, R2 w/ SP1 (standard, enterprise, datacenter, web server, HPC, embedded), Hyper-V R2 SP1. (64bit only)MS Windows Server 2012 (standard, datacenter, hyper-v, HPC pack)MS Windows Server 2012 R2 (standard, datacenter, hyper-V
RHEL 5.8, 5.9, 5.10 (64bit)RHEL 6.2, 6.3, 6.4 (64bit)SLES11SP2, SP3 (64bit)Ubuntu 12.04.03 LTSVMware ESXi MN 5.0 U3, 5.1 U2, 5.5
Big Data Storage Node
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Advantages* of HP Big Data Reference ArchitectureA new standard for Big Data delivery at scale
42
41
40
39
38
37
36
35
34
33
32
31
30
29
28
27
26
25
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
42
41
40
39
38
37
36
35
34
33
32
31
30
29
28
27
26
25
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
Green=10Gbps, Yellow=1Gbps SFP+
SYS
Management ConsoleACTLINK
Green=10Gbps, Yellow=1Gbps SFP+
21 43 65 87 109 1211 24232221201918171615141310/100/1000Base-T
HP 5920Series SwitchJG296A
Green=10Gbps, Yellow=1Gbps SFP+
SYS
Management ConsoleACTLINK
Green=10Gbps, Yellow=1Gbps SFP+
21 43 65 87 109 1211 24232221201918171615141310/100/1000Base-T
HP 5920Series SwitchJG296A
ProLiant
DL360p
Gen8
UIDSID
3
4
1
2
5
6 7 8
seria
l ata
5.4
k
60
GB
seria
l ata
5.4
k
60
GB
seria
l ata
5.4
k
60
GB
seria
l ata
5.4
k
60
GB
UID
28
30
29
31
33
21
34
36
35
37
39
38
40
42
41
43
45
44
1
3
2
4
6
5
7
9
8
10
12
11
13
15
14
16
18
17
19
21
20
22
24
23
25
27
26BA
Moonshot1500
UID
UID UID
19 257 13120 268 14221 279 15322 2810 16423 2911 17524 3012 186
19 257 13120 268 14221 279 15322 2810 16423 2911 17524 3012 186
ProLiant
SL4540
Gen8
UID
UID UID
19 257 13120 268 14221 279 15322 2810 16423 2911 17524 3012 186
19 257 13120 268 14221 279 15322 2810 16423 2911 17524 3012 186
ProLiant
SL4540
Gen8
ProLiant
SL4540
Gen8
UIDUID
UID
16 216 11117 227 12218 238 13319 249 14420 2510 155
UID
16 216 11117 227 12218 238 13319 249 14420 2510 155
42
41
40
39
38
37
36
35
34
33
32
31
30
29
28
27
26
25
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
42
41
40
39
38
37
36
35
34
33
32
31
30
29
28
27
26
25
24
23
22
21
20
19
18
17
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
Green=10Gbps, Yellow=1Gbps SFP+
SYS
Management ConsoleACTLINK
Green=10Gbps, Yellow=1Gbps SFP+
21 43 65 87 109 1211 24232221201918171615141310/100/1000Base-T
HP 5920Series SwitchJG296A
Green=10Gbps, Yellow=1Gbps SFP+
SYS
Management ConsoleACTLINK
Green=10Gbps, Yellow=1Gbps SFP+
21 43 65 87 109 1211 24232221201918171615141310/100/1000Base-T
HP 5920Series SwitchJG296A
ProLiant
DL360p
Gen8
UIDSID
3
4
1
2
5
6 7 8
seria
l ata
5.4
k
60
GB
seria
l ata
5.4
k
60
GB
seria
l ata
5.4
k
60
GB
seria
l ata
5.4
k
60
GB
UID
28
30
29
31
33
21
34
36
35
37
39
38
40
42
41
43
45
44
1
3
2
4
6
5
7
9
8
10
12
11
13
15
14
16
18
17
19
21
20
22
24
23
25
27
26BA
Moonshot1500
UID
UID UID
19 257 13120 268 14221 279 15322 2810 16423 2911 17524 3012 186
19 257 13120 268 14221 279 15322 2810 16423 2911 17524 3012 186
ProLiant
SL4540
Gen8
UID
UID UID
19 257 13120 268 14221 279 15322 2810 16423 2911 17524 3012 186
19 257 13120 268 14221 279 15322 2810 16423 2911 17524 3012 186
ProLiant
SL4540
Gen8
ProLiant
SL4540
Gen8
UIDUID
UID
16 216 11117 227 12218 238 13319 249 14420 2510 155
UID
16 216 11117 227 12218 238 13319 249 14420 2510 155
Traditional Architecture
Traditional Hadoop Architecture
HP Big Data
Reference Architecture
CPU Performance(SpecINT)
22% better
Data Capacity(TB)
Equivalent
Memory(GB)
20% greater
Power(W)
25% less
Rack Space 60% less
TCO/Performance
($ per MB/s)15% better
HP Big Data Reference Architecture
* Normalized on capacity
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.26
An exhaustive Big Data IT portfolio
HP TS Consulting Big Data Service Portfolio
Manage Advise
Transform
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.27
Benefits & Gains: Value customers will get• A unified transformation
reference model aligned to business
needs
• A powerful and structured tool
to present the initiative and gain stakeholders’ buy-in
Start-up
Roadmap
Scope and boundaries
Common visionand leadership
CustomersBig Data initiative
• Operational scope definition (AsIs vs. ToBe)
• Limits, challenges, key success factors
• Time bounded methodology helps focusing on key facts
• Their target vision
• Gaps in their ability to transform
• Initiatives unique to your customer’s requirements
• Actions that have been identified and captured during workshop(s)
• Recommended next steps based on HP’s experience and offering
• Pragmatic initiatives to reduce risk/cost etc.
• Additional considerations to improve efficiencies
• Correlation of business drivers and company vision, use cases and issues
• Quick wins to boost successful start up
• Common, shared strategies
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.28
Start your journey with a Transformation visionIdentify your path, your use case
Do you need to stabilize and secure your existing environment?
Do you need to transform your network?
Do you need to modernized your data center?
Do you need to embrace cloud?
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank you
Related Documents
