Bare Metal Provisioning for Big Data Vol.01 2016/12/1 Woosuk, Choi EC Core Technology Department http://www.rakuten.co.jp/
Bare Metal Provisioning for Big Data Vol.01 2016/12/1Woosuk, ChoiEC Core Technology Department http://www.rakuten.co.jp/
2
At first, Simple Comparison Virtualization VS Bare Metal
3
What’s the best one for Big Data System ? • We’re used to need enough resource every time in order to
processing huge data in the Rakuten • In terms of performance/cost, winner is Bare Metal
Bare Metal Virtualization ( Cloud )
InfraManagement Difficult Easy
Performance Best Effort Bottleneck Storage/Network ..
Solutions Many legacy way .. AWS , OpenStack ..
Virtualization VS Bare Metal
4
H/W Selection Approach
Low Price
Micro Server
Local Storage
A Large number of servers
DAS
High Performance Storage
5
Quanta S910-X31E
Micro Server
FEATURE HIGHLIGHTS• Chassis
• Nodes : 9 nodes • Network - Inner Switch
• 1Set, No redundancy • 10Gbps x 2 uplink• 1Gbps x 24 internal
• Power • 2 redundant PSUs
• Per Node• Processor
• Xeon E3-1200 v3 (4C/1S/8T)• Memory
• DDR3 16GB/32GB
• Network • 1Gbps x 2 via Inner Switch
• Storage - No Raid • Controller : SATAIII• SSD : 2.5” 120GB/480GB• HDD : 2.5” 1TB
6
Quanta 210-X12RS/D51B-1U
1U type Server
FEATURE HIGHLIGHTS• Processor
• Xeon E5-2620/2640 v3 • 8C/1S/32T
• Memory • DDR3/4 128GB
• Network • 10Gbps x 1 , No redundancy
• Power • 1 PSU , No redundancy
• Storage • Raid Controller : MegaRAID • SSD : 2.5” 480/600GB x 4 • HDD : 2.5” 1TB x 2
7
Quanta 210-X22RQ/D51B-2U
2U type Server for Hadoop
FEATURE HIGHLIGHTS• Processor
• Xeon E5-2620/2640 v3• 8C/2S/32T
• Memory • DDR3/4 128GB/256GB
• Network • 10Gbps x 1 , No redundancy
• PSU• 1 PSU , No redundancy
• Storage • Controller : SATAIII ( no RAID )• SSD : 2.5” 120GB x 2• HDD : 3.5” 4/6/8TB x 12
8
Bare Metal, But is it really good for everything ? So, we can make Bare Metal management system
which is most likely to cloud solution
9
2014
2014 3Q2015 2Q
2015 4Q2016
Bare Metal System Project for Big Data
1st Revision Launch
2nd Phase Start
2nd Revision Launch
HistoryFor Next ….• More Global• Not only Big Data• Expand for others
• Scratch• By Admin• GUI/No API
• More Open Source• By User• Resource Management• GUI/API
10
US Data Center200 servers
Boston Office
EU Data Center300 Servers
Paris Office
JP Data Center5000 Physical Servers
India Operation Center24x7 Trouble Tier1 Support
US Data Center500 servers
Tokyo Office
Where are we ?
11
• Racking/Cabling• HW Check
• Stress Test• BIOS Configuration• Error on BMC
Acceptance
• Registration to OS Provisioning System
• Test / Building OS image for each type
• Base Chef recipes
Ready to Provisioning • Provisioning at once
• OS Installation• Application
Deployment• Set up monitoring
ProvisioningConnected System
Server Assign
Physical Server Delivery Flow
Bare Metal Management Facility
Infra Administrator ApplicationPlatform Admin
12
Rack = SW NAME : XXXX
3
3
1
1
Racking Rule – for Infra Hostname
Infra Hostname Rule
Region SW Host = Rack Name
Port Num = Position
Server Type
AJP XXXX 01 2U
JP-XXXX-01-2u
BJP XXXX 03 1U
JP-XXXX-03-1u
B
A
Region : JP
Mapping Rack position and Port number in Switch
OS provisioning engine can make infra hostname automatically
Rack Position
Port Number
13
OCP Server
DHCPTFTPSCP BDD Servers1.Get IP address and boot loader file name
2.Get boot loader file(pxelinux.0)
3.Get boot option configuration(pxelinux.cfg/default)
4.Start kernel5.Get boot image
7.Perform BDD OCP Scripts
8.Scp logs file
8.Analyze logs and get certificate result
Certificate the whole rack servers as new delivery
6.Start boot image
HW Check - Data Flow
14
No. Script Name Note
1 Check for initial delivery Collect HW information and check whether it is meet our expectation.
2 HW full stress test for DOA Check stability of HW
3 Initial FW / BIOS update Check BIOS version, downgrade / upgrade BIOS version when necessary
4 Initial standard BIOS configuration Mandatory configuration in BIOSEx) Dedicated network for BMC
5 Over provisioning for SSD Better write performance
6 Initial L2 Switch configuration Basic configuration on ToR / Inner Switch
HW Check - Function
15
Ready to Provisioning - Bare Metal Provisioning System
16
Role Mission Detail
Redundancy / Capacity Plaining
Self Provisioning
All provisioning including application deployment should be handled by users at once.
Full Automation Fire and Forget.
Pool/Resource Management
All user can control their own resource from pool. It can help your capacity planning as well as trouble shooting
Platform Quality
Full Stack Management
All system management, all user can do that by themselves
Easy Operation Support not only GUI but also API for easy maintenance
Fast Delivery Faster faster faster, like cloud
Mission
17
Installation Engine - MAAS Managem
ent Tool - Chef
OS provisioning system
App
OS
App
OS
GSP Hadoop
Bare Metal Management
App
OS
App
OS
Cluster for Redundancy
Pool
Platform Admin / Application Owner
Infra Admin
Self provisioningResource management
Developing and Maintenance
Concept
18
Component Detail
Provisioning ControllerThis was built up by Rakuten in order to control all provisioning process and data.It store own organization data in Rakuten as well as control MAAS and Chef , Power DNS, for all automation.
MAAS Metal as a Service from Canonical
Chef Management tool
PowerDNS Master DNS
BIND Slave DNS
Shinken Alerting system is compatible with Nagios.
Graphite Time Series DB
Components with Open Source
19
Self Provisioning with Full Automation
Dash Board
Organization
Role/Recipe
Host name
Custom data
Provisioning System
OS Provisioning with Chef
Easy Install
API
Worker
Controller
Installation Engine
Management
Monitoring
Operation System
Configuration
All Operation by Chef
App Deploy
Monitoring
Kick OS Provisioning for your system
With Recipes
Recipe for Application
Full Automation
Control Control
ConnectChef
MAASPower DNSShinken Graphite
Operation by each User
20
Pool
APP 1
Deployed
Pool
APP 2
Deployed
After Commissionin
g
Assignment
Provisioning
Destroy OK
Take over
OK
OK OK
OK
NG
OK
OK
Resource/Pool Management : Reservation
Infra Admin
Admin User 1 ADMIN
Authority
Authentication
Chef Organization
= Tenant APP 1 APP2
Admin User 1 User 2Only Admin
NGNGNG
OK
NG
User 2
Only possible by build user
21
APP 1
Deployed
APP 2
Deployed
Provisioning
Destroy OK
Take over
OK
OK OK
OK
NG
OK
OK
Admin User 1 ADMIN
Authority
Authentication
Chef Organization
= Tenant APP 1 APP2
Admin User 1 User 2
NGNGNG
OK
NG
User 2
Only possible by build user
After Commissioning
Shared Pool
Quotation for each group/platform
Self pick up from shared Pool
Resource/Pool Management : Shared Pool
22
• Designed by ApplicationApp Monitoring
• Designed by Application• Custom Package by Application App Deployment
• Custom Configuration on OS by Application• Server Account based on Chef OrganizationOS Configuration
• Default OS monitoring Infra Monitoring
• Default Configuration on OS• Basic Packages OS Configuration
• Simple image• Pattern base Partitioning / Raid configuration OS Installation
• Detail H/W Spec• Custom Information for BDDInventory Data
Full Stack Management
Role/Recipe
Infra Base
App XX Application/Platform Admin
Infra Admin
Responsibility Chef Organization
OS Provisioning
Criteria
App YY
Role/Recipe
App ZZ
MAAS
OS Images
23
Easy Provisioning
1st Step• Chose Server
2nd Step
• Chose Action • Install• Destroy
3rd Step
• Hostname what you want• Unnecessary DNS operation
• OS distribution/version • Tenant yours and environment• Chose recipes of your application
Final, click and get it
Hey, I want new server
Just Do It
24
InstallOS SetupOS SetupEnv
Provisioning Process and Fast Delivery
Provisioning System
Default Infra Role
App Role
Manage recipes for app
Default Infra Monitoring
App Monitoring
Basic Install DNS entry
OS / APP Configuration
MonitoringConfigurationTask
Worker
Approximately 30 min
Request via GUI/API
Operation System
Chef ChefMAASPowerDNS
25
Chef-Client
CollectedYum
Nrpe
Collector
Graphite
Graphite Web
TimeSeriseDB
Inventory DB
OCS Inventory
InventoryGLPI agent
Shiken
Thruk
Monitoring
Config files
Docker
PagerDutyAlert
Repository
Package
JIRA
Issue Tracking
Chef Enterprise
Configuration
DataBag
Account
Organization
CookbookCookbook
Cookbook
Elastic Search
Graylog2
Log Analysis
GIT
Source Management
Jenkins
Graylog Web
Graphana
Dash Board
Core Engine
OS Provisioning
MAAS
PowerDNS
DNS
MySQL
OS Install
Entry
Finally, You can get server with one click
OperationSupport
26
What is Next ? Is it done ? No…
27
Global Hosting for BCP
US 1 US 2 EU JP 1 JP 2
Zone 1
Zone 2
Zone 3
Global Zone
ProvSystem
Global Dash Board
ProvSystem
ProvSystem
ProvSystem
Prov System
Local System
Data Center
Local Zone
28
Platform : APP 1Environment : DEV
Management
API
Application
Application
HaProxy
Network Layer
Float IP
NAT
Private Network for User
Platform : APP 2Environment : PROD
.
.
Network ControlBare Metal
Management
XAnother Application
ACLBlock by ACL
Permit
Permit
Permit
Permit
XUser
API
Application
Application
Permit
29
Data Base
DB 1
DB XX
ApplicationLayer
.
.
Storage Layer
API
App 1
App 2
Internal SLB
HaProxy
AccessLayer
Access
Web 1
Web2
External SLB
HaProxy
Permit
Permit
ACL
Separated Network
30
Separating Functionality
Like OpenStack Concept • Current system only focus OS provisioning as one worker• Current system had been designed for only dedicated DC • Functionality requirement is getting increase
31
Installation Engine Managem
ent Tool - Chef OS provisioning system
App
OS
Bare Metal App
Bare Metal Management
App
OS
App
OS
Cluster for Redundancy
Pool
New feature for Cloud
Docker
OS OS OS
VM
Cloud
OpenStack/Sw
arm/M
esos ..
Global Dash Board
Cost Allocation
32
Thank you
33
We are hiring!
Let’s join us for Building the core part of Rakuten
Just mail to me, I can help you for your [email protected]
http://global.rakuten.com/corp/careers/