STORAGE ARCHITECTURES FOR THE MODERN D ATA CENTRE Chris M Evans Langton Blue Ltd Copyright © 2016 Langton Blue Ltd 1
Apr 21, 2017
STORAGEARCHITECTURES FOR THE MODERNDATACENTRE
Chris M EvansLangton Blue Ltd
Copyright © 2016 Langton Blue Ltd 1
A WORDABOUTME
Copyright © 2009-‐2016 Langton Blue Ltd 2
§ 28 years experience in the IT industry across all platforms, including IBM mainframe, Windows & Open Systems.
§ Co-‐founder and independent consultant at Langton Blue Ltd, a specialist consulting company in the UK.
§ Blogger and part-‐time analyst at Architecting IT§ Twitter: @chrismevans, @architectingIT,
@langtonblue§ Web: www.architecting.it,
www.langtonblue.com
AGENDA
§ Evolution of The Data Centre§ Understanding Customer Needs§ Storage Architectures§ Product Marketplace§ Q&A
Copyright © 2009-‐2016 Langton Blue Ltd 3
EVOLUTION OF THE DATA CENTREA quick primer on storage heritage
Copyright © 2009-‐2016 Langton Blue Ltd 4
20TH CENTURYDESIGN
§ Monolithic§ Bespoke hardware§ Expensive to acquire, maintain &
manage§ Static designs and configuration
§ (Think time taken to run binfilechange)
§ Limited or no data services§ No RAID, replication, snapshots§ All features that developed over time
Copyright © 2009-‐2016 Langton Blue Ltd 5
20TH CENTURYARCHITECTURE
§ Custom hardware designs & components still reign§ Hosts own LUNs mapped from physical RAID groups§ Performance defined by RAID group size and disk
speed§ Wide striping for performance
§ Throughput & I/O latency influenced (and restricted) by controller capabilities
§ Lots of manual effort to rebalance systems, especially at scale§ LUN migration, rebuilds, reliance on host-‐level LVMs for a
lot of the work
§ Lots of risk fragmentation and orphan storage§ Wasted space in RAID groups
Copyright © 2009-‐2016 Langton Blue Ltd 6
CONTROLLER A CONTROLLER B
BED BED BED BED BED BED BED BED
21ST CENTURYTECHNOLOGY
§ Vendors have moved away from bespoke hardware designs§ x86 Standardisation§ Commodity, reliable components§ High performance processors & memory – Moore’s Law
§ Interoperability§ Components generally work together without problems§ Storage protocols are robust and mature (still also developing – NVMe)
§ New Technology§ Flash§ NVDIMM§ 3D Xpoint
§ Migration of “smarts” to software -‐ Software Defined Storage§ Features implemented in code, not microcode
Copyright © 2009-‐2016 Langton Blue Ltd 7
FLASH & NEW MEDIA
§ Designs§ SLC (Single Level Cell), MLC (Multi-‐level Cell), TLC (Triple Level Cell)§ 3D-‐NAND – 3-‐D rather than planar (2D) technology for higher density
§ 3D-‐Xpoint – faster and more scalable persistent storage than Flash, slower than DRAM and in between on price
§ Intel/Micron joint development§ NVRAM/NVDIMM – DIMM form factor products delivering
persistent flash memory on the motherboard§ NVMe – new connectivity standard for writing to persistent storage
§ PCIe and SDD type form factors (not yet widely adopted)
Copyright © 2009-‐2016 Langton Blue Ltd 8
PROCESSOR & MEMORYPERFORMANCE
Copyright © 2009-‐2016 Langton Blue Ltd 9
§ Processor, memory and bus speeds are increasing
§ Processor performance and bus speeds follow Moore’s Law
§ Bus speeds double with each release§ PCIe v1.x = 250MB/s (per lane) -‐ 2003§ PCIe v2.x = 500MB/s (per lane) -‐ 2007§ PCIe v3.0 = 985MB/s (per lane) -‐ 2010§ PCIe v4.0 = 1969MB/s (per lane) – 2014
§ Features much easier to deliver from software on x86§ New instruction sets like SSSE3 and AVX2
allow storage computations like erasure coding/Reed Solomon to be done in software
§ Fewer reasons to develop bespoke ASICs and FGPAs
SOFTWARE DEFINED STORAGE(KEY FEATURES)
§ Abstraction – I/O services should be delivered independent of the underlying hardware, through logical constructs like LUNs, volumes, file shares and repositories.
§ Automation – resources should be consumed using CLIs and APIs rather than manually allocated through a GUI.
§ Policy/Service Driven – the service received (IOPS, latency) should be established by policies that implement Quality of Service, availability and resiliency.
§ Scalable – solutions should enable performance & capacity scaling independent of I/O delivery.
Copyright © 2009-‐2016 Langton Blue Ltd 10
MAJOR STORAGETHEMES
§ More choice in products (both hardware and software than ever before)
§ Divergence – storage platforms for specific purposes (no single platform for all requirements)§ Object storage, cloud, backup, primary
§ Convergence – collapsing the storage/server hardware into a single unit integrating new technology (converged & hyper-‐converged)
§ Integration – VVOLs, APIs, platform drivers (e.g. Cinder), hypervisor extensions (VASA, VAAI)
§ New Media§ Flash and its successors
Copyright © 2009-‐2016 Langton Blue Ltd 11
UNDERSTANDING CUSTOMER NEEDSThe changing face of application deployment
Copyright © 2009-‐2016 Langton Blue Ltd 12
WHAT ARE CUSTOMERS’ PAIN POINTS
§ Cost § Performance § Operational Complexity § Reliability§ Floor Space§ Power Consumption
Copyright © 2009-‐2016 Langton Blue Ltd 13
Source: Tintri State of Storage Report, March 2015https://www.tintri.com/sites/default/files/fie ld/pdf/wh itepapers/state_of_storage_ infographic_150513t10216.pdf
WHAT DO CUSTOMERS CARE ABOUT
§ Efficiency§ Cost ($/GB, effective or raw)§ Physical media
§ Performance§ Latency, IOPS, throughput, Quality of Service
§ Management§ Multi-‐tenancy, APIs (Cinder), Automation, Abstraction (VVOLs)
§ Reliability§ Resiliency, Availability
§ Features§ Data Services, Data Protection
§ Breaking the buying cycle§ No forklift upgrades or 3-‐4 year refresh projects
Copyright © 2009-‐2016 Langton Blue Ltd 14
PREDICTABLYUNPREDICTABLE
§ Workloads becoming more random in nature§ I/O blender
§ Peaks and troughs in demand§ VDI (Bootstorms, logoff storms)
§ Increase in I/O Density§ Driven by virtualisation and soon containers
§ Agility§ Desire to move data around, spin up copies, create test dev environments, destroy
and recreate§ Persistence
§ Applications expecting to be transient in nature, but more need for storage to be 100% available across the entire infrastructure
Copyright © 2009-‐2016 Langton Blue Ltd 15
VIRTUAL FIRST STRATEGY
§ Provided the ability to consolidate over-‐provisioned server resources§ Systems typically 25% allocated
§ Reduced delivery time for (virtual) servers from days/months to hours§ Assuming sufficient capacity available
§ Enabled high availability through infrastructure§ No need to deploy clustered systems§ Centralised data protection (e.g. VADP)
§ Centralised storage and virtualisation don’t play well together§ Storage doesn’t understand VMs
§ Storage sees LUNs & volumes§ Virtualisation sees VMs§ LUNs encapsulate many volumes – no visibility of VMs§ Problems being worked around with VVOLs and other solutions that integrate directly
into hyper-‐converged platforms.
Copyright © 2009-‐2016 Langton Blue Ltd 16
TECHNICAL REQUIREMENTS– QUALITY OF SERVICE
§ Eliminate the “noisy neighbour” problem§ Provide storage resources based on policy§ Ensure storage isn’t under-‐delivering
§ Guarantee IOPS, performance minima§ Ensure storage isn’t over-‐delivering
§ 1st on the array gets best performance (until there are more)
§ Set limits and prioritisation§ Apply to VM/VMDK where possible
Copyright © 2009-‐2016 Langton Blue Ltd 17
TECHNICAL REQUIREMENTS -‐ API DRIVEN
§ Admin-‐based functionality driven by API§ Ability to create, manage, destroy LUNs
from code§ Integrate storage functions into
workflow§ Cloud, Automation, Self Service
§ Reduce dependence on manual process§ Agility
Copyright © 2009-‐2016 Langton Blue Ltd 18
TECHNICAL REQUIREMENT– SUPPORTVIRTUAL
§ Reduce waste from over-‐provisioning§ Support – object (e.g. VM-‐based)
management§ Apply policies at the VM level
§ Data location, performance, latency, backup§ VM friendly for data protection
(snapshots/replication)§ Offload heavy lifting tasks (e.g. VAAI)
Copyright © 2009-‐2016 Langton Blue Ltd 19
DIVERSITY OF STORAGEPLATFORMS
Copyright © 2009-‐2016 Langton Blue Ltd 21
HYBRID
COMMODITY OPEN-‐SOURCE
APP/DATA AWARE
SMART STORAGE
OBJECT STORES
SCALE-‐OUT SAN
SCALE-‐OUT NAS
ALL-‐FLASH
SOFTWARE DEFINED
SCALE-‐OUT BACKUP
ULTRA-‐FAST FLASH
HIGH-‐CAPACITY FLASH
HPC
INFRASTRUCTURE CONSUMPTION CONTINUUM
Copyright © 2009-‐2016 Langton Blue Ltd 22
As-‐a-‐Service Hyper-‐converged Converged Proprietary
ApplianceSoftware on Commodity
SMB/Start-‐ups Large Enterprise Hyperscale & SP
Ease of Implementation
Implementation Flexibility
Degree of Vendor Lock-‐in
Cost Efficiency
Harder
More Flexible
Less Lock-‐in
At Large Scale
Easier
Less Flexible
More Lock-‐in
At Small Scale
SCALE UP
§ Typically Dual controller implementation§ Active/Active & Active/Passive
§ Throughput limited by controller performance§ More throughput means better controller
capability (CPU/memory)§ I/O performance limited by media§ Flash based implementations failed to use
flash to fullest capability§ Older designs don’t cater for write endurance
issues§ Automated tiering capabilities to place data on
most appropriate technology
Copyright © 2009-‐2016 Langton Blue Ltd 23
CONTROLLER A CONTROLLER B
SCALE OUT (CLOSELYCOUPLED)
§ Scale out through adding nodes and/or capacity
§ Tight node integration either paired or with bespoke backplane (e.g. PCIe/Infiniband)
§ Symmetric design – scale with nodes of similar size/capacity
§ LUNs/Volumes delivered from specific nodes – not fully load balanced
§ Scale out limited by backplane traffic§ Add/remove nodes dynamically
Copyright © 2009-‐2016 Langton Blue Ltd 24
CTRL A CTRL B CTRL C CTRL D CTRL E CTRL F
SCALE OUT (LOOSELY COUPLED)
§ Fully distributed “Shared Nothing” architecture§ Loose node coupling (usually IP-‐based)§ All nodes contribute storage (no disk shelves)
§ At least one node of “spare” capacity required across the cluster§ More nodes means more efficiency in redundancy
§ Data resiliency based on replicating data between nodes§ Standard hardware (typically 1U server) design
§ Node power efficiency & size become important§ High scalability§ Smaller increments in adding capacity/performance§ More complex design
Copyright © 2009-‐2016 Langton Blue Ltd 25
NODE A
NODE B
NODE C
NODE D
NODE E
NODE F
HYPER-‐CONVERGED
§ Single hardware device for storage and server§ Node-‐based scale-‐out§ Integrated hypervisor & storage services
§ Storage implemented in VM or hypervisor kernel§ Distributed file system (scale-‐out shared nothing)
§ Efficient use of resources§ Simplified management§ At scale probably more expensive than separate
components§ Reduced VM density§ Issues managing performance vs capacity scaling
Copyright © 2009-‐2016 Langton Blue Ltd 26
NODE A
NODE B
NODE C
NODE D
NODE E
NODE F
HYPER-‐CONVERGENCE -‐ TCO
§ Simplified§ Less “plan” time -‐ no requirement to certify and test components§ Less “build” time – deploy a new node, power up and add to the
configuration§ Less “operate” time – single management console, typically
integrated with the hypervisor, tight hypervisor awareness§ Efficient
§ Scale per node; less wastage, no up-‐front hardware acquisition§ “Decommission” time almost eliminated; simply add new nodes
and evacuate/remove old nodes over time
§ Provides overall TCO savings (not storage specific)§ Not typically suited to high-‐scale environments
Copyright © 2009-‐2016 Langton Blue Ltd 27
STORAGEVIRTUALISATION -‐ EVOLUTION
§ Monolithic – single central controller§ Inline in the data path§ Controller maps logical to “physical” storage§ data management features built into the controller§ Not highly scalable
§ CentralisedMetadata – Distributed Data§ Central metadata functions§ Data distributed/replicated across many devices§ System not in the data path§ Separation of control and data planes
§ Totally Distributed§ No central metadata or data§ Data distributed across many devices§ Fully scalable architecture
Copyright © 2009-‐2016 Langton Blue Ltd 28
ADVANCED SERVICES
§ Space Efficiency§ Compression§ De-‐duplication§ Thin Provisioning
§ Quality of Service§ Object abstraction (VVOLS, per VM services)§ Application consistent snapshots/backup
Copyright © 2009-‐2016 Langton Blue Ltd 29
SPACE OPTIMISATION/EFFICIENCY
Copyright © 2009-‐2016 Langton Blue Ltd 30
Feature Compression De-‐duplication Thin Provisioning
Physical space reduction ✔ ✔ ❌
Architecture/metadata dependent ❌ ✔ ✔
CPU Friendly ❌ ✔ ✔
Traditional OLTP Database friendly ✔ ❌ ❌
VDI/VSI Friendly ❌ ✔ ✔
HDD Friendly ✔ ❌ ✔
Flash Friendly ✔ ✔ ✔
QUALITY OF SERVICE (AGAIN)
§ Assign service-‐based metrics to application workloads§ Reduce/eliminate the “noisy neighbour” effect§ Set application owner “expectations” on performance
§ Avoid first/last application syndrome from legacy arrays§ Metrics
§ Latency§ IOPS§ Throughput
§ Establish maximum/minimum/burst values & priority
Copyright © 2009-‐2016 Langton Blue Ltd 31
OBJECTABSTRACTION
§ Implement per-‐VM services§ Abstract away from LUNs/Volumes
§ Implement object-‐specific services§ Snapshots§ Replication§ Cloning
§ Application-‐consistent data management§ Examples
§ VVOLs§ Object Stores
Copyright © 2009-‐2016 Langton Blue Ltd 32
PUTTING IT ALL TOGETHER
§ Start with Requirements§ Don’t buy what you don’t need§ Focus on your sensitivities (cost, performance, scalability)
§ See the “bigger picture”§ Don’t just storage in isolation
§ Don’t buy a technology§ Look past the glamour of shiny boxes!
§ Evaluate your options§ Use tools to create an equitable test environment
§ Use online resources and other views§ Wisdom of the crowd applies
Copyright © 2009-‐2016 Langton Blue Ltd 35