DDN’s Vision for the Future of Lustre - OpenSFS: The …cdn.opensfs.org/.../04/LUG-Presentation-Final_Triendl.pdfDDN’s Vision for the Future of Lustre LUG2015 Robert Triendl ddn.com

ddn.com

DDN’s Vision for the Future of Lustre

LUG2015 Robert Triendl

ddn.com

ddn.com

Topics

1. The Changing Markets for Lustre

2. A Vision for Lustre that isn’t Exascale

3. Building Lustre for the Future

4. Peak vs. Operational Performance

5. Application Optimized Lustre

6. Why Conventional Storage Still Matters

3

ddn.com

Hyperscale Storage Markets

HPC Scratch

Petabytes Streaming Write

Large Files Infiniband

Single Location

Cloud WORM(N)

Billions of Files Random Read

Small Files Ethernet

Distributed

Big Data & Data

Analytics

4

ddn.com

Lustre Markets Today Work 31%

Data 33%

Mixed 22%

Cloud 2%

Archive 12%

5

ddn.com

Market Diversification 6

Work Data Mixed Use Archive Cloud

HPC Work

Weather Climate

CAE Chemical

General Academic

Genomics

Big Data Science Security

Finance

Energy

Tier 2 HPC Cloud

Cloud

ddn.com

Lustre Futures Beyond Exascale 7

CIFS/NFS Export, AD Integration, RAS Features, Snapshots, Data Management, etc. Manufacturing

Random Performance, Small File & Metadata Performance, Data Management, Security, etc. Genomics

Virtualization, Snapshots, Small File Read Performance, Data Distribution, etc. Cloud

Broad Application Support, Connectors, User Monitoring, User Access to Snapshot, etc.

General Academic

Archive Data Management Features, SMR Drive Use, Data Scrubs, Data Distribution, etc.

ddn.com

Market Evolution 8

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2012 2013 2014

Archive

Cloud

Mixed

Data

Work

ddn.com

Market Segments 9

0%  

10%  

20%  

30%  

40%  

50%  

60%  

70%  

80%  

90%  

100%  

2011   2012   2013   2014  

Industry  

Government  

University  

ddn.com

Disks: Throughput vs. IOPS 10 MB/sec

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

0

200

400

600

800

1000

1200

1400

1600

1800

Lustre 1.8 Lustre 2.4 ExaScaler 2.2 Lustre with BtrFS Next Gen SAS Drives

IOPS/10 GB/sec

ddn.com

Lustre Development at DDN

▶ Lustre Usability Features

▶ Build-in Reliability and Availability

▶ Lustre Recovery

▶ Features for a Broader Market

▶ Performance for Broad Set of Applications

▶ Application-optimized Lustre

11

ddn.com

Lustre Code Contributions 12

0

20

40

60

80

100

120

2.1 2.4.0 2.3.50-2.4.0 2.5.0 2.5.50-2.6.0

Other EMC SUSE Bull Cray Seagate DDN

ddn.com

DDN ExaScaler Software Stack 13

Intel DSS

DDN Block Storage with SFX Cache

ldiskfs OpenZFS btrfs

DDN Lustre Edition

Intel IML DDN ExaScaler

DDN IME NFS/CIFS/S3 DDN Clients

DDN DirectMon DDN ExaScaler Monitor

Intel Hadoop

ExaScaler Data Management Framework Fast Data Copy

Object (WOS) S3 Cloud Tape

DDN DDN & Intel Intel HPDD

Data Management

Monitoring & Management

Core FS

Storage HW Other HW

Other

ddn.com

Why BtrFS?

▶ Standard Local Filesystem in RHEL7

▶ Better Throughput Performance than ZFS

▶ Similar Feature Set, but all Linux

▶ No Possible Patent Infringement

▶ Simple Integration and Deployment

14

ddn.com

Application-Optimized Lustre

▶ Lustre for Specific Applications

▶ Workload Profiling

▶ Optimization Across I/O Calls

▶ Optimizing Application Runtime

▶ Working with Customers

15

ddn.com

Genome Pipeline Benchmarks 16

2.5.1 DDN Branch 0

10

20

30

40

50 Human Genetics samtools workflow

Run

time

(Hou

rs)

Lustre 2.5 Client Performance

Samtools 20% faster with DDN Lustre optimizations

ddn.com

SSD Pools and Caching

▶ DSS to Link File Layer to Block Layer

▶ Build into the File System

▶ Better use of SSDs for I/O Optimization

▶  Increased Small File Performance

▶  Increased Random Read to Large Files

▶ Additional specificity with fadvice()

17

ddn.com

ExaScaler Monitoring

Monitoring  Server collectd

Graphite  plugin

UDP(TCP)/IP    based  small    text  message    transfer

graphite

•  Lightweight  •  Near  real-­‐Rme  •  Massive  scale    

•  Filesystem, OSS, MDS, OST, MDT, etc. •  JOB ID, UID/GID, application stats, etc. •  Archive of data by policy

Burst    Buffer  

OSS,  MDS  Storage  

18

ddn.com

TITECH Examples 19

64 Billion of Lustre Stats in 15 days!

ddn.com

Why Block-Level Raid? 20

▶ Best Mixed I/O Performance

▶ Consistent Performance

▶ Hardware-optimized Performance

▶ Best Performance During Failure

▶  Integrated Storage Services

ddn.com

SFA RAID Stack Performance

▶ Above 1 Million 4K IOPS per 8 CPU Cores

▶ Above 10 GB/sec per 8 CPU Cores

▶ 8 Cores Sufficient for PCI Infrastructure

▶ More Cores for File System Services

▶ Additional Cores for More Functionality

21

ddn.com

SFA Random Read 22

0  

5  

10  

15  

20  

25  

30  

1   8   16   24   32   40   48   56  

512K  I/O  Size  

1M  I/O  Size  

2M  I/O  Size  

4M  I/O  Size  

MB/sec

ddn.com

Flexible SSU Design 23

S M L XL

ddn.com

SFA14K Performance SSU 24

Up to 45 GB/sec Up to 2950 TB External MDT

OST Storage

4-6 OSS

2-4 MDS

Monitoring

MDT Storage

ddn.com

“Wolfcreek” Hardware 25

ddn.com

“Wolfcreek” Hardware 26