1 FOR USE IN IEEL SOLUTIION CLASS ONLY Intel Confidential Intel® Enterprise Edition for Lustre* (IEEL) Introduction Implementing an IEEL Solution Session.

1 FOR USE IN IEEL SOLUTIION CLASS ONLYIntel Confidential

Intel® Enterprise Edition for Lustre* (IEEL) IntroductionImplementing an IEEL SolutionSession 01June 2014

* Other names and brands may be claimed as the property of others.

FOR USE IN IEEL SOLUTION CLASS ONLY2High Performance Data DivisionIntel Confidential

IEEL Introduction and Lustre ReviewObjectives

Review of:

• What is Lustre

• What is Intel’s position in the Lustre community

• What is IEEL and what are its main components

• What’s new in IEEL 2.0

• Roadmaps

• Competition


What is Lustre

Lustre powers

60% of the

Top100 supercomputers


What is Lustre

• Lustre is a multi-vendor global parallel file system• Parallel

• Shared

• Multi-vendor

• Open source

• Scalable

• Provided by Intel


What is Lustre Lustre is a Parallel File System


Lustre Storage Components


What does a Lustre Cluster Look Like?


• Lustre accelerates workflow and ‘time to solution• Performance – Up to 2TB/sec

• Global name space

• Highly scalable – Up to 512 PB in one file system

• Dominates the world’s largest computing systems

• Faster and more scalable than NFS or CIFS

• Lustre is supported on a wide range of vendor hardware for storage and compute.

• ‘build up’ for performance or ‘build out’ for capacity

Why do Companies use Lustre?


• Lustre is supported on a wide range of vendor hardware for storage and compute.

• Users can add storage capacity incrementally

• Can ‘build up’ for more performance

• Can ‘build out’ for more capacity

• High performance access via LAN and WAN

Why do Companies use Lustre?


Key Markets using Lustre TodayExamples of Sectors and Lustre Customers

Energy, oil and gas exploration– Total/Fina, Shell, BP, Chevron

Government and academic research– CEA, IU, ORNL, LLNL, ANU, SDSC

Financial services– Morgan Stanley, Central Bank of Italy, National Bank

of Poland

Life sciences, pharma and genomics– Sanger Institute, Lund University

Weather forecasting and climate modeling

– Australia Weather Bureau


Intel’s Role with Lustre From Whamcloud to Intel

Whamcloud was started July 16, 2010

• Brent Gorda – CEO• Eric Barton – CTO

Founded Whamcloud to keep Lustre* in play and vendor-neutral for HPC

• Recognized by OpenSFS and EOFS as the maintainer of open source repositories

Acquired by Intel in July 2012• Becomes the High Performance Data

Division• Same team, same mission, more

resources


Meeting Needs of Demanding Users

HPDD will continue to focus on HPC requirements…

– Drive open and collaborative development– Rigorously tested to ensure high quality,

predictable

Penetrate the Enterprise and Big Data’ markets

Intel provides Support Services to both Lustre communities

– Open source and Intel ® Enterprise Edition for Lustre

– Support the smallest academic users to DOE Fast Forward program


• Lustre file system backed by Intel

• 8x5 Enterprise support (24x7 for IEEL)

• Documentation from Intel HPDD

• Professional services and training

• Available from Intel HPDD partners

What Intel Provides for the Lustre Community


Lustre Community Edition


Lustre 2.5.x

Lustre 2.5.x is the actual maintenance release stream

• Centos 6.4/6.5 support

• Replaces SLES11 SP2 with SLES11 SP3 client support

• Interop with 2.4.x only; 2.1.x upgrade should be possible

• Fuller information at http://wiki.opensfs.org/Lustre_2.5.0

Feature Highlights

• HSM (LU-3608) • CEA Primary developers; enables data to be transferred

between different storage types

http://wiki.opensfs.org/Lustre_2.5.0


Lustre 2.5.0 Code Contributions (So Far)

Number of commits between 2.4.50 and Sept 7th

Number of Lines of Code Changed between 2.4.50 and Sept 7th

ANU 10 Bull 510

CEA 11266 Cray 28

DDN 114

EMC 3272

Fujitsu 11

gentoo 27

Intel 43454

IU 17

LLNL 224

ORNL 6460

Suse 68 TACC 193 Xyratex 3543ANU 1

Bull 27

CEA 27 Cray 6

DDN 12

EMC 10

Fu-jitsu

1

gen-too 1

Intel 223

IU 1

LLNL 8

ORNL 31

Suse 2 TACC 1Xyratex 20


Lustre 2.5.0 Testing Contributions (So Far)

Number of 2.x bugs reported in Jira between Feature Freeze and Sept 7th

?? 3 Aeon 1

Bull 6CEA 4

Cray 3

DDN 9 gentoo 1

Intel 122

LLNL 2

NASA 3ORNL 2

S&C 2 SSEC 1Xyratex 12


OpenSFS

OpenSFS is a community organization that represents a broad community of users.

• Intel is a board member of OpenSFS

• OpenSFS provides funding for Lustre community releases

• OpenSFS funds Lustre community development projects

• Sponsors annual Lustre User Groups in North America

• See opensfs.org for more information


OpenSFSCommunity Development Working

Group(CDWG)OpenSFS CDWG serves as a forum to discuss

Lustre development matters• Agreeing priorities for test matrix

• Maintaining roadmap

• Sharing test results

• Warning others of known issues

• Reminders of release deadlines

• CDWG wiki repository for development in progress

• http://wiki.opensfs.org/Community_Development_Working_Group

http://wiki.opensfs.org/Community_Development_Working_Group

http://wiki.opensfs.org/Community_Development_Working_Group


European Open File System (EOFS)

• European community group of Lustre users

• Intel is a member of EOFS

• Does not provide funding like EOFS does

• Sponsors annual Lustre Administrator and Developer Forum

• See eofs.org for more information


Intel® Enterprise Edition for Lustre*


Intel Enterprise Edition for Lustre

Leading open source HPC file system

packaged for commercial use

•What is Intel EE Lustre

– Lustre Overview

– Intel Manager

– Hadoop and Lustre

• Lustre in the Cloud


Intel Enterprise Edition for Lustre v 2.0

Hierarchical Storage Management

Tiered storage

Intel® Manager for Lustre CLI

REST API

Management and Monitoring Services

Hadoop Connectors

Lustre File SystemFull distribution of open source Lustre software v2.5

Storage Plug-InArray Integration

Global Technical Support from Intelopen sourceIntel value-add


Intel Lustre version


Tiered storage


REST API


Hadoop Connectors





Intel® Lustre version

Based on Lustre version 2.5.x

Includes single thread improvement patch (from 2.6)

Support for Xeon Phi lustre clients

RH, Centos 6.4/6.5

SuSE 11 SP3


Intel Manager for Lustre


Tiered storage


REST API


Hadoop Connectors





Intel® Manager for Lustre

Provisions and monitors Lustre file systems

Storage hardware neutral

Modern webapp built on REST API

Intuitive GUI

Fully featured CLI

Provides plugin interface for integration with storage and other software tools



The ‘dashboard’ canvas displays a variety of charts that illustrates performance levels and resource utilization. Administrators can easily view file systems, check resource consumption for jobs, and monitor performance.

In depth storage hardware reporting is possible when combined with optional hardware vendor plug-ins.

System status indictor provides the status for all managed file systems. Click to go to detailed information.

Configure servers, volumes and power controls. Optionally, enable HSM per file system

Intelligent, intuitive log files – understand how your storage is performing quickly and easily


Intel Manager for Lustre - REST API

Management services interface for Intel® Enterprise Edition for Lustre* software• GUI and CLI utilize the REST API

Exposed interface for integration with storage management tools and ISV applications

Standard interface• Administration• Alerts• Statistics and metrics

Documentation• REST API for Intel® Enterprise Edition for Lustre*

software


Hadoop Connectors for Lustre (HAL and HAM)


Tiered storage


REST API


Hadoop Connectors





Hadoop Adapter for Lustre (HAL)

Replace HDFS with Lustre

Plugin for Apache Hadoop 2.3 and CDH 5.0.1

No changes to Lustre needed

Enable HPC environments to use MapReduce v2 with existing data in place

Allow Hadoop environments to migrate to a general purpose file system


Hpc Adapter for Mapreduce (HAM)

Replace YARN with Slurm

Plugin for Apache Hadoop 2.3 and CDH 5.0.1

No changes to Lustre needed

Enable HPC environments to use Slurm as scheduler for MapReduce v2 jobs

Allow Hadoop environments to migrate to a more sophisticated scheduler


Why Use Lustre for Hadoop?

Convergence of HPC and data analytics

Desire for HPC systems to run Hadoop workloads• Hadoop is the most popular software stack for big data analytics

• Lustre is the file system of choice for HPC clusters

But, HDFS expects nodes with locally attached disks

Most HPC clusters are diskless compute nodes

Benefits of using Lustre for Hadoop applications• Improved application performance – without changes to app

• Management simplicity lowers costs

• More efficient and productive storage resources

• No data transfer overhead for staging inputs and extracting results

• No need to partition storage into HPC (Lustre) and Analytics (HDFS)

33




Tiered storage


REST API


Hadoop Connectors






• IEEL provides and supports tools to enable HSM solutions:

• Lustre* 2.5.x

• POSIX Copytool

• Robinhood (policy engine)

• HSM Monitoring is integrated into Intel Manager for Lustre* (IML)

• If you use a POSIX compliant archive storage target you can build a complete HSM solution


Intel Enterprise Edition for LustreChallenges

Lustre is the dominant file system for high-end HPC

Limited use by enterprise segments

Rapid adoption of HPC tools is driving Lustre• Into commercial and enterprise

• Downward into ‘missing middle” markets

Making Lustre simpler to manage is critical• Simpler to install and configure

• Easier to monitor and manage

• Worldwide, commercial grade support


Intel® EE for Lustre* (IEEL) solves major challenges moving Lustre into the Enterprise

MarketChallenge IEEL Feature

Simplified Management • Delivers easily managed Lustre file system• Smart default configuration• Intuitive alerts and reports

File system Health • Real Time System Monitoring• Capture, manages and reports statistics

System Integration • Rigorously tested for stability, efficiency and reliability

Vendor Differentiation • Open Source REST API for extensibility• Storage Plugin to create unique vendor specific options

Native Big Data support and Data Analytics

• HAL and HAM give to Hadoop jobs a native support of powerful parallel filesystem

Data Life Cycle • Complete POSIX solution for long term archiving• Storage tiering to lowering the TCO of solution

Long Term Support • Global 24x7 Support• Align with the Intel Ecosystem including the next generation of Intel Processor,

Coprocessor and Fabric




Intel® Manager for Lustre At-A-Glance version 2

39

The ‘dashboard’ canvas displays a variety of charts that illustrates performance levels and resource utilization. Administrators can easily view file systems, check resource consumption for jobs, and monitor performance.

In depth storage hardware reporting is possible when combined with optional hardware vendor plug-ins.

System status indictor provides the status for all managed file systems. Click to go to detailed information.

Configure servers, volumes and power controls. Optionally, enable HSM per file system

Intelligent, intuitive log files – understand how your storage is performing quickly and easily


Complete Hierarchical Storage Management Framework

• Hierarchical Storage Management provides a reliable mechanism for archiving data onto tiers of secondary, high-capacity, affordable secondary storage. HSM optimizes free space, preserving performance performance while providing additional assurance against data loss.

• Configure and manage using IML

• Functionally complete and ready for integration with ISV applications

• Benefit: Tiered storage lowers total cost of ownership and simplifies resource sharing, optimizing the overall costs of storage

*

ManagementNetwork

MetadataServers

MetadataTarget (MDT)

ManagementTarget (MGT)


Data Network(InfiniBand, 10GbE)

Lustre Clients

Policy EngineServer

HSM Agents(Copytools)

HSM Execute


Shared, parallel storage optimized for Hadoop applications

• I/O patterns for Hadoop applications are very different from traditional HPC – often open and close MANY files

• Metadata servers can become overloaded when run Hadoop jobs at scale

• New extended attributes caching improves metadata server performance

• Benefit: Scalable, fast and simple to manage storage for Hadoop applications

*Metadata operations during a Terasort experiment1 using a 3TB dataset and 240 Map-Reduce tasks

1 Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.


Native Lustre Client for Intel® Xeon Phi™

• Native Lustre client for Intel® Xeon Phi™

• Allows applications running on Phi to have direct access to fast, scalable storage resource

• Benefit: Improved I/O performance for Xeon Phi™ applications

*

NFS over Virtual Ethernet Lustre over Virtual IB0

50

100

150

200

250

300

IOZONE Benchmark using 32 threads

Write Read

MB/

sec

10X



Improved Single Client I/O Performance1

• Single client, single threaded applications are common in the enterprise market

• Improved single client and single stream performance levels

• Benefit: Improve the I/O performance of enterprise applications

• Example: MapReduce

*

1 2 4 8 160

500

1000

1500

2000

2500

3000

3500

IOR benchmark on a single client using QDR In-finiBand network

Lustre 1.8.x Lustre 2.x IEEL 2

n. threadsM

B/s

ec





Road Map - Highlights

• Targeting a shared nothing architecture• No more HA for OSS and MDS nodes

• RAID across OST’s

• Easier to do with data (OST’s) than metadata (consistency)

• Based on some studies certain strategic directions stand out

• Performance (love it, need it, got to have it)

• Data Protection (data integrity and no data loss)

• Data Access (different protocols and tools)

• Management (deploy, manage, monitor)45

Intel Confidential — Do Not Forward * Some names and brands may be claimed as the property of others.


Strategic Features for Product ImprovementsNote: No there is no time frame for these features

• Performance:• Improve single client performance (Bandwidth, IOPS, metadata)

• Storage pools within Lustre* file system (with quotas)

• Vastly improved small file performance

• Improved aggregate metadata performance• DNE – deployment, management, monitoring

• Data Protection:• Focus on a “shared nothing” architecture (e.g. Network RAID)

• Network RAID• RAID-1, RAID-1 with 3+ copies?, RAID-5/6• Initially on OST’s (MDT’s are much more difficult)

• Data integrity (T10 compliance and OpenZFS)

• Snapshots (taking them, managing them, mounting them)

• Disaster Recovery and Backups (processes)

• HSM (role in data management, backup, and DR)46



Strategic Features for Product ImprovementsNote: No there is no time frame for these features

• Data Access:• CIFS/NFS/SSHFS access (WebDAV is a target as well)

• Big Data Integration

• Auditing tools

• Management/Monitoring/Analytics:• Goal is a comprehensive tool (not a collection)

• Deployment

• Management• Giving the admin a complete clear picture to make decisions and tools to implement policies• Interoperability with other tools (e.g. Nagios)

• Monitoring

• Data mining (historical analysis)

• Deep auditing of data usage and configuration

• Stability and Reliability• Whether perceived or not, the HPC community and other communities

view Lustre* as being unstable and unreliable. This needs to be fixed.47



IEEL 2.2

• Minor release update

• Features:• OpenZFS:

• IML can build on top of OpenZFS• OpenZFS supported by Intel

• HPC Adapter for MapReduce (HAM)• Replaces YARN in Hadoop 2, allowing MR jobs to be run

like “regular HPC jobs”• Cloudera Distribution of Hadoop (CDH) will be supported• SLURM support

• 2H 2014

48Intel Confidential — Do Not Forward * Some names and brands may be claimed as the property of others.


IEEL 3

• GA is TBD (2015 is target)• More time to harden and test Lustre* version

• More time to develop best practices with new features

• Based on Lustre* 2.5 or later

• Features:• Lustre* QoS based on UID, GID, NID (based on NRS)

• Deployment of Worker nodes for various tasks (CIFS, NFS, SSHFS, lnet router)

• Single button for log gathering

• Better IML scalability (more OST’s)

• Addition of REST API’s for accessing IML database• Example: historical data gathering

• Features are subject to changes



Lustre* features in IEEL 3 - Lustre* 2.5+

• Performance improvements:• Single thread performance improvements

• MDS serialization improvements (better metadata performance)

• DirectIO performance improvement (useful for certain workloads)

• QoS control based on UID, GID, and NID• Based on NRS (throttle RPC rates)

• Uses TBF – Token Bucket Filter



IML features in IEEL 3

• VM deployment on “worker” nodes• Deploy, manage, monitor VM’s on worker nodes (can use OSS nodes)

• Goal is to use VM’s for various “roles”:• CTDB with NFS/CIFS (SSHFS)• lnet routers

• Management of NRS plugin• Control QoS parameters for UID, GID, NID throttling

• Single button in IML to capture logs and dumps for debugging• Phone home capability

• Manual storage configuration• Allows for auto-detect to be over-ridden in certain circumstances

• Used only in POC’s and not in production

• HA improvements





Primary HPC Storage Vendors (per IDC survey research, 2013)

IBM resells NetApp E Series into HPC

HP leads with IBRIX, will quickly shift to DDN

EMC positions Isilon for HPC use

NetApp E Series resold by IBM, DDN, SGI, HDS, Appro (Cray)


Primary File System Used at HPC Sites

Good growth for Lustre~60% of top 100 fastest systems use Lustre

GPFS usage in commercial market (37%)

Lustre use within commercial base: 5%

(Per IDC survey research, 2013)


Lustre and the CompetitionWhere to be aggressive• In NFS environments

– Lustre performs 5-10 times as well, and scales much better– Lack of NFS support for natively non Ethernet network

– (RDMA in NFS v4 only very immature implementation)

– Lack of NFS in cache coherency– A clear win for Lustre

Where to proceed with some caution• Very challenging as GPFS is mature and well marketed

– But, very expensive license based on number of servers, clients, functionalities and type of processor. Extra cost for support

– Block based filesystem very different design from Lustre (object based)

• Lustre succeeds here when:– Customer places a premium on performance– Multi-vendor no locked-in storage options are desirable– Lower TCO– Many support options– Vibrant and open community


Lustre and the CompetitionpNFS environments• Good design, lacking in the implementation

• Very slow adoption for the poor implementation of the client

• On RHEL 6.4 only the client is supported

• Lack in support of a native non Ethernet network

• Lack of cache coherency

Panasas environments• IEEL is less expensive and more open.

• Panasas equals vendor lock-in in on hardware and software but is easy to use. IML can fix this.

• NFS and CIFS support are provided by a gateway


Lustre and the Competition

In Fraunhofer (EMEA only) environments• Very easy to install and manage

• Lack of enterprise support features

• Restricted number of developers and users

In FUSE based filesystem (GLUSTER, CEPH)

• Designed for high latency network and the cloud.

• Doesn't fit well with HPC workloads and scalability

• Rich set of functionalities but dependent on FUSE

development

• CEPH is not POSIX compatible


Lustre and the Competition

Lustre 1.8 (as competition)

• Mature, widely used, difficult to upgrade

• Lack manageability and enterprise functionalities of

IEEL

• Lack in scalability with new multicore processors

• Lack of many features available in IEEL, see next

slide


Questions?

1 FOR USE IN IEEL SOLUTIION CLASS ONLY Intel Confidential Intel® Enterprise Edition for Lustre* (IEEL) Introduction Implementing an IEEL Solution Session.

Documents

solution performance

ieel solution class

intel slide

lustre powers

cifs lustre

lustre community

ieel solution session

ieel solutiion class