Cephfsglusterfs.talk

Cloud storage reloaded:Some aspects on operating distributed Linux file systems

like Ceph and GlusterFS

Udo Seidel

LinuxCon Europe 05-07/11 2012 2

Agenda

● Introduction/motivation● Distributed storage● Ceph and GlusterFS● Operational considerations● Summary


Introduction


Me ;-)

● Teacher of mathematics & physics● PhD in experimental physics● Started with Linux in 1996● Linux/UNIX trainer● Solution engineer in HPC and CAx environment● Head of a international team with people in

Erding, Miami and Sydney @Amadeus


Shared File Systems

● Normal 'business' for Linux● Several implementation approaches

● Network based, e.g. NFS, CIFS● 'clustered'

– Shared disk, e.g. OCFS2, GFS2– Parallel/distributed, e.g. Ceph, GlusterFS


Distributed File Systems

● 'Recent' attention on distributed storage● Cloud hype● Big Data

● Which solution to choose?● Just storage or file system too?

● Knock-on effects● Technical● Non-technical


Operations

● Important part of the life cycle● Geek vs. Enterprise● Technical challenges and more


Distributed Storage


History

● Reviewing storage task responsibilities● Block allocation● Space management

● Extension of SCSI standard● Object based storage● Meta-Data handling separated from data

management


Object based storage

● Storage objects quite general● Partition, file, ...● Unique identifier

● OSD (Object based Storage Device)● Hardware -> original trigger● Software -> common implementation

● Interfaces to existing technology● Part of the distributed storage solution● Additional development of existing approaches


Distributed storage: Paradigm changes

● Block -> Object● Central -> Distributed

● Few -> Many● Big -> Small


Distributed storage – Now what?!?

● Several implementations● Different functions● Support models● Storage vendors initiatives● Relation to Linux distributions


Ceph and GlusterFS


Ceph and GlusterFS

● Sufficient community presence● Picked up by Enterprise Linux vendors● Good amount of commonalities but still quite

different


Ceph and GlusterFS – High level

● Server part● Block storage -> storage server (OSD)● (POSIX) file system -> Meta data● HA -> replication and distribution

● Client part● (POSIX) file system● Storage layer


Ceph


GlusterFS


Ceph and GlusterFS – some detailsCEPH GLUSTERFS

Server implementation userspace userspace

Client implementation userspace/FUSE and native kernel

userspace/FUSE

File system access CEPH protocol suite GlusterFS and NFS protocol

Block device access yes RFP ongoing

QEMU integration yes Under development

Replication traffic OSD network Client-server network

Interconnect IP based IP based and/or Infiniband

Encryption no not really

Back-end file system XFS ..... BTRFS ... ext4 XFS ........ext4

Storage object CRUSH object file (except for striping)

GEO replication not really yes


Ceph and GlusterFS – more detailsCEPH GLUSTERFS

Online growth/shrink yes yes

POSIX yes alike

Concurrent workload technically yes technically yes

NAS replacement technically yes technically yes

SAN replacment technically yes no

Hadoop integration yes yes

Openstack integration yes yes

Community version available

yes yes

Production readiness Block layer File system layer

Service provider Vendor and other Vendor and other

... ... ...


Operational considerations


Status quo “pre distributed storage”

● Server and Storage● Clear responsibilities● Known area: SCSI, NFS, FC● Strict border lines

– Technically– organizationally

● Storage hardware and topology● Less focus on local drives● Dedicated/separated storage network


Changes with distributed storage

● New paradigm● Big overlap between server and storage

● Technical implication● Organizational structure

● Strong network dependency● Topology● physics

● Support and supported setups


Challenge one: server vs. storage

● Need for● Server admin with strong storage skills● Storage admin with strong O/S skills

● Review of● Current organizational setup● Strategy for traditional storage● Operation procedures● HA architecture


Challenge one: possible answers

● 'DevOps model' for O/S and storage admins● Best of both worlds● Better common understanding● Potential of process simplification● BUT organizational changes needed!


Challenge one: possible answers

● 'Appliance model' for O/S and storage admins● Less internal changes● Potential of reusing existing processes● BUT less control and visibility


Challenge two: infrastructure

● Dedicated vs. Shared● Responsibilities

● server vs. storage vs. network

● Transport medium● Local setup

● Spinning disks vs. SSD● RAID?● HA configuration of single server


Challenge two: possible answers

● Different flavour of performance vs. cost topic● Dedicated vs. Shared● Transport medium● Spinning disks vs. SSD● HA on server level


Challenge two: possible answers

● Team up with network organization● Single server full blown SPOF on purpose● Think big


Challenge three: support

● Service provider● Application support


Challenge three: possible answers

● Combine with Linux vendor support● Single point of contact ● Leverage vendor QA

● Hide the new technology● XaaS ● Appliance approach● File system level

● Reminder: Think big


Summary ... kind of ...


The message

● New world with several challenges ● Don't underestimate them● Don't overestimate them

● Technical revolution● requires non-technical changes● Don't underestimate the potential

● Think big! ;-)● Get started!


References

● http://www.ceph.com● http://www.gluster.org● “your business/company”


Thank you!

Cephfsglusterfs.talk

Business

storage storage objects

storage hardware

cloud storage

glusterfslinuxcon europe

reallyyeslinuxcon europe

amadeuslinuxcon europe

visibilitylinuxcon europe

introductionlinuxcon