Ceph scalable, unified storage files, blocks & objects Tommi Virtanen / @tv / DreamHostOpenStack Conference 2011-10-07.

Cephscalable, unified storage

files, blocks & objects

Tommi Virtanen / @tv / DreamHost OpenStack Conference 2011-10-07

Storage system 

Open SourceLPGL2

no copyright assignment

Incubated by DreamHost

started by Sage Weil at UC Santa Cruz, research group partially funded by tri-labs

50+ contributorsaround the world

Commodity hardware

No SPoF 

No bottlenecks 

Smart storagepeers detect, gossip, heal

Monitors

Object storage

pool, name    data (bytes),

           metadata: key=value, k2=v2, ...

librados (C)libradospp (C++)

PythonPHP

your favorite language here

Smart clienttalk to the cluster, not to a gateway

compound operationschoose your consistency (ack/commit)

Poolsreplica count,

access control,placement rules,

...

CRUSHdeterministic placement algorithm

no lookup tables for placementDC topology and health as input

balances at scale

zonerowrackhostdisk

Autonomousothers say: expect failurewe say: expect balancing

failure, expansion, replica count, ...

btrfs / ext4 / xfs / *really, anything with xattrs

btrfs is an optimizationcan migrate one disk at a time

process per XX = disk, RAID set, directory

tradeoff: RAM & CPU vs fault isolation

RADOS gatewayadds users, per-object access controlHTTP, REST, looks like S3 and Swift

i <3 botouse any s3 client

just a different hostnamewe'll publish patches & guides

RBDRADOS Block Device

Live migrationone-line patch to libvirt

don't assume everything is a filename

Snapshotscheap, fast

rbd create mypool/myimage@mysnap

Copy on Writelayering aka base image

soon

rbd map imagename

/dev/rbd0/dev/rbd/*

QEmu/KVM driverno root needed

shorter codepath

Ceph Distributed Filesystem

mount -t cephor FUSE

High Performance Computing

libcephfsno need to mount, no FUSE

no root access neededalso from Java etc

Samba, NFS etc gateways

Hadoop shimreplaces HDFS,

avoids NameNode and DataNode

devopsdevopsdevops

Chef cookbooksOpen Source

on Githubsoon

BarclampOpen Source

on Githubsoon

devving to help opsnew store node

hard drive replacementdocs, polish, QA 

ceph.newdream.netgithub.com/NewDreamNetwork

Questions?

[email protected]

P.S. we're hiring!

Bonus round 

Want iSCSI?export an RBD

potential SPoF & bottlenecknot a good match for core Ceph

your product here

s3-testsunofficial S3 compliance test suiterun against AWS, codify responses

Teuthologystudy of cephalopods

multi-machine dynamic testsPython, gevent, Paramiko

cluster.only('osd').run(args=['uptime'])

roles:- [mon.0, mds.0, osd.0]- [mon.1, osd.1]- [mon.2, osd.2]- [client.0]

tasks:- ceph:- trashosds:    op_delay: 1    chance_down: 10- kclient:- workunit:    all:      - suites/bonnie.sh

ceph-osd pluginsSHA-1 without going over the network

update JSON object contents

ceph.newdream.netgithub.com/NewDreamNetwork

Questions?

[email protected]

P.S. we're hiring!