Introduce LeoFS

You can rely on whenever you need

Yosuke Hara Oct 30, 2012

Tuesday, October 30, 12

1. Motivation

Tuesday, October 30, 12

Need to store and managehuge amount of “media files” at low-cost.

Tuesday, October 30, 12

1. Low ROI2. Possibility of SPOF3. Storage Expansion is difficult during increasing data

?

Tuesday, October 30, 12

X 3.5

Face the situation

*ZDNet: http://www.zdnet.com/blog/btl/amazon-web-services-hosts-more-than-762-billion-objects-in-s3/68308

Tuesday, October 30, 12

“Object Storage System”

Tuesday, October 30, 12

5MB 100MB a few GB

From Photo Storage To Cloud Storage

1st step as Cloud StorageSpecialize in “Photo”

Tuesday, October 30, 12

5MB 100MB a few GB

From Photo Storage To Cloud Storage

Aim to “DATA-HUB” in the CloudHandle various unstructured data

Tuesday, October 30, 12

S3FS-C

Tuesday, October 30, 12

Aim To

Tuesday, October 30, 12

2. Overview

Tuesday, October 30, 12

Storage

Gateway

Manager

ConcurrencyDistributionFault tolerance

Using in Telecom, Banking, e-commerce, Instant messaging,...

Tuesday, October 30, 12

LeoFS-Manager

LeoFS-Gateway

LeoFS-Storage

REST over HTTPRPC

Request from Web Application(s) or Browser

META Object Store

Storage Engine/Router

META Object Store META Object Store

RPC

Storage Engine/Router Storage Engine/Router

Load Balancer

S3-API

SNMP

GUI Console

LeoFS Overview

Gateway (Stateless Proxy)

HTTP Request/Response Handling+

w/Object Cache

ManagerSystem Management

Ring MonitorNode State Monitor

StorageObject Storage, Meta data Storage

+Replicator/Recoverer, Qeueue

Tuesday, October 30, 12

LeoFS-Manager

LeoFS-Gateway

LeoFS-Storage

REST over HTTP (80/443) RPC

(4369)

Request from Web Application(s) or Browser

META Object Store

Storage Engine/Router

META Object Store META Object Store

RPC (4369)

Storage Engine/Router Storage Engine/Router

Load Balancer

S3-API

Monitor(SNMP)

GUI Console

(4000,4010,4020)

(10020, 10021)

Tuesday, October 30, 12

1 - Minimum for Development

Manager x 1 Gateway x 1 Storage x 1

20 - 50TB Storage System (# of replicas = 3)

Manager x 2 Gateway x 3 .. Storage x 8 - 15

10TB .. 20TB / server

50 - 300TB Storage System (# of replicas = 3)

Manager x 2 Gateway x 4 .. Storage x 45 - 90

10TB .. 20TB / server

on XEN, VMWare

Tuesday, October 30, 12

Media Platform

Application / Log Collector

Search / Analysis

PaaS / IaaS

DATA-HUB

Tuesday, October 30, 12

Tuesday, October 30, 12

“DATA-HUB”

Tuesday, October 30, 12

Inside LeoFS

Tuesday, October 30, 12

LeoFS ArchitectureGatewayHTTP

Tuesday, October 30, 12

LeoFS ArchitectureGatewayHTTP

Storage Cluster

Erlang RPC

Erlang RPC

Manager Cluster

State/Process MonitorErlang RPC

Object Cache

Tuesday, October 30, 12

LeoFS Architecture - Gateway / Storage

Storage Engine

Object Storage Metadata

StorageRPC RPC

membership (fault-detection)

redundant-manager replicator

queue

read-repairer

Gateway REST over HTTP (S3-API)

redundant-manager membership (fault-detection)

get put delete head

RPC

Tuesday, October 30, 12

LeoFS Gateway

Tuesday, October 30, 12

*Cowboy: Erlang light-weight HTTP-Server - http://http://www.ninenines.eu/

Gateway

From ApplicationsS3-API

Object Cache

Replicate when using RPC

Consistent HashingHorizontal Distribution

Storage Nodes

[ LRU, Slab allocator, Skip graph ]“Cowboy”

“Stateless Proxy”

Tuesday, October 30, 12

LeoFS Storage

Tuesday, October 30, 12

...

LeoFS Storage

leo-object-storage

LeoFS Storage Engine

Metadata : Keeps an in-memory index of all data.Object Storage : Log structured (append-only) object store.

Request From Gateway

replicatorrepairer

queue...

Tuesday, October 30, 12

LeoFS Storage Engine - Data Structure

Offset Version Time-stamp{VNodeId, Key}

<Metadata>

Checksum

for Sync

KeySize CustomMeta Size File Size

for Retrieve an File (Object)

Footer (8B)

Checksum KeySize DataSize Offset Version Time-stamp

{VNodeId,Key} User-Meta Footer

<Object>

Header (Metadata - Fixed length) Body (Variable Length)

User-MetaSize

ActualFile

Supe

r-bl

ock

Obj

ect-1

Obj

ect-2

Obj

ect-3

<Object Container>

Obj

ect-4

Obj

ect-5

Tuesday, October 30, 12

ObjectID

Leo Object StorageTraditional File System

directory inode

owner infosize

timestampblocks

directory data

inodefilename

file inode

owner infosize

timestampblocks

File

LeoFS Storage Engine

Tuesday, October 30, 12

ObjectID

Leo Object Storage

LeoFS Storage Engine

Robustness

High Performance

Tuesday, October 30, 12

< META DATA >IDFilenameOffsetSizeChecksum

“Object Container”

Header

File

Footer

< META DATA >IdFilenameOffset, SizeChecksum (MD5)Version#

Storage Engine

LeoFS Storage Engine - Retrieve an object from the storage

Metadata Storage

Tuesday, October 30, 12

Storage Engine

Data

LeoFS Storage Engine - Insert an object into the storage

Insert a metadata

Append an object into the object container

Metadata Storage

Tuesday, October 30, 12

Compaction

OLD Object ContainerOLD Metadata

NEW Object ContainerNew Metadata

LeoFS Storage Engine - Remove unnecessary objects from the storage

Storage Engine

Tuesday, October 30, 12

Large object Support

Tuesday, October 30, 12

Large-object Support (LeoFS v0.12)

Gateway

chunk-0

chunk-1

chunk-2

chunk-3Client

1. Able to equalize disk usage of each storage node2. High I/O efficiency

Original fils’s Metadata #metadata{ ... dsize = 10485768, cnumber = 4, csize = 3670016, ...}

Storage Cluster

Tuesday, October 30, 12

LeoFS Manager

Tuesday, October 30, 12

LeoFS Manager

Manager

For Administrators

Monitor

Operate

RING, Node State

status, suspend,resume, detach, whereis, ...

TCP/IP

for continue operation

Tuesday, October 30, 12

LeoFS-Manager

LeoFS-Gateway

LeoFS-Storage

REST over HTTP (80/443) RPC

(4369)

Request from Web Application(s) or Browser

META Object Store

Storage Engine/Router

META Object Store META Object Store

RPC (4369)

Storage Engine/Router Storage Engine/Router

Load Balancer

S3-API

Monitor(SNMP)

GUI Console

(4000,4010,4020)

(10020, 10021)

Verticalized Storage System

Tuesday, October 30, 12

3. Future Works

Tuesday, October 30, 12

Improve Performance & Functionality

High I/O efficiency

1. Multi-layer Cache (Using SSD)

HIGH-Performance (Low latency)

Primary Cache

Secondary Cache

Gateway

Reduce traffic between Gateway and Storage

Storage

Tuesday, October 30, 12

Improve Performance & Functionality

2. Multi-Datacenter for Geographical Optimization

HIGH-Scalability

HIGH-Availability

Tuesday, October 30, 12

System Console

Log Analyzer / Searcher

3. LeoFS’s Tools

Improve Performance & Functionality

Tuesday, October 30, 12

4. Wrap Up

Tuesday, October 30, 12

LeoFS will realize 3-HIGHs, soonHIGH Cost PerformanceHIGH ReliabilityHIGH Scalability

Tuesday, October 30, 12

Thank you for your time

LeoFS - http://www.leofs.org

Tuesday, October 30, 12