Introduce LeoFS

You can rely on whenever you need

Yosuke Hara Oct 30, 2012

Tuesday, October 30, 12

1. Motivation


Need to store and managehuge amount of “media files” at low-cost.


1. Low ROI2. Possibility of SPOF3. Storage Expansion is difficult during increasing data

?


X 3.5

Face the situation

*ZDNet: http://www.zdnet.com/blog/btl/amazon-web-services-hosts-more-than-762-billion-objects-in-s3/68308


http://www.zdnet.com/blog/btl/amazon-web-services-hosts-more-than-762-billion-objects-in-s3/68308

http://www.zdnet.com/blog/btl/amazon-web-services-hosts-more-than-762-billion-objects-in-s3/68308

“Object Storage System”


5MB 100MB a few GB

From Photo Storage To Cloud Storage

1st step as Cloud StorageSpecialize in “Photo”


5MB 100MB a few GB

From Photo Storage To Cloud Storage

Aim to “DATA-HUB” in the CloudHandle various unstructured data


S3FS-C


Aim To


2. Overview


Storage

Gateway

Manager

ConcurrencyDistributionFault tolerance

Using in Telecom, Banking, e-commerce, Instant messaging,...


LeoFS-Manager

LeoFS-Gateway

LeoFS-Storage

REST over HTTPRPC

Request from Web Application(s) or Browser

META Object Store

Storage Engine/Router

META Object Store META Object Store

RPC

Storage Engine/Router Storage Engine/Router

Load Balancer

S3-API

SNMP

GUI Console

LeoFS Overview

Gateway (Stateless Proxy)

HTTP Request/Response Handling+

w/Object Cache

ManagerSystem Management

Ring MonitorNode State Monitor

StorageObject Storage, Meta data Storage

+Replicator/Recoverer, Qeueue


LeoFS-Manager

LeoFS-Gateway

LeoFS-Storage

REST over HTTP (80/443) RPC

(4369)


META Object Store



RPC (4369)


Load Balancer

S3-API

Monitor(SNMP)

GUI Console

(4000,4010,4020)

(10020, 10021)


1 - Minimum for Development

Manager x 1 Gateway x 1 Storage x 1

20 - 50TB Storage System (# of replicas = 3)

Manager x 2 Gateway x 3 .. Storage x 8 - 15

10TB .. 20TB / server

50 - 300TB Storage System (# of replicas = 3)

Manager x 2 Gateway x 4 .. Storage x 45 - 90

10TB .. 20TB / server

on XEN, VMWare


Media Platform

Application / Log Collector

Search / Analysis

PaaS / IaaS

DATA-HUB



“DATA-HUB”


Inside LeoFS


LeoFS ArchitectureGatewayHTTP


LeoFS ArchitectureGatewayHTTP

Storage Cluster

Erlang RPC

Erlang RPC

Manager Cluster

State/Process MonitorErlang RPC

Object Cache


LeoFS Architecture - Gateway / Storage

Storage Engine

Object Storage Metadata

StorageRPC RPC

membership (fault-detection)

redundant-manager replicator

queue

read-repairer

Gateway REST over HTTP (S3-API)

redundant-manager membership (fault-detection)

get put delete head

RPC


LeoFS Gateway


*Cowboy: Erlang light-weight HTTP-Server - http://http://www.ninenines.eu/

Gateway

From ApplicationsS3-API

Object Cache

Replicate when using RPC

Consistent HashingHorizontal Distribution

Storage Nodes

[ LRU, Slab allocator, Skip graph ]“Cowboy”

“Stateless Proxy”


http://www.ninenines.eu/

http://www.ninenines.eu/

LeoFS Storage


...

LeoFS Storage

leo-object-storage

LeoFS Storage Engine

Metadata : Keeps an in-memory index of all data.Object Storage : Log structured (append-only) object store.

Request From Gateway

replicatorrepairer

queue...


LeoFS Storage Engine - Data Structure

Offset Version Time-stamp{VNodeId, Key}

<Metadata>

Checksum

for Sync

KeySize CustomMeta Size File Size

for Retrieve an File (Object)

Footer (8B)

Checksum KeySize DataSize Offset Version Time-stamp

{VNodeId,Key} User-Meta Footer

<Object>

Header (Metadata - Fixed length) Body (Variable Length)

User-MetaSize

ActualFile

Supe

r-bl

ock

Obj

ect-1

Obj

ect-2

Obj

ect-3

<Object Container>

Obj

ect-4

Obj

ect-5


ObjectID

Leo Object StorageTraditional File System

directory inode

owner infosize

timestampblocks

directory data

inodefilename

file inode

owner infosize

timestampblocks

File



ObjectID

Leo Object Storage


Robustness

High Performance


< META DATA >IDFilenameOffsetSizeChecksum

“Object Container”

Header

File

Footer

< META DATA >IdFilenameOffset, SizeChecksum (MD5)Version#

Storage Engine

LeoFS Storage Engine - Retrieve an object from the storage

Metadata Storage


Storage Engine

Data

LeoFS Storage Engine - Insert an object into the storage

Insert a metadata

Append an object into the object container

Metadata Storage


Compaction

OLD Object ContainerOLD Metadata

NEW Object ContainerNew Metadata

LeoFS Storage Engine - Remove unnecessary objects from the storage

Storage Engine


Large object Support


Large-object Support (LeoFS v0.12)

Gateway

chunk-0

chunk-1

chunk-2

chunk-3Client

1. Able to equalize disk usage of each storage node2. High I/O efficiency

Original fils’s Metadata #metadata{ ... dsize = 10485768, cnumber = 4, csize = 3670016, ...}

Storage Cluster


LeoFS Manager


LeoFS Manager

Manager

For Administrators

Monitor

Operate

RING, Node State

status, suspend,resume, detach, whereis, ...

TCP/IP

for continue operation


LeoFS-Manager

LeoFS-Gateway

LeoFS-Storage

REST over HTTP (80/443) RPC

(4369)


META Object Store



RPC (4369)


Load Balancer

S3-API

Monitor(SNMP)

GUI Console

(4000,4010,4020)

(10020, 10021)

Verticalized Storage System


3. Future Works


Improve Performance & Functionality

High I/O efficiency

1. Multi-layer Cache (Using SSD)

HIGH-Performance (Low latency)

Primary Cache

Secondary Cache

Gateway

Reduce traffic between Gateway and Storage

Storage



2. Multi-Datacenter for Geographical Optimization

HIGH-Scalability

HIGH-Availability


System Console

Log Analyzer / Searcher

3. LeoFS’s Tools



4. Wrap Up


LeoFS will realize 3-HIGHs, soonHIGH Cost PerformanceHIGH ReliabilityHIGH Scalability


Thank you for your time

LeoFS - http://www.leofs.org


http://www.leofs.org

http://www.leofs.org

Introduce LeoFS

Technology

improve performance

leofs storage engine

manager s3

photo storage

monitor rpc

stateless proxy

api rest

web application