Managing 100TB of small files…

Managing 100TB of small files…

Prospero Media Storage

IGT –

July 2011

Event

Numbers

• 70TB used space• 700 million files• 200GB and 250,000 files uploaded every day• 1200Mbps bandwidth throughput in peak• 180TB of data is being served out monthly• 3700 Hits per second in peak • 40 storage node servers – 300TB raw space• $0.13 per GB

Motivation

• Web 2.0 content serving paradigm shift– Too many files

• 12M users x 1 file = very long tail– Too many connections

• 1M users + keepalive = 1M connections– Living with modern content in web 2.0

• 1 file x (thumbnail + iPhone + Mac) = 3 file copies

Traditional Architecture

Centralized Storage (NAS, SAN, DAS etc.)

HT TP

IO IO IO IO

Traditional Architecture

Centralized Storage (NAS, SAN, DAS etc.)

HT TP – TOO MANY CONNECTIONS

IO IO IO IO

Traditional Architecture

Centralized Storage (NAS, SAN, DAS etc.)

HT TP

IO IO IO IOIO IOIO

Traditional Architecture

To o m u c h I O

HT TP

IO IO IOIO IOIOIO

Traditional Architecture

Centralized Storage (NAS, SAN, DAS etc.)

HT TP

IO IO IO IOIO IOIO

Cache

“There are only two hard things in Computer Science: cache invalidation and naming things”.

-- Tim Bray quoting Phil Karlton

Architecture goals

• Symmetric identical server nodes– Simplified management and scaling– Linear scaling out

• No functional / role servers– No single point of failure– No performance bottlenecks

• Multiple datacenters support– DRP support– Geo load distribution

Meet Prospero

• Distributed Web content storage system

• Full blown HTTP support

• Runs on low cost commodity hardware

• Adjustable file level replication controls redundancy policy for every content type

• Provides dynamic image manipulation

How do we do it?

Designed to fail

• Fallback for every operation– Geographical, machine, storage medium

• Write never fails– All files will reach their destination

• Journaling– Tracking all uploaded files

• Pending jobs – Guaranteed file distribution

How do we achieve this

• Control the input– define the only unified API

• Functional process isolation– every function deserves its own process by default– watchdogs– monitors– alerts

5.static 7.static3.static1.static

0.static

00-1f

2.static

20-3f

6.static

60-7f

4.static

40-5f

HTTP HTTP HTTP

HTTP HTTP HTTP

get 37D815B5.jpg Go to 37 range servers Fallback if not found

Fallback Example

Node Architecture

Output – Front• lighttpd forkHTTP handler• Dynamic image processor• lighttpd forkVan Gogh• customCross Datacenter Distributer• customLocal Datacenter Distributer• Tornado web framework

applicationSupervisor HTTP handler

Input – Supervisor

Real Life

It’s all about performance

• Non blocking IO, readiness notification (epoll)• Asynchronous file IO (AIO)• Zero copy (sendfile)• Memory maps• Inter-process binary protocols• UNIX socket• Minimize dynamic memory allocation• lighttpd memory footprint: 50MB

Lessons learnt

• Be symmetric• Control the input• Design to failure• Performance matters again• Simple is hard but a must