Easy access to open stack object storage

Easy Access to OpenStack Object StorageJuan J. Martínez

@reidrachttp://www.usebox.net/jjm/

Merry OpenStack, LondonDec 2013

What is Swift?

Object Storage

● Not a real file system● RESTful HTTP API● Blob and metadata (name,

content type, date-time, size, MD5)

● Support for pseudo-hierarchical directories

Real File System

● open, read, write, seek, close● chdir, mkdir, rmdir● remove, rename● chmod, chown, stat, lock● etc

Open Source software for creating redundant, scalable object storage using clusters of standardized servers to store petabytes of accessible data.

CUSTOMERS

What we have

What we sell

What customers want

Solution: Proxies

Back in 2011...● Building a product based on OpenStack Object Storage● Started working with Bexar● Went live with Diablo

● Not many end-user tools using OpenStack Object Storage API(actually, it was Rackspace Cloud Files™)

● Customers won’t start building things straight away● Internal tools WIP (snapshot VM, re-image from snapshot)● It was hard to explain the product*:

What is Amazon S3? I want Dropbox!

* disclaimer: I’m a developer

Proxies: FTPftp-cloudfs

● Personal project of Chmouel Boudjnah (now @ https://github.com/cloudfs/ftp-cloudfs)● Open Source● Very cool, “almost working” (Jan 2011)● fs.py abstraction:

Swift API -> FTP

Memset started contributing to make it “production ready”:● Support for OpenStack Object Storage (was Cloud Files only)● Missing bits in fs.py, more tests, introduced some new bugs● Fixed performance problems: memcache all the things!● Finally migrated from python-cloudfiles to python-swiftclient (kept the name, it’s cool)● Keystone auth, resume downloads, listing +10k files, large file support (>5GB)

Proxies: SFTPsftp-cloudfs

● Open Source sponsored by Memset Hosting Ltdhttps://github.com/Memset/sftpcloudfs

● ftp-cloudfs’ fs.py + paramiko (ssh2 protocol for Python)● Secure: client -> SSH -> sftp-cloudfs -> SSL -> swift● Not as performant as FTP (1.6 MB/s vs pyftpdlib 4.2 MB/s), CPU intensive● Only password based authentication● Partial SCP support contributed by Koert van der Veer● + everything supported by ftp-cloudfs

Bigger than the API● +2 TB / mo traffic (in > out)● Small VMs running ftp-cloudfs, sftp-cloudfs and shared cache (memcache)● Load-balanced with LVS● Easy to scale (add more VMs!)● Use cases: backups, CMS plugins (CDN support), remote storage, ???● FTP > SFTP > API

Useful Open Source projects:● Numbers from PyPI

ftp-cloudfs: 1895 downloads last monthsftp-cloudfs: 1107 downloads last month

● Patches from several contributors (including hosting companies)● “Stable” and almost feature complete, a new release every now and then

Questions?Juan J. Martínez

@reidrachttp://www.usebox.net/jjm/

Merry OpenStack, LondonDec 2013

Easy Access to OpenStack Object Storage