ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов

MagnetoDB Key-Value Storageservice for OpenStack

Mirantis, 2014

Serge.Kovaleffgmail.comlinkedin.comfacebook.com

@

[email protected]

Odessa, GeeksLab 4/12/2014@

isviridov at FreeNode

WII FM

WII F

WII FM

What the heck is OpenStack?

What the heck is OpenStack?

to be continued ...

Is it Big?

Community

1227 active contributors in IceHousemore than 166 organizations

Communication

44 active official IRC channels2887 e-mails in March

Is it Big?

Is it about data?

CERN- 3 PB per custer- 50000 + 35000 core

PayPal 80000 VMs (according to Forbes)

OpenStack Big Data

OpenStack Sahara - Elastic Hadoop clusters provisioning and management on OpenStack and elastic data processing (on-demand Hadoop job workflow)

OpenStack MagentoDB - key-value storage service

What is MagnetoDB?

MagnetoDB - OpenSource Amazon DynamoDB API implementation for OpenStack, the key-value database service for storing any amount of data with seamless scalability and predictable performance.

MagnetoDB key features ● Easy-to-integrate REST-like API (AWS SDK, boto

clients)● OpenStack standards following REST API● Schemaless, non-relational table-based model● Put/get/query/scan item operations● Eventual and strong consistency reads● Local Secondary indexes● Batch read/write operations● Designed to handle any amount of data and any level

of request traffic● Seamless throughput and storage scaling● Fault tolerance

MagnetoDB use cases

● High request traffic level applications

● Time series data

● Queries are distributed across the data

● Any other applications

● Migration of workloads between AWS and

OpenStack

Architecture highlights

● Layered architecture

● Horizontally scalable

● Pluggable database backend

● Uses native OpenStack infrastructure

○ OpenStack HEAT for provisioning

○ OpenStack Neutron LoadBalancer

○ Integration with OpenStack Keystone planned

Overall architecture

Integration with OpenStack services

Current status

○ 2.0.2 version released

○ Open source, Apache 2 license

○ Launchpad homepage: https://launchpad.net/magnetodb

○ Code published on github: https://github.

com/stackforge/magnetodb

○ Deployable to Vanilla OpenStack

○ Compatible with existing DynamoDB clients

MagnetoDB Pilot Features

● Table CRUD API○ ListTables○ DescribeTable○ CreateTable○ DeleteTable

● Item CRUD API○ PutItem○ UpdateItem○ DeleteItem○ GetItem

● Data querying API○ Query○ Scan

● Error Handling API

MagentoDB use-case: dynamodb-geo

“Geo Library for Amazon DynamoDB allows to easily create and query geospatial data”

Next steps● General

○ Move the development to OpenStack Infra - done○ Integration with OpenStack Keystone - done

● MagnetoDB API, following OpenStack tenets - in progress● DynamoDB API

○ Atomic counters○ Asynchronous table creation○ Atomic conditional write operations○ Batch operations - in progress

● Other features○ Management API○ Quota management

Summary

● The 2.0.2 implementation is available on GitHub in public

repository

● MagnetoDB is compatible with AWS SDK and boto library

● It is now possible to run applications which use DynamoDB as

a storage on top of OpenStack using MagnetoDB

Links and Resources

○ Wiki: https://wiki.openstack.org/wiki/MagnetoDB

○ Launchpad: https://launchpad.net/magnetodb/

○ Source code: https://github.com/stackforge/magnetodb

○ IRC: #magnetodb @ FreeNode

○ AWS DynamoDB http://aws.amazon.com/dynamodb/

○ dynamodb-geo https://github.com/awslabs/dynamodb-geo

Q&A

Got Questions?