LEVERAGING BTRFS, LINUX AND OPEN SOURCE - SNIA · OVERVIEW Developing with BTRFS Server side API's with Django and DRF ZMQ for data replication Django and ZMQ for asynchronous APIs

LEVERAGING BTRFS, LINUX AND OPENSOURCE

Suman Chakravartula / rockstor.com @rockstorinc

SDC 2015

PROJECT BRIEF

PROJECT PRINCIPLESOpen SourceCommunity focusedHardware agnosticLinux/BTRFS basedSimple to install and manageUse boring technology whenever possibleContributor friendly

OVERVIEWDeveloping with BTRFSServer side API's with Django and DRFZMQ for data replicationDjango and ZMQ for asynchronous APIsBackbone.jsGevents, SocketIO and websocketsDocker

HIGH LEVEL ARCHITECTURE

BTRFSVolume Management features

mix and match drivesadd and remove driveschange raid profies

BTRFSCopy on Write features

SubvolumesSnapshotsClonesSend/Receive Snapshots

BTRFSOther cool features

Quota groupsCompressionDeduplicationSSD awareness

DEVELOPING WITH BTRFSbtrfs-progs user space utilitiesCutting edgeNot developer friendly yet.

BTRFS LAYOUT

OTHER OS LEVEL TOOLSNetworkManagersmartmontoolsudevadmSystemTapinotifyand more...

APPLICATION ARCHITECTURE

DJANGO AND FRIENDSDjango MVCPostgresqlDjango Rest Framework

STATE SYNCHRONIZATIONSystem state is the factApplication state is the truthMinimal application stateEventual consistency

BACKBONE AND FRIENDSBackbone MVCJQueryD3

ZMQJob schedulingAsynchronous replication of SharesSystemTap probes (On hold)

DJANGO + ZMQ = ASYNCHRONOUSLong running tasks: btrfs balance, scrub etc..Task manager uses the RESTful API

DJANGO + ZMQ

DOCKER + ZMQHosting Docker based applicationsZMQ for app installs and updates.

ROCK-ON PLAY STORE

SYSTEMTAP AND ANALYTICSZMQ for scheduling stap scriptsProbe management from the UIReports using D3

ASYNCHRONOUS REPLICATIONBTRFS send/receiveLAN or WANReplicate to multiple destinations (planned)Encryption (planned)

PUB-SUB FOR DATA TRANSFER

PUSH-PULL FOR META DATA

RESPONSIVE UI AND EVENT DRIVENBACKEND

Subscribe, not pollEvent driven message passingPush notifications

GEVENT-SOCKETIO + WEBSOCKETS

SUMMARYdeveloping with BTRFSserver side API's with Django and DRFZMQ for data replicationDjango and ZMQ for asynchronous APIsBackbone.jsGevents, SocketIO and websocketsDocker

OPEN SOURCE SOFTWARE ANDCOMMUNITY FEEDBACK LOOP

Community focused developmentVision versus community wants

QUESTIONS OR COMMENTS?Important links

Website: Community: Code: Docs:

rockstor.comforum.rockstor.com

github.com/rockstor/rockstor-corerockstor.com/docs