YALMS: A Group Drive API for Cloud-Based Classrooms

.

YALMS Basics

M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 2/26...

2/26

.

YALMS Basics

.YALMS stands for .....

.YetAnotherLectureManagementSystem

• next generation digital classroom 02

• several robots: video annotation, content packing, multimedia storage, etc.

• can run in clouds and cloud-based classrooms◦ specifically, in VM-per-student environments

• knowledge extraction and social components

02 "Homepage of the YALMS software presented in this paper" https://github.com/maratishe/yalms (2013)


3/26

.

YALMS : The Big PictureYALMS: Intro

YALMS.annotation

2013.07・ET研

2013.09・ET研

YALMS.streaming2014.01・ET研

YALMS.hiperf2013.10・ITE-MMS研

YALMS.cloud2013.11・IN研

YALMS.format

YALMS.done!

DASH Adaptive HTML5

PDF video

VM sync groupdrive

SSDmulticore

slices

stream recordreplay storage

2014.01・LOIS研

content


4/26

.

YALMS.groupdrive : The Specs


5/26

.

YALMS.groupdrive : Overall Idea

Master Copy

Slave Copy

Slave Copy

Slave Copy


6/26

.

YALMS.groupdrive : The Specs

1. realtime async. asymm. P2P sync -- but notdistributed storage

2. fully distributed, can work without a master

3. smart and adaptive, fit for any environment◦ remote master, fully distributed, throughput-limited, clouds, etc.

4. efficient with large files -- a sub-file data grain

5. API with cross-platform implementation◦ Linux, Windows, VMs


7/26

.

YALMS.groupdrive : Existing (bad)Solutions


8/26

.

Bad Solution : BitTorrent

• a very bad solution• needs central registry -- Tracker

• good only with many peers, too much overhead with few◦ rarest-first policy is not fit for the problem

• good for download, but what about upload?


9/26

.

Bad Solution : DSync

• a paper in 05 -- the closest rival

• aggregation frommultiple sources while optimizing load• not really distributed -- single point prospective -- o2m or o2o

• no practical considerations -- no measurement, ordered lists, etc.

05 H.Pucha+3 "Adaptive File Transfers for Diverse Environments" USENIX Annual Technical Conference (2008)


10/26

.

Bad Solution : P2P Streaming

Share

…

Content Provider (origin)

P2P Network

• good for videostreaming 04

06

• again, bad forbi-directionalsyncs

04 myself+0 "Multi-Source Stream Aggregation in the Cloud" Wiley Book on Content Delivery (ACDN) (2013)

06 myself+0 "Extremely Scalable and Low Demand Live P2P Streaming based on Variable Bitrate" CANDAR (2013)


11/26

.

Bad Solution : Group Virtualization

Service Provider

Virtual Wireless User

Wireless User

Wireless User

Wireless User

Application Service Service traffic • specific to

remoteservices 07

• does not workwell with fullylocal syncs

• not specific tosyncs

07 myself "Virtual Wireless User: A Practical Design for Parallel MultiConnect Using WiFi Direct..." Mobiquitous (2013)


12/26

.

Proposal and Design


13/26

.

Proposal : Fundamentals

.Objective..

.API + Implementation of the proposed group drive

• standard RESTful API• implemented in PHP

• cross-platform

• adaptive to e2e throughput -- response time

• avoid congesting the master


14/26

.

Design : PUSH/PULL Basics• simple: 1 PULL and X PUSH rounds, X peers in each round -- X is scale

.The PULL..

.

• regularly ask peers for changes, new content comes in replies• do not push the newly recieved content

.The PUSH..

.• push toX peers in X rounds for quick distribution

.Conflicts...... ignored for now, treat like in GoogleDrive, etc.


15/26

.

Design : The MFFxR Unit• basic list unit for PUSHes and PULLs -- ordered by decreasingthroughput

• Random Peer is basically for measurement/discovery• scale is a global parameter

Self

List:M List:MF

List:MFF List:MFF+R


16/26

.

Design : Peer Discovery Roadmap

• dynamically incorporate new peers

REP( peers)

REQ() Done Update

peers

Master


17/26

.

Design : PULL Roadmap• a roadmap rather than state machine

Align peers

MFFF… + R

REP( null)

REQ( state)

REP( block, epoch)

Scale (s)

Error, failure, timeout

Replace (if new block)

Done

Notify


18/26

.

Design : PUSH Roadmap

Align

peers MFFF… + R

REP( OK)

REQ(block)

REP( Collision)

Scale (s)

Done

I already have it! (double up)

If M or R

Pick next from MFFF… for 2nd round

Notify


19/26

.

Experiments


20/26

.

Experiments

• one 10Gb file

• initial state: only masterhas the file

• try various environments

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5log( time)

0

0.2

0.4

0.6

0.8

1

Peer

com

plet

ion

ratio

YALMS.groupdrive Local Only (10 peers and scale 2)YALMS.groupdrive Remote Master (10 peers and scale 2)Traditional (1 client)Traditional (10 clients)


21/26

.

Implementation and API


22/26

.

Implementation : Basic Unit of Info.Exchange

• standard unit of data exchange -- JSON -- part of API 03• some parts are not used -- blob only used for PUSHes/PULLs

Datatype Parametername Comments

string relpath Relative to top folderstring filenamelong blockpos Position of the head byte of the block in file

int blocksize Fixed for each sessiondouble mtime Epoch time of last update for that block at Master

string abspath Path for this block at Master. Not used by Slave Copies.blob content Blob: binary string. Used when needed.

03 "Homepage of the YALMS.groupdrive project" https://github.com/maratishe/yalms.groupdrive (2013)


23/26

.

Implementation : Actual API

• per-peer, MFFxR is outside of the API (part of implementation)

• state is a flat JSON of items (keys) and MD5 digests (values) 03

• APIs are standard RESTfull HTTP request/reply routines

• API1: pull(state)−−reply(dataunit)• API2: push(dataunit)−−reply(status)



24/26

.

Implementation : Software Design• prefers Linux but will work on Windows using portable TinyWeb 03

OS,

Sys

tem

OS,

Sys

tem

WebServer

wgetReqs

RepsWeb

Server

wgetReqs

Reps



25/26

.

That’s all, thank you ...


26/26

.

[01] (2013)The enPiT Projecthttp://www.enpit.jp

[02] (2013)Homepage of the YALMS software presented in this paperhttps://github.com/maratishe/yalms

[03] (2013)Homepage of the YALMS.groupdrive projecthttps://github.com/maratishe/yalms.groupdrive

[04] myself+0 (2013)Multi-Source Stream Aggregation in the CloudWiley Book on Content Delivery (ACDN)

[05] H.Pucha+3 (2008)Adaptive File Transfers for Diverse EnvironmentsUSENIX Annual Technical Conference


26/26

.

[06] myself+0 (2013)Extremely Scalable and Low Demand Live P2P Streaming based on Variable BitrateCANDAR

[07] myself (2013)Virtual Wireless User: A Practical Design for Parallel MultiConnect Using WiFiDirect...Mobiquitous

[08] (2013)Dropbox Core APIhttps://www.dropbox.com/developers/core


26/26

YALMS: A Group Drive API for Cloud-Based Classrooms

Technology

source stream

virtual wireless

group drive

adaptive file

yalms software

content delivery

based classrooms

bad solution