YALMS: A Group Drive API for Cloud-Based Classrooms

YALMS Basics

M.Zhanikeev, H.Koide -- maratishe@gmail.com -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 2/26...

YALMS Basics

.YALMS stands for .....

.YetAnotherLectureManagementSystem

• next generation digital classroom 02

• several robots: video annotation, content packing, multimedia storage, etc.

• can run in clouds and cloud-based classrooms◦ specifically, in VM-per-student environments

• knowledge extraction and social components

02 "Homepage of the YALMS software presented in this paper" https://github.com/maratishe/yalms (2013)

YALMS : The Big PictureYALMS: Intro

YALMS.annotation

2013.07・ET研

2013.09・ET研

YALMS.streaming2014.01・ET研

YALMS.hiperf2013.10・ITE-MMS研

YALMS.cloud2013.11・IN研

YALMS.format

YALMS.done!

DASH Adaptive HTML5

PDF video

VM sync groupdrive

SSDmulticore

slices

stream recordreplay storage

2014.01・LOIS研

content

YALMS.groupdrive : The Specs

YALMS.groupdrive : Overall Idea

Master Copy

Slave Copy

YALMS.groupdrive : The Specs

1. realtime async. asymm. P2P sync -- but notdistributed storage

2. fully distributed, can work without a master

3. smart and adaptive, fit for any environment◦ remote master, fully distributed, throughput-limited, clouds, etc.

4. efficient with large files -- a sub-file data grain

5. API with cross-platform implementation◦ Linux, Windows, VMs

YALMS.groupdrive : Existing (bad)Solutions

Bad Solution : BitTorrent

• a very bad solution• needs central registry -- Tracker

• good only with many peers, too much overhead with few◦ rarest-first policy is not fit for the problem

• good for download, but what about upload?

Bad Solution : DSync

• a paper in 05 -- the closest rival

• aggregation frommultiple sources while optimizing load• not really distributed -- single point prospective -- o2m or o2o

• no practical considerations -- no measurement, ordered lists, etc.

05 H.Pucha+3 "Adaptive File Transfers for Diverse Environments" USENIX Annual Technical Conference (2008)

Bad Solution : P2P Streaming

Content Provider (origin)

P2P Network

• good for videostreaming 04

• again, bad forbi-directionalsyncs

04 myself+0 "Multi-Source Stream Aggregation in the Cloud" Wiley Book on Content Delivery (ACDN) (2013)

06 myself+0 "Extremely Scalable and Low Demand Live P2P Streaming based on Variable Bitrate" CANDAR (2013)

Bad Solution : Group Virtualization

Service Provider

Virtual Wireless User

Wireless User

Application Service Service traffic • specific to

remoteservices 07

• does not workwell with fullylocal syncs

• not specific tosyncs

07 myself "Virtual Wireless User: A Practical Design for Parallel MultiConnect Using WiFi Direct..." Mobiquitous (2013)

Proposal and Design

Proposal : Fundamentals

.Objective..

.API + Implementation of the proposed group drive

• standard RESTful API• implemented in PHP

• cross-platform

• adaptive to e2e throughput -- response time

• avoid congesting the master

Design : PUSH/PULL Basics• simple: 1 PULL and X PUSH rounds, X peers in each round -- X is scale

.The PULL..

• regularly ask peers for changes, new content comes in replies• do not push the newly recieved content

.The PUSH..

.• push toX peers in X rounds for quick distribution

.Conflicts...... ignored for now, treat like in GoogleDrive, etc.

Design : The MFFxR Unit• basic list unit for PUSHes and PULLs -- ordered by decreasingthroughput

• Random Peer is basically for measurement/discovery• scale is a global parameter

List:M List:MF

List:MFF List:MFF+R

Design : Peer Discovery Roadmap

• dynamically incorporate new peers

REP( peers)

REQ() Done Update

Master

Design : PULL Roadmap• a roadmap rather than state machine

Align peers

MFFF… + R

REP( null)

REQ( state)

REP( block, epoch)

Scale (s)

Error, failure, timeout

Replace (if new block)

Notify

Design : PUSH Roadmap

peers MFFF… + R

REP( OK)

REQ(block)

REP( Collision)

Scale (s)

I already have it! (double up)

If M or R

Pick next from MFFF… for 2nd round

Notify

Experiments

• one 10Gb file

• initial state: only masterhas the file

• try various environments

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5log( time)

YALMS.groupdrive Local Only (10 peers and scale 2)YALMS.groupdrive Remote Master (10 peers and scale 2)Traditional (1 client)Traditional (10 clients)

Implementation and API

Implementation : Basic Unit of Info.Exchange

• standard unit of data exchange -- JSON -- part of API 03• some parts are not used -- blob only used for PUSHes/PULLs

Datatype Parametername Comments

string relpath Relative to top folderstring filenamelong blockpos Position of the head byte of the block in file

int blocksize Fixed for each sessiondouble mtime Epoch time of last update for that block at Master

string abspath Path for this block at Master. Not used by Slave Copies.blob content Blob: binary string. Used when needed.

03 "Homepage of the YALMS.groupdrive project" https://github.com/maratishe/yalms.groupdrive (2013)

Implementation : Actual API

• per-peer, MFFxR is outside of the API (part of implementation)

• state is a flat JSON of items (keys) and MD5 digests (values) 03

• APIs are standard RESTfull HTTP request/reply routines

• API1: pull(state)−−reply(dataunit)• API2: push(dataunit)−−reply(status)

Implementation : Software Design• prefers Linux but will work on Windows using portable TinyWeb 03

WebServer

wgetReqs

RepsWeb

Server

wgetReqs

That’s all, thank you ...

[01] (2013)The enPiT Projecthttp://www.enpit.jp

[02] (2013)Homepage of the YALMS software presented in this paperhttps://github.com/maratishe/yalms

[03] (2013)Homepage of the YALMS.groupdrive projecthttps://github.com/maratishe/yalms.groupdrive

[04] myself+0 (2013)Multi-Source Stream Aggregation in the CloudWiley Book on Content Delivery (ACDN)

[05] H.Pucha+3 (2008)Adaptive File Transfers for Diverse EnvironmentsUSENIX Annual Technical Conference

[06] myself+0 (2013)Extremely Scalable and Low Demand Live P2P Streaming based on Variable BitrateCANDAR

[07] myself (2013)Virtual Wireless User: A Practical Design for Parallel MultiConnect Using WiFiDirect...Mobiquitous

[08] (2013)Dropbox Core APIhttps://www.dropbox.com/developers/core

YALMS: A Group Drive API for Cloud-Based Classrooms

source stream

virtual wireless

group drive

adaptive file

yalms software

content delivery

based classrooms

bad solution

Technology

API 610, 11th Ed. S4) Truflo Pumps, Inc.€¦ · API 610,.....

GSPLF Magnetic Drive Pumps Low Flow API 685 - 2nd Edition

Transforming today’s classrooms for tomorrow’s...

PERL API Reference Guide for CiscoWorks Network … · PERL...

GSPV Range Vertical Inline Sealless Magnetic Drive Pumps API...

Drive new business opportunities through naturally RESTful.....

in K–12 Classrooms - ASCD: Professional Learning ... |...

Google Drive/Docs for Classrooms - a short introduction

Использование Google Drive API для...

Developing a communication supporting classrooms ... · PDF....

Creating a Successful API Program to Drive Digital...

JULY 2016 - API-Delta Chapter · API DELTA CHAPTER—JULY.....

GSP Range Sealless Metallic Magnetic Drive Pumps API · PDF....

Itinerant Pedagogical Advisors (API) Monitoring Note 1 ·.....

Intel® Rack Scale Design PSME API Specification® Rack...

API 610, 11th Ed....API 610, 11th Ed. Process Pump (VS6) API...