FolderShare: Building a data sharing cloud on Drupal 8 for ... · • Project website ( Drupal 5 : 50,000+ pages) • SeedMe2 cloud service ( Drupal 7: 150,000+ pages) • SeedMe2

AmitChourasia,DavidNadeau&MichaelNormanSanDiegoSupercomputerCenter,UCSanDiego

Projectwebsite:dibbs.seedme.org

Projectcode:dibbs.seedme.org/downloads

Trialwebsite:sandbox.seedme.org

FolderShare:BuildingadatasharingcloudonDrupal8forresearchers

Aboutme•  VisualizaKonscholar/evangelist•  Lecturer/Instructor•  Interests

–  HighperformancecompuKng–  Datastewardship–  Computergraphics

SanDiegoSupercomputerCenter@UCSanDiegoDrupalusersince~2006|version4.7.4•  Personalwebsite(Drupal4-8:150pages)•  Projectwebsite(Drupal5:50,000+pages)•  SeedMe2cloudservice(Drupal7:150,000+pages)•  SeedMe2pla@orm(Drupal8:AnEcipate1M+items)

Seeking:VeryproficientDrupal8PHPprogrammers

Desired:DeepknowledgeofDrupal8core

PresentaKonOverview1.  Background&moKvaKon2.  Architecture3.  SeedMe2pla\orm

FolderShare:Virtualfilesystem»  EnKtydatamodel&accesscontrol»  Filemanagement&security»  ViewsintegraKon»  UI&Commandplugins»  Fileforma_ers»  Webservices

SmalldatamoduleandAPI–  VisualizaKon

4.  Targetusers/Usecases5.  Screenshots6.   Demo

SeedMe1a.k.a.StreamencodeexploreanddisseminateMyexperiments•  BasedonDrupal7•  InproducKonasPla\ormasaService(PaaS)•  Videoencodingwasthemainfocus

SeedMe2:DatasharingbuildingblocksEvoluKonoftheoriginalSeedMeproject(Completerewrite)•  BasedonDrupal8•  IncorporatesuserfeedbackfromoriginalSeedMeproject•  BuiltfordistribuKonandextension•  Datasharinganddatamanagementisthemainfocus

SeedMeProject

SeedMe2’sfocus

EnablerapidaccesstodataConsumabledataCanbehandledbystockwebbrowser(Upload/Download,<2GBperfile)Displayableonmanydevices(PhonetoPC)

Datamanagementstumblingblocks

Transfer Storage CollaboraKon AutomaKonAccesscontrol

PresentaKonDiscovery

ByLexiLarson

Datamanagementstumblingblocks

Transfer Storage CollaboraKon AutomaKonAccesscontrol

Issuesduetocontentdispersion

DescripKoninsomeone’smindDataintheCloud Discussiononemails

ThreeD’s:Data,DescripEon,Discussion

FilesystembasedsoluEons

RelatedsoluKons

ContentmanagementsystembasedsoluEonsHubZeroFigShare

MiddlewareGlobusIRodsNEWT

SoYwarerepositoriesGitHubSVNCVS

FilehosEngClouddrives(Dropbox,etc)WebDAV

ToolsSCPFTP

LimitaEonsofexisEngsoluEons Lackextensibility Lacksupportforrichcontent(descripKon,discussion,etc…) Lackindependentdevelopersupport Lack3wayinteracKonviawebbrowser&commandline&API Resourcerestricted

Workflow

Update as desired

Description

CreateProject

Sharing

Add foldersView

SearchDownload

Web browser, Command line, REST or App

Sign In

1 2

3

4

Upload files

Users

Drupal8

Architecture

Modules•  Virtualfilesystem

•  Accesscontrol•  Hierarchicalstorage•  Commandplugins•  UIanddisplay•  Search/index•  Webservices

•  Fileforma_ers•  QuickVisualizaKon

Webserver(Apache+PHP)

Database(MySQL)

Webbrowser

RESTclients

Commandline

Smalldata(PHPlibrary)

Drupal8ContributedModules

e.g.Federated

AuthenEcaEonviaOIDCmodule

ProjectcontribuKons

SeedMePla\ormEcosystemDrupal(ContentManagementSystem)•  Widelyusedinindustry,academiaandgovernment(thirdmostpopularCMSonwebajerWordpress&Joomla)•  Modulararchitecturewithlargeecosystem(over1,000contributedmodules)•  Largedeveloper&supportcommunity(4,000contributorstocore+thousandsmore)•  Securityadvisoryandupdatesforcoreandstablecontributed

moduleseverymonth•  VersaKledeploymentopKons(personalhosKng,insKtuKonalhosKng,cloudhosKng)

FolderSharemodule•  Requireddependencies(12-AllinDrupalcore)

–  DateKme–  Field–  File–  Image–  Link–  Media–  OpKons–  Text

•  OpEonaldependencies(3-AllinDrupalcore)–  Comment–  RESTfulwebservices

•  HTTPbasicauthenKcaKon•  SerializaKon

–  Search

–  Help–  Filter–  System–  User–  Views

•  Coremodulesrecommended–  Texteditor–  FieldUI(maybe)–  ViewsUI(maybe)

•  Contributedmodulesrecommended–  Realname–  RESTUI(maybe)–  SmallData(forquickvisualizaEonofCSV,JSONfiles)

FolderSharemodule

•  Virtualfilesystem(fieldable):–  EnKtytype&API– Accesscontrols– Usagetracking–  Views,displays,breadcrumbs,forms–  Pluginsforfieldforma_ers,search,views,acKons,andqueueworkers

•  Configurablebysites–  e.g.Keywords,comments,flags,DOIs

•  Extensiblebydevelopers

Files&folders

•  Childrenpointtoparents– ParentIDsenablefastqueriesforallchildrenofafolder

– RootIDsenablefastqueriesforaccesscontrolsandbreadcrumbs

parenKdpointstoimmediateparent

rooKdpointstotopfolder

AbstractedphysicalstorageFoldersexistinthedatabaseEveryuploadedfilehasaFileID(sequenEal)FileIDsay1Bitpa_ern0000000000000001Storedas0000/0000/0000/0001FileIDsay100Bitpa_ern0000000001100100Storedas0000/0000/0110/0100

Accesscontrols

•  Drupalaccount-based•  Permissions+accesscontrollistontopfolders– Listofusersthatcanviewandauthor

•  TopfoldercontrolsenKrehierarchy– SimplerthandesktopOSes– Similartofilesharingservices– Fasttocheckaccess

Filestorage

•  Foldersonlyexistindatabase•  Filesdescribedindatabase&storedondisk

•  Diskdirectory!==folderhierarchy– Be_erforsecurityandloadbalancing– Fileshavegeneratednames

•  Avoidscharactersetandnamelengthlimits

– Fileshavenoextensions•  AvoidsaccidentalserverexecuKonof“.php”,etc.

Views

•  Listpersonal,public,andsharedfiles&folders– Pages&embeddedviewsinfolderpages

•  Integrateddesktop-likeUI– Selectfilesandfolders– Thenchoosemenucommand

•  ThreeUIvariants:– NoscripKng– ScripKngbutnoAJAX– ScripKngwithAJAX

Plugins

•  Fieldforma_ers– Foldernames,enKtyreferences,MIME-typeicons

•  Search–  Indexandpresentresults

•  Queueworker– Updatefolderhierarchysizesinbackground

•  AcKons&customcommands– MenuUIitemstoadd,delete,etc

TotallinesofphpcodeNode:25,639(Drupalcore8.5.0)Foldershare:50,156(Alphaversion)

0

20,000

40,000

60,000

80,000

100,000

120,000

140,000

v1 v2 v3 v4 v5 alpha

Line

sofcod

e

Release

Foldersharecodemain

PHP

Docs

0

2000

4000

6000

8000

10000

12000

v1 v2 v3 v4 v5 alpha

Line

sofcod

e

Release

Foldersharecodemisc

JS

YML

CSS

TWIG

TXT

Codetrivia

FoldershareAPIDocumentaKon

SmallDataAPI&Module

•  Structureddataparsers&writers– Tables,trees,andgraphs–  JSON,CSV,TSV,TXT,etc.

•  Fieldforma_ers– Light-weightvisualizaKon– Lineplots,barcharts,piecharts,etc.

SmallData

FORADMINISTRATORS

FolderShareconfiguraEonAdminmenuStructure>FolderShare

FolderShareconfiguraEonFieldsManagefields,forms&display

ConfiguraKonlocatedinadminmenuStructure>FolderShare

FolderShareconfiguraEonFilesStoragelocaKon&uploadrestricKons

FolderShareconfiguraEonInterfaceCommandplugins

FolderShareconfiguraEonListsManagelisKngoffileandfolders

FolderShareconfiguraEonSearch(opEonal)

FolderShareconfiguraEonSecurityManagesharingcapabiliKes

FolderShareconfiguraEonWebservices(opEonal)ManageRESTcapabiliKes

FolderShareRESTseengsRequiresRESTUIcontributedmoduleManageRESToperaKons

FolderShareUsageAdminmenuReports>FolderShareusage

FolderSharepermissions

FORUSERS

Topfoldersownedbyyou

Topfolderssharedwithyou

Publictopfolders

Menu

Theselistsdisplaytoplevelfolders

FoldersmayhaveadescripKon

MenuopKons–withnoselecKon

MenuopKons–withselecKon

Sharingformtorestrictaccess

SortablelisKngoffilesandfolders.

Differentusers

Breadcrumbsshowspath

EveryfolderandfilemayaddadescripKon.(Forma_edtextfieldakaBodyfieldinDrupal’sNode)

AddcustomsfieldssuchasCommentstoFolderShare.

TheFolderShareenEtyisfieldable.

MenuopKonschangeonselecKon

ViewsubfolderBreadcrumbsshowspath

VisualizaKonscanbeswitchedinteracKvelytodifferentcharttypes

QuickvisualizaKonofCSV&JSONfiles

SamplecommandlineinteracKonfoldershare --helpfoldershare --host http://demo.seedme.org --user dave --password ’cliRocks!'

help

SamplecommandlineinteracKonfoldershare --helpfoldershare --host http://demo.seedme.org --user dave --password ’cliRocks!'

help

ls --help ls / ls -l "/Classification Collection" ls -l "/Classification Collection/Preliminary Results"

mkdir --help mkdir "/test" mkdir "/test/data"

upload --help upload "plots/villi.png" "/test" upload "plots/composite.png" "/test” Upload "sample-small-data/OpenGL mesh memory use.csv" "/test/data" Upload "sample-small-data/Image classification breakdown schedule.json" "/test/data"

download --help download "/test" "/Users/amit/downloads"

rm --help rm -rf "/test”

SeedMe2–Targetusers/Usecases•  Researchers

–  CollaboraKonhuborpersonaldashboard•  Projectrepositories

–  IncludeprojectspecificcustomizaKon(e.g.taxonomy,keywords)•  Developers

–  IntegrateyourscienKficapplicaKon•  Sciencegateways

–  Datasharing–  Datapublishing–  Dataescrowservice

•  CyberInfrastructureproviders–  OfferSeedMepla\ormtoyouruserbase/communiKes

Realusagebyphysicists

Deployment/serviceopKonsDIY-Runowninstance(Yourbranding+yourdomain)•  Onyourownhardware•  CondohardwareProvider/vendorrunsaninstance•  YourinsKtuKon•  CIcenters•  Commercialvendors

Asacloudservice(Ourbranding&ourdomain)•  dibbs.seedme.org

Explorerswelcome(webbrowserneeded)Trialwebsite:sandbox.seedme.org

Tryityourself!

DevelopersinvitedDownloadcode:h_ps://dibbs.seedme.org/downloads

Team

AmitChourasia,DavidNadeau,DmitryMishin&MichaelNormanSanDiegoSupercomputerCenter|UniversityofCaliforniaSanDiego

SeedMe

AcknowledgementsAllusersandapplicaKonintegratorsfortheirvaluablefeedback

NaEonalScienceFoundaEon

ThismaterialisbaseduponworksupportedbytheNaKonalScienceFoundaKonunderGrantNo.ACI-1235505andACI-1443083

"Anyopinions,findings,andconclusionsorrecommendaKonsexpressedinthismaterialarethoseoftheauthor(s)anddonotnecessarilyreflecttheviewsoftheNaKonalScienceFoundaKon."

Talk to usamit sdsc.edu

•  Keen to learn about potential FolderShare uses cases in your work•  Seeking: Very proficient Drupal 8 PHP programmers Desired: Deep knowledge of Drupal 8 core •  Single sign-on: Checkout http://www.cilogon.org/drupal

Projectwebsite:dibbs.seedme.org

Trialwebsite:sandbox.seedme.org

MyquesEons1.  Howtodoaccesscontrolcheckonimagestyles?2.  HowtosavestateinformaKonincustomDrupalfileforma_ers?