Page 1
AmitChourasia,DavidNadeau&MichaelNormanSanDiegoSupercomputerCenter,UCSanDiego
Projectwebsite:dibbs.seedme.org
Projectcode:dibbs.seedme.org/downloads
Trialwebsite:sandbox.seedme.org
FolderShare:BuildingadatasharingcloudonDrupal8forresearchers
Page 2
Aboutme• VisualizaKonscholar/evangelist• Lecturer/Instructor• Interests
– HighperformancecompuKng– Datastewardship– Computergraphics
SanDiegoSupercomputerCenter@UCSanDiegoDrupalusersince~2006|version4.7.4• Personalwebsite(Drupal4-8:150pages)• Projectwebsite(Drupal5:50,000+pages)• SeedMe2cloudservice(Drupal7:150,000+pages)• SeedMe2pla@orm(Drupal8:AnEcipate1M+items)
Seeking:VeryproficientDrupal8PHPprogrammers
Desired:DeepknowledgeofDrupal8core
Page 3
PresentaKonOverview1. Background&moKvaKon2. Architecture3. SeedMe2pla\orm
FolderShare:Virtualfilesystem» EnKtydatamodel&accesscontrol» Filemanagement&security» ViewsintegraKon» UI&Commandplugins» Fileforma_ers» Webservices
SmalldatamoduleandAPI– VisualizaKon
4. Targetusers/Usecases5. Screenshots6. Demo
Page 4
SeedMe1a.k.a.StreamencodeexploreanddisseminateMyexperiments• BasedonDrupal7• InproducKonasPla\ormasaService(PaaS)• Videoencodingwasthemainfocus
SeedMe2:DatasharingbuildingblocksEvoluKonoftheoriginalSeedMeproject(Completerewrite)• BasedonDrupal8• IncorporatesuserfeedbackfromoriginalSeedMeproject• BuiltfordistribuKonandextension• Datasharinganddatamanagementisthemainfocus
SeedMeProject
Page 5
SeedMe2’sfocus
EnablerapidaccesstodataConsumabledataCanbehandledbystockwebbrowser(Upload/Download,<2GBperfile)Displayableonmanydevices(PhonetoPC)
Page 6
Datamanagementstumblingblocks
Transfer Storage CollaboraKon AutomaKonAccesscontrol
PresentaKonDiscovery
ByLexiLarson
Page 7
Datamanagementstumblingblocks
Transfer Storage CollaboraKon AutomaKonAccesscontrol
Issuesduetocontentdispersion
DescripKoninsomeone’smindDataintheCloud Discussiononemails
ThreeD’s:Data,DescripEon,Discussion
Page 8
FilesystembasedsoluEons
RelatedsoluKons
ContentmanagementsystembasedsoluEonsHubZeroFigShare
MiddlewareGlobusIRodsNEWT
SoYwarerepositoriesGitHubSVNCVS
FilehosEngClouddrives(Dropbox,etc)WebDAV
ToolsSCPFTP
LimitaEonsofexisEngsoluEons Lackextensibility Lacksupportforrichcontent(descripKon,discussion,etc…) Lackindependentdevelopersupport Lack3wayinteracKonviawebbrowser&commandline&API Resourcerestricted
Page 9
Workflow
Update as desired
Description
CreateProject
Sharing
Add foldersView
SearchDownload
Web browser, Command line, REST or App
Sign In
1 2
3
4
Upload files
Page 10
Users
Drupal8
Architecture
Modules• Virtualfilesystem
• Accesscontrol• Hierarchicalstorage• Commandplugins• UIanddisplay• Search/index• Webservices
• Fileforma_ers• QuickVisualizaKon
Webserver(Apache+PHP)
Database(MySQL)
Webbrowser
RESTclients
Commandline
Smalldata(PHPlibrary)
Drupal8ContributedModules
e.g.Federated
AuthenEcaEonviaOIDCmodule
ProjectcontribuKons
Page 11
SeedMePla\ormEcosystemDrupal(ContentManagementSystem)• Widelyusedinindustry,academiaandgovernment(thirdmostpopularCMSonwebajerWordpress&Joomla)• Modulararchitecturewithlargeecosystem(over1,000contributedmodules)• Largedeveloper&supportcommunity(4,000contributorstocore+thousandsmore)• Securityadvisoryandupdatesforcoreandstablecontributed
moduleseverymonth• VersaKledeploymentopKons(personalhosKng,insKtuKonalhosKng,cloudhosKng)
Page 12
FolderSharemodule• Requireddependencies(12-AllinDrupalcore)
– DateKme– Field– File– Image– Link– Media– OpKons– Text
• OpEonaldependencies(3-AllinDrupalcore)– Comment– RESTfulwebservices
• HTTPbasicauthenKcaKon• SerializaKon
– Search
– Help– Filter– System– User– Views
• Coremodulesrecommended– Texteditor– FieldUI(maybe)– ViewsUI(maybe)
• Contributedmodulesrecommended– Realname– RESTUI(maybe)– SmallData(forquickvisualizaEonofCSV,JSONfiles)
Page 13
FolderSharemodule
• Virtualfilesystem(fieldable):– EnKtytype&API– Accesscontrols– Usagetracking– Views,displays,breadcrumbs,forms– Pluginsforfieldforma_ers,search,views,acKons,andqueueworkers
• Configurablebysites– e.g.Keywords,comments,flags,DOIs
• Extensiblebydevelopers
Page 14
Files&folders
• Childrenpointtoparents– ParentIDsenablefastqueriesforallchildrenofafolder
– RootIDsenablefastqueriesforaccesscontrolsandbreadcrumbs
parenKdpointstoimmediateparent
rooKdpointstotopfolder
Page 15
AbstractedphysicalstorageFoldersexistinthedatabaseEveryuploadedfilehasaFileID(sequenEal)FileIDsay1Bitpa_ern0000000000000001Storedas0000/0000/0000/0001FileIDsay100Bitpa_ern0000000001100100Storedas0000/0000/0110/0100
Page 16
Accesscontrols
• Drupalaccount-based• Permissions+accesscontrollistontopfolders– Listofusersthatcanviewandauthor
• TopfoldercontrolsenKrehierarchy– SimplerthandesktopOSes– Similartofilesharingservices– Fasttocheckaccess
Page 17
Filestorage
• Foldersonlyexistindatabase• Filesdescribedindatabase&storedondisk
• Diskdirectory!==folderhierarchy– Be_erforsecurityandloadbalancing– Fileshavegeneratednames
• Avoidscharactersetandnamelengthlimits
– Fileshavenoextensions• AvoidsaccidentalserverexecuKonof“.php”,etc.
Page 18
Views
• Listpersonal,public,andsharedfiles&folders– Pages&embeddedviewsinfolderpages
• Integrateddesktop-likeUI– Selectfilesandfolders– Thenchoosemenucommand
• ThreeUIvariants:– NoscripKng– ScripKngbutnoAJAX– ScripKngwithAJAX
Page 19
Plugins
• Fieldforma_ers– Foldernames,enKtyreferences,MIME-typeicons
• Search– Indexandpresentresults
• Queueworker– Updatefolderhierarchysizesinbackground
• AcKons&customcommands– MenuUIitemstoadd,delete,etc
Page 20
TotallinesofphpcodeNode:25,639(Drupalcore8.5.0)Foldershare:50,156(Alphaversion)
0
20,000
40,000
60,000
80,000
100,000
120,000
140,000
v1 v2 v3 v4 v5 alpha
Line
sofcod
e
Release
Foldersharecodemain
PHP
Docs
0
2000
4000
6000
8000
10000
12000
v1 v2 v3 v4 v5 alpha
Line
sofcod
e
Release
Foldersharecodemisc
JS
YML
CSS
TWIG
TXT
Codetrivia
Page 21
FoldershareAPIDocumentaKon
Page 22
SmallDataAPI&Module
• Structureddataparsers&writers– Tables,trees,andgraphs– JSON,CSV,TSV,TXT,etc.
• Fieldforma_ers– Light-weightvisualizaKon– Lineplots,barcharts,piecharts,etc.
SmallData
Page 23
FORADMINISTRATORS
FolderShareconfiguraEonAdminmenuStructure>FolderShare
Page 24
FolderShareconfiguraEonFieldsManagefields,forms&display
ConfiguraKonlocatedinadminmenuStructure>FolderShare
Page 25
FolderShareconfiguraEonFilesStoragelocaKon&uploadrestricKons
Page 26
FolderShareconfiguraEonInterfaceCommandplugins
Page 27
FolderShareconfiguraEonListsManagelisKngoffileandfolders
Page 28
FolderShareconfiguraEonSearch(opEonal)
Page 29
FolderShareconfiguraEonSecurityManagesharingcapabiliKes
Page 30
FolderShareconfiguraEonWebservices(opEonal)ManageRESTcapabiliKes
Page 31
FolderShareRESTseengsRequiresRESTUIcontributedmoduleManageRESToperaKons
Page 32
FolderShareUsageAdminmenuReports>FolderShareusage
Page 33
FolderSharepermissions
Page 35
Topfoldersownedbyyou
Topfolderssharedwithyou
Publictopfolders
Menu
Theselistsdisplaytoplevelfolders
FoldersmayhaveadescripKon
Page 36
MenuopKons–withnoselecKon
Page 37
MenuopKons–withselecKon
Page 38
Sharingformtorestrictaccess
Page 39
SortablelisKngoffilesandfolders.
Differentusers
Page 40
Breadcrumbsshowspath
Page 41
EveryfolderandfilemayaddadescripKon.(Forma_edtextfieldakaBodyfieldinDrupal’sNode)
Page 42
AddcustomsfieldssuchasCommentstoFolderShare.
TheFolderShareenEtyisfieldable.
Page 43
MenuopKonschangeonselecKon
Page 44
ViewsubfolderBreadcrumbsshowspath
Page 45
VisualizaKonscanbeswitchedinteracKvelytodifferentcharttypes
QuickvisualizaKonofCSV&JSONfiles
Page 46
SamplecommandlineinteracKonfoldershare --helpfoldershare --host http://demo.seedme.org --user dave --password ’cliRocks!'
help
Page 47
SamplecommandlineinteracKonfoldershare --helpfoldershare --host http://demo.seedme.org --user dave --password ’cliRocks!'
help
ls --help ls / ls -l "/Classification Collection" ls -l "/Classification Collection/Preliminary Results"
mkdir --help mkdir "/test" mkdir "/test/data"
upload --help upload "plots/villi.png" "/test" upload "plots/composite.png" "/test” Upload "sample-small-data/OpenGL mesh memory use.csv" "/test/data" Upload "sample-small-data/Image classification breakdown schedule.json" "/test/data"
download --help download "/test" "/Users/amit/downloads"
rm --help rm -rf "/test”
Page 48
SeedMe2–Targetusers/Usecases• Researchers
– CollaboraKonhuborpersonaldashboard• Projectrepositories
– IncludeprojectspecificcustomizaKon(e.g.taxonomy,keywords)• Developers
– IntegrateyourscienKficapplicaKon• Sciencegateways
– Datasharing– Datapublishing– Dataescrowservice
• CyberInfrastructureproviders– OfferSeedMepla\ormtoyouruserbase/communiKes
Page 49
Realusagebyphysicists
Page 50
Deployment/serviceopKonsDIY-Runowninstance(Yourbranding+yourdomain)• Onyourownhardware• CondohardwareProvider/vendorrunsaninstance• YourinsKtuKon• CIcenters• Commercialvendors
Asacloudservice(Ourbranding&ourdomain)• dibbs.seedme.org
Page 51
Explorerswelcome(webbrowserneeded)Trialwebsite:sandbox.seedme.org
Tryityourself!
DevelopersinvitedDownloadcode:h_ps://dibbs.seedme.org/downloads
Page 52
Team
AmitChourasia,DavidNadeau,DmitryMishin&MichaelNormanSanDiegoSupercomputerCenter|UniversityofCaliforniaSanDiego
SeedMe
AcknowledgementsAllusersandapplicaKonintegratorsfortheirvaluablefeedback
NaEonalScienceFoundaEon
ThismaterialisbaseduponworksupportedbytheNaKonalScienceFoundaKonunderGrantNo.ACI-1235505andACI-1443083
"Anyopinions,findings,andconclusionsorrecommendaKonsexpressedinthismaterialarethoseoftheauthor(s)anddonotnecessarilyreflecttheviewsoftheNaKonalScienceFoundaKon."
Page 53
Talk to usamit sdsc.edu
• Keen to learn about potential FolderShare uses cases in your work• Seeking: Very proficient Drupal 8 PHP programmers Desired: Deep knowledge of Drupal 8 core • Single sign-on: Checkout http://www.cilogon.org/drupal
Projectwebsite:dibbs.seedme.org
Trialwebsite:sandbox.seedme.org
MyquesEons1. Howtodoaccesscontrolcheckonimagestyles?2. HowtosavestateinformaKonincustomDrupalfileforma_ers?