Building Blocks of Mayan: Componentizing the eScience Workflows Through Software-Defined Service Composition Pradeeban Kathiravelu*, Tihana Galinac Grbac + , Luís Veiga* *INESC-ID Lisboa & Instituto Superior Técnico, Universidade de Lisboa, Portugal + University of Rijeka, Croatia 23rd IEEE International Conference on Web Services (ICWS 2016) June 27 - July 2, 2016, San Francisco, USA. Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 1 / 28
28
Embed
Building Blocks of Mayan: Componentizing the eScience Workflows Through Software-Defined Service Composition
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Building Blocks of Mayan:Componentizing the eScience Workflows Through
*INESC-ID Lisboa & Instituto Superior Técnico, Universidade de Lisboa, Portugal+University of Rijeka, Croatia
23rd IEEE International Conference on Web Services (ICWS 2016)June 27 - July 2, 2016, San Francisco, USA.
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 1 / 28
Overview
1 Introduction
2 Mayan Approach
3 Evaluation
4 Conclusion
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 2 / 28
Introduction
Introduction
eScience workflowsComputation-intensive.Execute on highly distributed networks.
Complex service compositions aggregating web servicesTo automate scientific and enterprise business processes.
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 3 / 28
Introduction
Motivation
Increasing demand forData quality and Quality of Service (QoS).Better Performance (Shorter completion times and higher throughput).Geo-distribution (workflows and compositions).
Need for additional control and flexibility.Exploring Trade-off: Efficiency vs. Accuracy.Leveraging Software-Defined Approaches (from SDN).
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 4 / 28
Multi-Tenanted Environments.Isolation Guarantees.Differentiated Quality of Service (QoS).
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 5 / 28
Introduction
Contributions
Support for,Adaptive execution of scientific workflows.Flexible service composition.Reliable large-scale service composition.Efficient selection of service instances.
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 6 / 28
Mayan Approach
Mayan
Extensible SDN approach for cloud-scale service composition
Driven by:Loose couplingMessage-oriented Middleware (MOM)Availability of a logically centralized control plane
Leveraging OpenDaylight SDN controller as the core.Modular, as OSGi bundles.Additional advanced features.
State of executions and transactions stored in the controller distributeddata tree.Clustered and federated deployments.
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 7 / 28
Mayan Approach
Services as the building blocks of Mayan
Prototypical Example:
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 8 / 28
Mayan Approach
Software-Defined Service Composition
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 9 / 28
Mayan Approach
Multiple Implementations and Deployments of a Service
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 10 / 28
Mayan Approach
Software-Defined Service Composition
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 11 / 28
Mayan Approach
Services as the building blocks of Mayan
Prototypical Example:
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 12 / 28
Mayan Approach
Too many requests on the fly?
Prototypical Example:
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 13 / 28
Mayan Approach
Alternative Deployment/Implementation
Prototypical Example:
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 14 / 28
Mayan Approach
Mayan Services Registry: Modelling Language
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 15 / 28
Web services frameworks.Apache Hadoop MapReduce.Hazelcast In-Memory Data Grid.
OpenDaylight SDN Controller.
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 21 / 28
Evaluation
Preliminary Assessments
A workflow performing distributed data cleaning andconsolidation [PK 2015].
A distributed web service composition.vs.Mayan approach with the extended SDN architecture.
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 22 / 28
Evaluation
Speedup and Horizontal Scalability
No negative scalability in larger distributions.100% more positive scalability for larger deployments.
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 23 / 28
Evaluation
Memory consumption in the Service Nodes
Initial coordination overhead in memory for smaller deployments.Minimal overhead for larger deployments.
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 24 / 28
Conclusion
Related Work
MapReduce for efficient service compositions [SD 2014].
But we should not forget the registry!
Palantir: SDN for MapReduce performance with the network proximitydata [ZY 2014].A multi-domain deployment of SDN for communitynetworks [PK 2016].
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 25 / 28
Conclusion
Conclusion
SDN-based approach that enables large scale flexibility withperformance
Components in eScience workflows as building blocks of a distributedplatform.Service composition with web services and distributed executionframeworks.Multi-tenanted multi-domain executions.
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 26 / 28
Conclusion
Conclusion
SDN-based approach that enables large scale flexibility withperformance
Components in eScience workflows as building blocks of a distributedplatform.Service composition with web services and distributed executionframeworks.Multi-tenanted multi-domain executions.
Future WorkMayan should further be deployed and evaluated on physicalgeo-distributed nodes.Extending Software-defined service composition for the networkfunctions in service composition of middlebox actions.
Load balancing.Firewalls.
Adapting as an NFV framework for service function chaining.
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 27 / 28
Conclusion
References
PK 2015 Kathiravelu, Pradeeban, Helena Galhardas, and Luís Veiga. "∂u∂u Multi-Tenanted Framework: DistributedNear Duplicate Detection for Big Data." On the Move to Meaningful Internet Systems: OTM 2015Conferences. Springer International Publishing, 2015.
SD 2014 Deng, Shuiguang, et al. "Top-Automatic Service Composition: A Parallel Method for Large-Scale ServiceSets." Automation Science and Engineering, IEEE Transactions on 11.3 (2014): 891-905.
ZY 2014 Yu, Ze, et al. "Palantir: Reseizing network proximity in large-scale distributed computing frameworks usingsdn." 2014 IEEE 7th International Conference on Cloud Computing (CLOUD). IEEE, 2014.
PK 2016 Kathiravelu, Pradeeban, and Luıs Veiga. "CHIEF: Controller Farm for Clouds of Software-DefinedCommunity Networks." Software Defined Systems (SDS), 2016 IEEE International Symposium on. IEEE,2016.
Thank you!Questions?
Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 28 / 28