Open Source based C-DAC Cloud Infrastructure & Servicesnisg.org/files/documents/UP1395901015.pdf · Open Source based C-DAC Cloud Infrastructure & Services 19 Mar 2014 NISG DeitY

Open Source based C-DAC Cloud Infrastructure &

Services

19 Mar 2014NISG DeitY Industrial Cloud Workshop

New Delhi

Centre for Development of Advanced Computing(C-DAC) Bangalore

19 Mar 2014 1

Dr. Prahlada Rao B.BJoint Director

C-DAC Cloud Services 2 EGov

C-DAC

• CDAC was launched as India’s initiative to Design & Develop HighPerformance Computers Indigenously in 1988

• Premier R&D organization under the Department of Electronics andInformation Technology (DeitY), Govt. of India

• C-DAC is engaged in R&D and Development of PARAM Series of High Performance Machines and also Specialized in :

– Cloud/Grid Computing Technologies– E-Governance Applications– Multilingual , Ubiquitous Computing

– Health Informatics– Embedded, Mobile computing– GIS…– Educational Research: ACTS

• Centre for Development of Advanced Computing : http://www.cdac.in

19 Mar 2014 2C-DAC Cloud Services 2 EGov

Presentation Plan

• About C-DAC• C-DAC Cloud Computing Activities

o C-DAC Cloud Infrastructureo Scientific Cloud Computingo SuMegha-Cloud Lab Kit

• Service Delivery Model• Cloud Disseminations• Conclusions

19 Mar 2014 C-DAC Cloud Services 2 EGov3

Supercomputing Evolution @ C-DAC

• Param 8000

• Param 9000

• Param 10000

• ParamPadma

• GARUDA Grid Comp

• Param Yuva II

• Cloud Comp

• PetaFlop Comp

1 Gbps NKN connectivity

2012

419 Mar 2014 C-DAC Cloud Services 2 EGov

70 + Partners 10000 CPUs –550TF

EGI, CHAIN-REDS

220TB Storage

1500 +Certificates

NKN


Focus Areas

• PAN C-DAC Cloud Infrastructure• C-DAC Cloud Stack • Enablement of multi-centric applications on C-DAC Cloud

• Cloud Interoperability


Highlights

• Infrastructure as a service • Cloud Stacks :

– Meghdooth– SuMegha (Scientific Cloud)

• Storage as a Service • Enablement of Multi centric Applications


HPC as a Service:

MPI/Hadoop Virtual Cluster(VC)Problem Solving Environments

Bangalore:HPC Applications

PAN C-DAC Cloud Initiatives: Overview

Chennai: Meghdooth Cloud Stack

Bangalore: SuMegha: Scientific Cloud Stack

Noida:HMS Deployment

Mohali:Courses

Kolkatta:NE Courses

TVM:HealthcareApplications

Mumbai:DisasterRecovery

Hyderabad:E-learning e-shikshak

Pune:Machine Translation MANTRA

Bangalore: PAN C-DAC Cloud Infrastructure

PSE NGS

PSE SFM

SaaSIaaS StaaS

19 Mar 2014 C-DAC Cloud Services 2 EGov 8

Pan C-DAC Cloud Computing

C-DAC local afforts to setup cloud with limited infrastructure. Cloud Stacks implemented

Based on open source. Eucalyptus, Nimbus, OpenStack as the core engine. Many extensions to meet production requirement.

Monitoring, metering and billing supported. Elasticity based on runtime parameters. Scientific Cloud Implemented (IaaS, SaaS, PaaS). Scientific Cloud Stack-SuMegha Released. Cloud Storage as a Service (CloudVault) Developed SuMegha_Cloud LabKit (to Build Privite Clouds) Developed MeghaSikshak Cloud based eLearning Tool Developed Currently Applications or Enabled on local infrastructure.

19 Mar 20149


Pan C-DAC Cloud Infrastructure• Physical Cloud infrastructure Set up • Host Cloud Middleware Stacks: Meghdoot, Sumegha-Scientific Cloud • Availability of the Cloud Infrastructure for applications.• Based on Meghdoot/SuMegha Cloud Stacks.• Physical Security, Power, Air Conditioning Available.• C-DAC Centres able to access resources for building and experimenting

with cloud applications.• Repository: for Application Snapshots, Tools, and common tools.• Provide tools for repositories, cloud management. • Disaster Recovery, backup for projects.


PAN C-DAC Cloud: Applications

Pan C-DAC Cloud Computing Supports: Pan C-DAC Cloud infrastructure Operating with: Meghdoot /

Sumegha Stacks. Cloud Enable Applications of Pan C-DAC Cloud Project

o Hospital Information System(HIS) C-DAC, Noidao HIS- Health Care Applications C-DAC, Thiruvananthapuramo Learning Management System (e-Sikshak) C-DAC, Hyderabado MANTRA (MAchiNe assisted TRAnslation tool) C-DAC Puneo Disaster Recovery as a Service (DRaaS) C-DAC Mumbai o eSanjeevani (A web based health care solution) C-DAC Mohali

1119 Mar 2014C-DAC Cloud Services 2 EGov

Private C-DAC network Public network (Internet) over VPN


Highlights

• Infrastructure as a service • Cloud Stacks :

– Meghdooth– SuMegha (Scientific Cloud)

• Storage as a Service • Enablement of Multi centric Applications


C-DAC Scientific Cloud


C-DAC Scientific CloudScientific Cloud is a model for on-demand access to a shared pool of HPC resources (e.g.,networks, servers, storage, applications) that can be easily provisioned as and when needed by the researchers and scientists.Benefits of Scientific Cloud• Easy access to large infrastructures• On demand access to HPC resources• Virtual ownership of cloud resources• Ease of deployment• Scientific Cloud provide an alternative for HPC outreach


SuMegha: Scientific Cloudwww.sumegha.in

Provides on demand access to Compute, Storage resources and enables parallel environments.Services Supported : Infrastructure as a service(IaaS) Storage as a service(StaaS) Software as a service(SaaS)

Stakeholders Faculty and Students from engineering collegesResearchers


SuMegha: Scientific Cloudwww.sumegha.in

• Cluster as an Infrastructure Service (IaaS)• Provisioning of on demand HPC clusters• Provisioning of Map reduce clusters to process large data sets • Support for MPI as a tightly-coupled applications category. • Design methodology for providing/supporting installation of MPI

libraries in multiple machines simultaneously• Storage as a Service (StaaS): CloudVault• SaaS:

– Job Submission Portal (C_JSP) , Cloud Access Portals– Monitoring tools

• PaaS: Program development and analysis Tools• Applications as Service (AaaS) PSEs

– PSE for Climate Modelling– PSE’s for Bio Informatics

• Types of applications supported– Sequential & Concurrent Applications– Data Parallel Applications - Map Reduce– Parallel Programs: OpenMP ,Pthreads, MPI

19 Mar 2014C-DAC Cloud Services 2

EGov17

SuMegha StackManagemen

tSCIENTIIFC APPLICATIONS

(Bioinformatics, Climate modeling)

Cloud Middleware (NIMBUS, Openstack Swift)

SLA & Policy Mana-gement

Monitoring

(NAGIOS+ Ganglia)

Accounting, Metering & Billing

Operating System (RHEL)Hypervisors (XEN)

AU

T

H

&

S

E

C

U

R

I

T

Y

)

SaaS(Job submission portal)

PaaS(HADOOP, MPI with High PerfIO)

IaaS (VM, Virtual Clusters , storage)

Interconnect ( Ethernet & infiniband)

Cluster Virtualization (Contextualization)

Scientific Cloud portal and Problem Solving Environments

Computational and Storage resources

18

StaaS(Storage as a Service)


SciCloud Service Offerings

19 Mar 2014


GUIIaaSPaaSSaaS

Cloud Portal( Selects the Resource)

Virtual Machine

(OS, Time, Type)

Virtual Cluster

(No. of nodes, type,Time)

HadoopCluster

(No. of nodes, Time)

MPI Cluster(No. of nodes, Time)

Cloud Vault(StaaS)

Web interface

Small

Medium

Large

PSEs forBioinformatics

Climate Modelling

Job Submission Portal

19

Infrastructure as a Service(IaaS)

• Accessible using Self service portalhttp://scientificcloud.cdacb.in/• Infrastructure as a Service :

Provisioning of virtual machines(VM) & virtual clusters (VC)– VM Specs

• Small, medium, large images• Access to cloud storage(cloud vault)• Access to additional elastic storage, through CIFS

gateway– VC Specs

• Upto 8 nodes in a cluster• Each node: Small, medium, Large• Type of cluster : MPI, HADOOP• Access to cloud storage(cloud vault)• Access to additional elastic storage, through CIFS

gateway

• Secure Access :Secure ssh based access to the VMs & VCs through internet using secure key

VM Size Details Size Detail

Size No. Of Virtual CPU RAM (GB)

Small 1 1

Medium 2 2

Large 4 4


Job Submission Portal for Virtual Clusters

• Web based interface to provide an easy & user friendly mechanism for job submission to MPI based virtual clusters.

• Enables users to run Sequential & Parallel jobs

Features•Job submission through Torque job scheduler.•Job monitoring to know the Status of submitted jobs•Output/Error files can be Viewed/Downloaded from the Portal•User Management on Virtual Cluster


Cloud Vault

End-User

Desktop ApplicationWeb Application

Mobile Application

Swift

Storage and Data Center

Physical Storage

OpenStack Object Storage

Cloud Vault is your personal hard drive in thecloud. Store any kind of file like Documents,Music, Videos, Photos, and Virtual MachineImages on CDAC Cloud Vault secure servers.


Cloud Vault provides access to storage space for storing and accessing data from anywhere, anytime

http://cloudvault.cdacb.in:8084

Features

– Easy access to Storage : Supports easy accessmechanisms such as Web interface, Linux clients

– File and directory operations : list, upload, download,delete

– Data Sharing: Enables users to share their data withthe other registered users of cloud vault in acontrolled way

– Interfacing with the Virtual Machines & VirtualClusters

– Reliability & Availability: Provides 99.5% up-timeguarantee and 3-way redundancy for data.

– Huge file support : Allows users to upload files fromrange of MBs, GBs (upto 40GB)

– Data isolation and security: Data of one user isisolated & secured from other

Storage as a Service : Cloud Vault


PSE for SFM• Web based problem

solving environment torun seasonal forecastmodel on MPI basedvirtual cluster provisionedby SuMegha

• Enables on linevisualization of the results

• Facility is provided to runGrADS commands fromthe browser, whichvisualize the necessaryforecast parameters.


PSE-SFM: JOB SUBMISSION:


PSE-SFM: JOB MONITORING


PSE-SFM: VISUALIZATION


PSE for NGS• Problem Solving Environment for Next Generation

Sequencing (PSE for NGS) • Salient Features:

– Executes the complete workflow Preprocessing(Quality of input data),Indexing, Alignment , post processing, Visualization

– Run-Time Logs for debugging and Directory Trees to navigate easily among the projects or different output files/directories.

– Huge Datasets uploading via Internet (More than 10GB supported).

– Common View window for visualisations and other textual outputs.

– Notification Centre for the user to know which step is going on at any given time and other information like which project is active and which is pending.


Next Generation Sequencing


SuMegha Cloud Kit • Packaged software kit to enable the automated deployment of

private scientific clouds on the hardware.• Suitable for academic institutes & research organizations who

want to setup cloud labs.Benefits Automated deployment of cloud setup for Infrastructure as a service Provides web based tools that supports easy and quick access to the

Virtual machines and Virtual HPC clusters Simple and quick installation of all the cloud components. Provides Pre-built OS images with complex parallel environments like MPI

and Map Reduce to cater the HPC needs. Provides Golden images with preloaded HPC applications like Seasonal

Forecast Model ,Cloud Next generation pipeline with the required visualization tools and job submission portal for easy parallel job submission

Enables the users to setup cloud lab without having expertise of system administration.


SuMegha Software Stack & Supported ImagesComponents : Hypervisor

Xen Cloud Middleware

Nimbus, Openstack Swift Storage

Glusterfs Portal

SuMegha portal Job Submission Portal Cloud Vault Portal

PSE PSE for SFM PSE for NGS Pipeline

Supported Images: Virtual Machines

CentOS (small, large, medium)

Job Submission Portal

Virtual Clusters MPICH enabled with job

submission portal HADOOP enabled

PSE enabled images SFM NGS Pipeline


Enabling Applications on Meghrajusing SuMegha Cloud Lab Kit


SuMegha Cloud Lab kitSophisticated Packaged software Kit to enable the automateddeployment of private clouds on the hardware.

Highlights– Open source components: Xen Hypervisor , Nimbus(cloud middleware) and GlusterFS– Cloud tools & Dashboard: Cloud Portal, Job Submission Portal and Problem Solving

Environment (PSE), Cloud monitoring.– Automated Installer: Installation and configuration of private cloud on the

recommended hardware to setup Cloud Lab without needing expert knowledge ofSystem Administration.

– Secured Access to VMs through SSH using private keys– Golden Images: VM’s of different sizes - small, medium, large with CentOS operating

system– Parallel Programming environments with MPI libraries and HADOOP framework.

Advantages• Provides a hand holding mechanism for moving existing applications & building new

applications on cloud• Lower cost to application developers :Application developer need not pay for the

cloud resources at the application development & testing phase• Enhanced cloud adoption in several domains


Phases of Cloud Application Enablement using SuMegha Lab kit

• Enabling Existing Application– Setup cloud testing environment using SuMegha lab kit auto

installer– Test application on local cloud setup using VMs– Deploy application on Meghraj Cloud

• Developing New Application– Developing Application on SuMegha Development Environment

(SDE) using Cloud APIs(openstack, eucalyptus,etc)– Deploy & Test Application on local setup– Deploy Application on Meghraj cloud


Elastic Framework for Cloud Applications

Elasticity Framework allows cloud resources (VMs, Storage) to dynamically scale-up or down as required by the application at run-time

Building Blocks• User defined application configurations.• Cloud monitoring.• Cloud Scalar.


TENANTAPP

STORE

SuMegha Elasticity

Framework

Monitoring Configuration

Deployment Configuration

and Scripts

Cloud API Production

Cloud A

Production Cloud B

Developer Private Cloud (Development Environment) SuMegha Lab

kit1

2

3

1

2

3

4

4


• Developer Of cloud Application :-1. Create his application 2. Provide Deployment configuration and scripts to the Elasticity

frame work3. Test on private cloud Created Using SuMegha Cloud Labkit4. List that in AppStore

• User of Cloud Application 1. Choose the cloud application and provide the monitoring

threshold listed in app store2. The deployment configuration will be taken from Elasticity

Framework.3. Setting the threshold in the motoring component of Elasticity

Framework.4. Deployed on the cloud chosen by user.


Steps to Cloud Application Enablement using SuMegha Lab kit

Points to Ponder

• Licensing for commercial cloud platforms supported by SuMegha

• Billing & Pricing based on cloud platform chosen by the users


Cloud Publications/Patents• Provisional patent on “A CLOUD STORAGE SERVICE FOR CLOUD COMPUTING”, Indian

Patent Office, Provisional Patent Application No. 1392/CHE/2013, Patent Filed on: 28 March 2013.

• Shivay Veer Sharma, Anil Bhargava, Payal Saluja, Prahlada Rao B. B., Accessing E-Infrastructures using CDAC Scientific Cloud (CSC) Services , 2nd IEEE CCEM 2013, Bangalore (Poster)

• Deepanshu S, Aman A, Ramesh Naidu Laveti, Arunachalam B, Vineeth Simon Arackal, Prahlada Rao B B; Problem Solving Environment for Seasonal Forecast Model on CDAC Scientific Cloud; ACC 2013, Bangalore

• AmanArora, Pavan Kumar, Vineeth Simon Arackal, Janaki Chintalapati, ArunachalamB,Prahlada Rao B B, Cloud NGS Tool: An Online Pipeline for Next Generation Sequencing Data; ACC 2013, Bangalore

• Vineeth Simon Arackal, AmanArora, Deepanshu Saxena, Arunachalam B, Prahlada Rao B B; SciInterface: A Web-Based Job Submission Mechanism for Scientific Cloud Computing; IEEE CCEM 2013, Bangalore

• Payal Saluja, Prahlada Rao BB, Ankit Mittal, Rameez Ahmad, C-DAC Scientific Cloud: On demand provisioning of HPC resources for Scientific Applications, paper published & presented at 18th International conference of parallel and distributed processing techniques and applications(PDPTA 2012) 6th-19th July 2012 at Las Vegas, Nevada, USA

• Prahlada Rao B. B, Payal Saluja, Neetu Sharma, Ankit Mittal, Shivay Veer Sharma, Cloud Computing for Internet of Things & Sensing Based Applications poster paper published and presented at 6th International conference on Sensing Technology (ICST 2012), 18-21st Dec 2012, Kolkata, India


Dissemination: Workshops/Trainings

Workshops Conducted: (3)o Indo_Taiwan International Workshop on Cloud Computing & Its Applications, 7-8th Nov 2013, Bangaloreo Prahlada Rao B. B. , National Workshop on Cloud Computing, conducted at CDAC Knowledge Park,

Bangalore, 21st May 2011 .(80+ participants from Industry, Academia)o National Work Shop on Virtualization, Feb 2011, C-DAC Bangalore ((70+ participants from Industry,

Academia)Tutorialso Prahlada Rao B.B., Payal Saluja, Vineeth Simon Arackle, Half-Day Tutorial “Open Source Platforms for

Scientific Clouds” CSI’s Annual Conference on Advances in Cloud Computing (ACC2012), 26th July 2012, Bangalore.India (60+ participants from Industry, Academia)

o Prahlada Rao, Vineeth Simon Arackal, Payal Saluja Tutorial on “Cloud Computing for HPC” C-DAC’s National Conference on Parallel Computing Technologies (PARCOMPTECH 2013) 21-23 Feb , Bangalore, India

o Prahlada Rao B.B , Tutorial on “Scientific Clouds” International Conference on Advances in Computing, Communications and Informatics (ICACCI-2013) , Mysore

o Prahlada Rao B.B , Invited Talk on “Trends in Cloud Computing” International conference on Advances in Cloud Computing (ACC 2013), Bangalore, 19-20 Oct 2013, India

o Cloud Computing Talks: 15+ Invited /KeyNote Talks in Intl Conferences/ Workshops, and TEQiPprograms


Dissemination: Technical Talks Given• Prahlada Rao B.B., “Resent Trends in Cloud Computing” Technology day Talk C-DAC Bangalore

10th May 2013.• Prahlada Rao B.B., “CDAC Cloud Initiatives” Workshop on National Cloud Innovation Council, C-

DAC Bangalore 29th April 2013.• Prahlada Rao B.B., “HPC for BigData” International Workshop on Data Analytics and Applications

(IWDAA) 2013, BITS, Goa Campus, 1st Mar2013.• Prahlada Rao B.B, Payal Saluja, Neetu Sharma, Ankit Mittal, Shivay Veer Sharma, “Cloud Computing

for IOT & Sensing Based Applicationss”, in IEEE International Conference on Sensing Technologies, ICST 2012, Kollkata, India, Dec 2012.

• Prahlada Rao B.B., Tutorial On “Cloud Computing”, International Conference on Trends in Knowledge and Information Dynamics, ICTK 2012, organized by ISI, NIMHANS Conv Centre, Bangalore, India, 10 July 2012.(150 participants from Industry, Academia)

• Payal Saluja, Prahlada Rao, Ankit Mittal, Rameez Ahmad, “CDAC Scientific Cloud: On Demand Provisioning of Resources for Scientific Applications”, in Proceedings of 18th International Conference PDPTA 2012, Las vegas, Nevada, USA, 16-19 July 2012.

• Dr. Prahlada Rao BB, Cloud Computing for Telemedicine, National Workshop on Telemedicine Today & Tomorrow, C-DAC Mohali, An Invited Talk, 6th July 2012.

• Vineeth Simon Arackal, Arunachalam B, Payal Saluja, Prahlada Rao B B, Co-existence of Cloud and Grid: A Case Study in Service Oriented Grid GARUDA. The International Conference on Information Technology- New Generations (ITNG 2011), Las Vegas, USA. 11-13April 2011.

• Vineeth Simon Arackel, “ Coexistence of Grid and Cloud” at "Grid To Cloud (G2C)- Challenges andApplications" Workshop at VIT, Vellore, 2nd April 2011.


Conclusions


• Middleware Development for Cloud, Grid & HPC:C-DAC Strength• C-DAC Cloud Activities

o Open Source based C-DAC Cloud Testbedo Cloud Applications of National Importanceo Scientific Cloud Computing Researcho SuMegha- Cloud LabKit for Privite Cloud Buildingo Cloud Interoperability Researcho Elastic Cloud Applications Development

• C-DAC Founding member of IntercloudTestbed-IEEE • Active member CCICI (Cloud Computing Innovation Council for India)

Thank You
