Cloud Architectures & Platforms CWIN17, September 27 th 2017 Presenter : SHUVADEEP DUTTA, Insights Platform, Insights & Data
Jan 21, 2018
Cloud Architectures & PlatformsCWIN17, September 27th 2017
Presenter : SHUVADEEP DUTTA, Insights Platform, Insights & Data
2Copyright © Capgemini 2017 All Rights Reserved 2Copyright © Capgemini 2017. All Rights Reserved
AGENDA
Platform Architecture Views
Cloud Platform Security
Platform Logging
Platform DevOps
3Copyright © Capgemini 2017 All Rights Reserved 3Copyright © Capgemini 2017. All Rights Reserved
Future Looking Data Lake – Conceptual Architecture
4Copyright © Capgemini 2017 All Rights Reserved 4Copyright © Capgemini 2017. All Rights Reserved
Big Data and Analytics Platform Logical Architecture – Cloud & Technology Agnostic View
5Copyright © Capgemini 2017 All Rights Reserved 5Copyright © Capgemini 2017. All Rights Reserved
Big Data and Analytics Cloud Platform – MS Azure / AWS Native Services
6Copyright © Capgemini 2017 All Rights Reserved 6Copyright © Capgemini 2017. All Rights Reserved
Big Data and Analytics Cloud Platform – Custom Built Stack (Opensource Hortonworks)
7Copyright © Capgemini 2017 All Rights Reserved 7Copyright © Capgemini 2017. All Rights Reserved
Cloud Deployment Considerations
Data topology, governance & security capabilities need to be evaluated taking into consideration the key considerations to define a target state hybrid cloud platform architecture
Focus Area Consideration
Data Location Location of data storage in alignment with regional regulatory compliance directives
Analytics Use Cases Type of analytical workloads going to be executed on the data repository located at a specific region / country
Data Access, Authentication and Authorization Mechanism of data and underlying platform resource access based on specific user role, user location, time of access etc.
Network Latency Volume of data transfer over network ensuring appropriate bandwidth and SLAs being met
Data Ingestion Type of data ingestion mechanisms used to transfer data from on-premise to cloud and vice-versa
Security and Regulatory Compliance Implementation of controls and mechanisms to meet platform security (perimeter, data and application), regional regulatory compliance directives such as PCI, SOX etc.
Platform Governance Platform wide metadata management, audit logging, master and reference data management capabilities; considerations
for bottom-up / top-down /Hybrid approaches: Data Catalog driven data discovery and knowledge sharing (bottom-up) vs. Data Governance Council (Data Stewardship) driven information asset management and knowledge sharing (Top-down)
Platform Infrastructure Considerations for High Availability, Fault Tolerant and Disaster Recovery capabilities
Considerations from platform performance and scalability based on specific Big Data Analytics workload use cases
Considerations for Platform resource management and orchestration
Data and Application Portability Considerations for data and application portability across different platform environments e.g. On-premise, Private, Public cloud
The information contained in this presentation is proprietary.
Copyright © 2016 Capgemini. All rights reserved.
Rightshore® is a trademark belonging to Capgemini.
www.capgemini.com
8Copyright © Capgemini 2017. All Rights Reserved
Capgemini’s DaaS (Data as a Service) Solution Framework
Meet all data and analytics management service needs from data ingestion, preparation, discovery, through till data analysis using opensource or commercial tools
Leverage client’s current investments and integrate with technology of client choice or extend / customize existing framework capabilities
10Copyright © Capgemini 2017 All Rights Reserved 10Copyright © Capgemini 2017. All Rights Reserved
Big Data and Analytics Platform – Cloud Security Framework
Physical Asset Security Management
Protection for physical assets and locations
including networks and data centers
Cloud Governance
Cloud specific security governance including directory
synchronization and geo locational
support
Information Asset Protection
Protection of data at rest or in transit
Governance & Compliance
Security governance, maintenance of
security policy, audit
and complianceThreats and Vulnerability Management
Management of vulnerabilities and
mitigations with
Network and endpoint protection
Incident Management
Management and responding to expected and
unexpected events
Identity and Access Management
Authentication of users and
management of identity
11Copyright © Capgemini 2017 All Rights Reserved 11Copyright © Capgemini 2017. All Rights Reserved
Cloud Security Capability Framework – Shared Responsibility Model
12Copyright © Capgemini 2017 All Rights Reserved 12Copyright © Capgemini 2017. All Rights Reserved
AWS VPC Security Reference Architecture
VPC Security Architecture Scenarios
Scenario 1 : VPC with a Single Public Subnet only
• Instances run in a private, isolated section of the AWS cloud with direct access to
the Internet
• Network ACLs and security groups can be used to provide strict control over
inbound and outbound network traffic to EC2 instances
Scenario 2 : VPC with Public and Private Subnets and NAT
• In addition to public subnet, a private subnet is added whose instances are not
addressable from the Internet
• Instances in the private subnet can establish outbound connections to the Internet
via the public subnet using Network Address Translation (NAT Gateway or Instance)
Scenario 3 : VPC with Public and Private Subnets and hardware VPN access
• IPsec VPN connection between Amazon VPC and customer data center, while also
providing direct access to the Internet for public subnet instances in Amazon VPC
• VPN appliance on customer corporate data center side
Scenario 4 : VPC with Private Subnet only and hardware VPN access
• Instances run in a private, isolated section of the AWS cloud with a private subnet
whose instances are not addressable from the Internet
• Private subnet is connected to customer data center via an IPsec VPN tunnel
13Copyright © Capgemini 2017 All Rights Reserved 13Copyright © Capgemini 2017. All Rights Reserved
AWS VPC Security Reference Architecture – Security Groups
VPC security groups to firewall each EC2 instance
Each instance can be in up to 5 security groups
Separate security groups for applications and management
Security groups are stateful with ingress and egress rules
Max. 50 rules per security group
VPC Router will allow any subnet to route to another in VPC
Network Access Control Lists are used to restrict internal VPC traffic
Elastic load balancers are used to distribute traffic between instances
Elastic load balancers are also placed in security groups
Platform security can scale up and down with solution; instances can be added into security groups during launch time
Use NAT instances to provide internet connectivity for Private Subnets; allow backend servers to route to AWS APIs – Ex. storing logs in S3
bucket or using DynamoDB, SES
Access AWS API endpoints through the Internet Gateway like S3, SES, DynamoDB, SNS etc.
14Copyright © Capgemini 2017 All Rights Reserved 14Copyright © Capgemini 2017. All Rights Reserved
AWS Identity and Access Management
Securely control access to AWS services and resources
Fine grained control of user permissions, resources and actions
Support for RunInstances
Multi Factor Authentication – Hardware token or Smartphone Apps
Segregation of roles using IAM
AWS Account Owner (Master)
Network Management
SecurityManagement
ServerManagement
StorageManagement
15Copyright © Capgemini 2017 All Rights Reserved 15Copyright © Capgemini 2017. All Rights Reserved
AWS Identity and Access Management Role Based Security
16Copyright © Capgemini 2017 All Rights Reserved 16Copyright © Capgemini 2017. All Rights Reserved
AWS Identity and Access Management using Enterprise’ existing Active Directory
Flow
1
2
3
4
The enterprise user accesses the identity broker
application
The identity broker application authenticates the users
against the corporate identity store
The identity broker application has permissions to
access the AWS Security Token Service (STS) to
request temporary security credentials
Enterprise users get a temporary URL that gives them
access to the AWS APIs or the Management Console
AWS Identity Federation with Temporary Security Credentials
17Copyright © Capgemini 2017 All Rights Reserved 17Copyright © Capgemini 2017. All Rights Reserved
AWS Data Storage Security – Capabilities
AWS S3 Capabilities –
Access controls at bucket and object level
Cryptographic capabilities such as SSL for data in motion, Server/Client side encryption, MD5 checksums
AWS Redshift Capabilities –
Full disk encryption; CloudHSM to store keys
Back-up access logs to S3 for analysis
Security groups and VPC for deployment; data loading using SSL from S3 with restricted access to S3
SSL encryption for data accessed over internet
AWS RDS Capabilities –
Restricted access to RDS instances using Security groups and IAM permissions
Data encryption (Data at rest and in motion)
Automatic patching for minor updates
AWS DynamoDB Capabilities –
Fine grained security access to columns and rows using IAM role and access policies
AWS EBS Volume Capabilities –
Option to use own encryption or commercial solutions Ex. Windows BitLocker or Linux LUKS for encrypted volumes and TrueCrypt for containers; Commercial : Safenet Protect-V,
Trend Secure Cloud etc.
18Copyright © Capgemini 2017 All Rights Reserved 18Copyright © Capgemini 2017. All Rights Reserved
Securing AWS Applications Process Flow
20Copyright © Capgemini 2017 All Rights Reserved 20Copyright © Capgemini 2017. All Rights Reserved
Platform Monitoring – Centralized Troubleshooting, Security, Audit and Monitoring
21Copyright © Capgemini 2017 All Rights Reserved 21Copyright © Capgemini 2017. All Rights Reserved
Platform Monitoring – Solution Options
Shippers Queue Logstash ElasticsearchElasticsearch
Kibana
Log Parsing Indexing & Curation Dashboards & ReportsRedis, RabbirMQ, KafkaSyslog, Rsyslog, Logstash, Fluentd etc.
JSONEvents Extract
Option 1 – Custom Built using ELK stack (Elasticsearch, Logstash and Kibana)
Option 2 – SaaS Solution – Loggly (alternate leading tool : Sumologic)
Shippers
Syslog, Rsyslog, Logstash, Fluentd, cloud plug-ins etc.
Application Framework
INGEST PROCESS INDEX
Search & Other
Services
• Cloud-based SaaS for easy central log collection, aggregation, management
• Easy set-up• Dynamic parsing - Real-time, JSON support, parsing/tagging,
self-documenting• Regular Expressions based• Dashboards, pre-configured and customizable, shareable• Anomaly Detection• Alerts that can be sent to HipChat, Slack, PagerDuty, HTTP
endpoints, others• JIRA Software integration, point-and click ticket creation
without leaving Loggly
23Copyright © Capgemini 2017 All Rights Reserved 23Copyright © Capgemini 2017. All Rights Reserved
DevOps Reference Architecture for Big Data Analytics Workloads
The information contained in this presentation is proprietary.
Copyright © 2015 Capgemini. All rights reserved.
Rightshore® is a trademark belonging to Capgemini.
www.capgemini.com
About Capgemini
With more than 145,000 people in over 40 countries, Capgemini is one of the world's foremost providers of
consulting, technology and outsourcing services. The Group reported 2014 global revenues of EUR 10.573
billion.
Together with its clients, Capgemini creates and delivers business and technology solutions that fit their
needs and drive the results they want. A deeply multicultural organization, Capgemini has developed its
own way of working, the Collaborative Business Experience™, and draws on Rightshore®, its worldwide
delivery model
Learn more about us at www.capgemini.com.