Cloudera & Deloitte Tech Summit€¦ · Prepare your AWS Environment – Scripted (VPC, Subnet, Security Groups) 17 Applying your Free Credit to your AWS Code 21 DEPLOY YOUR FULL

Cloudera & Deloitte Tech Summit

cloudera.com Con Page 1 of 54

Cloudera & Deloitte Tech Summit TABLE OF CONTENTS

ARCHITECTURE 3

PRE-REQUISITES 3 Enroll for AWS Free Tier Account 3 Download Automation Scripts from Github 4

SETUP YOUR AWS ENVIRONMENT 6 Log onto AWS Console 6 Choose your AWS Region 6 Create AWS IAM User 8 Create AWS Access Key 12 Import your AWS Key Pair 13 Prepare your AWS Environment – Scripted (VPC, Subnet, Security Groups) 17 Applying your Free Credit to your AWS Code 21

DEPLOY YOUR FULL EDH CLUSTER 23 Connect to your Altus Director Instance 23 Change your Altus Director Admin Password 24 Create an Environment 26 Create your Templates 28

Select your AWS Environment 28 Create your Cloudera Manager Template 31 Create your Cloudera Master Template 33 Create your Worker Template 35 Create your Edge Template 36

Deploy your EDH Cluster 38 Deploy your Cloudera Manager instance 38 Deploy your EDH Cluster 40

Checking all services are running 43

CLEANUP YOUR DEPLOYMENTS 47 Tear down your EDH Cluster 47

Troubleshooting 48 AWS Users with existing VPC, Subnet and Security-Group 48 Setup your SOCKS Proxy to secure all communications 51 Socks proxy 52 Make sure SOCKS Proxy is selected and running 52 Putty for Windows Laptops 53 Using Command Line Interface on your Terminal or Putty on Windows 53



ARCHITECTURE We will be deploying a Cloudera EDH cluster with 1 Master Node + 3 Worker Node + 1 Edge Node. Please note that this is a standard Cloudera Deployment for Pilots and POC For HA you will need 3 masters, 5 worker and 2 edge nodes

● 1 Master Nodes ● 3 Worker Nodes will support the 3 x HDFS Replication Factor. ● Usually Edge nodes are deployed by pair, to segregate Data Ingestion (input)

versus Data Visualization (output) flux.

PRE-REQUISITES

1.1 Enroll for AWS Free Tier Account Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides secure, resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers. (ref: https://aws.amazon.com/ec2/ ) If you do not have an AWS account, please enroll for a AWS Free Tier account on https://aws.amazon.com/free/. If you have an AWS account please skip over the next section and goto Download Automation Scripts from GitHub

Note: If you have been provided a company AWS account, please make sure you either have a AWS Access / Secret Key, either check with your company AWS administrator to have the permissions to create one (see section 1.5)


https://aws.amazon.com/ec2/

https://aws.amazon.com/free/


Once the AWS form is updated, you will be provided AWS User / Password for your first connection.

1.2 Download Automation Scripts from Github For the entire Hands On training, we will need first to gather some automation scripts on your local laptop.



Click on the URL: https://github.com/heibalvin/Cloudera. Then on the right panel click on Clone or Download and retrieve the ZIP file. Please unpack your ZIP file in your [HOME-DIRECTORY] (for ex: /Users/heibalvin/Github/Cloudera-master). for windows users use c:\Users\YOURUSERNAME\ You will find a Director directory which includes bootstrap scripts, as well as AWS CloudFormation scripts. Also, there will be a tech-summit private/public keys in the EMEA-Partner-Tech-Summit directory. You will need to change the permissions for these keys before being able to make use of them.


https://github.com/heibalvin/Cloudera


SETUP YOUR AWS ENVIRONMENT 1.3 Log onto AWS Console

You can now connect to the AWS Console on https://eu-west-1.console.aws.amazon.com with your newly received credentials.

1.4 Choose your AWS Region Amazon EC2 is hosted in multiple locations world-wide. These locations are composed of regions and Availability Zones. Each region is a separate geographic area. Each region has multiple, isolated locations known as Availability Zones. Amazon EC2 provides you the ability to place resources, such as instances, and data in multiple locations. Resources aren't replicated across regions unless you do so specifically. (ref: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html)


https://eu-west-1.console.aws.amazon.com/

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html


On the upper right corner of your AWS Console, please select an AWS Region. In our case, we will run the entire Hands-On on the EU (Ireland) Datacenter (eu-west-1). In order to be able to use the centos 7 AMI you need to have subscribed to it. The usage of that image costs nothing, but still you require to subscribe to it. Open a new browser tab and open https://aws.amazon.com/marketplace/pp/B00O7WM7QW. Click the orange “Continue to Subscribe” button.


https://aws.amazon.com/marketplace/pp/B00O7WM7QW


1.5 Create AWS IAM User

Note: Some accounts will come with default users, with your login, with Administrator permissions. If this is the case, you can skip this section.

Under Security, Identity & Compliance select IAM service. Then select Users from the menu on the left.



Click the Add user button. Select the options as shown in the screenshot below:


Cloudera & Deloitte Tech Summit Check Programmatic Access as well as AWS Management Console access. Indeed, both of these options are necessary for a user to connect to the AWS portal as well as use Cloud Formations industrialisation scripts.

Select Administrators group.



Click on Create User.



Note: Please keep this file secure, since you have all permissions on your account with this information.

Use the contents of this file to update the relevant fields in the /Users/heibalvin/Github/Cloudera-master/Director/Config/director-aws.conf file.: export AWS_ACCESS_KEY_ID=AKIAILPR4EQ5ZRULMBCA export AWS_SECRET_ACCESS_ID=1SGe**********************xKC7vYHg

You now could skip the next section, since you already have access keys. Please note down both of these access keys as they are very important

1.6 Create AWS Access Key An AWS Access Key is comprised of an AWS Access Key Id along with an AWS Secret Access Key. This key is required to allow Director to log into your AWS account and request instances etc. This step creates a new key although if you already have a key then you can use it instead. Your decision.

Select your current User.



Click on the Create Access Key button and download the csv file. This CSV file will contain both parts of the Access Key (ID and Secret)

Note: Please keep this file secure, since you have all permissions on your account with this information.

Use the contents of this file to update the relevant fields in the /Users/heibalvin/Github/Cloudera-master/Director/Config/director-aws.conf file.: export AWS_ACCESS_KEY_ID=AKIAILPR4EQ5ZRULMBCA export AWS_SECRET_ACCESS_ID=1SGe**********************xKC7vYHg

1.7 Import your AWS Key Pair A Key Pair will be needed so that you can have ssh access to machines in the cluster. Select AWS EC2 Services:



and then Key Pairs in left side menu.



Then click on Import Key Pair button.



Now let’s choose the tech-summit.pub public key to be imported from your [HOME-DIRECTORY] (for ex: /Users/heibalvin/Github/Cloudera-master/EMEA-Partner-Tech-Summit). Windows Users c:\Users\YOURUSERNAME\Github\Cloudera-master\EMEA-Partner-Tech-Summit



You will have successfully imported the public key from the tech-summit Key Pair within AWS Database.

1.8 Prepare your AWS Environment – Scripted (VPC, Subnet, Security Groups)

AWS CloudFormation provides a common language for you to describe and provision all the infrastructure resources in your cloud environment. CloudFormation allows you to use a simple text file to model and provision, in an automated and secure manner, all the resources needed for your applications across all regions and accounts. This file serves as the single source of truth for your cloud environment. (ref: https://aws.amazon.com/cloudformation/) We will benefit from CloudFormation capabilities to deploy our entire AWS environment. Select the AWS CloudFormation services by searching under the ‘Services’ menu item:


https://aws.amazon.com/cloudformation/

Cloudera & Deloitte Tech Summit From there click on Create New Stack. (or Create Stack, if you already have stacks in your account)

Upload a template to Amazon S3 (storage) and choose the [HOME-DIRECTORY]/Director/AWS/aws-quickstart-cloudera-director.template file. Windows Users c:\Users\YOURUSERNAME\Director\AWS\aws-quickstart-cloudera-director.template



You will then need to fill in several fields:



Stack Name Director

KeyPair tech-summit

Username YOUR USER NAME

Select Next then Before you Click Create ASK YOUR INSTRUCTOR FOR YOUR AWS COUPON



1.9 Applying your Free Credit to your AWS Code If you are a customer please make sure you have this coupon code. Ask your instructor. If you are Deloitte please use your own company AWS account. Step 1 - Logon to your AWS Account

Step 2 – Goto My Billing Dashboard In the right hand menu item select My Billing Dashboard

Step 3 – Goto Credits and add coupon below In the left hand side goto credits and apply coupon below Promo Code = You will be given this on the day



After applying your credit go back to the Create Stack Window in your browser


Cloudera & Deloitte Tech Summit Once the deployment is completed,click on the Output Tab. The Director instance: ec2-34-240-115-134.eu-west1.compute.amazonaws.com This is the URL to connect you need to copy your own version of this You will have all the information required to launch & deploy your EDH Cluster. A good way to keep this information handy is to update it in a file for example [HOME-DIRECTORY]/Director/Config/director-aws.conf file:

export AWS_OWNER=heibalvin export AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE" export AWS_SECRET_ACCESS_ID="wJalr***************EXAMPLEKEY" export AWS_REGION=eu-west-1 export AWS_AMI_ID=ami-4c457735 export AWS_KEY_PATH=/home/centos/Cloudera-master/EMEA-Partner-tech-summit/tech-summit export AWS_SUBNET_ID=subnet-0847cf7fd1f966e24 export AWS_SECGROUP_ID=sg-0356afdba5069ec40

DEPLOY YOUR FULL EDH CLUSTER

1.10 Connect to your Altus Director Instance On your newly created chrome browser, please type in URL the above link you copied from your outputs tab from your CloudFormation

You might have to wait 1-2 minutes for this to work. If you are having difficulties see SOCKS proxy in troubleshooting guide or ask your instructor The default username / password for director is: admin / admin



Accept the terms and conditions and hit Continue (you won’t see this screen again):

1.11 Change your Altus Director Admin Password



First thing to do is to Change Password from the upper-right menu.

You should provide your current password (admin) and a new password (for ex: Cloudera_123).



1.12 Create an Environment An Environment is the secure relationship between Director and an account in a Cloud Provider. Although Director is running in AWS in this instance it can talk to multiple other accounts in multiple Cloud Providers (Azure and GCP) at the same time. An Environment captures a specific relationship between this Director Instance and an account in a Cloud Provider.

From the Director Main menu please select Add Environment.



For the Key please remember to select upload the file from your download zip

This Environment will be used for deploying your resources:

● Environment Name: AWS ● Cloud Provider: AWS ● Access Key ID: [AWS_ACCESS_KEY_ID] (for ex: AKIAIOSFODNN7EXAMPLE) ● Secret Key ID: [AWS_SECRET_ACCESS_ID] (for ex: wJalr*********EXAMPLEKEY) ● EC2 Region: eu-west-1 ● SSH Username: centos ● SSH Private Key: tech-summit (could be found under

[HOME-DIRECTORY]/EMEA-Partner-Tech-Summit/tech-summit)

Hit Continue and you should get to this screen:



We will not use Simple Setup or Advanced Setup. In fact, we will design our own template, by selecting our newly created AWS Environment.

1.13 Create your Templates

1.13.1 Select your AWS Environment



Then select the ‘Templates’ menu at the top.



And then click the button ‘Create Instance Template’:



1.13.2 Create your Cloudera Manager Template After clicking on the Create Instance Template, you must now enter the main parameters needed, using the values you saved from the CloudFormation output stack:

● Instance Template Name: Cloudera Manager ● Instance Type: m4.2xlarge ● Image (AMI) ID: ami-4c457735 ● Tags: owner = [YOUR NAME] ● Security Group ID: [AWS_SECGROUP_ID] ● VPC Subnet ID: [AWS_SUBNET_ID]


Cloudera & Deloitte Tech Summit Make sure you are also updating the Advanced Options:

● Instance Prefix: CM ● Bootstrap Script: director-centos-7-installation-script.sh

(could be found at location [HOME-DIRECTORY]/Director/)

This bootstrap script is required for all instanced.

This template is designed for Cloudera Manager instance, with a Virtual Machine sized with 4 vCPU / 16 GB RAM with 50 GB SSD disk for the Operating System.



1.13.3 Create your Cloudera Master Template For the Templates Tab you now are able to create other templates based on your newly created Cloudera Manager template. Click on the Copy Template button.





Update your Master Template by replacing these parameters:

● Instance Template name: Master ● Instance Type: m4.2xlarge ● Instance Prefix: Master

This template is designed for Master Node instances, with a Virtual Machine sized with 2 vCPU / 8 GB RAM with 50 GB SSD disk for the Operating System. Select Save Changes

1.13.4 Create your Worker Template



From the Templates Tab , please click on Copy Template button from Cloudera Manager template. Update your Template by replacing these parameters:

● Instance Template name: Worker ● Instance Type: m4.xlarge ● Instance Prefix: Worker ● EBS Volume Count: 1

This template is designed for Worker Node instances, with a Virtual Machine sized with 4 vCPU / 16 GB RAM with 50 GB SSD disk for the Operating System. In addition to the Master Nodes, we will attach a 500 GB HDD DISK for data storage purpose.

1.13.5 Create your Edge Template




Cloudera & Deloitte Tech Summit From the Templates Tab, please click on Copy Template button from Cloudera Manager template. Update your Template by replacing these parameters:

● Instance Template name: Edge ● Instance Type: m4.2xlarge ● Instance Prefix: Edge ● EBS Volume Count: 2 ● EBS Volume Type: GP2 ● Root Volume Size: 100

This template is designed for Edge Node instances (specifically for Cloudera Data Science Workbench), with a Virtual Machine sized with 8 vCPU / 32 GB RAM with 100 GB SSD disk for the Operating System. In addition to other Nodes, we will attach a 2 * 500 GB SSD DISK for Application Block Device and Docker Image Block Device.

1.14 Deploy your EDH Cluster

1.14.1 Deploy your Cloudera Manager instance

Select Deployment Tab and click on Add Deployment (Cloudera Manager + Cluster).



Update the parameters as follow:

● Deployment Name: CM ● Instance Template: Cloudera Manager ● License Type: Cloudera Enterprise Trial ● CM Admin Username: admin ● CM Admin Password: Cloudera_123

Leave the other values defaulted like Embedded Database

Before continuing, we will need to setup the JAVA_HOME variable on all hosts.

Select the Cloudera Manager Configurations button



Select Cloudera Manager Configurations and add variable

java_home = /usr/java/jdk1.8.0_121-cloudera

Then Hit on ‘Continue’

Then a new Pop Up window appears click ‘OK’

1.14.2 Deploy your EDH Cluster



Update the parameters as follow:

● Cluster Name: EDH ● Services: All Services

Update the Instance Groups: Change the default instance count for workers to 3

● masters group name with Master template, instance count 1 and minimum 1. ● workers group name with Worker template, instance count 3 and minimum 1. ● gateway group name with Edge template, instance count 1 and minimum 1.



Click OK

Click on Continue.



Checking all services are running

In your Cloudera Manager your Cluster Name EDH should have a Status of Good Health Lets goto Cloudera Manager In your Cloudera Director

On the right hand side you can View Properties Click this and a drop down appears



This Public IP is the IP for Cloudera manager Open your browser and copy and paste this in with :7180

The logon screen appears



Type in User = admin Password = Cloudera_123 You should then see the next screen


Cloudera & Deloitte Tech Summit Let’s Check HUE Click on HUE-1 service If you are having problems connecting you will have to configure your SOCKS proxy see troubleshooting guide below

User = admin Password = Cloudera_123

YOU HAVE NOW COMPLETED THIS LAB



CLEANUP YOUR DEPLOYMENTS DO NOT DO THIS UNTIL AFTER WEDNESDAY 17th 6:00pm At the end of day 2 please tear down your EDH cluster. You need to do this on Thursday 18th October after the labs.

1.15 Tear down your EDH Cluster

Connect back to your Director instance. Select your AWS created environment. You will see a Cloudera Manager CM and a cluster EDH. On the left hand side click on the action button and select Terminate Cluster. The Cloudera Manager will remain for other Cluster Deployments.



Troubleshooting

1.16 AWS Users with existing VPC, Subnet and Security-Group CloudFormation provides a common language for you to describe and provision all the infrastructure resources in your cloud environment. CloudFormation allows you to use a simple text file to model and provision, in an automated and secure manner, all the resources needed for your applications across all regions and accounts. This file serves as the single source of truth for your cloud environment. (ref: https://aws.amazon.com/cloudformation/) We will benefit from CloudFormation capabilities to deploy our entire AWS environment. Select the AWS CloudFormation services by searching under the ‘Services’ menu item:

From there click on Create New Stack. (or Create Stack, if you already have stacks in your account)


https://aws.amazon.com/cloudformation/


Upload a template to Amazon S3 (storage) and choose the [HOME-DIRECTORY]/Director/AWS/aws-quickstart-with-VPC-cloudera-director.template file.



You will then need to fill in several fields:

Stack Name Director

KeyPair tech-summit

SecurityGroup For ex: sg-038785ea0b759e3a4

Subnet For ex: subnet-05c5e7372a587c484

Username YOUR USER NAME



Select Next then Before you Click Create ASK YOUR INSTRUCTOR FOR YOUR AWS COUPON.

1.17 Setup your SOCKS Proxy to secure all communications A SOCKS proxy server allows a client (such as your web browser) to connect directly and securely to a server (such as your Cloudera Director server web UI) and, from there, to the web UIs on other IP addresses and ports in the same subnet, including the Cloudera Manager and Hue web UIs. The SOCKS proxy provides you with access to the Cloudera Director UI, Cloudera Manager UI, Hue UI, and any other cluster web UIs without exposing their ports outside the subnet. (ref: https://www.cloudera.com/documentation/director/2-8-x/topics/director_security_socks.html#concept_y11_nm3_wr ) Retrieve your initially opened terminal window, or open a new session. Proceed as follow: > cd [HOME-DIRECTORY] > cd Cloudera-master/EMEA-Partner-Tech-Summit > nohup ssh -i tech-summit -CND 8157 centos@[DIRECTOR-URL] &


https://www.cloudera.com/documentation/director/2-8-x/topics/director_security_socks.html#concept_y11_nm3_wr


Cloudera & Deloitte Tech Summit This will create a SSH secure tunnel for communicating between your laptop and the Altus Director instance. Moreover, your laptop will be considered as a virtual machine deployed in the private subnet of the AWS deployment. You will be able to reach private IPs within the private subnet. Now let’s create a specific chrome web browser session with the SOCKS Proxy predefined. For a MacBook laptop please proceed as follows: "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \

--user-data-dir="$HOME/chrome-with-proxy" \

--proxy-server="socks5://localhost:8157"

Note: For other Operating Systems (Linux or Windows) please refer to the Altus Director official documentation (ref: https://www.cloudera.com/documentation/director/2-8-x/topics/director_security_socks.html#concept_y11_nm3_wr ).

Socks proxy

Make sure SOCKS Proxy is selected and running





select the ec2-.... below in your Chrome Browser

Putty for Windows Laptops

If you need Command Line Interface on Linux/Centos for windows you need to download and install Putty

https://www.putty.org/

Install the 64-bit version to your windows PC

Using Command Line Interface on your Terminal or Putty on Windows

If you then need to connect > cd [HOME-DIRECTORY] > cd Cloudera-master/EMEA-Partner-Tech-Summit


https://www.putty.org/

Cloudera & Deloitte Tech Summit > nohup ssh -i tech-summit -CND 8157 centos@[DIRECTOR-URL] & For example nohup ssh -i tech-summit -CND 8157 centos@http://ec2-52-213-184-245.eu-west-1.compute.amazonaws.com


Cloudera & Deloitte Tech Summit€¦ · Prepare your AWS Environment – Scripted (VPC, Subnet, Security Groups) 17 Applying your Free Credit to your AWS Code 21 DEPLOY YOUR FULL

Documents