Bd class 2 complete

BUMPER

Topic 1

HDFS – Hands On (Part – 1)

Class 2 – Hadoop Distributed File System

AGENDA

• What is Big Data?• Hadoop Distributed File System• MapReduce• Understanding Hadoop Ecosystem• Setting up a Hadoop Cluster• HDFS – Hands On• MapReduce-Hands On

Pre-requisites

HDFS – Hands On

Virtual Machine is up and running.

Connected to your Virtual Machine using putty as ‘hduser’.

Command Syntax

HDFS – Hands On

hadoop fs –ls / (To list directory contents)

Command Syntax

HDFS – Hands On


hadoop fs -<command> <args>

Command Syntax

HDFS – Hands On



hadoop: This is the binary executable.

Command Syntax

HDFS – Hands On




fs: Invokes the Hadoop file system, which is the HDFS.

Command Syntax

HDFS – Hands On





<command>: Indicates what is the purpose of the statement and always preceded by a ‘-‘.

Command Syntax

HDFS – Hands On





<command>: Indicates what is the purpose of the statement and always preceded by a ‘-‘.

<args>: Indicates the arguments that are applicable for the command.

Where do DataNodes store data?HDFS – Hands On


hadoop.tmp.dir = /tmp/hadoop


hadoop.tmp.dir = /tmp/hadoop dfs.data.dir = ($hadoop.tmp.dir)/dfs/data


hadoop.tmp.dir = /tmp/hadoop dfs.data.dir = ($hadoop.tmp.dir)/dfs/data = /tmp/hadoop/dfs/data


hadoop.tmp.dir = /tmp/hadoop dfs.data.dir = ($hadoop.tmp.dir)/dfs/data = /tmp/hadoop/dfs/data

VERSION >> Java properties fileblk_********* >> Raw data of a fileblk_******.meta >> Metadata of the blockHow come there is a block when we have not loaded any file?

jobtracker.infoHDFS – Hands On

fsckHDFS – Hands On

Generates a summary report that lists the overall health of the filesystem.


Total size: Indicates the size of the directory (root directory in our case). Does not account for replication.

Total dirs: Indicates the number of directories in HDFS

Total files: Indicates the number of files in HDFS

Total blocks: Indicates the number of blocks

Default replication factor:Average replication factor:Corrupt blocks:Missing replicas: Number of data nodes:Number of racks:

Edit .bashrc

HDFS – Hands On

Navigate to the home directory.

cd

List hidden files.

ls -a

Edit the .bashrc file.

vi .bashrc

Update HADOOP paths using ‘export’ command.

export HADOOP_CONF=/home/hduser/hadoop/confexport HADOOP_PREFIX=/home/hduser/hadoop

# Add Hadoop bin/ directory to path

export PATH=$PATH:$HADOOP_PREFIX/bin

Execute the updated contents of the .bashrc file.

source ~/.bashrc

copyFromLocalHDFS – Hands On

Copies file from local file system to HDFS.

hadoop fs –copyFromLocal <Path to source file on Local File System> <Target path in HDFS>

hadoop fs –copyFromLocal NOTICE.txt noticehdfs.txt

copyFromLocalHDFS – Hands On

copyFromLocal commands internally results in:

a file getting split into multiple blocks.

the client contacting the NameNode to find out where each block should be copied in the cluster.

replication of blocks to nodes assigned by NameNode.

How many blocks were created?HDFS – Hands On

RECAP

HDFS Commonly used commandsHDFS Concepts

BUMPER

BUMPER

Topic 2

HDFS – Hands On (Part – 2)


AGENDA


Load a file larger than the block sizeHDFS – Hands On

Load a 200 MB file and see how many blocks were created.

Command to generate a 200 MB dummy file.dd if=/dev/zero of=file.txt count=1024 bs=204800

hadoop fs –copyFromLocal file.txt file.txtcd /tmp/hadoop/dfs/data/currentls –lrt

Load a file larger than the block sizeHDFS – Hands On

Block 1 = 64 MB

Block 2 = 64 MB

Block 3 = 8 MB

Block 4 = 64 MB


fsck after loading 2 additional files.

Total size has increased.Total dirs: 7. Additions - /user and /user/hduser directories.Total files: 3. Additions - 2 newly loaded files.Total blocks: 6. Additions - 1 block of the 1st file and 4 blocks of the 2nd file.

catHDFS – Hands On

Displays contents of file on the command prompt.

hadoop fs –cat <Path of file in HDFS>

hadoop fs –cat noticehdfs.txt

copyToLocalHDFS – Hands On

Copies file from HDFS to local file system.

hadoop fs –copyToLocal <Path of file in HDFS> <Path of file in Local File System>

hadoop fs –copyToLocal noticehdfs.txt noticelocal.txt

mkdirHDFS – Hands On

Creates a directory inside HDFS.HDFS paths are relative.

Creates directory in current user’s home directoryhadoop fs –mkdir newdir

Creates new directory under roothadoop fs –mkdir /newdir

rmHDFS – Hands On

Removes file (s).

hadoop fs –rm <File Name>

Removes file and empty directories.hadoop fs –rm noticehdfs.txt

Trash featureHDFS – Hands On

Prevents accidental deletion of files and directories.Disabled by default.To enable, configure the fs.trash.interval property in core-site.xml file.

RECAP

HDFS Commonly used commandsHDFS Concepts

BUMPER

BUMPER

Topic 3

HDFS – Web UI


AGENDA


NameNode Web Interface

HDFS – Hands On

HDFS Web Interface URL.

http://<namenode_host>:50070/

From the Virtual Machine:

http://localhost:50070/

From outside the Virtual Machine:http://<IP Address of VM or Hostname of VM>:50070/Example- http://192.168.234.135:50070/

NameNode Web Interface

HDFS – Hands On

Server Name and Port

Last start time of the NameNode

Hadoop Version, followed by subversion source code repository

To browse the files in HDFS View NameNode log files

Number of files, directories and blocks. Heap memory utilized/available.

Storage capacity of machines in the clusterHow much space utilized in HDFSSpace utilized by O/S, Applications etc.Amount of space available on HDFS

How many blocks have replicas less than Replication Factor

Nodes that are active and in contact with NameNodeNodes that are NOT in contact with NameNodeNodes administratively removed from the cluster

RECAP

HDFS Web UI

BUMPER

BUMPER

Topic 4


MapReduce – Hands On (Part – 1)

AGENDA


How does MapReduce work?

MapReduce


MapReduce

Map Input List

Map Output List

Reduce Input List

Reduce Output List

Mapping Phase

Reducing Phase


MapReduce

Map Input List

Map Output List

Reduce Input List

Reduce Output List

Mapping Phase

Reducing Phase


MapReduce

Map Input List

Map Output List

Mapper

Reduce Input List

Reduce Output List

Mapping Phase

Reducing Phase


MapReduce

Map Input List

Map Output List

Mapper

Reduce Input List

Reduce Output List

Mapping Phase

Reducing Phase


MapReduce

Map Input List

Map Output List

Mapper

Reduce Input List

Reduce Output List

Mapping Phase

Reducing Phase


MapReduce

Map Input List

Map Output List

Mapper

Reduce Input List

Reduce Output List

Mapping Phase

Reducing Phase


MapReduce

Map Input List

Map Output List

Mapper

Reduce Input List

Reduce Output List

Reducer

Mapping Phase

Reducing Phase


MapReduce

Map Input List

Map Output List

Mapper

Reduce Input List

Reduce Output List

Reducer

Mapping Phase

Reducing Phase


MapReduce

Map Input List

Map Output List

Mapper

Reduce Input List

Reduce Output List

Reducer

Mapping Phase

Reducing Phase

Hadoop MapReduce

MapReduce

<1, King Queen King>

<King, 1><Queen, 1><King, 1>

<2, Minister King Soldier>

<3, Queen Soldier King>

<Minister, 1><King, 1><Soldier, 1>

<Queen, 1><Soldier, 1><King, 1>

<King, 1><King, 1><King, 1><King, 1>

<Minister, 1>

<Queen, 1><Queen, 1>

<Soldier,1><Soldier,1>

<King, (1,1,1,1)><Minister, 1>

<Queen, (1,1)><Soldier, (1,1)>

<King, 4><Minister, 1>

King Queen King

Minister King Soldier

Queen Soldier King

Input SplittingMap Shuffling Reduce Result

<Queen, 2><Soldier, 2>

Map Output

Hadoop MapReduce

MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Hadoop MapReduce

MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Hadoop MapReduce

MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Hadoop MapReduce

MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Hadoop MapReduce

MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Hadoop MapReduce

MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Hadoop MapReduce

MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Hadoop MapReduce

MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Hadoop MapReduce – Roles: User vs. Framework

MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output


MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Load data into HDFS


MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Load data into HDFS

Specify Path & Input Format


MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Load data into HDFS



MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Load data into HDFS


Create ‘Input Splits’


MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Load data into HDFS



Create individual Records


MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Load data into HDFS





MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Load data into HDFS




User Defined Logic


MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Load data into HDFS




User Defined Logic


MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Load data into HDFS




User Defined Logic


MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Load data into HDFS




User Defined Logic

User Defined Logic


MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Load data into HDFS




User Defined Logic

User Defined Logic


MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Load data into HDFS




User Defined Logic

User Defined Logic Specify Path &

Output format


MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Load data into HDFS




User Defined Logic


Output format


MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Load data into HDFS




User Defined Logic


Output format

Replication, Rack Awareness etc.


MapReduce








<Minister, 1>






King Queen King


Queen Soldier King



Map Output

Load data into HDFS




User Defined Logic


Output format

Replication, Rack Awareness etc.

MapReduce Execution FrameworkMapReduce


Mapper Process


Reduce Process

Mapper Process


Reduce Process

Mapper ProcessDriver


Reduce Process

Mapper Process

Input HDFS File - inputFile.txt

Driver


Reduce Process

Mapper Process

Input HDFS File - inputFile.txtBlock A Block B Block C

Driver


Reduce Process

Mapper Process


Driver

InputFormat


Reduce Process

Mapper Process


Driver

Input Split 1

InputFormat


Reduce Process

Mapper Process


Driver

Input Split 1 Input Split 2 Input Split 3 Input Split 4

InputFormat


Reduce Process

Mapper Process


Driver


InputFormat

Calculates


Reduce Process

Mapper Process


Driver


InputFormat

Mapper Process

Calculates


Reduce Process

Mapper Process


Driver


InputFormat

Mapper Process

Calculates


Reduce Process

Mapper Process


Driver

Record Reader


InputFormat

Mapper Process

Record Reader

Calculates


Reduce Process

Mapper Process


Driver

Record Reader


InputFormat

Mapper Process

Record Reader

Reads Reads

Calculates


Reduce Process

Mapper Process


Driver

Record Reader


InputFormat

Mapper Process

Record Reader

Reads Reads

Calculates

Defines


Reduce Process

Mapper Process


Driver

Record Reader


InputFormat

Mapper Process

Record Reader

Reads

Passes <K,V> pairs

Reads

Calculates

Defines

Passes <K,V> pairs


Reduce Process

Mapper Process


Driver

Mapper

Record Reader


InputFormat

Mapper Process

Mapper

Record Reader

Reads

Passes <K,V> pairs

Reads

Calculates

Defines

Passes <K,V> pairs


Reduce Process

Mapper Process


Driver

Mapper

Record Reader


InputFormat

Mapper Process

Mapper

Record Reader

Reads

Passes <K,V> pairs

Reads

Calculates

Defines

Passes <K,V> pairs

<K, V> pairs <K, V> pairs


Reduce Process

Mapper Process


Driver

Mapper

Record Reader


InputFormat

Reduce Process

Mapper Process

Mapper

Record Reader

Reads

Passes <K,V> pairs

Reads

Calculates

Defines

Passes <K,V> pairs



Reduce Process

Mapper Process


Driver

Mapper

Record Reader


InputFormat

Reduce Process

Mapper Process

Mapper

Record Reader

Reads

Passes <K,V> pairs

Reads

Calculates

Defines

Passes <K,V> pairs



Reduce Process

Mapper Process


Driver

Mapper

Record Reader


InputFormat

Reduce Process

Mapper Process

Mapper

Record Reader

Reads

Passes <K,V> pairs

Reads

Calculates

Defines

Passes <K,V> pairs



Reduce Process

Mapper Process


Driver

Mapper

Record Reader


InputFormat

Reduce Process

Mapper Process

Mapper

Record Reader

Reads

Passes <K,V> pairs

Reads

Calculates

Defines

Passes <K,V> pairs


Shuffle


Reduce Process

Mapper Process


Driver

Mapper

Record Reader


InputFormat

Reduce Process

Mapper Process

Mapper

Record Reader

Reads

Passes <K,V> pairs

Reads

Calculates

Defines

Passes <K,V> pairs


Partition Shuffle


Reduce Process

Mapper Process


Driver

Mapper

Record Reader


InputFormat

Reduce Process

Mapper Process

Mapper

Record Reader

Reads

Passes <K,V> pairs

Reads

Calculates

Defines

Passes <K,V> pairs


Partition ShuffleSort


Reduce Process

Mapper Process


Driver

Mapper

Record Reader


InputFormat

Reduce Process

Mapper Process

Mapper

Record Reader

Reads

Passes <K,V> pairs

Reads

Calculates

Defines

Passes <K,V> pairs




Reduce Process

Mapper Process


Driver

Mapper

Reducer

Record Reader


InputFormat

Reduce Process

Mapper Process

Mapper

Reducer

Record Reader

Reads

Passes <K,V> pairs

Reads

Calculates

Defines

Passes <K,V> pairs




Reduce Process

Mapper Process


Driver

Mapper

Reducer

Record Reader


InputFormat

Reduce Process

Mapper Process

Mapper

Reducer

Record Reader

Reads

Passes <K,V> pairs

Reads

Passes <K,V> pairs

Calculates

Defines

Passes <K,V> pairs

Passes <K,V> pairs




Reduce Process

Mapper Process


Driver

Mapper

Reducer

Record Reader


InputFormat

Reduce Process

Mapper Process

Mapper

Reducer

Record Reader

Reads

Passes <K,V> pairs

Reads

Passes <K,V> pairsOutputFormat

Calculates

Defines

Passes <K,V> pairs

Passes <K,V> pairs




Reduce Process

Mapper Process


Driver

Mapper

Reducer

Record Reader


InputFormat

Output Data

Reduce Process

Mapper Process

Mapper

Reducer

Record Reader

Output Data

Reads

Passes <K,V> pairs

Reads


Calculates

Defines

Defines

Passes <K,V> pairs

Passes <K,V> pairs




Reduce Process

Mapper Process


Driver

Mapper

Reducer

Record Reader


Writer

InputFormat

Output Data

Reduce Process

Mapper Process

Mapper

Reducer

Record Reader

Writer

Output Data

Reads

Passes <K,V> pairs

Reads


Calculates

Defines

DefinesDefines

Passes <K,V> pairs

Passes <K,V> pairs




Reduce Process

Mapper Process


Driver

Mapper

Reducer

Record Reader


Writer

InputFormat

Output Data

Reduce Process

Mapper Process

Mapper

Reducer

Record Reader

Writer

Output Data

Reads

Passes <K,V> pairs

Reads


Calculates

Defines

DefinesDefines

Passes <K,V> pairs

Passes <K,V> pairs




Reduce Process

Mapper Process


Driver

Mapper

Reducer

Record Reader


Writer

InputFormat

Output Data

Reduce Process

Mapper Process

Mapper

Reducer

Record Reader

Writer

Output Data

Reads

Passes <K,V> pairs

Writes

Reads

Passes <K,V> pairs

Writes

OutputFormat

Calculates

Defines

DefinesDefines

Passes <K,V> pairs

Passes <K,V> pairs




Reduce Process

Mapper Process


Driver

Mapper

Reducer

Record Reader


Writer

InputFormat

Output Data

Reduce Process

Mapper Process

Mapper

Reducer

Record Reader

Writer

Output Data

Reads

Passes <K,V> pairs

Writes

Reads

Passes <K,V> pairs

Writes

OutputFormat

Defines

Calculates

Defines

DefinesDefines

Passes <K,V> pairs

Passes <K,V> pairs




Reduce Process

Mapper Process


Driver

Mapper

Reducer

Record Reader


Writer

InputFormat

Output Data

Reduce Process

Mapper Process

Mapper

Reducer

Record Reader

Writer

Output Data

Reads

Passes <K,V> pairs

Writes

Reads

Passes <K,V> pairs

Writes

OutputFormat

Defines

Defines

Calculates

Defines

DefinesDefines

Passes <K,V> pairs

Passes <K,V> pairs




Reduce Process

Mapper Process


Driver

Mapper

Reducer

Record Reader


Writer

InputFormat

Output Data

Reduce Process

Mapper Process

Mapper

Reducer

Record Reader

Writer

Output Data

Reads

Passes <K,V> pairs

Writes

Reads

Passes <K,V> pairs

Writes

OutputFormat

Defines

Defines

Calculates

Defines

Defines

DefinesDefines

Passes <K,V> pairs

Passes <K,V> pairs




Reduce Process

Mapper Process


Driver

Mapper

Reducer

Record Reader


Writer

InputFormat

Output Data

Reduce Process

Mapper Process

Mapper

Reducer

Record Reader

Writer

Output Data

Reads

Passes <K,V> pairs

Writes

Reads

Passes <K,V> pairs

Writes

OutputFormat

Defines

Defines

Calculates

Defines

Defines

Defines

DefinesDefines

Passes <K,V> pairs

Passes <K,V> pairs



RECAP

MapReduce Execution Framework

BUMPER

BUMPER

Topic 5


MapReduce – Hands On (Part – 2)

AGENDA


Java MapReduce Programming

MapReduce

Hello World of MapReduce >> Word Count program

Eclipse – Integrated Development Environment (IDE)

https://www.eclipse.org/downloads/



RECAP

Part two of Java MapReduce program

BUMPER

Bd class 2 complete

Education

recap hdfs

list directory

datanodes

local file

copyfromlocal

fsck hdfs

hadoop file

hdfs hands