Presented By- SHAILJA DALMIA 13IT252 BIG DATA ANALYTICS USING HADOOP
Presented By- SHAILJA DALMIA 13IT252
BIG DATA ANALYTICS USING HADOOP
INTRODUCTIONEra of digitilized WorldChallenges to cutting edge businessesGFS and MapReduceIn 2006,Mike Caferella & Doug Cutting
working under Nutch project implemented Hadoop.
Open Source Framework for writing and running distributed applications.
WHAT IS BIG DATA?
WHY DFS?
What is Distributed File System?
What is Hadoop?
Hadoop Core Components
What is HDFS?
Design of HDFS
Area where HDFS is not a good fit
HDFS COMPONENTS
NameNodeDataNodes
Job Tracker and Task Tracker
HDFS Architecture
Map Reduce• Framework that assigns task to each
datanodes. Map Step-master node takes the input ,partition
it up into smaller sub problem,leading to multi level tree structure.
Reduce Step-Combine the results and generate the output
Each mapping operation is independent of other,Key value pair is generated ,sorters and shufflers are applied .
Parallelism offer fault tolerance,if one nodes fails ,still the work can be rescheduled.
Similar to Divide and Conquer technique. Does task in parallel to accomplish work in less
time.
Hadoop Key Features:
AccessibleRobustnessSimpleScalableCost EffectiveFlexibleFault Tolerant
Differences Between Hadoop and RDBMS
Hadoop Designed to scale out
architecture.Key value pairsFunctional
Programming(scripts and codes),can build complex models
Offline processing (WORA)
RDBMSScaling is expensiveTables having relational
structureDeclarative queriesOnline Processing.(work
for random reading and writing few records.
Hadoop Related TechnologiesAvro-Data Serialization System,rich data
structures,container file,compact fast binary data format.
Chukwa-powerful toolkit for analyzing data.
Hbase-Distributed database,provides big table like capabilities.
Hive-data warehouse useful for data summarization .Uses HiveQL language.
ConclusionHadoop had gained huge momentum
Technologies around are evolving really fast
There is no “One size fits all”
Valuable ,powerful tool.
More targeted businesses.