Ruby ruby ruby

HadoopEdit by Cassell Hsu

2013.04.19

Hadoop

Master

Slave

Slave

Slave

Hadoop

•Master

•NameNode

•JobTracker

•SecondaryNameNode

NameNode•Where is NameNode?

•Master

•HDFS(Hadoop Distributed File System)

•What is NameNode?

•資料之位置資訊

•資料之屬性

？？

NameNode

•位置資訊 ?

•所有資料皆存放在 – DataNode

DataNode

•What is DataNode

•存放資料•Where is DataNode

•HDFS

•Slaves (and Master)

UserUser

DataNodeDataNode

NameNodeNameNode

DataNodeDataNode

DataNodeDataNode

128Mb

B64Mb

A64Mb

Check hdfs-site.xml

B64Mb

A64Mb

Hadoop

DataNodeDataNodeNameNodeNameNode

MasterDataNodeDataNode

DataNodeDataNode

DataNodeDataNode

Slaves

Hadoop

•Master

•NameNode

•JobTracker

•SecondaryNameNode

JobTracker

•What is JobTracker?

•排程工作•Where is JobTracker?

•Master

誰來工作？

JobTracker & TaskTrackerJobTracker TaskTracker

Where Master Slaves

What 排程工作 執行工作

Hadoop

DataNodeDataNodeNameNodeNameNode

MasterDataNodeDataNode

Slaves

JobTrackerJobTracker

TaskTrackerTaskTracker

Hadoop

•Master

•NameNode

•JobTracker

•SecondaryNameNode

SecondaryNameNode

•What is SecondaryNameNode?

•NameNode發生錯誤時補救

•Where is SecondaryNameNode?

•Master

Hadoop

DataNodeDataNodeNameNodeNameNode

MasterDataNodeDataNode

Slaves

JobTrackerJobTracker

TaskTrackerTaskTracker

SecondaryNodeSecondaryNode

MapReduce

User Master

Slave2

Slave1

A

A1

A2Task

NameNode

Task

Result1

Result2

Reduce

Final Result

HDFS

MapReduce

•檔案切割•Hadoop上區塊切割

•程式指定