Hadoop Monitoring best Practices

• 1. How to monitor the$H!T out of Hadoop Developing a comprehensive open approach to monitoring hadoop clusters

2. Relevant Hadoop Information

• From 3 3000 Nodes

• Hardware/Software failures common

• Redundant Components DataNode, TaskTracker

• Non-redundant Components NameNode, JobTracker, SecondaryNameNode

• Fast Evolving Technology (Best Practices?)

3. Monitoring Software

• Nagios

• • Red Yellow Green Alerts, Escalations

• • Defacto Standard Widely deployed

• • Text base configuration

• • Web Interface

• • Pluggable with shell scripts/external apps

• • • Return 0 - OK

4. Cacti

• Performance Graphing System

• RRD/RRA Front End

• Slick Web Interface

• Template System for Graph Types

• Pluggable

• • SNMP input

• • Shell script /external program

5. 6. hadoop-cacti-jtg

• JMX Fetching Code w/ (kick off) scripts

• Cacti templates For Hadoop

• Premade Nagios Check Scripts

• Helper/Batch/automation scripts

• Apache License

7. Hadoop JMX 8. Sample Cluster P1

• NameNode & SecNameNode

• • Hardware RAID

• • 8 GB RAM

• • 1x QUAD CORE

• • DerbyDB (hive) on SecNameNode

• JobTracker

• • 8GB RAM

• • 1x QUAD CORE

9. A Sample Cluster p2

• Slave (hadoopdata1-XXXX)

• • JBOD 8x 1TB SATA Disk

• • RAM 16GB

• • 2x Quad Core

10. Prerequisites

• Nagios (install) DAG RPMs

• Cacti (install) Several RPMS

• Liberal network access to the cluster

11. Alerts & Escalations

• X nodes * Y Services = < Sleep

• Define a policy

• • Wake Me Ups (SMS)

• • Dont Wake Me Ups (EMAIL)

• • Review (Daily, Weekly, Monthly)

12. Wake Me Ups

• NameNode

• • Disk Full (Big Big Headache)

• • RAID Array Issues (failed disk)

• JobTracker

• SecNameNode

• • Do not realize it is not working too late

13. Dont Wake Me Ups

• Or Wake someone else up

• DataNode

• • Warning Currently Failed Disk will down the Data Node (see Jira)

• TaskTracker

• Hardware

• • Bad Disk (Start RMA)

• Slaves are expendable (up to a point)

14. Monitoring Battle Plan

• Start With the Basics

• • Ping, Disk

• Add Hadoop Specific Alarms

• • check_data_node

• Add JMX Graphing

• • NameNodeOperations

• Add JMX Based alarms

• • FilesTotal > 1,000,000 or LiveNodes < 50%

15. The Basics Nagios

• Nagios (All Nodes)

• • Host up (Ping check)

• • Disk % Full

• • SWAP > 85 %

• * Load based alarms are somewhat useless389% CPU load is not necessarily a bad thing in Hadoopville

16. The Basics Cacti

• Cacti (All Nodes)

• • CPU (full CPU)

• • RAM/SWAP

• • Network

• • Disk Usage

17. Disk Utilization 18. RAID Tools

• Hpacucli not a Street Fighter move

• • Alerts on RAID events (NameNode)

• • • Disk failed

• • • Rebuilding

• • JBOD (DataNode)

• • • Failed Drive

• • • Drive Errors

• Dell, SUN, Vendor Specific Tools

19. Before you jump in

• X Nodes * Y Checks * = Lots of work

• About 3 Nodes into the process

• • Wait!!! I need some interns!!!

• Solution S.I.C.C.T.Semi-Intelligent-Configuration-cloning-tools

• • (I made that up)

• • (for this presentation)

20. Nagios

• Answers IS IT RUNNING?

• Text based Configuration

21. Cacti

• Answers HOW WELL IS IT RUNNING?

• Web Based configuration

• • php-cli tools

22. Monitoring Battle Plan Thus Far

• Start With the Basics

• • Ping, Disk !!!!!!Done!!!!!!

• Add Hadoop Specific Alarms

• • check_data_node

• Add JMX Graphing

• • NameNodeOperations

• Add JMX Based alarms

• • FilesTotal > 1,000,000 or LiveNodes < 50%

23. Add Hadoop Specific Alarms

• Hadoop Components with a Web Interface

• • NameNode 50070

• • JobTracker 50030

• • TaskTracker 50060

• • DataNode 50075

• check_http + regex = simple + effective

24. nagios_check_commands.cfg

• Component Failure

• (Future) Newer Hadoop will have XML status

define command { command_namecheck_remote_namenode command_line$USER1$/check_http -H$HOSTADDRESS$ -u http://$HOSTADDRESS$:$ARG1$/dfshealth.jsp -p $ARG1$ -r NameNode } define service { service_description check_remote_namenode use generic-service host_name hadoopname1 check_command check_remote_namenode!50070 } 25. Monitoring Battle Plan

• Start With the Basics

• • Ping, Disk (Done)

• Add Hadoop Specific Alarms

• • check_data_node (Done)

• Add JMX Graphing

• • NameNodeOperations

• Add JMX Based alarms

• • FilesTotal > 1,000,000 or LiveNodes < 50%

26. JMX Graphing

• Enable JMX

• Import Templates

27. JMX Graphing 28. JMX Graphing 29. JMX Graphing 30. 31. Standard Java JMX 32. Monitoring Battle Plan Thus Far

• Start With the Basics !!!!!!Done!!!!!

• • Ping, Disk

• Add Hadoop Specific Alarms !Done!

• • check_data_node

• Add JMX Graphing !Done!

• • NameNodeOperations

• Add JMX Based alarms

• • FilesTotal > 1,000,000 or LiveNodes < 50%

33. Add JMX based Alarms

• hadoop-cacti-jtg is flexible

• • extend fetch classes

• • Dont call output()

• • Write your own check logic

34. Quick JMX Base Walkthrough

• url, user, pass, object specified from CLI

• wantedVariables, wantedOperations by inheritance

• fetch() output() provided

35. Extend for NameNode 36. Extend for Nagios 37. Monitoring Battle Plan

• Start With the Basics !DONE!

• • Ping, Disk

• Add Hadoop Specific Alarms !DONE!

• • check_data_node

• Add JMX Graphing !DONE!

• • NameNodeOperations

• Add JMX Based alarms !DONE!

• • FilesTotal > 1,000,000 or LiveNodes < 50%

38. Review

• File System Growth

• • Size

• • Number of Files

• • Number of Blocks

• • Ratios

• Utilization

• • CPU/Memory

• • Disk

• Email (nightly)

• • FSCK

• • DSFADMIN

39. The Future

• JMX Coming to JobTracker and TaskTracker (0.21)

• • Collect and Graph Jobs Running

• • Collect and Graph Map / Reduce per node

• • Profile Specific Jobs in Cacti?