Managing Enterprise Hadoop Clusters with Apache Ambari

Managing Enterprise Hadoop Clusters with

Apache Ambari

Jayush Luniya @ Hortonworks Apache Ambari PMC

Agenda

Ambari Overvie

Ambari Features Demo Q&A

What’s Apache Ambari?

100% open-source platform for simplifying

Hadoop cluster management and

Highly extensible.

It’s a wild zoo out there!Gotta manage this

efficiently.

Apache Ambari Themes

• Deliver the core operational capabilities to provision, manage and monitor Hadoop clusters at scale.

Operate Hadoop at Scale

• Robust API for integration with existing enterprise systems, such as Microsoft SCOM and Teradata Viewpoint.

Integrate with the Enterprise

• Provide extensible platform for Customers, Partners and the Community (Stacks, Views)

Extend for the Ecosystem

Apache Ambari

Open Source Activity

Inception: AMBARI-1 (Sept, 2011)

Fast forward 5 years to today…

Latest JIRA: AMBARI-16131 150+ Contributors 60+ Committers 16131 JIRAs filed 14254 JIRAs fixed

At 1.5 day per JIRA ~ 90 person years!

Used by hundreds of companies

Ambari – 3rd Biggest Project* @ Apache

* Based on total JIRAs filed on a project basis as of April 26, 2016

#2: Hadoop at ~32k as it is split across multiple JIRA Projects

#1#3#4#5

Timeline

Ambari 1.6.*May 2014908 JIRAs

Ambari 1.5.*Apr 2014

1218 JIRAs

Ambari 1.7.*Dec 2014

1620 JIRAs

Ambari 2.0.* April 20151804 JIRAs

Current GA Version (2.2.2)

Ambari 2.1.*July 2015

2674 JIRAs

Ambari Stacks

Resolution of 9k+ JIRAs

Ambari Blueprints Ambari Views

Alerts FrameworkMetrics SystemRolling UpgradeKerberos Automation

Enhanced DashboardsSmart Configs

Ambari 2.2.*Dec 2015

1542 JIRAs

Express UpgradeAMS Grafana

Agenda

Ambari Overvie

Extensibility Features

• To add new Services (ISV or otherwise) beyond HDP stack• To customize a Stack for customer specific environmentsStacks

• To use Ambari for automating cluster installations.• To share best practices on layout and cluster configurationBlueprints

• To extend and customize the Ambari Web UI• Add new capabilities, customize existing capabilitiesViews

Anatomy of Ambari Extension Points

Ambari Stacks

Stack Terminology

Term Definition Examples

STACK Defines a set of Services, where to obtain the software packages and how to manage the lifecycle.

HDP-2.3, HDP-2.2

SERVICE Defines the Components that make-up the service. HDFS, NAGIOS, YARN

COMPONENT The building-blocks of a Service, that adhere to a certain lifecycle.

NAMENODE, DATANODE, OOZIE_SERVER

CATEGORY The category of Component. MASTER, SLAVE, CLIENT

REPO Repository metadata where the artifacts reside http://public-repo-1.hortonworks.com/HDP/centos6/2.x/GA/2.3.0.0

Ambari Stack Stacks define Services + Repo

– What is a stack, and where to get the bits

Each service has a definition– What components are part of the Service

Each service has defined lifecycle commands– start, stop, status, install, configure

Lifecycle is controlled via command scripts Ability to define “custom” commands

Ambari Server

Service Definitions

Command Scripts

xml python

Ambari Agents

Stacks Support Inheritance

HDP 2.1 Stack

HDP 2.0 Stack

Overrides any Service definitions, commands and configurations Adds new Services specific to this Stack

Defines a set of Service definitions Default service configurations and command scripts

Blueprints

Automated Cluster Deployment

Deploy clusters of any scale with ease Two REST API calls is all it takes to provision a clusterWho uses it? HDInsight (Microsoft Azure) Hortonworks QA

Example: Create a 100-node Cluster

{ "configurations" : [ { ”hdfs-site" : {

"dfs.datanode.data.dir" : ”/hadoop/1,/hadoop/2,/hadoop/3" } } ], "host_groups" : [ { "name" : ”master-host", "components" : [ { "name" : "NAMENODE” }, { "name" : "RESOURCEMANAGER” }, … ], "cardinality" : "1" }, { "name" : ”worker-host", "components" : [ { "name" : ”DATANODE” }, { "name" : ”NODEMANAGER” }, … ], "cardinality" : "1+" }, ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.0" }}

{ "blueprint" : ”my-blueprint", "host_groups" :[ { "name" : ”master-host", "hosts" : [ { "fqdn" : ”master001.ambari.apache.org”

} ] }, { "name" : ”worker-host", "hosts" : [ { "fqdn" : ”worker001.ambari.apache.org”

}, { "fqdn" : ”worker002.ambari.apache.org”

}, … { "fqdn" : ”worker099.ambari.apache.org”

} ] } ]}

1. POST /api/v1/blueprints/my-blueprint 2. POST /api/v1/clusters/my-cluster

Cluster Replication

{ "configurations" : [ { ”cluster-env" : {

”user_group" : ”hadoop" } ”hdfs-site" : {

"dfs.datanode.data.dir" : ”/hadoop/1,/hadoop/2,/hadoop/3" } } ], "host_groups" : [ { "name" : ”master-host", "components" : [ { "name" : "NAMENODE” }, { "name" : "RESOURCEMANAGER” }, … ], "cardinality" : "1" } ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.0" }}

GET/api/v1/clusters/my-cluster?format=blueprint

Export blueprint from an existing cluster Import blueprint to replicate the cluster

Blueprint Features

Ambari 2.0: High availability (HA) cluster deployments Adding hosts using blueprints (AMBARI-8458)Ambari 2.1: Advanced cluster creation options (AMBARI-10750)Ambari 2.2: Kerberized cluster deployments (AMBARI-13431) Stack advisor recommendations (AMBARI-13487)

Stack Upgrades

Stack Upgrades Rolling vs Express Upgrade modes Side-by-Side Bits and Configs

Bits:/usr/hdp/2.2.0.0-2041/usr/hdp/2.2.4.2-2/usr/hdp/2.3.0.0-3000

Configs:/etc/hive/conf/ (initial)/etc/hive/conf/v0 (HDP 2.2.4.2)/etc/hive/conf/v1 (HDP 2.3)

2.2.0.0 2.2.4.2 2.3.0.0minor jump major jump

Express vs Rolling Upgrade

Rolling Upgrade Services are up the entire time Upgrade one component at a time Robust and fault-tolerant Service checks performed frequently during the upgradeExpress Upgrade All services are brought down, upgraded and restarted Faster upgrade mode Planned service downtime Relatively service checks performed less frequently during the upgrade.

Stack Upgrade – Install Version

Install new version in parallel on all agents No downtime

Stack Upgrade – Orchestration

Not necessarily “one-click” but fully guided

Stack Upgrade – Upgrade Catalog

Upgrades are driven by upgrade catalogs defined in stack definitions. Defines upgrade groups and upgrade order Provides ability to modify configurations

– Set, move, delete, transform Upgrade steps can be marked as skippable and retryable Supports executing custom scripts during upgrade

Stack Upgrade – Upgrade Catalog

Stack Downgrade

Can trigger downgrade at any stage of the stack upgrade Cannot downgrade once stack upgrade has been finalized

Smart Configurations

Hadoop Configuration Challenges

Too many configurations– Which ones are important?

Too easy to mess up– What are valid/reasonable values?– What are the units?– Ok, what about dependencies?

Gets harder with combinations of services, host assignments, enabled features, CPU/RAM/disks, etc– Any recommendations? What am I doing wrong?

Smart Configurations

Ambari Smart Configs UI

Customizable layout

- Tabs- Sections- Sub-sections- Simple grid layout

(Advanced Tab contains remaining configurations)

New Widgets

- Sliders- Recommended- Minimum- Maximum- Increment Step

- Combos- Enumerated values

- Toggles- Binary options

- Spinners- Splits value into multiple

controls. Time in milliseconds split into days, hours, minutes.

- Lists- Enumerated values- Single select- Multi select

Implemented- HDFS- YARN- MapReduce- Hive- HBase

Stack Driven Layouts

Stack has theme.json file

Layout Tabs Sections Sub-sections

Placement Configs placement in sub-sections

Widgets Widget type Optional Units Bytes (B, KB, MB, GB, TB, PB) Time (Millis, Seconds, Minutes, Hours, Days, Months,

Years)

{ "name": "default", "description": "Default theme for HBASE service", "configuration": { "layouts": [ { "name": "default", "tabs": [ { "name": "settings", "display-name": "Settings", "layout": { "tab-columns": "3", "tab-rows": "3", "sections": [ ... ] } } ] } ], "placement": { "configuration-layout": "default", "configs": [...] }, "widgets": [ { "config": "hbase-env/hbase_master_heapsize", "widget": { "type": "slider", "units": [ { "unit-name": "GB" } ] } }, ... ] }}

Config Metadata and Dependencies

Extended Metadata Defined in property_value_attributes Hold non-UI metadata about value range,

increment, unit, etc

Dependencies Models bi-directional relationship between configs Depends On (property_depends_on)

Answers “which configs do I depend on?”

Depended By (dependencies) Answers “which configs are dependent on me?”

Ambari automatically updates dependencies

{ "StackConfigurations": { "final": "false", "property_depends_on": [ { "type": "yarn-site", "name": "yarn.nodemanager.resource.memory-mb" } ], "property_description": “The minimum allocation for every", "property_display_name": "Minimum Container Size (Memory)", "property_name": "yarn.scheduler.minimum-allocation-mb", "property_type": [], "property_value": "512", "property_value_attributes": { "type": "int", "maximum": "5120", "minimum": "0", "unit": "MB", "increment_step": "256" }, "type": "yarn-site.xml" }, "dependencies": [ { "StackConfigurationDependency": { "dependency_name": "hive.tez.container.size", "property_name": "yarn.scheduler.minimum-allocation-mb” } }, { "StackConfigurationDependency": { "dependency_name": "mapreduce.map.memory.mb", "property_name": "yarn.scheduler.minimum-allocation-mb” } }, { "StackConfigurationDependency": { "dependency_name": "mapreduce.reduce.memory.mb", "property_name": "yarn.scheduler.minimum-allocation-mb” } }… ]}

Metrics

Ambari Metrics Service (AMS) - Goals

Ability to collect metrics from Hadoop and other Stack services Ability to collect system level metrics Ability to retain metrics at a high precision for a configurable time period Ability to automatically purge metrics after retention period Provide integration point for metrics collection and retention by external system Trigger alerts based on metrics in Ambari

Ambari Metrics System - Architecture

AMS Grafana

Ambari 2.2.2 Powerful dashboard builder integrated with AMS Pre-built Grafana dashboards for host-level and service-level metrics User can build and save custom dashboards

AMS Grafana

Alerts

Alert – Types

Type Description Status ThresholdsConfigurable?

PORT Watches a port based on a configuration property such as the URI. OK, WARN, CRIT Yes (seconds)

WEB Watches an HTTP or HTTPS endpoint and determines connectivity and HTTP status code. OK, WARN, CRIT No

AGGREGATE Aggregate of status for another alert definition. OK, WARN, CRIT Yes (percentage)

METRIC Watches a metric or series of metrics in JMX and compares a mathematical result against a threshold. OK, WARN, CRIT Yes (variable)

SCRIPT Uses a custom script to handle checking. OK or CRIT No

UI – Current Alerts

Configured by default; managed via the the web client

UI – Host Alerts

Automatically refreshes Query alert history

UI– Customization & Instances

Status text, thresholds, and interval

Ambari Views

View Framework Provide various applications accessible from Ambari Web UI – interact with the cluster via a

browser from a single place for all users (cluster operators, data analysis, developers, etc)

Easy to develop No need to understand Ambari core code – view development is just like creating any other web

application

Easy to deploy Packaged as a single jar file Auto create / auto configure

CS Queue Manager for Cluster Operators

Capacity Scheduler Queue Manager

HDFS File Browser for General Users

HDFS File Browser

Job Analysis for Developers

Troubleshoot Tez JobsTroubleshoot / Improve Hive queries

Query Editors for Data Analysts

Create, edit, execute, and analyze Hive queries Create, edit, and execute Pig scripts

Ambari Server in Views-Only mode

AmbariServer Cluster managed by Ambari

AmbariServer “Views-only” mode

(aka “Stand-alone” mode)Cluster not managed by Ambari

Management

Use Views

Use Views on existing clusters not managed by Ambari Can use Views against multiple clusters

Kerberos Automation

Ambari 2.0 Ambari manage Kerberos principals and keytabs Works with existing MIT KDC or Active Directory Once Kerberized, seamlessly handle:

Adding new hosts Adding new components to existing hosts Adding new services Moving components to different hosts

Agenda

Ambari Overvie

Agenda

Ambari Overvie

Thank You!

Try Ambari Follow the Ambari Quick Start Guide https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide

Learn more Visit the project website http://ambari.apache.org/

Get Involved User Mailing List: user-subscribe@ambari.apache.org

Developer Mailing List: dev-subscribe@ambari.apache.org

Use JIRA to file bugs and improvement requests https://issues.apache.org/jira/browse/AMBARI/

Jayush Luniya @ Hortonworks (Apache Ambari PMC)

Future Roadmap

AMS Grafana Integration Ambari Management Packs Ambari Logsearch Patch Upgrades Multi Service Versions Multi Service Instances

Largest production clusters managed by Ambari ~1600 nodes, ~800 nodes

Largest test cluster for Ambari scale testing ~400 nodes

Largest test cluster where rolling upgrade was performed ~400 nodes~40 hours

Managing Enterprise Hadoop Clusters with Apache Ambari

Technology

Dynamic Hadoop Clusters -...

Network Management Operations - INPM€¦ · • Apache...

Multi-tenant Apache Hadoop Clusters - BI Consulting

Hortonworks Data Platform€¦ · 1.2. Viewing the Ambari.....

Protecting your Critical Hadoop Clusters Against Disasters

Hadoop Personal Project: Apache...

Adaptive Preshuffling in Hadoop Clusters - Auburn...

StreamlineHadoop DevOpswith* ApacheAmbari ·...

Apache Ambari BOF - OpenStack - Hadoop Summit 2013

Streamline Hadoop DevOps with Apache Ambari

Administering Ambari - Cloudera...Using the Ambari Admin...

Dynamic Hadoop Clusters

Scheduling scheme for hadoop clusters

Understanding Hadoop Clusters and the Network

Multi-tenant Apache Hadoop Clusters - BI...

Performance Issues on Hadoop Clusters