Streamline Apache Hadoop Operations with Apache Ambari and SmartSense

1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

Streamline Apache Hadoop Operations with Apache Ambari and SmartSense

2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

Streamlining Apache Hadoop Operations with Ambari & SmartSense

Roni FontaineDirector Product

Marketing

Paul CoddingDirector Product

Management


Agenda

Newest features and highlights in Ambari 2.5

How to double Hadoop performance using SmartSense 1.4

Flexible enterprise support model options


What’s New in Ambari 2.5

Service Auto Start (AMBARI-2330)

DB Inconsistency Self-Healing (AMBARI-18990)

Simplified Log Rotation Configuration (AMBARI-16880)

HDFS TopN User & Operation Visualization (AMBARI-19320)

Download All Client Configurations (AMBARI-19275)

Configuration Change Communication (AMBARI-19572)

Add/Remove JournalNodes (AMBARI-7748)

Ignore Host Pre-Check when adding Hosts (AMBARI-18817)

Grafana dashboard for Ambari (AMBARI-17589)

AMS Collector High Availability TP (AMBARI-15901)

Password Credential Store Management (AMBARI-18650)

Post-user-creation script hook (AMBARI-18722)

ZK ACL and SASL Configuration (AMBARI-17324)

Ambari SPNEGO support (AMBARI-18365)

Core Features

Security Features

HDP “Fenton” Support

Port Preserving HS2 Rolling Upgrade (AMBARI-18591)

Log Search TP Update (AMBARI-18821)

Built-In SNMP MIB (AMBARI-19257)

SmartSense Mandatory Install (AMBARI-18346)

RegionServer GC Configuration Optimization (AMBARI-19573)

Core Features Continued


Ambari 2.5 + HDP Support Matrix

Added support for HDP 2.6

Deprecated support for HDP 2.4, and removed support for HDP 2.2

HDP 2.6 HDP 2.5 HDP 2.4 HDP 2.3 HDP 2.2

Ambari 2.5

Ambari 2.4

Ambari 2.2.1

Ambari 2.2

deprecated

deprecated deprecated

deprecated

deprecated


Ambari 2.5 + OS Support Matrix

Added support for Ubuntu 16 (for HDP 2.6 Only)

Dropped support for Ubuntu 12

RHEL 6 RHEL 7 Debian 7 SLES 11 SLES 12 Ubuntu 12 Ubuntu 14 Ubuntu 16

Ambari 2.5

HDP “Fenton” Only

Ambari 2.4

HDP “Erie” Only

Ambari 2.2


AgendaWhat’s New in Ambari 2.5.0

Feature Highlights: Service Auto Start


How it works

Ambari Agent

Host

Ambari Agent

Host

Ambari Agent

Host

Ambari Server

Ambari DB

LDAPAuthN

Host Restart

• Agent checks in with Server• Server asks agent to check status of

deployed components• Agent reports back state• Server asks agent to start components

not in the desired state

HDP ComponentComponent Stops



Feature Highlights: Add/Remove JournalNodes



Feature Highlights: Simplified Log Rotation Configuration



Feature Highlights: Log Search Improvements


Log Search Demo



Feature Highlights: Configuration Change Communication


Stack Advisor Behavioral Changes

Add Service

Delete Service

Add Host Component

Move Master

HA Wizards

Delete Host

Add/Remove ZooKeeper Server

Goal: Ensure users are aware of configuration changes related to the activity they are performing.

What we found: Identified multiple locations in which configurations were being changed without notifying the user explicitly

What Changed: New pop-ups and more communication when changes are required










Feature Highlights: Download All Client Configurations




Client Configuration



Feature Highlights: HDFS TopN User & Operation Visualization


HDFS TopN User & Operation Visualization*

Operations

Users

Users / Operations

*Only works with HDP 2.6



Feature Highlights: AMS Collector High Availability TP


AMS Current state

AMBARICollector API

GRAFANA

HBASE

PHOENIX

SYSTEM

MO

NIT

OR

S

HDPSERVICES SI

NK

S

METRICS COLLECTOR

API consumers

Sinks – HDFS, YARN, HBASE, STORM, KAFKA, FLUME, ACCUMULO, LogSearch, Hive, Nifi

Monitors – System metrics

Grafana

Ambari




Feature Highlights: Post User Creation Hook


Post User Creation Hook

Disabled by default

Enabled with two properties– ambari.post.user.creation.hook.enabled=true

– ambari.post.user.creation.hook=/var/lib/ambari-

server/resources/scripts/post-user-creation-hook.sh

Works with manual user creation as well as LDAP sync

Can be run as a one-off command as well



Hortonworks Connected Data Platforms and Solutions

Data Services

Hortonworks Solutions

Enterprise DataWarehouse Optimization

Cyber Security andThreat Management

Internet of Thingsand Streaming Analytics

Data CenterHortonworks Data Suite

HDFHDP

HortonworksConnection

CloudHortonworks Data Cloud

AWS HDInsight

Hortonworks Connection

Enablement Subscription

SmartSense™Premier Operational Support

Educational Services

Professional Services

Community Connection


Accelerate Case Resolution

Prevent Issues

Understand Your Cluster


SmartSense Today – Accelerate Case Resolution

SmartSense provides Hadoop Operators with an Ambari Integrated tool to quickly capture diagnostic information for specific services and hosts into a single “Bundle” that’s automatically uploaded to Hortonworks Support.

Significantly reduces the back-and-forth nature of troubleshooting issues.

A M B A R I

O P SH O R T O N W O R K S

S U P P O R T

S U P P O R TC A S E

S m a r t S e n s eS E R V E R

B U N D L E

G AT E W AY



Prevent Issues



SmartSense Today – Prevent Issues

SmartSense analyzes Bundles for configuration issues – recommendations are produced and made available for each cluster in the Hortonworks Support Portal

Recommendations prevent operational issues, and improve performance and overall cluster throughput.


After Applying only 3 SmartSense Recommendations

They can now run 1200 concurrent jobs

...with only 350 waiting jobs at peak hours

Issue: YARN @ capacity, struggling to add more use cases

Before SmartSense

Could only run 500 jobs concurrently

1100 jobs would be pending waiting for

resources at peak hours

With SmartSense = 2X Throughput Improvement



Prevent Issues



“Who’s creating all of these small files in HDFS!?”

“What are my top 10 most active users, and longest running jobs?”

“How much should I charge users for their cluster resource use?”

SmartSense Today – Understand Your Cluster



Chargeback Reporting




HDFS Dashboards




HDFS Dashboards

YARN Dashboards


Impact of Hortonworks SmartSense

0200400600800

100012001400

WithoutSmartSense

WithSmartSense

Concurrent Jobs

B U N D L E

2X Throughput Improvement

Address 30% of Issues

Configuration Issues

Avoid 10% of Sev1 Issues

Production Down

Single-Bundle Case Resolution 25% of the Time

SmartSense

Troubleshooting Bundle


Hortonworks Connection Ensures Success of Your Big Data Journey


New Hortonworks Flex Support Subscription

Universal, Usage-based Support Subscription

Cloud & On-Prem

HDCloud

IaaS

On-Prem

Single, flexible, portable support for transition to cloud

New support offering for Spark Data Science, ETL, EDW-Analytics in the Cloud

Performance optimization with Hortonworks SmartSense


Hortonworks Data Cloud for AWS

Pre-tuned for use with AWS

Powered by HDP

Focused on business agility

Prescriptive, ephemeral use cases

• Data Science (Apache Spark & Zeppelin)• Analytics (Apache HIVE)• ETL (Apache HIVE & Spark)• Business Intelligence (OLAP/Druid)


EASE OF USE: Choose from a set of pre-tuned and pre-configured templates.


Choose from a Set of Prescriptive, Ephemeral Workload Clusters

Data Science

Spark 1.6, 2.1

Business intelligence

(OLAP) Druid TP

Analytics & Reporting with

HIVE 2.2 & LLAP

ETL with Spark


Try it! Hortonworks Data Cloud for AWS Marketplace

How –to Video: https://hortonworks.com/video/hd-cloud-aws/

Try it Now: https://aws.amazon.com/marketplace/pp/B01LXOQBOU

https://hortonworks.com/video/hd-cloud-aws/

https://aws.amazon.com/marketplace/pp/B01LXOQBOU


Flex Support Subscription Components


Access to Expertise Plus Freedom, Agility and Flexibility

Expertise for Data Science, ETL and Analytics in HDCloud

Match support costs to usage patterns

Migrate infrastructure on your own terms

Support for Dev, QA & ephemeral production

Built-In Performance Optimization

https://hortonworks.com/products/cloud/aws/


Questions?References

https://hortonworks.com/apache/ambari/ hortonworks.com/services/support/smartsense/ hortonworks.com/services/support/flex/ hortonworks.com/products/cloud/aws/

Streamline Apache Hadoop Operations with Apache Ambari and SmartSense

Technology