Hadoop Security Features that make your risk officer happy

HADOOP SECURITY FEATURES

That make your risk officer happy

By Anurag Shrivastava, ING Commercial Bank, Amsterdam

@shri2201

Security for Hadoop

Source: http://blogs.gartner.com/merv-adrian/2014/01/21/security-for-hadoop-dont-look-now/

Hadoop Security Features 2

Hadoop in Enterprise

Data Lake – an important information assets for enterprise

Data from System of

Records and Logs are stored

in Hadoop

Significant cost

savings for

Enterprise

Diverse types of

users

Picture Source: http://arunkottolli.blogspot.nl/2014/03/understanding-data-in-big-data.html


Operational Security in Enterprise• User Access Management

• Security Event Monitoring

• Application State Monitoring

• Security Testing

• Patch Management

• Data Protection

• Backup and restore


User Access Management

Requirements

Privileged, group and generic accounts

Separation of technical and business users

Separation of environments (DTAP)

Separation of admins and other users

Separation of users in different business roles

Application of four eyes principle when entering or

changing the data


Security Event Monitoring• Definition of application specific events

• All login attempts failed or successful

• Unauthorized attempt to access a table or file

• Operational performance of application

• Name node performance

• CPU, Disk

• Integration with Master Control Room

• Alerting the asset manager


Data Protection (1/2)• Confidentiality

• Protect information from unauthorized

disclosure

• Integrity

• Ensure the accuracy, completeness and

timeliness of information and prevent data

tempering

• Availability

• Ensure that information and service is

available when required

Picture Source:

http://www.attix5.co.uk/thought-

leadership/why-data-protection-software-

essential-good-nights-sleep


Data Protection (2/2)• Confidentiality

• Logon

• Access Control

• Malicious code protection


• Encryption

• Integrity

• Message authentication code

• Data Lineage

Picture Source:

http://www.attix5.co.uk/thought-

leadership/why-data-protection-software-

essential-good-nights-sleep


Security under spotlight in Data Lake

• All kinds of enterprise data – structured,

semi-structured and unstructured

• Many groups of users – Data Scientists,

Analysts, Engineers, Marketers,

Managers

• Long term retention of data

• Different types of workloads

• Value of data grows as the data from

different sources are combined in Data

Lake

Picture source: http://beyondplm.com/2014/05/05/plm-downstream-usage-and-future-information-rivers/


Data Lake Risks

• Data Lake is an attractive target of inside and outside attackers

• Security compromise in Data Lake can have major or catastrophic

business impact

IT Risk assessment gives Hadoop implementation

the highest risk rating for Data Lake use case.


Lab Like Security is not Enough

Play Area Big Data Predictive Analytics

Lab

Production

System


Predictive Analytics Lab

Stepping Stone

(Citrix)

18 x Hadoop

Nodes

GIT, Libraries,

Build Tools

Monitoring

Services

Data Files in

Batches

Dedicated VLAN Shared ServicesShared Services

SMTP Relay

Internet via

Corporate

Infrastructure

Firewall Rules

Guard the

Perimeter

Security

Of Hadoop

Cluster

18 x Hadoop

Nodes

Lab like security works for a small group of people


Limitations of Hadoop• No “Data at Rest” Encryption

• A Kerberos-Centric Approach

• Limited Authorization Capabilities

• Complexity of the Security Model and Configuration

Unfortunately this is not sufficient for Data Lake that ingests all the

data and caters to thousands of users.


Hadoop Security

Hadoop Security Solutions from Major Vendors

Hortonworks acquires XASecure to

bring ACLs in Hadoop

Apache Ranger

Apache Knox

Apache Falcon

Cloudera is working on Project Rhino Project Rhino

Apache Sentry


HDP-Apache Ranger


Apache RangerApache Ranger currently supports authorization, auditing and security administration of limited

number of HDP components

Hive

HBase

StormKnox

HDFS


Apache Ranger Goals

1. Centralized security administration to manage all security related tasks in

a central UI or using REST APIs.

2. Fine grained authorization to do a specific action and/or operation with

Hadoop component/tool and managed through a central administration tool

3. Standardize authorization method across all Hadoop components.

4. Enhanced support for different authorization methods - Role based access

control, attribute based access control etc.

5. Centralize auditing of user access and administrative actions (security

related) within all the components of Hadoop.


Apache Knox and Hadoop Services

Hadoop Services

Covered

• WebHDFS (HDFS)

• Templeton

(HCatalog)

• Stargate (HBase)

• Oozie

• Hive/JDBC


Apache Falcon• Visualize Data Pipeline Lineage

• Track Data Pipeline audit logs

• End to End Monitoring of Data

Pipeline

• Policies for Data Replication and

Retention


Apache Sentry and Project Rhino


Goals of Project Rhino

• Provide encryption with hardware-enhanced performance

• Support enterprise-grade authentication and single sign-on for

Hadoop services

• Provide role-based access control in Hadoop with cell-level

granularity in HBase

• Ensure consistent auditing across essential Apache Hadoop

components


Apache Sentry and Project Rhino


Making Risk Officer Happy

• Hadoop security has

more to offer

• Role based access

• Audit logging

• Data encryption

• User Access Management


• Application State Monitoring

• Security Testing

• Patch Management

• Data Protection

• Backup and restore

Overlapping efforts of vendors, Lack of complete coverage for all products,

Varying commitment to open source would slow down the adoption of Hadoop.


THANK YOUAnurag Shrivastava

@shri2201

Hadoop Security Features that make your risk officer happy

Technology

hadoop source

operational security

hadoop attractive

biggest data security

payment cardshadoop

data stores

enterprise data lake

important information