Trusted Analytics as a Service (BDT209) | AWS re:Invent 2013

Trusted Analytics as a Service

Vin Sharma, Intel Corporation

November 12, 2013

Data-Driven discoveries depend on analytics

Consumer Behavior Security & Risk Management

Operational Efficiency

Location Aware Ad Placement

Buyer Protection Program

Personalized Preventive Care

Claim Fraud Reduction

Traffic Optimization

Smart Energy Grid

Machine-generated data requires end-to-end analytics

3

1990 2000 2010

Traditional Analytics

• Descriptive analysis, business intelligence, and reporting

• Internally sourced, relatively small, structured data

• Analysts and Quants huddled in back-rooms

Big Data Analytics

• Interactive analysis, complex queries, and data-intensive models

• Fast and large amounts of poly-structured data from multiple sources

• Data Scientists at the fore

End-to-End Analytics

• Real-time analysis of streaming data from IoT

• Predictive and prescriptive analysis integrated into organizational processes

• Widespread access to tools

End-to-end analytics for the Internet of things era

4

Servers Storage Network

Data Platform

Analytics Platform

Verticals Help build lighthouse solutions for targeted

verticals

Enable horizontal platform for e2e

analytics

Accelerate evolution of Apache Hadoop

Catalyze architectural transitions to drive

growth

End-to-end analytics needs software-defined infrastructure

Compute Storage Network

Thermals Power Location

Compliance Orchestration Service Assurance

Datacenter Operating Systems

Intelligent Workload Placement

Composable Resource Pools

Datacenter Facilities

Scheduler File System Processing API Security

Apache Hadoop as a Datacenter Operating System

6

Scheduler

File Systems

Process Mgmt

API

Memory Mgmt

I/O

Future NVM

HDFS, LustreFS, GlusterFS, Ceph + Kafka

Future Fabric Controller

Hadoop, Storm,

GraphLab, Spark, Shark,

MPI

YARN + SLURM | Moab

Expressway

Security

TXT, AES-NI Rhino Data Governance

Intel leadership in foundational technologies of big data

Enabling technical computing on

massive data sets

Helping organizations build open

interoperable clouds

Contributing code and fostering ecosystem

HPC Cloud Open Source

* Other names and brands may be claimed as the property of others.

Intel employs over 10,000 software developers

Hadoop in a virtualized infrastructure

• Good – Agility: Lets you bring up and tear

down resources quickly on demand. – Fault Tolerance: Protect against

SPOF in Hadoop/HDFS (NN, JT, Zookeeper) and reduce downtime for planned updates.

– Resource Efficiency: Run multiple Hadoop clusters or other applications

– Security: Isolate clusters or nodes – Simpler management of datacenter

• Bad – Performance hit of virtualization is

indeterminate and hard to optimize – Storage configuration with SAN

and NAS is very different from the disk attached storage of typical Hadoop

– Nested virtualization with JVM in a VM is philosophically uncomfortable

Hadoop in the cloud

• Good – If your data is stored in a cloud

provider's storage infrastructure, moving compute to data is logical.

– If your analytics jobs are infrequent, you can rent the cluster only when you need it.

– Isolation offers security. – Easy to use. Easy to expand. – Pay as you go.

• Bad – Cost of storage rises at the rate

of ingest and storage. – Cost of compute rises with

cluster time. There is no "spare cluster time" for low priority work.

– Hadoop makes assumptions about running in a fixed physical infrastructure.

Deploying IDH on AWS • Use a hop machine to connect into the VPC (private network)

for IDH. This is the only machine that allows inbound SSH connections from clients on the internet. You must SSH into the hop machine to gain access to machines in the VPC.

• The hop machine hosts the aws_system scripts.

• Although data may be retained on AWS, do not expect data to always be saved. Assume machines and data will removed at any time. Save any needed data or results to another location.

Deploying IDH on AWS createIDHCluster.sh • Picks a management node. This should be the first IP address in the

list of IPs that you specify in the nodeips argument. • After the nodes are running, verifies it can SSH in as the root user

on the management node and as either the root user or some other non-root user on the other nodes.

• Checks that IDH is NOT installed on any of the nodes. If it cannot SSH in or IDH is installed, the script exits with a failure.

• Copies over the IDH tarball and the idhscripts.tar to the management node.

• On the management node, sets up the yum repository and installs intel manager. Then installs and configures IDH on all the nodes.

Script options

bash ./createIDHCluster.sh --nodeips= 10.0.20.240,10.0.20.241,10.0.20.242,10.0.20.243

--idhtarball= /share/dev_builds/intelhadoop-3.0+19555-en-commercial-without-reg.el6.x86_64.tar.gz

--scripttarball= /home/vin/idhscripts.tar

Why Intel Distribution for Apache Hadoop

Intel® Distribution for Apache Hadoop* software

Hardware-enhanced performance & security Enables partner innovation in analytics Strengthens Apache Hadoop* ecosystem

Intel employs over 300 people developing and supporting big data software

Hadoop Security and Compliance Challenges Hadoop is an ecosystem of loosely coupled components

HiveQL Interactive Query

Sqoo

p

RDB

Data

Col

lect

or

Flum

e Lo

g Da

ta C

olle

ctor

Zook

eepe

r Co

ordi

natio

n

Pig Data manipulation

YARN (MRv2) Distributed Processing Framework

HDFS 2.0 Hadoop Distributed File System

R connectors statistics

Giraph Graph analysis framework

HCatalog Metadata

Hive (compiler, planner, driver)

HBase Coprocessors Data execution engine

HBase Real-time Distributed BigTable

Mahout Data mining

Oozie Data flow

Hadoop Security and Compliance Challenges Components sharing an authentication framework

HiveQL Interactive Query

Sqoo

p

RDB

Data

Col

lect

or

Flum

e

Log

Data

Col

lect

or

Zook

eepe

r Co

ordi

natio

n

Pig Data manipulation

YARN (MRv2) Distributed Processing Framework

HDFS 2.0 Hadoop Distributed File System

R connectors statistics

Giraph Graph analysis framework

HCatalog Metadata

Hive (compiler, planner, driver)

HBase Coprocessors Data execution engine

HBase Real-time Distributed BigTable

Mahout Data mining

Oozie Data flow

Hadoop Security and Compliance Challenges Components capable of access control

HiveQL Interactive Query

Sqoo

p

RDB

Data

Col

lect

or

Flum

e

Log

Data

Col

lect

or

Zook

eepe

r Co

ordi

natio

n

Pig Data manipulation

YARN (MRv2) Distributed Processing Framework

HDFS 2.0 Hadoop Distributed File System

R connectors statistics

Giraph Graph analysis framework

HCatalog Metadata

Hive (compiler, planner, driver)

HBase Coprocessors Data execution engine

HBase Real-time Distributed BigTable

Mahout Data mining

Oozie Data flow

Hadoop Security and Compliance Challenges Components capable of admission control

HiveQL Interactive Query

Sqoo

p

RDB

Data

Col

lect

or

Flum

e

Log

Data

Col

lect

or

Zook

eepe

r Co

ordi

natio

n

Pig Data manipulation

YARN (MRv2) Distributed Processing Framework

HDFS 2.0 Hadoop Distributed File System

R connectors statistics

Giraph Graph analysis framework

HCatalog Metadata

Hive (compiler, planner, driver)

HBase Coprocessors Data execution engine

HBase Real-time Distributed Big Table

Mahout Data mining

Oozie Data flow

Hadoop Security and Compliance Challenges Components capable of (transparent) encryption

HiveQL Interactive Query

Sqoo

p

RDB

Data

Col

lect

or

Flum

e

Log

Data

Col

lect

or

Zook

eepe

r Co

ordi

natio

n

Pig Data manipulation

HDFS 2.0 Hadoop Distributed File System

R connectors statistics

Giraph Graph analysis framework

HCatalog Metadata

Hive (compiler, planner, driver)

HBase Coprocessors Data execution engine

HBase Real-time Distributed Big Table

Mahout Data mining

Oozie Data flow

YARN (MRv2) Distributed Processing Framework

Hadoop Security and Compliance Challenges Components sharing a common policy engine

HiveQL Interactive Query

Sqoo

p

RDB

Data

Col

lect

or

Flum

e

Log

Data

Col

lect

or

Zook

eepe

r Co

ordi

natio

n

Pig Data manipulation

HDFS 2.0 Hadoop Distributed File System

R connectors statistics

Giraph Graph analysis framework

HCatalog Metadata

Hive (compiler, planner, driver)

HBase Coprocessors Data execution engine

HBase Real-time Distributed Big Table

Mahout Data mining

Oozie Data flow

YARN (MRv2) Distributed Processing Framework

Hadoop Security and Compliance Challenges Components sharing a common audit log format

HiveQL Interactive Query

Sqoo

p

RDB

Data

Col

lect

or

Flum

e

Log

Data

Col

lect

or

Zook

eepe

r Co

ordi

natio

n

Pig Data manipulation

HDFS 2.0 Hadoop Distributed File System

R connectors statistics

Giraph Graph analysis framework

HCatalog Metadata

Hive (compiler, planner, driver)

HBase Coprocessors Data execution engine

HBase Real-time Distributed Big Table

Mahout Data mining

Oozie Data flow

YARN (MRv2) Distributed Processing Framework

Project Rhino

• Strategic Objectives • Framework support for encryption and key management • Token based authentication and SSO for internal cluster services • Role-based access control for simpler administration of authorizations • A common authorization framework, optional but easy to adopt • Consistent audit logging, enhanced for compliance support

• Current Projects

• Develop crypto framework in Hadoop Common • Enable transparent encryption in HBase • Extend HBase support for ACLs to the cell level

HDFS | Lustre | GlusterFS Hadoop Compatible File Systems

YARN (+MapReduce) Distributed Processing Framework

HBa

se

Zook

eepe

r C

oord

inat

ion

Flum

e Lo

g C

olle

ctor

Sq

oop

Dat

a Tr

ansf

er

Hive Query

Ooz

ie

Wor

kflo

w

Mahout Machine Learning

Pig Scripting

R Stats

Hcatalog Metadata

Deployment

Upgrade

Configuration

Unified Logging

Tuning

Alerts

Resource Monitor

Job Profiler

Security Controls

Heat Map

Rhino (Security) [Encryption, Authentication, Authorization, Auditing]

High Availability and Disaster Recovery

HBase Explorer

Recommendation Engine Behavior Model Vertical Accelerators

Analytics Workbench

Connectors Netezza, Oracle, SAP, SQLServer,

Teradata, DB2 Kafka

Event Bus Lucene, Solr

Search Tribeca

Graph Mining Gryphon

Low-latency SQL-92

SLURM Scheduler

Intel Distribution: Security

23 All external names and brands are claimed as the property of others.

Enterprise data requires defense in depth

Firewall

Gateway

Authn

AuthZ

Encryption

Audit & Alerts

Isolation

Intel Expressway protects Hadoop APIs

Authn

RBAC

Encryption

Containment • Enforces consistent security policies across all Hadoop services

• Serves as a trusted proxy to Hadoop, Hbase, and WebHDFS APIs

• Complies with Common Criteria EAL4+, HSM, FIPS 140-2

certifications

• Deploys as software, virtual appliance, or hardware appliance

Hcatalog

Stargate

WebHDFS

Firewall

REST APIs

Kerberos authenticates Hadoop services

Encryption

Containment

Firewall

APIs

Authentication

KDC request ticket

send service ticket

request service

send respose

validate ticket

4

1 2 3 5 Intel

Manager

• Wizard enables setup of secure cluster with encrypted key exchange

• Manager generates principal and keytab for Hadoop services

• Manager enables batch upload of keytab files

Intel Manager simplifies role-based access control

Firewall

AuthZ

• File, table, and service-level controls

• Intel Manager pushes ACLs to each node

Intel Distribution provides HDFS encryption

Firewall

RBAC

• Extends compression codec into crypto codec

• Provides an abstract API for general use

MapReduce RecordReader

Map Combiner Partitioner

Local Merge & Sort Reduce

RecordWriter

HDFS

Decrypt

Encrypt

Derivative Encrypt

Derivative Decrypt

Crypto Codec Framework

• Extends compression codec and establishes a common abstraction of the API level that can be shared by all crypto codec implementations as well as users that use the API

CryptoCodec cryptoCodec = (CryptoCodec) ReflectionUtils.newInstance(codecClass, conf); CryptoContext cryptoContext = new CryptoContext(); ... cryptoCodec.setCryptoContext(cryptoContext); CompressionInputStream input = cryptoCodec.createInputStream(inputStream);

...

• Provides a foundation for other components in Hadoop* such as MapReduce or HBase* to support encryption features

Crypto Codec Framework: Class Hierarchy <<Java Interface>>

Compressor

<<Java Interface>>

Compression Code

<<Java Interface>>

Decompressor

<<Java Interface>>

Encryptor

<<Java Interface>>

Decryptor <<Java Interface>>

Crypto Codec

<<Java Class>>

Crypto Context

<<Java Interface>>

Key Provider

<<Java Interface>>

Key ProfileResolver <<Java Class>>

Key

<<Java Class>>

KeyProfile

0..1 0..1 0..1

Crypto Codec: API Example The usage is aligned with compression codec but with context supporting Configuration conf = new Configuration(); CryptoCodec cryptoCodec = (CryptoCodec) ReflectionUtils.newInstance(AESCodec.class, conf); CryptoContext cryptoContext = new CryptoContext(); cryptoContext.setKey(Key.derive(password)); cryptoCodec.setCryptoContext(cryptoContext); DataInputStream input = inputFile.getFileSystem(conf).open(inputFile); DataOutputStream outputStream = outputFile.getFileSystem(conf).create(outputFile); CompressionOutputStream output = cryptoCodec.createOutputStream(outputStream); // encrypt the stream writeStream(input, output); input.close(); output.close();

Crypto Codec: A Simple MapReduce Example

The usage is aligned with compression codec usage in MapReduce job but with context resolving Job job = Job.getInstance(conf, "example"); JobConf jobConf = (JobConf)job.getConfiguration(); FileMatches fileMatches = new FileMatches( KeyContext.refer("KEY00", Key.KeyType.SYMMETRIC_KEY, "AES", 128)); fileMatches.addMatch("^.*/input1\\.intelaes$", KeyContext.refer("KEY01", Key.KeyType.SYMMETRIC_KEY, "AES", 128)); String keyStoreFile = "file:///" + secureDir + "/my.keystore"; String keyStorePasswordFile = "file:///" + secureDir + "/my.keystore.passwords"; KeyProviderConfig keyProviderConfig = KeyProviderCryptoContextProvider.getKeyStoreKeyProviderConfig( keyStoreFile, "JCEKS", null, keyStorePasswordFile, true); KeyProviderCryptoContextProvider.setInputCryptoContextProvider( jobConf, fileMatches, true, keyProviderConfig);

Key Distribution and Protection for MapReduce • Targets

– A framework at MapReduce side for enabling crypto codec in MapReduce job such as key context resolving, distribution and protection

– Enabling different key storage or management systems to plug-in for providing keys

– Satisfying the common requirements that stage and file of a single job may use different keys

• A complete key management system is not part of Intel® Distribution for Apache Hadoop* software – An API to integrate with an external key manage system is included

Secrets Distribution

Shared storage or distributed in each

node

Node A

IM Agent

task

task

task

task

Job credentials & data

encryption key

Node B

IM Agent

task

task

task

task

Job credentials & data encryption key

1 2

3

IM Agent: Intel® Manager for Apache Hadoop* is a service resident in each cluster node.

Pig* & Hive* Encryption: Overview Client

Pig* Hive*

MapReduce

HDFS*

Secrets P

rotection S

ervice

Intel ® Manager for

Apache H

adoop*

software

Local Disk

Cluster

Intel

Encrypted Intermediate data

Encrypted Job input/output data

Encrypted secrets Encrypted secrets

Decrypt secrets

https for uploading master key

Master key also be encrypted

Pig* & Hive* Encryption

• Pig* Encryption Capabilities – Support of text file and Avro* file format – Intermediate job output file protection – Pluggable key retrieving and key resolving – Protection of key distribution in cluster

• Hive* Encryption Capabilities – Support of RC file and Avro file format – Intermediate and final output data encryption – Encryption is transparent to end user without changing existing SQL

HBase* Encryption • Transparent table/CF encryption – HBase-7544 • Transparent encryption for ZooKeeper* commit log – ZooKeeper-1688

Crypto Software Optimization

Multi-Buffer • Process multiple independent

data buffers in parallel • Improves cryptographic

functionality up to 2-9X

Intel® Data Protection Technology

Advanced Encryption Standard New Instructions (AES-NI) • Processor assistance for performing AES

encryption

• Makes enabled encryption software faster and stronger

Internet

Data in Motion Secure transactions used pervasively in ecommerce, banking, etc.

Data in Process Most enterprise and cloud applications offer encryption options to secure information and protect confidentiality

Data at Rest Full disk encryption software protects data while saving to disk

AES-NI -

AES-NI Accelerated Encryption

Non Intel®

AES-NI With Intel®

AES-NI Intel® AES-NI Multi-Buffer

Enc

rypt

ion

Dec

rypt

ion

Enc

rypt

ion

Dec

rypt

ion

AES-NI - Advanced Encryption Standard New Instructions See slide in backup for test environment

hadoop.intel.com

Legal Disclaimer INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm Intel, Xeon, Look Inside and the Intel logo are trademarks of Intel Corporation in the United States and other countries.

*Other names and brands may be claimed as the property of others. Copyright ©2013 Intel Corporation.

Legal Disclaimer

• Intel® AES-NI requires a computer system with an AES-NI enabled processor, as well as non-Intel software to execute the instructions in the correct sequence. AES-NI is available on select Intel® processors. For availability, consult your reseller or system manufacturer. For more information, see Intel® Advanced Encryption Standard Instructions (AES-NI).

• Software Source Code Disclaimer: Any software source code reprinted in this document is furnished under a software license and may only be used or copied in accordance with the terms of that license.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Risk Factors The above statements and any others in this document that refer to plans and expectations for the third quarter, the year and the future are forward-looking statements that involve a number of risks and uncertainties. Words such as “anticipates,” “expects,” “intends,” “plans,” “believes,” “seeks,” “estimates,” “may,” “will,” “should” and their variations identify forward-looking statements. Statements that refer to or are based on projections, uncertain events or assumptions also identify forward-looking statements. Many factors could affect Intel’s actual results, and variances from Intel’s current expectations regarding such factors could cause actual results to differ materially from those expressed in these forward-looking statements. Intel presently considers the following to be the important factors that could cause actual results to differ materially from the company’s expectations. Demand could be different from Intel's expectations due to factors including changes in business and economic conditions; customer acceptance of Intel’s and competitors’ products; supply constraints and other disruptions affecting customers; changes in customer order patterns including order cancellations; and changes in the level of inventory at customers. Uncertainty in global economic and financial conditions poses a risk that consumers and businesses may defer purchases in response to negative financial events, which could negatively affect product demand and other related matters. Intel operates in intensely competitive industries that are characterized by a high percentage of costs that are fixed or difficult to reduce in the short term and product demand that is highly variable and difficult to forecast. Revenue and the gross margin percentage are affected by the timing of Intel product introductions and the demand for and market acceptance of Intel's products; actions taken by Intel's competitors, including product offerings and introductions, marketing programs and pricing pressures and Intel’s response to such actions; and Intel’s ability to respond quickly to technological developments and to incorporate new features into its products. The gross margin percentage could vary significantly from expectations based on capacity utilization; variations in inventory valuation, including variations related to the timing of qualifying products for sale; changes in revenue levels; segment product mix; the timing and execution of the manufacturing ramp and associated costs; start-up costs; excess or obsolete inventory; changes in unit costs; defects or disruptions in the supply of materials or resources; product manufacturing quality/yields; and impairments of long-lived assets, including manufacturing, assembly/test and intangible assets. Intel's results could be affected by adverse economic, social, political and physical/infrastructure conditions in countries where Intel, its customers or its suppliers operate, including military conflict and other security risks, natural disasters, infrastructure disruptions, health concerns and fluctuations in currency exchange rates. Expenses, particularly certain marketing and compensation expenses, as well as restructuring and asset impairment charges, vary depending on the level of demand for Intel's products and the level of revenue and profits. Intel’s results could be affected by the timing of closing of acquisitions and divestitures. Intel's results could be affected by adverse effects associated with product defects and errata (deviations from published specifications), and by litigation or regulatory matters involving intellectual property, stockholder, consumer, antitrust, disclosure and other issues, such as the litigation and regulatory matters described in Intel's SEC reports. An unfavorable ruling could include monetary damages or an injunction prohibiting Intel from manufacturing or selling one or more products, precluding particular business practices, impacting Intel’s ability to design its products, or requiring other remedies such as compulsory licensing of intellectual property. A detailed discussion of these and other factors that could affect Intel’s results is included in Intel’s SEC filings, including the company’s most recent reports on Form 10-Q, Form 10-K and earnings release.

Rev. 7/17/13

We are sincerely eager to hear your feedback on this presentation and on re:Invent. Please fill out an evaluation form when you have a chance.