HPE Reference Architecture for Fast Data Analytics on ... · Apache Mesos. When faced with these challenges, enterprises tend to typically migrate towards a public cloud model that

HPE Reference Architecture for Fast Data Analytics on Mesosphere DC/OS Accelerating fast data deployments with HPE Elastic Platform for Big Data Analytics

Reference Architecture

http://www.hpe.com/

Reference Architecture

Contents Executive summary ................................................................................................................................................................................................................................................................................................................................ 3 Introduction ................................................................................................................................................................................................................................................................................................................................................... 3 Solution overview ..................................................................................................................................................................................................................................................................................................................................... 7 Solution components ............................................................................................................................................................................................................................................................................................................................ 8

Software ..................................................................................................................................................................................................................................................................................................................................................... 8 Hardware ................................................................................................................................................................................................................................................................................................................................................... 9 HPE EPA building-block model for fast data analytics ............................................................................................................................................................................................................................... 11

Design principles .................................................................................................................................................................................................................................................................................................................................. 15 Deployment options and scenarios for the HPE EPA Analytics block ......................................................................................................................................................................................... 15

Best practices and configuration guidance for the solution ......................................................................................................................................................................................................................... 16 Framework configuration guidance .............................................................................................................................................................................................................................................................................. 16

Capacity and sizing ............................................................................................................................................................................................................................................................................................................................ 17 HPE EPA for fast data analytics example configurations ......................................................................................................................................................................................................................... 17 Proof of Concept Workload Demos .............................................................................................................................................................................................................................................................................. 18 Analysis and recommendations ....................................................................................................................................................................................................................................................................................... 22

Summary ...................................................................................................................................................................................................................................................................................................................................................... 23 Implementing a proof-of-concept .................................................................................................................................................................................................................................................................................. 24

Appendix A: Bill of materials ...................................................................................................................................................................................................................................................................................................... 24 Appendix B: Deploying/Configuring DC/OS Workload Frameworks ..................................................................................................................................................................................................... 29

DC/OS Cassandra .......................................................................................................................................................................................................................................................................................................................... 29 DC/OS Kafka ...................................................................................................................................................................................................................................................................................................................................... 31 DC/OS Elastic .................................................................................................................................................................................................................................................................................................................................... 33 DC/OS Zeppelin .............................................................................................................................................................................................................................................................................................................................. 34

Resources and additional links ................................................................................................................................................................................................................................................................................................ 35

Reference Architecture Page 3

Executive summary Enterprises continue to search for ways to accelerate big data deployments while adopting new analytic applications that enable analytics processing in place – from the data center to the edge to the cloud. While bare-metal clusters constitute the majority of existing big data deployments, enterprises are increasingly searching for new approaches to reduce their total cost of ownership (TCO) through the consolidation of data, compute and workloads. Virtualizing big data deployments with Linux® containers is a new alternative gaining momentum as a key enabler for driving business, infrastructure, and development agility.

The public cloud offers an attractive option for enterprises launching new big data Proof of Concept (POC) virtual clusters, with the objective of either offloading non-business critical or transient analytics workloads, or to host development and test clusters that can be quickly provisioned and scaled up or down. As these big data POCs evolve into mainstream business critical applications for the enterprise, hybrid deployments will likely become the norm. Additionally, big data processing frameworks that were originally constructed on batch oriented frameworks like MapReduce on Hadoop, will need to evolve to support what is now referred to as “Fast Data”, streaming data processing pipelines based on new technologies like Apache Spark, a modern unified processing engine that enables diverse analytic workloads such as Machine Learning and Internet of Things (IoT) analytics to a broader user audience.

HPE Elastic Platform for Big Data Analytics (EPA) is designed as a logical, modular infrastructure foundation that delivers a scalable and elastic multi-tenant big data platform for on-premises deployment. HPE EPA offers a wide selection of building blocks based on density, capacity, and performance, designed to reduce TCO and data center footprint while optimizing performance for a variety of big data workloads. This would include Extract, Transform, Load (ETL) processing offloaded from traditional data warehouses (DW), SQL based interactive analytics, near-real-time event processing of data streams, and machine / deep learning applications with GPUs. Customers can select modular building blocks of compute (Analytics block), storage (Density Optimized Storage block), and networking (Networking block) from Hewlett Packard Enterprise’s diverse product portfolio, and integrate these blocks with software that enables an on-demand and elastic infrastructure foundation for big data. These blocks can be deployed with co-located compute and storage on the same block, or as blocks of compute and storage disaggregated over a high bandwidth network.

The combination of HPE EPA architecture with Analytics blocks built on HPE Synergy and Mesosphere DC/OS (Data Center / Operating System) is designed to deliver all the benefits of the cloud, enabling customers to rapidly provision Fast Data infrastructure on-premises with the benefits of a secure, scalable, and high performance architecture for Fast Data Analytics. This can dramatically reduce deployment complexity while improving business agility by providing an elastic self-service infrastructure for the big data services made available via the Mesosphere Catalog. The time-to-value for big data deployments can be reduced from weeks to days, while reducing overall costs compared to traditional bare-metal deployments.

For an in-depth analysis of the Hewlett Packard Enterprise EPA architecture for scalable and shared enterprise analytics platforms, along with the benefits of separating compute and storage, review the HPE Elastic Platform for Big Data Analytics technical white paper at http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=4AA6-8931ENW.

Target audience: This paper is intended for decision makers, system and solution architects, big data administrators, system administrators and experienced users that are interested in simplifying the deployment of their fast data infrastructure and applications.

Document purpose: This white paper describes a solution that enables containerized deployments of fast data infrastructure, combining the Mesosphere DC/OS software platform with HPE Elastic Platform for Big Data Analytics (EPA). In addition to outlining key solution components, the paper also provides guidelines for configuring and deploying this combined solution using the HPE Synergy Composable Infrastructure platform.

Introduction As organizations look to rapidly accelerate time-to-value from their big data investment, the infrastructure required to support new workloads and use cases is changing to meet those needs. The big data ecosystem is evolving from a traditional model of a dedicated bare-metal cluster with co-located compute and storage for batch analytics, into a Spark-based ecosystem 1 serving many different use cases requiring different processing and storage requirements for each workload. The big data ecosystem offers an almost unlimited palette of technologies that you can pick from to build your data processing pipeline – depending on your specific use case, processing requirements (from batch to real-time), and the existing systems needing integration (Figure 1). The complexity of the big data ecosystem continues to increase as new frameworks, new

1 More information about the Spark ecosystem can be found at https://www.infoq.com/articles/apache-spark-introduction.

http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=4AA6-8931ENW

https://www.infoq.com/articles/apache-spark-introduction


versions, and new innovations are constantly being introduced. Additionally, business requirements for analytics use cases are also evolving as enterprises look to enable analytics in place – at the core on-premises, in the cloud, and at the edge.

Figure 1. The diversity of components available to build big data processing pipelines

With the advent of containers to virtualize workloads at scale without the associated performance penalty of virtual machines compared to bare-metal, enterprises are exploring ways to design modern microservice based applications. However, one of the many challenges for the enterprise, in a rapidly evolving container space, is the fact that legacy and big data applications are stateful, and persistent; whereas, containerized applications typically are stateless and transient in nature. As many organizations attempt to build their own containerized big data environments with open source container orchestration tools like Kubernetes and Mesos with Mesosphere DC/OS, they soon realize that there are many other components like security, networking, build management, persistent storage, etc. that need to be integrated manually to have a functioning big data environment. At the same time, the evolution of big data based on Hadoop to fast data based on Spark enables separation of the data processing and data storage layers without being tied into a specific resource management framework like YARN, which allows for greater freedom not only in selecting and scaling of compute and storage, but also in the selection of resource management frameworks like Apache Mesos.

When faced with these challenges, enterprises tend to typically migrate towards a public cloud model that allows the infrastructure to be provisioned on-demand with all the necessary components required for building a data processing pipeline. However, data gravity tends to dictate a hybrid approach to big data, with some workloads in the cloud and others on-premises or at the edge. To address these needs, Hewlett Packard Enterprise and Mesosphere have developed a joint solution leveraging Mesosphere Enterprise DC/OS software and the HPE Elastic Platform for Big Data Analytics to build a cost-effective and flexible on-premises architecture for deploying scalable Fast Data analytics workloads.


Mesosphere Enterprise DC/OS is an enterprise grade, data center operating system, providing a platform for securely provisioning containers at production scale to support next generation analytics workloads and frameworks (Figure 2).

Figure 2. Fast Data Analytics with Mesosphere DC/OS

Mesosphere DC/OS is based on the production proven Apache Mesos, and allows operators and data scientists to accelerate deployment and operation of advanced data services while maximizing utilization and reducing infrastructure and cloud cost. The HPE Elastic Platform for Big Data Analytics is a modular and logical framework that allows customers to choose the right blocks for the workload to achieve optimum performance and speed of deployment, while reducing big data cluster footprint by optimizing for density. Server building blocks range from standard 2U two-socket HPE ProLiant series to HPE Apollo range of density optimized servers for HPC, and object storage, or HPE Synergy Composable Infrastructure for hybrid platform deployments.

This solution includes the introduction of a new Analytics building-block for the HPE EPA based on HPE Synergy. HPE Synergy provides a Composable Infrastructure that yields an additional level of flexibility for provisioning nodes to support different analytics framework requirements. Compute and storage nodes are integrated into a frame and can be configured to support different requirements for DC/OS node types. For example, NoSQL frameworks such as Cassandra may require additional SSD drives per node and be composed from available drives on the storage modules. For even higher performance NVMe drives are available in each node. As new frameworks need to be added or replaced in a fast data pipeline, such as Apache Storm instead of Apache Spark, the DC/OS nodes can be recomposed as needed to meet the requirements of the new framework.

By combining the flexibility of the HPE EPA architecture with the self-service provisioning and elastic scalability benefits of Mesosphere DC/OS, customers have a platform to rapidly provision new frameworks and provide a Fast-Data-as-a-Service platform in an on-premises deployment model. Enterprises can significantly accelerate the time to deploy new Fast Data analytics infrastructure deployments, putting the tools in place for the business to drive new innovation and analytics workload development. Leveraging Mesosphere Enterprise DC/OS and the power of Linux containers, data scientists and analysts can create their own Fast Data analytics clusters leveraging Spark, Cassandra, and dozens of other Fast Data analytics frameworks on-demand within minutes. Users can spin up on demand instances for their big data analytics tools of choice, selecting from over 100 applications in the DC/OS workload service catalog 2 (see Figure 3 below).

2 Found at https://www.mesosphere.com/ as of 10/23/2017

https://www.mesosphere.com/


Users can easily try out new application package versions, new applications, and new big data frameworks – without waiting for additional physical infrastructure to be provisioned by IT.

Figure 3. Mesosphere DC/OS Universe: One-Click installation of over 100 Platform Services

The HPE Elastic Platform for Big Data Analytics (EPA) is designed as a modular infrastructure foundation to address the need for a scalable multi-tenant platform, by enabling independent scaling of compute and storage through infrastructure building blocks that are optimized for density and workloads.

With Mesosphere DC/OS and the HPE Elastic Platform for Big Data Analytics, customers have a more flexible, agile, and cost-effective approach to building new infrastructure to support the next generation of analytics workloads. Enterprise customers can leverage the self-service, agility and elasticity benefits of the public cloud operating model while keeping their data and analytics workloads on-premises. Key benefits include:

• Accelerate time to market through single-click deployments of containerized distributed systems for a wide range of different business intelligence, analytics, visualization, and data preparation tools.

• Simplify on-premises or hybrid deployments of Fast Data infrastructure and other data services available as packages in the Mesosphere Universe.

• Increase business agility by empowering data scientists and analysts to quickly create big data clusters running in Linux containers, in a matter of minutes with just a few mouse clicks.

• Automated and Resilient Operations by enabling dynamic and automatic scaling of infrastructure and placement of workloads with built-in resiliency from failures, as well as zero-downtime upgrades of data services.

• Infrastructure Agility and Efficiency allowing traditional and modern applications to run on a shared / hybrid infrastructure with built-in and fully automated security.


For additional general information on the HPE Elastic Platform for Big Data Analytics and the modular building block architecture design, read the “HPE Elastic Platform for Big Data Analytics – Modular building blocks of compute and storage optimized for modern workloads” white paper available on HPE.com at http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=4AA6-8931ENW.

Solution overview Figure 4 below provides a solution diagram for the HPE Fast Data Analytics platform architecture.

Figure 4. DC/OS environment 3

The foundation of the solution includes the HPE Elastic Platform for Big Data Analytics and Mesosphere Enterprise DC/OS version 1.10 software, an enterprise grade data center operating system, providing a platform for running containers, big data, and applications in production. Mesosphere Enterprise DC/OS is an operating system that spans all of the machines in your data center and provides a powerful abstraction layer for your computing resources. DC/OS is a highly scalable two-level scheduler that pools your infrastructure, automatically allocating resources and scheduling tasks based on demands and policy. DC/OS provides a highly elastic and scalable way of developing and deploying applications, services, and big data infrastructure on shared resources.

3 Source: Mesosphere. https://www.mesosphere.com/product

http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=4AA6-8931ENW

https://www.mesosphere.com/product


HPE Elastic Platform for Big Data Analytics (EPA) is designed to be a modular infrastructure foundation that can be scaled in a building block fashion to support different workloads and requirements. Why is this elastic design model important? By building a platform based on an elastic architecture design you can achieve the following benefits:

• Optimize for workload disaggregation and varied resource requirements (e.g., GPU requirements, high density storage, etc.)

• Supports independent scale of compute and storage nodes

• Optimize for density, performance, and cost in the architecture

As customers are looking to build next-generation analytics capability and the infrastructure to support new workloads, there are three types of workloads that drive different underlying resource requirements:

• Fast Data Analytics (e.g., Streaming data analytics using Kafka, Spark, NoSQL, SMACK™ 4, Elastic Search, etc.)

– Memory and Compute intensive

– Low latency storage (SSDs) in the 50-100s of TB capacity range

• Data Intensive Analytics (e.g., Batch Analytics using Hadoop, Hive, etc.)

– Batch analytics workloads using MapReduce processing frameworks

– Spinning media storage devices in the PB capacity range

– Cost effective “data-at-rest” storage

• Deep Learning Analytics (e.g., TensorFlow)

– GPU-enabled nodes

– Large capacity remote storage (e.g., S3, Object, or HDFS)

For modern fast data analytics workloads, customers have two deployment options – bare-metal or container-based. For petabyte scale deployments requiring high performance, customers typically gravitate towards rack servers like HPE ProLiant DLs and HPE Apollos for density and cost optimization.

For terabyte scale big data environments and those requiring transient clusters for DevOps and data science workbenches, Hewlett Packard Enterprise has introduced an HPE Synergy-based Analytics block into the HPE EPA architecture. The Synergy Analytics block is a composable system that allows for rapid provisioning of compute, storage, and networking.

If the customer wants to manage large PB capacity datasets or add GPU-enabled platforms to run deep learning workloads as part of the DC/OS environment, the HPE EPA Analytics blocks can be coupled with existing Density-optimized Storage and Accelerated Compute blocks based on the HPE Apollo platforms. This highlights the elasticity and workload-optimized design benefits that the combination of Mesosphere DC/OS and the HPE EPA architecture bring together.

Solution components Software Mesosphere Enterprise DC/OS Enterprise DC/OS is a distributed operating system based on the Apache Mesos distributed systems kernel. It enables the management of multiple machines as if they were a single computer. It automates resource management, schedules process placement, facilitates inter-process communication, and simplifies the installation and management of distributed services. Its included web interface and available command-line interface (CLI) facilitate remote management and monitoring of the cluster and its services.

4 Acknowledgement of SMACK, which is a trademark of By the Bay, LLC. The SMACK trademark notice can be found here: https://smack.ai/

https://smack.ai/


Figure 5 below highlights the Mesosphere Enterprise DC/OS software stack and enterprise hardened features that build on the open source Apache Mesos distro and add critical elements to improve security and support for multi-tenancy.

Figure 5. A New Multi-Tenant Deployment Model using Shared Infrastructure

DC/OS has three kinds of nodes, boot server, master nodes and agent nodes. The Mesosphere boot server is the server that’s used to configure and start up the cluster. Master nodes control the Mesos layer of the cluster and run many key processes for DC/OS. Master nodes have the following characteristics:

• The minimum number of Master nodes is 1, but the recommended number of master nodes is 3 or 5.

• It is recommended to size up memory and CPU for larger clusters. Additionally, the use of SSDs is recommended for larger clusters.

• Master nodes need to communicate with and update each other often to run the various processes of DC/OS.

Agent nodes perform the actual work within the cluster, they run the containers, frameworks, services, jobs, etc. There are two types of DC/OS agent nodes. They are public nodes and private nodes.

The following, Table 1, lists the differences between public and private nodes within DC/OS.

Table 1. Features and differences between public agents and private agents

Public Agents Private Agents

Run the services that facilitate connections from outside the cluster, typically a load balancer. The default agent and run most everything else within the cluster.

Exist on a network that is accessible from outside the cluster. Exist on a network that is accessible only within the cluster.

The solutions described in this Reference Architecture include the necessary infrastructure allocation to support the master, public and private agent node requirements.

Hardware Composable Infrastructure: HPE Synergy The HPE Synergy platform is designed to bridge traditional and cloud-native applications with the implementation of Composable Infrastructure. Composable Infrastructure combines the use of fluid resource pools, made up of compute, storage, and fabric with software-defined intelligence. The resource pools can be flexed to meet the needs of any business application based on load demands. HPE Synergy lets IT administrators and


developers use infrastructure as code to deploy and manage their data center environments. Developers and ISVs can programmatically control a Composable Infrastructure through a single, open API that is native in HPE Synergy powered by HPE OneView.

HPE OneView, automates, provisions and configures resources according to application needs. This makes it easy for administrators to deploy and elastically scale a cluster on bare-metal servers with just a few clicks; saving time, reducing the opportunity for error, and streamlining auditing.

HPE Synergy Image Streamer is a new approach to deployment and updates for Composable Infrastructure. This new approach for Composable Infrastructure combines true stateless computing with rapid deployment and updates.

HPE Synergy Platform The HPE Synergy Frame is the basic building block into which all other Synergy components are inserted. It is a 10u enclosure with 12 slots that can be consumed with compute or storage modules.

HPE Synergy Composable Compute resources create a pool of flexible compute capacity that can be configured almost instantly to rapidly provision infrastructure for a broad range of applications. The HPE Synergy 480 Gen10 Compute Module takes a single slot within the HPE Synergy Frame.

The HPE Synergy D3940 Storage Module provides a fluid pool of storage resources for the Composable Infrastructure. Additional capacity for compute modules is easily provisioned and intelligently managed with integrated data services for availability and protection. The HPE Synergy D3940 Storage Module takes two, side-by-side slots within the HPE Synergy Frame.

HPE Synergy Composer HPE Synergy Composer provides the enterprise-level management to compose and deploy system resources for your application needs. This management appliance uses software-defined intelligence with embedded HPE OneView to aggregate compute, storage, and fabric resources in a manner that scales to your application needs, instead of being restricted to the fixed ratios of traditional resource offerings. The following Figure 6 is a snapshot of the HPE Synergy Frame from the HPE Synergy Composer.

Figure 6. HPE Synergy Composer view


HPE Synergy Image Streamer HPE Synergy Image Streamer is a new approach to deployment and updates for Composable Infrastructure. This management appliance works with HPE Synergy Composer for fast, software-defined control over physical compute modules with operating system provisioning. HPE Synergy Image Streamer enables true stateless computing combined with capability for quick deployment and updates.

Making the HPE Synergy environment even more flexible is the HPE Synergy Composer and the HPE Synergy Image Streamer combination, both of which are powered by HPE OneView.

The HPE Synergy Composer provides the mechanism to enable compute as code and as such, enables a customer to set and modify the personality of each HPE Synergy compute module. The HPE Synergy Composer is used to associate drives contained in the HPE Synergy D3940 Storage Module and networking resources with a particular HPE Synergy compute module.

Further enhancing the HPE Composer’s feature set, is the HPE Synergy Image Streamer. For HPE EPA, the HPE Synergy Image Streamer is the appliance that provides the operating system image to all of the HPE Synergy compute modules under its direction. This means that if a different compute module is required to perform a task, all that’s required is to associate the profile and operating system image with the new compute module and that new compute module now has the exact same personality as the older one, that’s being replaced.

HPE EPA building-block model for fast data analytics The premise of the HPE EPA architecture is to provide a set of building blocks that are designed to support different workload requirements, be flexible, and simplify the infrastructure decision making process for customers. Building off this premise, the solution for Fast Data Analytics is composed of three core building blocks:

• Rack block

• Top of rack (ToR) Network block (Optional depending on customer network requirements)

• Analytics block

These three blocks form the foundation on which the Mesosphere DC/OS software is installed. Depending on the network requirements for the customer, the solution may consist of only two core blocks (Rack and Analytics). This building block model provides flexibility in the number of compute and storage modules in the frame and provides a foundation for scaling from a single-frame test and development sandbox environment up through a multi-rack production ready DC/OS cluster (Figure 7).

Figure 7. Evolution from a single-frame Test/Dev environment to an enterprise wide cluster


HPE EPA Rack block The HPE Rack block consists of a single rack and its accessories (PDUs, cables, etc.). This can be either an HPE or third-party rack. Depending on the server types either a 1075mm or 1200mm deep rack may be used. Please consult your server documentation to determine if a deeper rack is required. PDU requirements should be evaluated based on the final EPA configuration and your data center power requirements.

The following, Table 2, specifies the individual components of the Rack block.

Table 2. Rack block components

Component Configuration

Rack model HPE 642 1200mm Shock Intelligent Rack or equivalent

Rack model HPE 642 1075mm Shock Intelligent Series Rack or equivalent

PDU HPE PDUs or equivalent third-party PDUs

HPE EPA Network block The Network block is an optional block in the rack-level Analytics cluster solution. Whether this block is required is dependent on the particular customer scenario and upstream network connectivity requirements. In a three-frame HPE Synergy design the network is aggregated and routes upstream through the two master frames as shown in Figure 8.

Figure 8. Three frame networking diagram

This design provides the capability to have up to 10 x 40Gb ports of connectivity upstream from the three HPE Synergy Frames. This yields a northbound oversubscription ratio between 3:1 and 4:1 depending on the ratio of compute to storage modules in each Synergy Frame (3:1 is the default). However, there is no east:west oversubscription due to the interconnect bandwidth fabric connecting each of the frames together.


The following, Table 3, shows the detail of the Networking block components.

Table 3. Network bock components


Switch model (2) HPE FlexFabric 5950 32QSFP28 Switches (40/100Gb ports)

Switch model (1) HPE 5900AF-48G-4XG-2QSFP+ Switch

Scenarios where customers may want to include the ToR Network block would include the following:

• Additional aggregation of the 10 x 40Gb uplink ports to fewer, higher bandwidth 100Gb links

• Multi-rack connectivity

• Connectivity to external HPE Apollo based Density-optimized or GPU enabled servers

The minimum quantity for this block is 0 and the maximum quantity is 1 per Rack block.

HPE EPA Analytics block The Analytics block is the core building block of the Fast Data analytics platform solution.

The default recommended configuration for the Analytics block is to provision ten HPE Synergy 480 Gen10 Compute Modules and one HPE Synergy D3940 Storage Module per HPE Synergy Frame. This provides a baseline configuration that can support a balanced configuration of 3 to 4 drives per node for workloads requiring fast, SSD-based storage (e.g., Cassandra). If a customer finds they need a larger amount of storage then up to four HPE D3940 Storage Modules can be placed in the frame providing up to 40 drives per compute node in a storage optimized design. Placing four HPE D3940 Storage Modules in a frame would leave space for four HPE Synergy 480 Gen10 Compute Modules. For additional storage per Compute Module, Hewlett Packard Enterprise recommends placing two 1.92TB SATA SSDs in each of the available drive slots in the front of the Compute Module. These drives can be used for containerized application storage. For additional performance, those two SSD drives can be converted to NVMe storage, in which case, Hewlett Packard Enterprise recommends the 1.6TB Mixed Use NVMe drives. Hewlett Packard Enterprise recommends that drives placed inside each compute module be mirrored. This is because the applications homed on these drives contain Mesos and Docker images. In the event of a drive failure, those images would become unavailable and much like an operating system image, would render the node unusable.

In addition to the number of compute modules per frame, the variability for HPE Synergy compute nodes is the number of and clock speed of the processors, the number of cores per processor, and the amount of memory per node. Hewlett Packard Enterprise recommends configuring memory in increments of 6 DIMMs per processor of the Intel® Scalable family of processors. This is to maximize memory access speeds.

Variability for the HPE Synergy D3940 Storage Module, in addition to the number of HPE D3940s to be installed in each individual frame, is the number of and size of the SSDs to be placed in the Storage Module. The D3940 has a capacity of 40 drives. Those drives may be attached to any compute module installed in the same frame as the HPE D3940. This composability benefit allows the Synergy environment to be changed if the application framework requirements change and additional drives need to be presented to compute nodes. Removing the constraints of a certain number of drive slots per server allows for a much more fluid architecture. And if you find that you need more storage on this server, reconfiguration is as simple as reassigning a profile and then rebooting.

The operating system is provided to each of the Compute Modules by HPE Image Streamer. This allows all of the storage, both in the Compute Modules, as well as in the Storage Modules to be used for application deployment purposes, because none of the space required by the OS is consumed on the storage.


Table 4 and Table 5 below contain the detailed requirements and recommendations for the Analytics block in a POC sandbox and in a rack-level configuration. The differences in the configuration are driven by the networking benefits when integrating the Synergy Frames together in a multi-frame design.

Table 4. Analytics block in a single frame configuration “Analytics Sandbox”


Model (1) HPE Synergy Frame

Networking HPE Virtual Connect F8 Module

Storage Connect Module (2) HPE Synergy 12Gb SAS Connect Module

Module (2 - 10) HPE Synergy 480 Compute Modules

Storage (1 - 4) HPE Synergy D3940 Storage Modules

Drives per Storage Module 40 SFF SSD options (drive options and count are flexible. (Recommend 1.92TB MU SAS SSDs)

Per Compute Module

Processor (2) Intel® Xeon-G® 6130 16-core Scalable Family processors (default)

Memory 384GB memory (default). Range of memory 128GB-1TB

Controller HPE Smart Array P416ie-m (for connectivity to the D3940)

Containerized application Disks (2) SATA SSDs installed in front drive bays (RAID-1 because it’s the Docker image)

Network card Synergy 3820c 10/20GbE Converged Network Adapter

Table 5. Analytics Block in a rack-level configuration “Analytics Cluster”


Model (1) Rack block

Model (3) HPE Synergy Frames

Networking (2) HPE Virtual Connect F8 Module

Networking (4) HPE 20Gb Interconnect Link Module

Storage Connect Module (6) HPE Synergy 12Gb SAS Connect Module

Storage (3-12) HPE Synergy D3940 Storage Module

Drives per Storage Module 40 SFF SSD options (drive options and count are flexible) (Recommend 1.92TB MU SSDs)

Module (6-30) HPE Synergy 480 Compute Modules

Per Compute Module

Processor (2) Intel Xeon-G 6130 16-core Scalable Family processors (default)

Memory 384GB memory (default). Range of memory 128GB-1TB

Controller HPE Smart Array P416ie-m (for connectivity to the D3940)

Containerized application Disks (2) SATA SSDs installed in front drive bays (RAID-1 because it’s the Docker image)

Network card 1 Synergy 3820c 10/20GbE Converged Network Adapter

HPE Density-Optimized Storage block When a large amount of storage is required, a customer can opt for one or more Density-Optimized Storage blocks. This block is based on the HPE Apollo 4200 Gen9 server, typically configured with 28 SATA large form factor drives.

Variability in this block is the number of processors, typically 2 (a customer may opt for only one processor), the clock speed of the processor, and number of cores per processor. This block starts out with 256GB of memory, but this is also customizable by the customer from 128GB to 768GB. The number and size of the hard disk drives is a choice left up to the customer.


Table 6 contains the detailed information on which the Density-Optimized Storage block is based.

Table 6. Density Optimized Storage block.


Model (1) HPE Apollo 4200 Gen9 24 LFF server plus optional LFF rear 4HDD cage

Processor (2) Intel E5-2660 v4 14-core processors (default)

Memory 256GB memory (range 128GB-768GB)

Controller HPE Smart Array P840ar/2G FIO controller

OS Disks HPE dual 150GB RI Solid State M.2 kit

Data Disks 28 X HPE 8TB 6G SATA 7.2K RPM 3.5in MDL LP HDD (drive options and count are flexible)

Network Controller HPE 10/25Gb 2P 640FLR-SFP28 for connection to a switch

Design principles Deployment options and scenarios for the HPE EPA Analytics block Single frame development, test and POC environment For use in non-production environments, such as dev/test and proofs of concept, Hewlett Packard Enterprise recommends a single frame. The frame contains one Composer, one Image Streamer, nine or ten HPE Synergy 480 Gen10 Compute modules and one HPE Synergy D3940 Storage Module. This would provide an environment that mimics the more robust production environments outlined next and allows for all of the functionality of Mesosphere to be deployed, tested and characterized in a specific end-user environment. In this environment, we will have 1 Mesosphere boot server, 1 or 3 Mesosphere Master nodes, which leaves 5 to 8 agent nodes, which can be divided between public agents and private agents based on specific requirements. The bill of materials for this environment may be found in Appendix 1b.

Production starting point environment As a production starting point, Hewlett Packard Enterprise recommends three HPE Synergy 12000 frames within a Rack Block. This provides redundancy within the cluster, eliminates any single point of failure and optimizes the network topology.

Hewlett Packard Enterprise recommends that each frame be populated with 3 Compute Modules and 1 Storage Module. The storage module is then populated with twelve 1.92TB MU SSDs, 4 per Compute Module.

In this configuration, there will be 1 Mesosphere boot server, 3 Mesosphere Master nodes and 5 agent nodes.

This configuration can be scaled one compute server at a time, by adding the compute server to a HPE Synergy 12000 frame and four 1.92TB SSDs to the corresponding HPE Synergy D3940 Storage module. As the environment is scaled, Hewlett Packard Enterprise recommends placing additional Compute Modules into the three frames in a round robin fashion. For example, going from 9 Compute Modules to 10 Compute Modules would mean adding the new Compute Module and associated 4 SSDs into the first frame. Growing the environment from 10 Compute Modules to 11 Compute Modules would mean adding the new Compute Module and associated SSDs to the second frame.

A bill of materials for this environment may be found in Appendix 1c.

Production full rack environment A full rack deployment would have 3 frames fully populated. The default population recommendation would have 10 Compute Modules and 1 Storage Module populated with 40 x 1.9TB SSDs per frame, giving the entire environment 30 Compute Modules, 3 Storage Modules and 120 x 1.9TB SSDs for the cluster. In this scenario, there would be 1 Mesosphere boot server, 3 or 5 Mesosphere Master nodes and 24 or 26 agent nodes which can be deployed as public agents and private agents as desired.

A customer can, of course, place more D3940 Storage modules into each frame if additional high speed storage is required, by eliminating Compute Modules. Each Storage Module takes the space of 2 Compute Modules.

The bill of materials for this environment may be found in Appendix 1a.

Growth beyond the Production full rack environment If additional resources are required beyond the full rack environment, Hewlett Packard Enterprise recommends growing the environment as outlined in the Production Starting Point section. Growing the environment 3 frames at a time optimizes the networking topology and ensures


that over-provisioning of network bandwidth is kept to a minimum. An additional factor with scaling is that a single instance of HPE OneView can manage up to 21 frames, which is 7 racks with 3 frames each. Additional instances of OneView can extend the management reach and still allow for management via a single pane of glass.

Best practices and configuration guidance for the solution Hewlett Packard Enterprise recommends that the two drives in each compute module be mirrored to ensure the Mesos and Docker images do not become unavailable should a drive failure occur.

The Mesosphere boot server is not needed to run the cluster. But rather, it is needed for the initial installation and updating of the cluster. Because of this feature, it can be recreated. However, Hewlett Packard Enterprise recommends retaining the boot server and use its storage as a place for downloaded artifacts, for example, the Cassandra image, etc. Hewlett Packard Eenterprise recommends the OS image for the boot server be supplied by the HPE Image Streamer. This leaves all storage assigned to this node available for images. Storage assigned to this node can be internal or external. External storage allows easier reassignment of this node, should it become unavailable for any reason.

Hewlett Packard Enterprise recommends HPE Synergy D3940 based storage be used for persistent storage requirements. For example, Hewlett Packard Enterprise recommends placing the Cassandra database on HPE Synergy D3940 storage. This is for reassignment purposes. Should a particular node become unavailable, a different node can replace the unavailable node by reassigning the HPE Synergy D3940 storage to the new node. Hewlett Packard Enterprise also recommends that log file storage be placed on HPE Synergy D3940 storage.

Should additional storage be required, two internal SSD drives are recommended in each of the HPE Synergy 480 Gen10 compute modules. Hewlett Packard Enterprise recommends that this storage be used for Docker and Mesos images. While, placing images on internal storage devices makes the compute modules stateful, recreating the images will allow for movement between nodes.

If additional storage performance is required beyond what is available with internal SSDs, Hewlett Packard Enterprise recommends converting the internal drive bays to NVMe drives and placing 1.6TB mixed use NVMe drives in those slots. HPE recommends not RAIDing NVMe drives, but they can be used for inbound streaming of data, for example with Kafka. These drives would hold the data until it can be pushed to and persisted by Cassandra.

Mesosphere DC/OS Master Agents are designated and created when the cluster is initially configured. Subsequent to that, Master Agents can neither be added, nor can they be deleted from the cluster. Because Master Agents need to keep each other updated continuously, they are relatively chatty. This means that more Master Agents perform more slowly than fewer Master Agents. Additional Master Agents are there only for availability. Should all of the Master Agents become unavailable, the cluster is unavailable. Hewlett Packard Enterprise recommends that thorough planning be performed to determine the number of Master Agents needed for a particular cluster. HPE recommends 3 or 5 Master Agents for production environment. For dev/test and POC environments, Hewlett Packard Enterprise recommends 3 Master Agents, however, a customer may opt for only 1. Even customers with clusters that have more than 10,000 nodes have configured only 7 Master Agents.

Framework configuration guidance As part of the function evaluation for this RA we ran several analytics workloads based on common frameworks in the SMACK architecture that run on Mesosphere DC/OS which included the Tweeter application and the Taxi-source application.

Before deploying services, it is recommended that you review the latest Mesosphere deployment service guides.

Services like Cassandra and Kafka you will likely want to configure persistent storage for your data. With DC/OS you can configure Mesos disk resources across your cluster simply by mounting storage resources on agents using a well-known path. When a DC/OS agent starts, it scans for volumes that match the pattern /dcos/volume<N> where <N> is an integer. The DC/OS agent is then automatically configured to offer these disks resources to services.

When configuring a service for the attribute disk_type if “MOUNT” is specified DC/OS will use the /dcos/volume<N>.

Cassandra • Set the number of cores used by each Cassandra instance to 8.

• Set the amount of memory used by each Cassandra instance to 32768MB

• Use storage allocated from the D3940 for persistent storage.

• For persistent storage set the attribute Disk_Type to “MOUNT”

https://docs.mesosphere.com/service-docs/


Kafka • Use mounted volumes for consistent performance

• Ensure brokers are distributed to different nodes (by default they will run on any node with sufficient resources which may result in colocation)

Elastic • Use mounted volumes for consistent performance

Note More information regarding deployment parameters and how to set them can be found in Appendix B.

Capacity and sizing HPE EPA for fast data analytics example configurations This section highlights three example configurations based on the Mesosphere Enterprise DC/OS software layer and HPE EPA architecture. The solutions highlighted in this section are representations of example configurations for the following three use-cases:

• Analytics sandbox – Data science starter environment for customers looking to investigate the use of containers for deploying next-generation analytics

• Analytics cluster – Rack-level cluster for deploying analytics frameworks and hosting multiple development environments

• Extended analytics cluster with density-optimized storage – Multi-Rack cluster design for deploying production-sized fast data and data intensive application frameworks and multiple development / test environments.

Example analytics sandbox The starting point for a test/dev or POC environment is the single frame that was presented in detail in Table 4 in the prior section above. This is a single HPE Synergy Frame and is designed for customers looking to build a basic sandbox for understanding DC/OS functionality. This is not a recommended configuration for production deployments or large scale testing.

Building a next-generation analytics application with the SMACK framework The SMACK architecture is one of the core frameworks that customers deploy when building analytics workloads on top of Enterprise DC/OS. The SMACK framework is an acronym that is composed of multiple services including Spark, Cassandra, and Kafka among the most popular. These services are commonly used when needing to ingest, analyze, store, and react on multiple streaming inputs. Examples prevail in credit card fraud detection, healthcare, and customer sentiment to name just a few. In this Reference Architecture, both the DC/OS Tweeter application and the IoT Taxi demos created by Mesosphere were loaded in an Analytics Sandbox configuration in HPE labs to demonstrate the core functionality of the HPE EPA with the HPE Analytics block based on HPE Synergy.

The following diagram, Figure 9 is a graphical depiction of the Analytics Sandbox environment that HPE used to enable this proof of concept.

Figure 9. Graphical depiction of the sandbox environment that was used for this POC


The following diagram, Figure 10 is a graphical depiction of how the storage was consumed during the POC. HPE used some of the storage on the HPE Synergy D3940 as boot disks for the Master Nodes. This was done so that the Master Nodes could be easily moved between compute modules should the need arise.

Figure 10. Graphical depiction of the D3940 storage consumption utilized during this POC

Proof of Concept Workload Demos To showcase how DC/OS can dramatically reduce deployment complexity by providing elastic self-service infrastructure for the big data services in the Mesosphere Catalog, two proof of concept demos were deployed on the sandbox environment shown above.

Tweeter Data application The Tweeter Data application is an application that analyzes incoming tweets and creates reports highlighting the most active tweeter users. The end-to-end application is made up of the following software and services.

• Marathon-LB: A high availability proxy based load balancer. In this instance, it is being used to intercept all of the incoming tweets and distribute them to one of the Tweeter web services that was maintained within three containers across three private agent nodes.

• Tweeter: Sends the tweets to Cassandra for storage and also streams them to the Kafka message bus.

• Cassandra: A distributed database. Cassandra maintains its high availability by storing each bit of data 3 times on 3 different nodes. In this case, Cassandra is used to store the tweets for persistence and for later review.

• Apache Kafka: A messaging system built to scale for big data. Kafka enables applications built on different platforms to communicate via asynchronous message passing. In the sandbox environment, Kafka was used to stream the tweets to Spark.

• Zeppelin: An interactive analytics notebook that is used to enable interactive analytics and visualization.

• Spark: In this environment, Spark was used to analyze the tweets in near real-time using Spark Streaming APIs. In the demo, Spark was used under the Zeppelin framework.


Figure 11 below shows the data pipeline for the Tweeter application.

Figure 11. Tweeter application flow chart

The proof of concept demo is meant to demonstrate the full data-processing pipeline architecture for Fast-Data Analytics. A containerized Web application, called Tweeter, is used as shown in Figure 11. The Tweeter app is similar to Twitter where a web page allows a user to post short messages that can be seen on the Web page. To simulate a massive data load into the pipeline, another containerized application also known as a Tweet Bot, and named PostTweets is used. PostTweets reads messages from an input file and posts them onto the web page in a rapid fashion. As the web application containers are run on private agent nodes, DC/OS Marathon-LB service is used as a proxy and load balancer providing access to a public web page.

The messages from the Web application and the Bot are ingested into DC/OS Kafka framework which acts as a message broker that reliably delivers the messages to any real-time stream processing system. In the demo pipeline, DC/OS Spark framework is used with Spark Streaming to process the messages in real time. The messages are simultaneously sent to the Cassandra database for reliable storage and for further retrieval and analysis. To enable interactive real-time analytics, the DC/OS Zeppelin service with the Spark backend is used. As the Spark Streaming application is run using Zeppelin, it starts reading messages from Kafka, stores them in an in-memory temporary table. A Spark SQL query is then run against this table to aggregate the message counts and group them by the user to show the top tweeters. The table is updated in real-time as the messages are streamed in from Kafka and the results for top tweeters can be seen changing in real time.

IoT application This proof of concept demo is meant to validate a full stack geo-enabled Internet of Things (IoT) solution using DC/OS with containerization and the frameworks Kafka, Spark, and ElasticSearch. As more and more IoT sensors and devices are deployed throughout the world, solutions like these enable opportunities to gain location-based insights using these sensors. To process thousands or even millions of events per second, the solution can scale with DC/OS providing needed scalability and performance. This solution demonstrates the use of micro-services that perform specific tasks that can be deployed on DC/OS. These include:

• Scala application based source ingestion tasks that are responsible for ingesting high-velocity sensor feeds.

• Apache Kafka based broker tasks that are responsible for routing sensor events from sources to streaming analytics tasks.

• Spark streaming based real-time analytic tasks that enable users to perform geospatial analytics in the stream.


• The Spark streaming tasks also store the high-velocity sensor feeds in ElasticSearch where events are indexed in a way that for rapid retrieval by space, time, or the combination of space and time.

• Map Application Tasks that access the data in ElasticSearch to visualize the data and provide temporal playback of the data on a map.

The control flow of the demo solution is depicted below in Figure 12.

Figure 12. IoT application control flow chart

Taxi-Source Scala Application

The Taxi-Source is a containerized Scala application that acts as a source for emitting events to be consumed by Kafka brokers. The events simulate Taxi position and time observations in New York City. The source partitions the data as evenly as possible across the available brokers. It plays back a file based on the time values supplied in the file, e.g. play back each unique timestamp at an interval of 3 seconds.

Kafka Brokers

The DC/OS Kafka framework provides support for topics that are published by Taxi-Source application and delivers the messages to any real-time stream processing system. In the demo pipeline, the messages are consumed by Taxi-Stream real-time analytic tasks using Spark Streaming.

Taxi-Stream Real-time Analytic Tasks with Spark Streaming

The Taxi-Stream task uses Spark Streaming to consume events from Kafka brokers, perform analytics and synchronize the results to ElasticSearch. These tasks are also containerized for easy deployment and scalability.

ElasticSearch

The DC/OS ElasticSearch framework is used to efficiently index events by space, time, and all the other attributes of the event. Here the ElasticSearch framework is used for storing the events and for searching them.

Map Web App

The Map Web App is a containerized JavaScript Web app that periodically queries the ElasticSearch application to reflect the latest state of events on a map. The map application can be accessed on the public agent node at /map/index.html. The JavaScript web app queries for statistics from ElasticSearch every second and the map is automatically updated with the latest taxi movement statistics that are visualized as rectangles on the map or raw observations visualized as symbols on the map. The Map Web App has the ability to enable 'replay' of the spatiotemporal observations and also supports the ability to generate a heat map based on content being queried from ElasticSearch.


Figure 13 shows a screenshot of the services used in this demo solution.

Figure 13. DC/OS Dashboard showing services used by the demo solution

Figure 14 shows a screenshot of aggregation visualization of the near real-time taxi movement.

Figure 14. Aggregation visualization of taxi movement.


Figure 15 shows a screenshot of taxi movement density changes with time using heat map as the visualization.

Figure 15. Visualization of taxi movement heat map.

Analysis and recommendations Example analytics cluster The following, table 7, breaks out the number of Bootstrap nodes, the number of Master Agent nodes, the number of Private Agent nodes, the amount of raw storage, and the number of 40Gb connections to the backbone network as the environment is scaled from a single frame to a single rack and then a two rack environment.

Table 7. Analytics block scaling

Test/dev or POC Single Full Rack Production Two Full Rack Production

Bootstrap nodes 1 1 1

Number of Analytics blocks 1 3 6

Number of Master Nodes 3 3 - 5 3 - 5

Number of public agents 1 3 6

Number of private agents 5 21 - 23 48 - 50

Amount of raw storage 76 PB 5 230 PB4 460 PB4

Number of external 40Gb connections 7 12 24

Example analytics cluster with density-optimized storage The following, Table 8, represents how the Analytics blocks can be scaled along with the Density Optimized Storage blocks. In this table, we assume the requirement to scale the Analytics rack at the same rate as the Density Optimized racks. This is left to the customer. It might be that more than one Analytics rack is required for every one Density Optimized Storage rack. Or it might be that more than one Density Optimized Storage rack is required for every Analytics rack.

5 Using 1.92TB Mixed Use (MU) SSDs


Table 8. Scaling the Analytics blocks, also using the Density Optimized Storage blocks

One Analytics block rack

One Density Storage block rack

Two Analytics block racks

Two Density Storage block racks

Three Analytics block racks

three Density Storage block racks

Number of racks 1/1 2/2 3/3

Number of blocks 3/18 6/36 9/54

Amount of raw storage in Analytics block rack

230 PB5 460 PB5 691 PB5

Amount of raw storage in Density Optimized Storage rack block

4032 PB 6 8064 PB6 12096 PB6

Number of 1Gb Management switches 1 1 2

Number of 40/100Gb switches 2 2 4

Summary With Mesosphere DC/OS on the HPE EPA building block architecture and the underlying HPE Composable Infrastructure, you can apply the right focused resource at the right problem at the right time. The fluidity that composability brings, means that just because you focused and deployed a server for a specific task doesn’t mean that you can’t repurpose those same resources for a different set of tasks. And repurposing a module means nothing more than pointing it at a different profile and rebooting. This provides the flexibility to adapt your Mesosphere DC/OS platform to accommodate shifting application framework requirements and extend to support additional analytics workloads.

For proof of concept environments, a single Analytics block frame is available. Once the environment is to go to production, Hewlett Packard Enterprise recommends deploying Rack blocks fully populated with three Analytics blocks, because of the networking topology and ease of integrating a rack into an existing network.

If especially dense storage is required, Hewlett Packard Enterprise recommends the Density Optimized Storage block which employs Apollo 4200 Gen9 24LFF servers, with optional drive cage, for a total of 28 X 8TB SATA hard disk drives for a total raw capacity of 224TB or 203TiB. Up to 18 of the Density Optimized Storage blocks can be added to a Rack block.

Mesosphere DC/OS allows assembly of the software components required to build a cluster more easily by abstracting each component. This reduces the amount of initial, as well as the ongoing, system administration time.

It is the combination of DC/OS with HPE EPA, enabled by HPE Synergy composability, that creates an environment that can scale from a single frame to an enterprise wide cluster in a seamless manner and without wasting resources as the environment is enhanced.

The use of Linux containers also allows for the seamless movement from development to departmental clusters to enterprise wide deployments by including everything required within the container’s footprint. This removes the issue of missing a key component when making the transition which causes the application to either not run or run in a manner that is inconsistent from environment to environment.

Hewlett Packard Enterprise has blended the best in class software with Mesosphere DC/OS, HPE OneView with the fluidity of HPE Synergy, and the key design principal of HPE EPA. Now, instead of a uniform server to address all facets of the cluster, we can create unique resources that are focused specifically on the job they are asked to perform. No longer do we need to have a node within a cluster have too many compute resources, simply to acquire additional storage. This combined with HPE Synergy’s Composability, the ability to repurpose compute, storage, and networking on the fly, means that organizations can shuffle resources between tasks quickly and easily, which reduces the number of resources required to accomplish the overall task at hand.

All of this means fewer resources are required to perform the job. This results in less spent on infrastructure, which leaves more to complete the deployment project. Organizations no longer need to use a crystal ball to determine what the end result of their sandbox or development and test environment will be in order to purchase equipment to satisfy that end state. They can buy the resource they need, when they need it with the assurance that no resource will be wasted in the transition. This reduces the initial as well as the overall spend when deploying Mesosphere DC/OS through all facets of implementation.

The combination of the HPE Elastic Platform for Big Data Analytics architecture with its modular, building block approach in concert with Mesosphere DC/OS software is designed to reduce the time from concept to implementation from months to days. It is meant to give customers 6 Using 8TB 7.2K RPM HDDs


a cloudlike experience. Further, the ability to reconfigure hardware quickly, means that the configuration can adjust to the needs as they surface rather than having to predict what the needs will be in the future.

This Reference Architecture describes solution testing completed in December 2017.

Implementing a proof-of-concept As a matter of best practice for all deployments, HPE recommends implementing a proof-of-concept using a test environment that matches as closely as possible the planned production environment. In this way, appropriate performance and scalability characterizations can be obtained. For help with a proof-of-concept, contact an HPE Services representative (hpe.com/us/en/services/consulting.html) or your HPE partner.

Appendix A: Bill of materials

Note Part numbers are at time of publication/testing and subject to change. The bill of materials does not include complete support options or other rack and power requirements. If you have questions regarding ordering, please consult with your HPE Reseller or HPE Sales Representative for more details. hpe.com/us/en/services/consulting.html

Table 1a. Bill of materials for a full rack Analytics Block

Qty Part number Description

Rack and Network Infrastructure (Rack Block)

1 BW908A HPE 42U 600x1200mm Enterprise Shock Rack

1 BW908A 001 HP Factory Express Base Racking Service

1 BW932A HPE 600mm Rack Stabilizer Kit

1 BW909A HPE 42U 1200mm Side Panel Kit

1 AF521A HPE Intelligent 8.3kVA/CS8265C/NA/J PDU

1 861413-B21 HPE CAT6 10Ft Cbl

2 861412-B21 HPE CAT6A 4ft Cbl

8 804101-B21 HPE Synergy Interconnect Link 3m AOC

2 720199-B21 HPE BLc 40G QSFP+ QSFP+ 3m DAC Cable

Analytics Block #1 (Master Frame 1)

1 797740-B21 HPE Synergy12000 CTO Frame 1xFLM 10x Fan

1 804353-B21 HPE Synergy Composer

1 804937-B21 HPE Synergy Image Streamer

10 871942-B21 HPE SY 480 Gen10 CTO Prem Cmpt Mdl

10 873381-L21 HPE SY 480/660 Gen10 Xeon-G 6130 FIO Kit

10 873381-B21 HPE SY 480/660 Gen10 Xeon-G 6130 Kit

120 815100-B21 HPE 32GB 2Rx4 PC4-2666V-R Smart Kit

20 877788-B21 HPE 1.92TB SATA MU SFF SC DS SSD

10 777430-B21 HPE Synergy 3820C 10/20Gb CNA

10 804428-B21 HPE Smart Array P416ie-m SR Gen10 Ctrlr

10 875242-B21 HPE 96W Smart Storage Battery 260mm Cbl

1 794502-B23 HPE VC SE 40Gb F8 Module

1 779218-B21 HPE Synergy 20Gb Interconnect Link Mod

2 755985-B21 HPE Synergy 12Gb SAS Connection Module

1 798096-B21 HPE Synergy 12000F 6x 2650W AC Ti FIO PS

http://www.hpe.com/us/en/services/consulting.html




1 804942-B21 HPE Synergy Frame Link Module

1 804923-B21 HPE Synergy12000 Frame Compute Hlf Shelf

1 804943-B21 HPE Synergy 12000 Frame 4x Lift Handle

1 804938-B21 HPE Synergy 12000 Frame Rack Rail Option

1 835386-B21 HPE Synergy D3940 CTO Storage Module

1 757323-B21 HPE Synergy D3940 IO Adapter


1 859493-B21 Synergy Multi Frame Master1 FIO

























Analytics Block #3 (Satellite Frame)






















Table 1b. Bill of materials for a single frame POC starter Analytics block










Analytics Block Frame, Compute Modules and storage






















Table 1c. Bill of materials for a starter production rack Analytics Block








1 861413-B21 HPE CAT6 10Ft Cbl


8 804101-B21 HPE Synergy Interconnect Link 3m AOC




















































Analytics Block #3 (Satellite Frame)





















Appendix B: Deploying/Configuring DC/OS Workload Frameworks Before deploying services, it is recommended that you review the latest Mesosphere deployment service guides.

Services like Cassandra and Kafka you will likely want to configure persistent storage for your data. With DC/OS you can configure Mesos disk resources across your cluster simply by mounting storage resources on agents using a well-known path. When a DC/OS agent starts, it scans for volumes that match the pattern /dcos/volume<N> where <N> is an integer. The DC/OS agent is then automatically configured to offer these disks resources to services. When configuring a service for the attribute disk_type if “MOUNT” is specified DC/OS will use the /dcos/volume<N>.

DC/OS Cassandra DC/OS Cassandra can be deployed two ways, using CLI, or Using DC/OS web interface, as outlined below.

Using CLI Use the following command on the node where DC/OS CLI was installed:

• dcos package install cassandra

You can use a custom configuration file to change the default configuration settings:

• dcos package install cassandra –options=<options-json-file>

The default framework name is “cassandra”. If you chose to change the framework name make sure you also change the name in all services using this instance of Cassandra. This can be changed by changing the “name” property in <options-json-file> as shown:

{

"service": {

"name": "cassandra2",

"user": "nobody",

…

}

Using DC/OS web interface Navigate to Catalog > Packages and search for Cassandra. Click on the Cassandra (certified) icon in the list of results that were returned. Use the following command to install the DC/OS Cassandra CLI subcommands:

• dcos package install cassandra --cli

The default framework name is “cassandra”. This can be changed by configuring the framework name in the service section of the advanced installation section.

The following describes the most commonly used features of the Cassandra framework and how to configure them via the DC/OS CLI and from the DC/OS web interface.

By default, Cassandra will be installed on 3 nodes. Cassandra will require at least 3 nodes to support the default data replication of 3. To change the number of nodes on which Cassandra is installed, set the following value:

• In DC/OS CLI <options-json-file>: nodes.count: integer (default: 3)

{

"nodes": {

"count": 3,

…

}

• DC/OS web interface: NODES: integer

https://docs.mesosphere.com/service-docs/


By default, Cassandra framework will use 0.5 CPU cores on each node where it is installed. To change the number of cores, set the following value:

• In DC/OS CLI <options-json-file>: nodes.cpus: number (default: 0.5)

{

"nodes": {

"cpus": 8,

…

}

• DC/OS web interface: CASSANDRA_CPUS: number

By default, Cassandra framework will use 1024 MB of memory on each node where it is installed. To change the amount of memory used by Cassandra, set the following value:

• In DC/OS CLI <options-json-file>: nodes.mem: integer (default: 1024)

{

"nodes": {

"mem": ,32768

…

}

• DC/OS web interface: CASSANDRA_MEMORY_MB: integer

Cassandra framework supports two disk volume types:

• ROOT volumes are effectively an isolated directory on the root volume, sharing IO/spindles with the rest of the host system. This is the default choice.

• MOUNT volumes are a dedicated device or partition on a separate volume, with dedicated IO/spindles.

To ensure reliable and consistent performance in a production environment, you should configure MOUNT volumes on the machines that will run the framework in your cluster and then configure the following as MOUNT volumes:

• In DC/OS CLI<options-json-file>: nodes.disk_type: MOUNT (default: ROOT)

{

"nodes": {

“disk_type”: MOUNT,

…

}

• DC/OS web interface: CASSANDRA_DISK_TYPE: MOUNT

Using DC/OS CLI<options-json-file>, you can change other framework settings, such as storage ports, RPC ports, virtual network etc. Please refer to the Cassandra Service Guide for the details.

Using DC/OS CLI<options-json-file>, you can also change Cassandra settings (normally specified in cassandra.yaml configuration file) such as cluster_name, num_tokens, partitioner, key_cache_save_period, row_cache_size_in_mb, row_cache_save_period, commitlog_sync_period_in_ms, commitlog_segment_size_in_mb, commitlog_total_space_in_mb, concurrent_reads, concurrent_writes, concurrent_counter_writes,


concurrent_materialized_view_writes, memtable_allocation_type etc. Please refer to Apache Cassandra documentation for the details about all the Cassandra settings.

DC/OS Kafka DC/OS Kafka can be deployed two ways, using CLI, or Using DC/OS web interface, as outlined below.


• dcos package install kafka


• dcos package install kafka –options=<options-json-file>

The default framework name is “kafka”. This can be changed by changing the “name” property in <options-json-file>.

Using DC/OS web interface Navigate to Catalog > Packages and search for Kafka. Click on the Kafka (certified) icon in the list of results that were returned. Use the following command to install the DC/OS Kafka CLI subcommands:

• dcos package install kafka --cli

The default framework name is “kafka”. This can be changed by configuring the framework name in the service section of the advanced installation section.

The following describes the most commonly used features of the Kafka framework and how to configure them via the DC/OS CLI and from the DC/OS web interface.

By default, Kafka Brokers will be installed on 3 nodes. To change the number of brokers, set the following value:

• In DC/OS CLI <options-json-file>: brokers.count: integer (default: 3)

{

"brokers": {

"count": 4,

…

}

• DC/OS web interface: BROKER_COUNT: integer

By default, Kafka framework will use 1 CPU core for each broker. To change the number of cores, set the following value:

• In DC/OS CLI <options-json-file>: borokers.cpus: number (default: 1)

{

"brokers": {

"cpus": 4,

…

}

• DC/OS web interface: BROKER_CPUS: number


By default, Kafka broker will use 1024 MB of memory on each node where it is installed. To change the amount of memory used by the broker, set the following value:

• In DC/OS CLI <options-json-file>: brokers.mem: integer (default: 1024)

{

"brokers": {

"mem": 2048,

…

}

• DC/OS web interface: BROKER_MEM: integer

Kafka framework supports two disk volume types:




• In DC/OS CLI <options-json-file>: disk_type: MOUNT (default: ROOT)

{

"brokers": {

"disk": 50000,

“disk_type”: MOUNT,

…

}

• DC/OS web interface: BROKER_DISK_TYPE: MOUNT

By default, brokers to be placed on any node with sufficient resources. To ensure that all brokers within a given Kafka cluster are never collocated on the same node, set the following value:

• In DC/OS CLI <options-json-file>: placement-strategy: NODE (default: ANY)

{

"brokers": {

placement-strategy: “NODE”,

…

}

• DC/OS web interface: BROKER_PLACEMENT_STRATEGY: NODE

By default the Kafka framework uses the ZooKeeper ensemble made available on the Mesos masters of a DC/OS cluster. You can configure an alternate ZooKeeper at install time by set the following value:

• In DC/OS CLI <options-json-file>: kafka_zookeeper_uri: <URI e.g. myhost:2181>


• DC/OS web interface: KAFKA_CUSTOM_ZOOKEEPER_PATH: <URI e.g. myhost:2181>

Using DC/OS CLI<options-json-file>, you can change other framework settings, such as service account, placement constraint, virtual network etc. Please refer to the Kafka Service Guide for the details.

Using DC/OS CLI<options-json-file>, you can also change Kafka broker settings (normally specified in services.properties configuration file) such log_retention_hours, log_roll_hours, log_segment_bytes etc. These same values are also represented as environment variables in the form KAFKA_OVERRIDE_LOG_RETENTION_HOURS etc. and may be modified through the DC/OS web interface Please refer to Apache Kafka documentation for the details about all the broker settings.

DC/OS Elastic DC/OS Elastic can be deployed two ways, using CLI, or Using DC/OS web interface, as outlined below.


• dcos package install elastic


• dcos package install elastic –options=<options-json-file>

The default framework name is “elastic”. This can be changed by changing the “name” property in <options-json-file>.

Using DC/OS web interface Navigate to Catalog > Packages and search for Elastic. Click on the Elastic (certified) icon in the list of results that were returned. Use the following command to install the DC/OS Elastic CLI subcommands:

• dcos package install elastic --cli

The default framework name is “elastic”. This can be changed by configuring the framework name in the service section of the advanced installation section.

The following describes the most commonly used features of the Elastic framework and how to configure them via the DC/OS CLI and from the DC/OS web interface.

By default, Elastic uses 3 Master Nodes and this can’t be changed. However, you can change the CPU cores, memory, disk size and type for the master nodes as shown below.

By default, Elastic framework will use 2 Data Nodes and 1 Coordinator Node and no (zero) Ingest Nodes. To change the number of Data Nodes, Coordinator Nodes and Ingest Nodes, set the following value:

• In DC/OS CLI <options-json-file>: count: integer

– Under ”data_nodes”, ”ingest_nodes”, ”coordinator_nodes” sections as appropriate

• DC/OS web interface: NODES: integer

By default, Elastic framework will use 1 CPU core for Master Node, 2 CPU cores for Data Nodes, 0.5 CPU cores for Ingest Nodes and 1 CPU core for Coordinator Node. To change the number of cores for Master Nodes, Data Nodes, Coordinator Nodes and Ingest Nodes, set the following value:

• In DC/OS CLI <options-json-file>: cpus: number

– Under “master_nodes”, ”data_nodes”, ”ingest_nodes”, ”coordinator_nodes” sections as appropriate

• DC/OS web interface: CPUS: number

By default, Elastic framework will use 2048 MB of memory for Master Node, 4096 MB of memory for Data Nodes, 2048 MB of memory for Ingest Nodes and 2048 MB of memory for Coordinator Node. To change the amount of memory used by Elastic, set the following value:


• In DC/OS CLI <options-json-file>: mem: integer

• DC/OS web interface: MEM: integer

Elastic framework supports two disk volume types for Master Nodes, Data Nodes, Coordinator Nodes and Ingest Nodes:




• In DC/OS CLI<options-json-file>: disk_type: MOUNT (default: ROOT)

– Under “master_nodes”, ”data_nodes”, ”ingest_nodes”, ”coordinator_nodes” sections as appropriate

• DC/OS web interface: DISK_TYPE: MOUNT

Using DC/OS CLI<options-json-file>, you can change other framework settings, such as service account, placement constraint, virtual network etc. Please refer to the Elastic Service Guide for the details.

Using DC/OS CLI<options-json-file>, you can also change Elastic settings (normally specified in Elastic.yaml configuration file) such as path of the data and logs directories etc. Please refer to Apache Elastic documentation for the details about all the Elastic settings.

DC/OS Zeppelin Zeppelin DC/OS Zeppelin can be deployed two ways, using CLI, or Using DC/OS web interface, as outlined below.


• dcos package install zeppelin


• dcos package install zeppelin –options=<options-json-file>

The default framework name is “zeppelin”. This can be changed by changing the “name” property in <options-json-file> as shown:

"service": {

"name": "zeppelin2",

"zeppelin_java_opts": "-Dspark.mesos.coarse=true -Dspark.mesos.executor.home=/opt/spark/dist"

},

…

Using DC/OS web interface Navigate to Catalog > Packages and search for Zeppelin. Click on the Zeppelin (Community) icon in the list of results that were returned.

You can change the number of CPU cores and the amount of memory used by underlying Spark executor by using the spark.cores_max and spark.executor_memory properties in <options-json-file> as shown:

"spark": {

"cores_max": 8

"executor_memory": 4096

}


Sign up for updates

© Copyright 2018 Hewlett Packard Enterprise Development LP. The information contained herein is subject to change without notice. The only warranties for Hewlett Packard Enterprise products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. Hewlett Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein.

Intel and Xeon are trademarks of Intel Corporation in the U.S. and other countries. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries. SMACK™ is a trademark of By the Bay, LLC.

a00039256enw, January 2018

Resources and additional links HPE Synergy, hpe.com/synergy

HPE Apollo, hpe.com/apollo

HPE Big Data Solultions, hpe.com/bigdata

HPE Reference Architectures, hpe.com/info/ra

HPE Servers, hpe.com/servers

HPE Storage, hpe.com/storage

HPE Networking, hpe.com/networking

HPE Technology Consulting Services, hpe.com/us/en/services/consulting.html

To help us improve our documents, please provide feedback at hpe.com/contact/feedback.

http://www.hpe.com/info/getupdated

http://www.facebook.com/sharer.php?u=https://www.hpe.com/h20195/V2/GetDocument.aspx?docname=a00039256enw

http://twitter.com/home/?status=HPE%20Reference%20Architecture%20for%20Fast%20Data%20Analytics%20on%20Mesosphere%20DC/OS+@+https://www.hpe.com/h20195/V2/GetDocument.aspx?docname=a00039256enw

http://www.linkedin.com/shareArticle?mini=true&ro=true&url=https://www.hpe.com/h20195/V2/GetDocument.aspx?docname=a00039256enw&title=HPE%20Reference%20Architecture%20for%20Fast%20Data%20Analytics%20on%20Mesosphere%20DC/OS+&armin=armin

http://www.hpe.com/

http://www.hpe.com/synergy

http://www.hpe.com/apollo

http://www.hpe.com/bigdata

http://www.hpe.com/info/ra

http://www.hpe.com/servers

http://www.hpe.com/storage

http://www.hpe.com/networking


http://www.hpe.com/contact/feedback

HPE Reference Architecture for Fast Data Analytics on ... · Apache Mesos. When faced with these challenges, enterprises tend to typically migrate towards a public cloud model that

Documents