Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. DMITRIY SETRAKYAN GridGain Founder & Chief Product Officer Apache Ignite PMC Apache Ignite TM - In-Memory Data Fabric Fast Data Meets Open Source http ://ignite.apache.org @apacheignite @dsetrakyan
21
Embed
August 2016 HUG: Better together: Fast Data with Apache Spark™ and Apache Ignite™
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
• Very Active Community• Great Way to Learn Distributed Computing• How To Contribute:
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Apache IgniteTM In-Memory Data Fabric: Strategic Approach to IMC
• Supports Applications of various types and languages
• Open Source – Apache 2.0• Simple Java APIs• 1 JAR Dependency• High Performance & Scale• Automatic Fault Tolerance• Management/Monitoring• Runs on Commodity Hardware
• Supports existing & new data sources
• No need to rip & replace
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Apache Ignite In-Memory Data Fabric
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
• Long Running Applications– Passing State Between Jobs
• Disk File System (HDFS?)– Convert RDDs to Disk Files and Back– Argh#$%
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
• In-Memory Key-Value Store– Good for Caching Tuples
• Foundation for Shared Memory State – IgniteRDD is based on Data Grid– Ignite File System is based on Data Grid
• On-Heap & Off-Heap Memory• In-Memory Indexes
– Fast SQL• Built for High Throughput and Low Latencies
Why Data Grid?
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Data Grid: Distributed Caching
Partitioned Cache Replicated Cache
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
• ANSI-99 SQL• Always Consistent• Fault Tolerant• In-Memory Indexes (On-Heap and Off-Heap)• Automatic Group By, Aggregations, Sorting• Cross-Cache Joins, Unions, etc.• Ad-Hoc SQL Support
Data Grid: Ad-Hoc SQL (ANSI 99)
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
SQL Cross-Cache GROUP BY Example
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Apache Ignite for Spark and Hadoop
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
• Automatic Resource Management• Easy Data Center Installation• Easy Data Center Configuration• On-Demand Elasticity
DevOps: Integration with Yarn and Mesos
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
• IgniteRDD Deployment Modes– Share RDD across tasks on the host– Share RDD across tasks in the application– Share RDD globally– Embedded vs External Deployments
• Faster SQL– In-Memory Indexes– SQL on top of Shared RDD
Share RDDs Across Spark Jobs
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
• Main Entry Point from Spark to Ignite• Specify Different Ignite Configurations• Embedded vs External Deployments
– Client vs Server Modes
IgniteContext
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
• Implementation of SparkRDD• Mutable (unlike native RDDs)• Partitioned over Ignite Partitioned Caches• Indexed SQL
– Spark only does Full Scans– Indexes are 1000x faster
IgniteRDD
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
• Ignite In-Memory File System (IGFS)– Hadoop-compliant– Easy to Install– On-Heap and Off-Heap– Caching Layer for HDFS– Write-through and Read-through HDFS– Performance Boost
Ignite In-Memory File System
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Ignite In-Memory Map Reduce• In-Memory Native
Performance• Zero Code Change• Use existing MR code• Use existing Hive queries• No Name Node• No Network Noise• In-Process Data Colocation• Eager Push Scheduling
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
• More SQL– Non-Collocated Joins– Data Modification Language (DML)– Dada Definition Language (DDL)
• More Drivers– JDBC (already in Ignite 1.5)– ODBC (Ignite 1.6)
Apache Ignite Roadmap
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Interactive SQL with Apache Zeppelin
Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
ANY QUESTIONS?
Thank you for joining us. Follow the conversation.