Apache Igniteand Apache Spark - events.static.linuxfound.org · Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the

Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSoftwareFoundationintheUnitedStatesand/orothercountries.

DenisMagdaGridGain ProductManager

ApacheIgnitePMC

ApacheIgnite andApacheSparkWhereFastDataMeetstheIoT

http://ignite.apache.org #apacheignite #denismagda


Agenda• IoT DemandstoSoftware• IoT SoftwareStack• DeviceOS/RTOS• DataCollectionandEnrichment• HTAPPlatform• ApplicationAPIs

• Demo


IoT DemandstoSoftware


IoT DemandstoSoftware• Real-timeingestion• Real-timeprocessing• Time-seriessupport• FlexibleQueryingAPIs– SQL– Full-textsearch– Geo-spatial

• Analytics– BI– MachineLearning

• High-availability• Simplescalability


IoT SoftwareStack


HTAP Platform

Data Collection and Enrichment

Device OS / Real-Time OS

Application APIs

IoT SoftwareStack


HTAP Platform

Data Collection and Enrichment

Device OS / Real-Time OS

Application APIs

ApacheIoT SoftwareStack


DeviceOS/RTOS


ApacheMyNewt• OpenSourceRTOS– CortexM0-M4– MIPS&RISC-V

• Networking– BluetoothLowEnergy– Wi-Fi– TCP/IP&UPD

• Build&PackageManagement• Securebootloaderandsigned

images• RemoteFirmwareUpgrade


DataCollectionandEnrichment


DataCollectionandEnrichment• SparkStreaming– Fault-TolerantStreams

Processing• DataCollection– Sockets– Kafka– Flume

• DataEnrichment– SparkAPI

• DataStorage– ApacheIgnite


HTAPPlatform


UseCase: Smart Metering and Utilities – delivers a comprehensive IOT platform

50+ Million Meters

• SilverSpringRequirements– Migratetoin-memoryprocessing– Addscalability&elasticity– Useopensourcetechnologies

• SilverSpring Solution– SaaSArchitecture

• Multi-Tenancy• AdvancedSecurity

– Strongcomputecapabilities• Co-locatedin-memoryprocessing

– Demonstratedbest• On-demandelasticity&scalability• ANSI-99SQLSupport• Transactionalconsistency

GridGainSecurity

SilverSpring IoTPlatform


ApacheIgniteIn-MemoryDataFabric

• Supports Applications of various types and languages

• Open Source – Apache 2.0• Simple Java APIs• 1 JAR Dependency• High Performance & Scale• Automatic Fault Tolerance• Management/Monitoring• Runs on Commodity Hardware

• Supports existing & new data sources• No need to rip & replace


In-MemoryDataGrid• DistributedKey-ValueDataStore

• DataReliability• High-Availability– Activereplicas,automaticfailover

• DataConsistency– ACIDdistributedtransactions


In-MemoryDataGrid:Partitioning

PartitionedCache ReplicatedCache


StreamingtoIgnite• IgniteDataStreamer– FastestDataInjection– Automaticdatapartitioning

• StreamReceivers– Customlogicexecution– Additionaldatatransformation

• StreamAdapter– Socket– Kafka– Flink– RocketMQ– etc.


ApplicationAPIs


ApacheIgniteSQLGrid• ANSI-99SQL• Geo-spatialQueries• Full-textSearch• AlwaysConsistent• FaultTolerant• Cross-Platform– JDBCandODBCdrivers– DML(INSERT,UPDATE,etc.)– DDL(CREATE,DROP,etc.)


ApacheIgniteSQLGrid:Queries• DistributedJoins• AutomaticGroupBy,Aggregations,Sorting

• Cross-CacheJoins,Unions,etc.• Ad-HocSQLSupport


ApacheIgniteComputeGrid• DirectAPIfor

MapReduce• DirectAPIforForkJoin• ZeroDeployment• StateCheckpoints• LoadBalancing• AutomaticFailover


SparkSharedRDDs• IgniteRDD– ShareRDDacrossjobson

thehost– ShareRDDacrossjobsin

theapplication– ShareRDDglobally

• FasterSQL– In-MemoryIndexes– SQLontopofSharedRDD


MachineLearningGrid


Demo


Resources• Documentation:– ApacheIgnite:https://apacheignite.readme.io/docs– ApacheSparkStreaming:http://spark.apache.org/streaming/– IgniteandSparkIntegration:https://apacheignite-

fs.readme.io/docs/ignite-for-spark– ApacheMyNewt:http://mynewt.apache.org

• DemoSourceCode:– https://github.com/dmagda/IgniteSparkIoT


ANYQUESTIONS?Thankyouforjoiningus.Followtheconversation.

http://ignite.apache.org

#apacheignite #denismagda

Apache Igniteand Apache Spark - events.static.linuxfound.org · Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the

Documents