Introducing Apache Geode and Spring Data GemFire

Introducing Apache Geode with Spring Data GemFireJohn Blum / Luke Shannon

24 June 2015

Agenda

What is Apache Geode? Apache Geode Technical Specification High-level Implementation Details Spring Data GemFire Q&A

Pivotal GemFireWhat is Apache Geode?

In a nutshell…– Open Source core of Pivotal GemFire– Apache Incubator

▪ https://wiki.apache.org/incubator/GeodeProposal

What is Apache Geode?One definition is…A distributed, in-memory compute and data management platform that elastically scales to achieve high-throughput, low-latency access to big, fast data in order to power business critical, analytical applications in real-time.

Ops / SecLinear scalability

Elastic capacity +/-

Latency optimizeddata distribution

What is Apache Geode? Technically…

– Distributed, in-memory database (IMDG) with ACID properties

– Transactional Key/Value Store (Cache)

– Glorified-version of ConcurrentHashMap

– Message Bus with guaranteed message delivery▪ Reliable, asynchronous event notifications

What Apache Geode Does? Enables new or existing Java applications to operate at cloud-scale

with predictable, low-latencies and be highly-available without sacrificing consistency.

Applications can transact and analyze Big Data in real-time to garner insights and achieve meaningful and impactful business results.

Apache Geode Characteristics In-Memory

Distributed

Consistent

Highly-Available / Fault Tolerant

High-Throughput / Low, Predictable Latency

High Scalability

Data-aware Parallel Compute

How are Apache Geode’s features implemented?

Memory Geode stores data in memory…

– JVM Heap– Off-Heap (Native Memory)

Geode (optionally) stores data to disk– Persistence + Overflow

ManagementMemory

Eviction (LRU)

Expiration (Region & Entry TTL/TTI)

Automatic resource management (critical & eviction Heap %)

Region Compression (Snappy)

JVM GC Tuning

IMDG An In-Memory Data Grid (IMDG), such as Apache Geode…

– Pools system resources across multiple nodes in a cluster to manage both application state and behavior.

– Memory, CPU, Network and (optional) Disk

High-Throughput / Low-Latency Geode uses concurrent, in-memory data structures with a highly

optimized distribution infrastructure to minimize context switching and contention.– synchronous/asynchronous replication for high read throughput

– partitioning for both high read and write throughput

Shared-Nothing Disk Persistence Architecture Manages data in local disk files on each node in the cluster

independently from other nodes– Members are self-sufficient; Members do not share memory or disk;– Data/Transactions appended to Oplog files– Compaction

Design goals (all about In-Memory)…– Minimize IO operations (Write-Behind vs. Write-Through)– Minimize locking– Minimize serialization

REPLICATE & PARTITION RegionsHigh Availability

Data is partitioned across nodes in the cluster and distributed uniformly– Rebalanced as nodes join/leave– Redundant copies

Persistence – Shared NothingServer 3Server 2

Bucket 3*primary*

Bucket 1secondary

Bucket 3secondary

Bucket 2*primary*

Server 1

Bucket 2secondary

Bucket 1*primary*

Persistence – Shared NothingServer 1 Server 3Server 2

Bucket 2secondary

Bucket 1*primary*

Bucket 3*primary*

Bucket 1secondary

Bucket 3secondary

Bucket 2*primary*

What if a server crashes?

Persistence – Shared NothingServer 1 Server 3Server 2

Bucket 2secondary

Bucket 1*primary*

Bucket 3*primary*

Bucket 1*primary*

Bucket 3secondary

Bucket 2*primary*

Bucket 2secondary

Primary Failover

RestoreRedundancy

Bucket 1secondary

Network Partitions

Partition RegionsConfigurable Consistency

Redundant Copies – 0 to 3

Colocation

Total Number of Buckets– 113 (Default)

Local Max Memory– 90% of Heap

Total Max Memory– All Members, Both Primary/Secondary

Copies

REPLICATE vs. PARTITIONConfigurable Consistency

Partition Regions– Updates routed through primary bucket– Holds lock on key when distributing updates

Replicate Regions– Possible conflicts due to out-of-order updates– Conflict checking (--concurrency-checks-enabled)

ReplicatedPartitioned Disabled

Apache Geode Topologies

Peer-to-Peer (P2P)Apache Geode Topologies

Configuration with the Cache embedded in the application process– Close proximity of application logic and data

Client/ServerApache Geode Topologies

Single-Hop Data Access

Continuously load balanced between servers to get the best response-times

Automatic fail-over when servers crash or become unresponsive

WANApache Geode Topologies

Site-to-Site Replication for DR

Use CasesApache Geode

Persistent OLTP/OLAP– System of Record (SOR)

Caching with JSR-107 (JCache) or Spring Cache Abstraction HTTP Session State Management

– Caching and Replication

Distributed L2 Caching for Hibernate Memcached Server (Gemcached)

Additional FeaturesApache Geode

Functions– Application logic is distributed, routed and run in parallel with close proximity to the data required by the

application.

Serialization (PDX)

Delta-propagation – difference of new/old value

JTA-compliant Transactions

Security – SSL, Auth

RoadmapApache Geode

HDFS persistence Off-Heap storage Lucene search/indexing Spark integration Cloud Foundry service

InterfaceApache Geode

Language Support– Native: Java, C/C++, C#– Non-native: JavaScript, Python,

Ruby, Scala, etc

API– Java HashMap (Region)– OQL / Function– Memcache client– REST API– Spring Data GemFire

Data Structure– User-defined Classes / POJOs– Portable Data Exchange (PDX) – Documents (JSON)

Powerful Data Access– Object Query Language (OQL)– Data-aware, Scatter-Gather

Functions– Publish/Scribe framework for RI/CQs

with reliable, async event queues

Spring Data GemFirehttp://projects.spring.io/spring-data-gemfire/

So… Why Spring?

“Simple things should be simple; complex things should be possible”

– Alan Kay

Spring Data GemFire (SDG)Consistently applies Spring's powerful, non-invasive programming model to simplify configuration and development with Apache Geode.

Spring Ecosystem Integration…– Spring Cache Abstraction / Transaction Management– Spring Data Commons + REST– Spring Integration (Inbound/Outbound Channel Adapters)– Spring XD (Input Source and Sinks)

Apache Geode combined with Spring Data GemFire can be used as JSR-107 (JCache) caching provider

GRAILS

Full-stack, Web

Stream, Taps, Jobs

Bootable, Minimal, Ops-Ready

Big, Fast,

FlexibleData Web,

Integration,Batch

Controllers, REST,WebSocket

INTEGRATION

Channels, Adapters,Filters, Transformers

Jobs, Steps,Readers, Writers

BIG DATA

Ingestion, Export,Orchestration, Hadoop

NON-RELATIONALRELATIONAL

GROOVYFRAMEWORK SECURITY REACTOR

Use Cases / ConfigurationsSpring Data GemFire

Apache Geode Configuration / Bootstrapping– Replacement for cache.xml; Can be used with Cluster Configuration

Application Peer (Cache) Application Client (ClientCache)

Examples…SDG-based Configuration

with SpringBootstrapping Apache Geode

@SpringBootApplication@ImportResource("/spring-data-gemfire-cache.xml")@EnableGemfireRepositories@EnableTransactionManagementpublic class SampleDataGemFireApplication { public static void main(String[] args) { SpringApplication.run(SampleDataGemFireApplication.class, args); }}

Or…gfsh>start server –name=Example … --spring-xml-location=“<classpath-to-spring-application-context.xml>”

Basic (Region) Data AccessData Access with Spring

@Repositorypublic class CustomerDao {

@Resource(name = “Example”) private Region<?, ?> example; …}

Example…

GemfireTemplateData Access with Spring Data GemFire

Advantages…– Simple, Convenient Data Access (CRUD, OQL, Function)– Protects developer from GemFire/Geode API changes – Exception Translation into Spring DAO Exception Hierarchy– Transaction Management

GemfireRepositoryData Access with Repositories

public interface CustomerRepository extends GemfireRepository<Customer, Long> {

// Spring context.xml<gfe-data:repositories base-package="org.example.app.customer.dao.repo"> <repo:include-filter type="assignable" expression=”org.example.app.customer.dao.repo.CustomerRepository"/></gfe-data:repositories>

Example…

Implementation

Annotation-based Function Implementation/Execution

package org.example.app.functions;

import …

@Componentpublic class ExampleFunctions {

@GemfireFunction public Customer update(…) { … } …}

Example…

(Optional)…

Execution

package org.example.app.function.executions;

import …

@OnRegion(“Example”)public interface ExampleFunctionExecution { Customer update(…);}

And…<gfe:annotation-driven/>

<gfe-data:function-executions base-package="org.example.app.function.executions">

Execution

Additionally…– @OnServer(s), @OnMember(s)

Examples…

JSR-107 (JCache), Spring Cache AbstractionGeode Caching Provider

With @Cacheable And @CachePut, @CacheEvict(allEntries=[true|false]), @CacheConfig

– http://docs.spring.io/spring/docs/current/spring-framework-reference/htmlsingle/#cache-jsr-107

Or with JSR-107 using…– @CacheResult, @CachePut, @CacheRemove, @CacheRemoveAll,

@CacheDefaults

DemoBy Luke Shannon

On the horizon…

RoadmapSpring Data GemFire

Support for GemFire 8.2 & 9.0– HDFS, Off-Heap, Apache Lucene, Spark

Support for pre-defined data sets using Snapshot Service– SGG-408

Expiration annotation support with CustomExpiry– SGF-404

Spring Session Adapter for GemFire/Geode– SGF-373

Roadmap Continued…Spring Data GemFire

Repository Query method annotation extensions...– SGF-392

public interface ExampleRepository implements GemfireRepository<Customer, Long> {

@Trace @Limit(20) @Import(“org.example.app.domain.SomeOtherType”) List<Customer> findByLastname(String lastName);

Get Involved…

Apache GeodeContribute

In addition to developing Spring applications with Apache Geode, now is a perfect time to get involved in the Apache Geode community and contribute growing Apache project from the ground-up.– Ways to participate…

http://geode.incubator.apache.org/contribute/

Finally…

Washington, DC – September 14 – 17, 2015SpringOne2GX

Early Bird price registration ends August 14th, Register Now!– https://2015.event.springone2gx.com/register

Building High-Scalable Spring Applications with Distributed In-Memory Data Grids.– Luke Shannon & John Blum– Tuesday, September 15th, 2015– https://2015.event.springone2gx.com/schedule/sessions/

building_highly_scalable_spring_applications_with_in_memory_distributed_data_grids.html

References http://geode.incubator.apache.org/ http://geode.incubator.apache.org/contribute/ https://github.com/project-geode/docs/wiki https://github.com/apache/incubator-geode https://pivotal.io/big-data/pivotal-gemfire http://gemfire.docs.pivotal.io/latest/userguide/index.html#getting_started/topics/gemfire_overview.html http://gemfire.docs.pivotal.io/latest/javadocs/japi/index.html http://stackoverflow.com/questions/tagged/geode+or+gemfire http://projects.spring.io/spring-data-gemfire/ http://docs.spring.io/spring-data-gemfire/docs/1.6.0.RELEASE/reference/html/ http://docs.spring.io/spring-data-gemfire/docs/current/reference/html/ http://docs.spring.io/spring-data-gemfire/docs/current/api/ https://github.com/spring-projects/spring-data-gemfire https://github.com/spring-projects/spring-gemfire-examples http://stackoverflow.com/questions/tagged/spring-data-gemfire

Q&AAny questions…

Thank you!

Introducing Apache Geode and Spring Data GemFire

Technology

Building Apps with Distributed In-Memory Computing Using...

Apache Geode で始めるSpring Data Gemfire

VMware GemFire® .NET® Client 10 · GemFire Native Client....

An introduction to Apache Geode

Pivotal GemFire® Native Client 10 · 2020-05-05 ·...

Apache Geode Offheap Storage

ApacheConBigData - Introducing Apache Geode - final · PDF.....

1372 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED...

Apache Geode/Pivotal GemFire - itdks.su.bcebos.com

Implementing a Highly Scalable Stock Prediction System with....

Pivotal's effort on Apache Geode

#GeodeSummit: Democratizing Fast Analytics with Ampool...

Getting Started with Apache Geode

Using Apache Calcite for Enabling SQL and JDBC Access to...

VMware GemFire® C++ Client 10€¦ · Pivotal Cloud...

Apache Geode Clubhouse - WAN-based Replication