Nosql Ug Paris Imdg in Action With Coherence Without Transactions Chapter 110525091639 Phpapp02

7/28/2019 Nosql Ug Paris Imdg in Action With Coherence Without Transactions Chapter 110525091639 Phpapp02

http://slidepdf.com/reader/full/nosql-ug-paris-imdg-in-action-with-coherence-without-transactions-chapter-110525091639 1/76

In Memory Data Grid in Action

with Oracle Coherence

for Paris NoSQL User Group

Cyrille Le Clerc

Transactions chapter will be presented

during another session

Wednesday, May 25, 2011



Speaker

2

Cyrille Le Clerc

@cyrilleleclerc

blog.xebia.fr

Open Source(Apache CXF, ...)

In Memory Data Grid

Large Scale

“you build it, you run it”




3

Once upon a time...




4

- Released Coherence in 2001

- Started as a distributed cache

- Released Gigaspaces XAP in 2001

- Started as a data grid

On the Financial side

• Very low latency

• Rich queries & transactions

• Scalability

• Data consistency

Needs within financial market :




5

Let’s define an In Memory Data Grid ...




Let’s define an In Memory Data Grid

6

eXtreme Scale

This is an In Memory Data Grid





7

This is Network Attached Memory




Similarities with NoSQL document orientedPartitioned, distributed Hastable, schema-less, value is not opaque,scale-out scalability

Very fast

In memory (persistence coming), business logic inside the data

Consistent and AvailableTransactional, redundant

Written in Java, data are POJOsNot necessary

Clients in Java, Microsoft, etc

8





9

Use cases for this presentation




Train Booking System

10

trains, stations,seats, booking and

passengers




eCommerce Web Site

11

warehouse stocks

231

2

canon-eos: 1ipod : 1headphone : 1

iphone: 1

...

ipad : 1

iphone: 1

barbie : 1

iphone: 1cabbage-doll: 1

121

311

12

264

637

{

"name": "Barbie Computer",

"stock": 637,

"weigth" : 200

}

warehouse &customers shopping carts




12

In Memory Data Grids Key Principles




Store Everything in a Mainframe !

13

3 To of RAM

80 x 5.2 GHtz cores Much more than $1,000,000

IBM z11

http://ibm.com/


http://ibm.com/

http://ibm.com/



Spread on Inexpensive Servers

14

Mainframe Cheap Servers !http://1userverrack.net/

http://ibm.com/


http://ibm.com/

http://ibm.com/



Partition Data

15

MainFrame

Small servers

Partition gamma

Partition beta

Partition alpha

Partition for scalability




Duplicate Data

16

s y n c

s y

n c h ro ni z a t i o n

Duplicate data for high availability

Partition alpha

Master

Standby Backup




17

Data Access Patterns





This is not traditional Java EE coding style !

Can apply very complex business logic inside thedata

18

Stored Procedures Style

Change management challenge !




19

Pattern : Targeted Operation




Pattern: Targeted Operation

20

Partition gamma

Search Trains

Partition beta

Search Trains

Partition alpha

Search Trains

{

"train-id": "tgv-3071-20110512",

"time" : 2011/05/12 12:15,

"departure" : "Paris",

"arrival" : "Marseille",

"seats" : 3,

}

Book Train Tickets

“train-id” is indexed




21

Pattern : Map Reduce Style Operation




Pattern: Map Reduce

22

Partition gamma

Search Trains

Partition beta

Search Trains

Partition alpha

Search Trains

{

"departure": "Paris",

"arrival": "Marseille",

"time" : 2011/05/12 12:00,

"seats" : 3,

}

Distributed “Search Train Ticket” Wednesday, May 25, 2011



Pattern: Map Reduce

23

Partition gamma

Search Trains

Partition beta

Search Trains

Partition alpha

Search Trains

{

"Paris -> Marseille : 12:15",

"Paris -> Marseille : 13:15"

}

Distributed “Search Train Ticket”

{ #NONE# }

{

"Paris -> Lyon -> Marseille : 12:40"

}




Pattern: Map Reduce

24

Partition gamma

Search Trains

Partition beta

Search Trains

Partition alpha

Search Trains

Distributed “Search Train Ticket”

{

"Paris -> Marseille : 12:15",

"Paris -> Lyon -> Marseille : 12:40",

"Paris -> Marseille : 13:15"

}





This is not traditional Java EE coding style

Don’t forget “Map Reduce” = “Distributed TableScan”

25

Use Indexes

Change management




26

CAP Theorem & In Memory Data Grids




CAP Theorem and In Memory Data Grid

27

Consistency

Availability

Partition

Tolerance

Only 2 of these 3 properties can be

achieved at any given

moment in timeBrewer’s Conjecture

http://lpd.epfl.ch/sgilbert/pubs/BrewersConjecture-SigAct.pdf






CAP Theorem and In Memory Data Grid

28

Consistency

Availability

Partition

Tolerance

Only 2 of these 3 properties can be

achieved at any given

moment in timeBrewer’s Conjecture


Data Grids






Cross Data Center Data Consistency

29

TokyoNew York

London

World wide replication

for financial marketWednesday, May 25, 2011




30

West Coast

East Coast

{


"stock": 147,

"weigth" : 200

}

{ "name": "Barbie Computer",

"stock": 147,

"weigth" : 200

}

Warehouse stocks





31

propagation delay !

West Coast

East Coast

{


"stock": 147,

"weigth" : 200

}

set stock to 146


"stock": 147,

"weigth" : 200

}





32

West Coast

East Coast

{


"stock": 147,

"weigth" : 200

}

set stock to 146


"stock": 147,

"weigth" : 200

}

set weight 175

reconciliation API needed !





33

West Coast

East Coast

{


"stock": 147,

"weigth" : 200

}

set stock to 146


"stock": 147,

"weigth" : 200

}

set weight 175

Network partitioning




34

Data Modeling




Data Modeling

Dominant Question Driven Design

Constrained Tree Schema

Denormalized

35

Opposite to Relational which is Domain Driven Design

Because RPC matters

Due to dominant questions and CTS




Data Modeling

36

TrainStop

date

TrainStation

code

name

Traincode

type

Seat

number

price

Booking

reduction

Passenger

name

Typical relational data model




Data Modeling

37

Find the root entity and denormalize

TrainStop

date

Seat

number

price

Booking

reduction

Passenger

name

R e f e r e n c e d a t a

D u p l i c a t e d i n e a c h g r i d n o d e

TrainStation

code

name

R o o t e n t i

t y

Partitioning readyentities tree

Train

code

type




Data Modeling

38

Remove unused data

TrainStop

date

Seat

number

price

Booking

reduction

Passenger

name

booked

TrainStation

codename

Train

code

type

Partitioned

Replicated




Data Modeling

39

TrainStop

date

TrainStation

code

name

Seat

number

price

bookedTrain

codetype

Data Grid Ready data structure

Partitioned

Replicated




40

Data Modeling is Hard !





41

Two root entities for thesame MoneyTransfer !

from to

CashWitdrawal

date

amount

MoneyTransfer

id

date

amount

Account

number

CashWitdrawal

date

amount

Account

number





42

CashWitdrawal

date

amount

CashWitdrawal

date

amount

MoneyTransferIn

id

date

amount

MoneyTransferOut

id

date

amount

Account

number

Account

number

Split MoneyTransfer





43

CashWitdrawal

date

amount

MoneyTransferOut

id

date

amount

Account

number

CashWitdrawal

date

amount

MoneyTransferIn

id

date

amount

Account

number

Split MoneyTransfer





44

CashWitdrawal

date

amount

MoneyTransferOut

id

date

amount

MoneyTransferIn

id

date

amount

Account

number

Data Grid Ready data structure




45

Grid Internals




Data Serialization

Used for data transfer and byte oriented storage

Hot topic like Apache Thrift, Apache Avro, GoogleProtocol Buffer

46

Must support evolvable data structure




Data Storage

Store Java Beans in the grid

Store byte arrays in the grid

47

No need to unmarshall for inprocess operations

Beware of garbage collector !

Pay unmarshalling at each read and write

Slightly more garbage collector friendly

Low-level / byte-oriented APIs to read data




Communication Protocols

UDP Multi Cast (Coherence, Gigaspaces)

TCP/IP (Websphere eXtreme Scale)

48




Topology

Partitions made of shards : 1 primary + 0..*backups)

Dynamic shards location (changes at runtime andat restart)

Can use dedicated “directory servers” or embed it

in the “data nodes”

49




JVM and Memory

Many editors recommend tiny 1.4 Go JVM !

More than ten JVM per server

50

Garbage collector hell

Management hell

More and more IMDG support large heaps




51

APIs




Raw Java Mapping with Oracle Coherence

52

hand-coded serialization JUnit is your friend !

public class Train extends AbstractEvolvable implements PortableObject {

enum Type {

HIGH_SPEED, NORMAL

}

/** Key of the Cache */

String code;

/** Indexed */

String name;

Type type;

List<Seat> seats = new ArrayList<Seat>();

int version;

List<TrainStop> trainStops = new ArrayList<TrainStop>();

@Override

public int getImplVersion() {

return 1;

}

@Override

public void readExternal(PofReader pofReader) throws IOException {

this.code = pofReader.readString(0);

this.name = pofReader.readString(1);

this.type = (Type) pofReader.readObject(2);

pofReader.readCollection(3, this.seats);

pofReader.readCollection(4, this.trainStops); this.version = pofReader.readInt(5);

}

@Override

public void writeExternal(PofWriter pofWriter) throws IOException {

pofWriter.writeString(0, this.code);

pofWriter.writeString(1, this.name);

pofWriter.writeObject(2, this.type);

pofWriter.writeCollection(3, this.seats, Seat.class);

pofWriter.writeCollection(4, this.trainStops, TrainStop.class);

pofWriter.writeInt(5, this.version);

}}

TrainStopdate

Seat

number

price

bookedTrain

code

type


JPA Style Mapping with Websphere eXtreme



JPA Style Mapping with Websphere eXtremeScale

53

sub entities can havecross relations

@Entity(schemaRoot=true)

public class Train {

@Id

String code;

@Index

@Basic

String name;

@OneToMany(cascade=CascadeType. ALL)

List<Seat> seats = new ArrayList<Seat>();

@Version

int version;

...}

TrainStopdate

Seat

number

price

bookedTrain

code

type




Map API with Oracle Coherence

54

NamedCache trainCache = CacheFactory.getCache("train-cache");

/** Save */

void persist(Train train) {

trainCache.put(train.getCode(), train);

}

/** Find by key */

Train findByCode(String code) {

return (Train) trainCache.get(code);

}

/** Find by Query Language */

Train findByTrainName(String name) {

Filter filter = QueryHelper.createFilter("name = :name" ,

Collections.singletonMap("name", name));

Set<Map.Entry<String, Train>> trainEntrySet = trainCache.entrySet(filter);

if (trainEntrySet.isEmpty()) {

return null;} else {

return trainEntrySet.iterator().next().getValue();

}

}

Map API




JPA Style with Websphere eXtreme Scale

55

/** Save */void persist(Train train) {

entityManager.persist(train);

}

/** Find by key */

Train findByCode(String code) {

return (Train) entityManager.find(Train.class, code);}

/** Query Language */

Train findByTrainName(String name) {

Query q = entityManager.createQuery("select t from Train t where t.name=:name");

q.setParameter("name", name);

return (Train) q.getSingleResult();

}

JPA Style Entity Manager




Creating Indexes

56

Map reduce (without index) = Distributed Table Scan !




Indexes with Oracle Coherence

57

class Train {

String name;

Collection<String> getTrainStationsCodes() {

return Collections2.transform(trainStops, ...);

}

...

}

{

NamedCache trainCache = CacheFactory.getCache("train-cache");

trainCache.addIndex(new ReflectionExtractor("getName"), false, null);

trainCache.addIndex(new ReflectionExtractor("getTrainStationsCodes"), false, null);

}


d i h b h S l



Indexes with Websphere eXtreme Scale

58

@Entity(schemaRoot=true)

class Train {

@Index

@Basic

String name;

@Index

Collection<String> getTrainStationsCodes() {

return Collections2.transform(trainStops, ...);

}

...

}

Query query = em.createQuery("select t from Train t where t.name=:name");

query.getPlan();

eXtreme Scale

for q2 in Train ObjectMap using INDEX on name = ( ?name)

filter ( q2.c[0] = ?name )

returning new Tuple( q2 )

This is an execution plan


M API



More APIs

Another Java EE versus Spring battle ?

JSR 347 Data Grids vs. Spring Data

59

Unified API ontop of NoSQL stores ?

Serialization / Object to Tuple Mapping API ?




60

Data Grid <-> Relational Database Interactions


D t G id R l ti l D t b



Data Grid <-> Relational Database

61

Data Grids are “In Memory” -> we need to persist data on disk !


D t G id < > R l ti l D t b




62

update / insert / delete

“select directly modified in DB”






63

backend DB

Highly available write behind queues + SQL batched statements

Data Grid -> Relational Database






64

TrainStop

date

TrainStation

code

name

Seat

number

price

bookedTrain

code

type

Constrained Tree Schema <-> Relational Impedance Mismatch

Data Grid -> Relational Database


Data G id < > Relational Database




DB writes MUST succeed !

65

Align the database on the Data Grid model !

Denormalize the database

Remove the foreign keys, use same PKs in DB and data grid

Support unordered SQL statements

Prefer raw SQL rather than reused business logic


Data Grid < > Relational Database




66

backend DB

Data Grid Originated Scheduled Refresh (Oracle System Change Number, etc)

select * from train where last_modif > ?

Relational Database -> Data Grid






67

backend DB

Database Originated Push JMS = durable subscription

(Oracle Database Change Notification, etc)

Relational Database -> Data Grid






In Memory -> prepare for reloading aftermaintenance operations !

Prepare consistency checkers

68

Need for “graceful shutdown with disk persistence”




69

Transactions




70

We didn’t have the time to talk abouttransaction.

Another session is planned at Paris No SQLUser Group for this.




71

Let’s go live !


Data Grids and Operations




Standard packaging?

Limited Management

Limited debugging tools

JVM pandemia

72

Do It Yourself (layout, scripts, etc)

Do It Yourself (stop/start, detecting data loss, etc)

Dozens of JVM to manage !

Do It Yourself (debugging consoles, troubleshooting agents)






Dev / Ops collaboration is required

Experts only !

73




74

The right tool for the right job






Incredibly fast ! Even with transactions !

Scalable

Good at data replication (when it implements it)

Very geeky on both dev and ops side

“Quite” expensive

75

Not an enterprise grade data store

Reconciliation api, etc

Requires very skilled people + change management

If you solve the data loading issue


Questions / Answers



?

Questions / Answers

Nosql Ug Paris Imdg in Action With Coherence Without Transactions Chapter 110525091639 Phpapp02

Documents