Viadeo - The Kasper way

Viadeo PlatformArchitecture

Viadeo LegacyThe analysis

Viadeo

website

DATA

Graph APIAcquisition

Mobile Widgets

Backoffices

One shared code to rule them all... (or not !)

The new architecture has been designed understanding the root issues of the current system :

➔ monolithic system, designed for the web only➔ difficult to test and ensure the quality of the products➔ unability to extract business logic from the code➔ extreme coupling of the components and effective porosity of the layers➔ designed from the data, not from the product➔ no common way to analyze produced data, our treasure, heavily heterogeneous

structures and data➔ not driven by metrics➔ not documented➔ no common interfaces between layers➔ Many ways to do the same thing

It was important to know from where we start..

Viadeoplatform

DATA

Partners API AcquisitionMobile Widgets BackofficesWebsite

One business platform to rule them all...

for well designed products generating usable data from standardized components

➔ true MVPs➔ fast web development➔ maintenable code base➔ tested code➔ easy to remove / delete /

refactor➔ separation command / query

(fast products / different concerns)

➔ multiple rollouts/day

➔ our treasure➔ normalized way to produce

data➔ event-oriented architecture➔ the correct database system

for the correct data➔ BI team involved into the new

system from the beginning

➔ One way to develop➔ One developer can switch of

team➔ Ensure maintainability over

years➔ Auto-documentation➔ Auto-exposition to clients➔ Encourage a knowledge

community➔ Encourage good development

patterns

We want a platform...

DATAR/W

A simple services platform

DATA

W

R

(COMMANDS)

(QUERIES)

Command Query Responsibility Segregation

DATA

COMMANDS

QUERIES PROJ ETL* * *

* E,T,L : Extract, Transform, Load

CQRS & Query indexes

EVENT/ENTITYSTORE

COMMANDS

QUERIES PROJ ETL

EVENTLOG

E

Event-Driven Architecture / Event-Sourced

COMMANDS

QUERIES PROJ ETL

EVENTLOG

E

EVENT/ENTITYSTORE

Domain Driven Design

CQRS-DDD-ES/EDASample use case

Let’s take an example“Companies who recruit”

Viadeo Platform

GetCompaniesWhoRecruitGetTopCompaniesWhoRecruitGetLatestRecruitmentsOfCompanyGetCountiesWhereCompaniesRecruit GetCompanyContactsForRecruitment

…

about 10 services for this project

Webapp

Queries Commands

1- Identifying services

GetTopCompaniesWhoRecruit ?

Viaduct - MySQL

?

2- Identify the data location

A dedicated ElasticSearch Index

Optimized for● search company by name● search by facets

Viaduct - MySQL

3- Choose the right Query data storage system

?

Option A - Classical "viadeo-batch" quering MySQL

Viaduct - MySQL

batch

4- Implements the batch script

Option B "Hadoop batch" over HDFS

Viaduct - MySQL

batch HDFS

4- Implements the batch script (bis)

➔ Full update each time➔ Load charge on MySQL if no HDFS➔ Tight coupling with MySQL model➔ Jobs Hell

Limitations

MySQL

Add Me A Position

Check if OK ?

Here !

AddMeAPositionService

Is there a smarter way ?

MySQL

Add Me A Position

member_added_a_new_position !

The BUS

Use events !

member_added_a_new_position

is OK.But what we want is

member_moved_into_a_new_company

Define clearly your business needs

or maybe…

company_has_a_new_employee

Define clearly your business needs

MySQL

Add Me A Position

member_added_a_new_position !

The BUS

When there is a new position added,

is it a new employee for a company ?

Implement your access points and their logic

member_added_a_new_position

MySQL

company_has_a_new_employee

WatchForNewRecruitmentListenerin the Company Domain

Then complete your business logic..

AddMeAPositionGetTopCompaniesWhoRecruit

listener

service

listener

company_has_a_new_employee member_added_a_new_position

WatchForNewRecruitment

MySQL

Finally use events to update the queried data

➔ An easy real-time implementation

➔ No load and scalability issues

➔ Monitoring and metrics

➔ Reuse company_has_a_new_employee

➔ Simple and cool to develop

Many gains...

Kasper PlatformThe new architecture

COMMAND COMMANDHANDLER

ENTITY

REPOSITORY

EVENT

QUERY QUERYHANDLER

EVENTLISTENER

EVENTLISTENER

handles

spawn

interact

s

interacts

listens

generates

stores

handles

BUS

listens

- Event Store- Entity Store- Business Indexes

persists

Query Indexesqueries indexes

published to

COMMAND AREA (domain X)

QUERY AREA (domain X)

submit

submit

listens domain Y

Hands on Kasper components

DOMAIN

COMMAND

EVENT

QUERY

Commands, Queries and Events are immutable POJOSused to transfer information accross different channels

COMMAND

QUERY

Hands on Kasper componentsC/Q/E : the domain API

x

x

x

APICOMMAND

QUERY

ENTITIES - REPOSITORIES - HANDLERS - LISTENERS

HANDLERS - LISTENERS - INDEXERS - DATA ACCESS

DOMAIN-

COMMANDS-

QUERIES-

RESULTS

Hands on Kasper componentsrecommended modules split w/ dependencies


ENTITY

REPOSITORY

EVENT

QUERY QUERYHANDLER

EVENTLISTENER

EVENTLISTENER

handles

spawn

interact

s

interacts

listens

generates

stores

handles

BUS

listens


persists


published to



submit

submit

listens domain Y


COMMANDHANDLER

QUERYHANDLER

COMMAND

QUERY

Status (OK, ERROR, REFUSED) (+ opt. security token)

RESULT

Interceptor

Interceptor

Hands on Kasper componentshandling commands and queries


ENTITY

REPOSITORY

EVENT

QUERY QUERYHANDLER

EVENTLISTENER

EVENTLISTENER

handles

spawn

interact

s

interacts

listens

generates

stores

handles

BUS

listens


persists


published to



submit

submit

listens domain Y


Entity (identified) != Value Object (valued)An Aggregate

- can be composed of several entities

- is managed by a root entity (the aggregate root)

- is persisted and retrieved through a Repository

- is an atomic data composition (persisted and retrieved as a whole)

- all accesses to the aggregate are done through the aggregate root

- logical consistency and coherency of the aggregate is then ensured at any time

Hands on Kasper componentsentities / aggregates

Kasper defines two kind of aggregates : the Concept aggregate and the Relation aggregate

The Relation aggregate is used to indicates an instance of relation between two Concept aggregates where the two linked concepts can exists independently of this relation.

Member RecruiterwasRecruitedBy

Member MemberisConnectedTo

Status Creator

bi-directional relation

createdBy

unidirectional relationconcept

concept

concept

concept

concept

concept

component relation (LinkedConcept<Member>)

Hands on Kasper componentstwo kind of aggregates

String conceptValue;

void setConceptValue(String strValue) {

if ( ! strValue.isEmpty()) { apply(ValueChangedEvent(strValue)); }

}

@EventHandlervoid onConceptValueChanged( ValueChangedEvent event) {

this.conceptValue = event.getValue();

}Event Bus

ValueChangedEvent handlersValueChangedEvent handlersValueChangedEvent handlersValueChangedEvent listeners

1: handle command

2: call the domain

3: auto apply event

4: generalize event

DoSomethingCommand

DoSomethingCommandHandler

ValueChangedEvent

Hands on Kasper componentsevent-sourced aggregate

new()apply(MyAggregateCreatedEvent)

changeMyProperty()apply(MyAggregatePropertyChangedEvent)

HANDLER

UNIT

OF

WORK

MyAggregate

Repository

BUS

doSave()

myAggregateCreatedEvent

myAggregatePropertyChangedEvent

Event store

Entity store

Business index

persists

loadadd()

Hands on Kasper componentsunit of work

load(id)save(id, aggregate)

delete(aggregate)

RepositoryEvent store

Entity store

Business index

has(id)business

specificBusinessMethod()

The Repository is the component of the Command area where all accesses to data are concentrated.

The standard Repository interface is a simple key/value store API.

If an access to some Business index is required, the repository ensures the main persistence database keeps synchronized with the business index.

A Repository can either access an Entity store or an Event store depending on the way the aggregate has to be saved regarding the company strategy and legacy migration possibilities.

note: Use this.getRepository().business().specificBusinessMethod() in your command handlers

get(id)

Hands on Kasper componentsanatomy of a repository


ENTITY

REPOSITORY

EVENT

QUERY QUERYHANDLER

EVENTLISTENER

EVENTLISTENER

handles

spawn

interact

s

interacts

listens

generates

stores

handles

BUS

listens


persists


published to



submit

submit

listens domain Y


Event StoreEvents

Entity StoreEntity

Entity

Entity

Entity store vs Event store


ENTITY

REPOSITORY

EVENT

QUERY QUERYHANDLER

EVENTLISTENER

EVENTLISTENER

handles

spawn

interact

s

interacts

listens

generates

stores

handles

BUS

listens


persists


published to



submit

submit

listens domain Y


The available event responses are:

➔ SUCCESS : All is fine, the event has been correctly handled.

➔ ERROR : An error occured which is an expected part of normal operations, the event is consumed and the system continues to operate.

➔ FAILURE : An unexpected error occured, which can require intervention before the system can resume at the same level of operation. This does not mean that failures are always fatal, rather that some capacity of the system will be reduced. Event will be requeued.

➔ TEMPORARILY_UNAVAILABLE : A managed error occured, identified by the developer as a particular failure. Event will be replayed then requeued.

Event listeners

domain A

domain B

listener for x

listener for y

listener for z : parent of x and y

kasper rabbitmq kasper

x 2

x 1

x 1queue xqueue yqueue z (x + y)

exchange

better isolation between consumerssimple acknowledgement logiconly consume what needednumber of queues to maintains

RabbitMQ implementation


ENTITY

REPOSITORY

EVENT

QUERY QUERYHANDLER

EVENTLISTENER

EVENTLISTENER

handles

spawn

interact

s

interacts

listens

generates

stores

handles

BUS

listens


persists


published to



submit

submit

listens domain Y


From simple events to business transactions

action → event → listener → actionaction → event → listener → actionaction → event → listener → action

events, timeouts → workflow → actions

Kasper SAGAs

EventSAGA

start

endstep

Kasper SAGAs

SAGA ?- A SAGA tells a STORY

- A SAGA permits to aggregate several LISTENERS in a coherent WORFLOW

- A SAGA is a COMMAND component (its goal is to WRITE/CREATE something)

- A SAGA is better for long-lived STORIES (to store a state over time)

- A SAGA can be a good pattern to replace BATCHES (not all)

- A SAGA is started by an EVENT

- A SAGA can have several STEPS triggered by EVENTS or SCHEDULES

- A SAGA have at least one ENDING STEP

- A SAGA is stored in the database, so it can live “indefinitely”

Kasper SAGAs

ANewCardAsBeenAddedOnAccountEvent

CardExpirationMonitoringSaga

D-30D-1

Card expired since 10 days

CardHasBeenRenewedEvent

(PremiumSubscriptionEnded)

send emai

l

send emai

l

send emai

l


ENTITY

REPOSITORY

EVENT

QUERY QUERYHANDLER

EVENTLISTENER

EVENTLISTENER

handles

spawn

interact

s

interacts

listens

generates

stores

handles

BUS

listens


persists


published to



submit

submit

listens domain Y


Spark & Spark streamingfor fun and profit

HDFS / S3sqoop COMMAND AREA (domain X)


Query Indexes

Spark batch

EVENT

BUS

Spark streaming

REPOSITORY

ENTITY

reads

writes writes

listen

publish

writes


ENTITY

REPOSITORY

EVENT

QUERY QUERYHANDLER

EVENTLISTENER

EVENTLISTENER

handles

spawn

interact

s

interacts

listens

generates

stores

handles

BUS

listens


persists


published to



submit

submit

listens domain Y


- Every command or query sent to the platform is sent with an associated context which specifies informations like :

➔ the name of the calling client➔ a security token used to authenticate, identify and authorize the member➔ some correlation ids to be used in platform logs

- Any HTTP/JSON client can access the platform once it provides the calling context

- Any JVM-compliant client can call the platform using the JAVA Kasper client


A Kasper platform : the big picture

Kasper PlatformAdditional stuff

Auto-documented platform

Auto-documented platform

Auto-measured platform

for well designed products generating usable data from standardized components

Remember: A platform..

AppendixFrom the code

@XKasperCommand(description = "Send an Hello to a buddy name")public class SendHelloToBuddyCommand extends CreateCommand {

@NotNull( message = NOT_PROVIDED_HELLO_MSG ) private final String message;

@NotNull( message = NOT_PROVIDED_BUDDY_MSG ) private final String forBuddy;

public SendHelloToBuddyCommand( final KasperID id, final String message, final String forBuddy ) { super(id); this.message = message; this.forBuddy = forBuddy; }

public String getMessage() { return this.message; }

public String getForBuddy() { return this.forBuddy; }}

Optional Kasper annotation used to document your APIInstead of simply implementing the Command interface you must give additional information on commands which concerns entities mutation to ensure a normalized API

Uses JSR-303 validation annotation

Kasper domain API components must be immutables

Immutable component with its big constructor (use builders when its necessary)

Getters to allow component properties read

Hands on Kasper componentsanatomy of a domain API component

Hands on Kasper componentshandling commands

@XKasperCommandHandler( domain = HelloDomain.class, description = "Submit a new Hello message to a specified buddy")public class SendHelloToBuddyCommandHandler extends EntityCommandHandler<SendHelloToBuddyCommand, Hello> {

@Override public CommandResponse handle(final SendHelloToBuddyCommand command) throws Exception {

final Hello newHello = new Hello( command.getIdToUse(), command.getMessage(), command.getForBuddy() );

this.getRepository().add(newHello);

return CommandResponse.ok(); }

}

You have to specify the attached domain and some optional documentation

Helper abstract class providing an easy way to access the repository

The Command class to process

The command to process

The only method to implement in order to handle the command

Create a new aggregate

Add the aggregate to the repository

Return ok

@XKasperQueryHandler( domain = HelloDomain.class, filters = { NormalizeBuddyQueryFilter.class })public class GetAllHelloMessagesSentToBuddyQueryHandler extends QueryHandler<GetAllHelloMessagesSentToBuddyQuery, HelloMessagesResult> {

private KeyValueStore store = HelloMessagesIndexStore.db;

@Override public QueryResponse<HelloMessagesResult> retrieve( final GetAllHelloMessagesSentToBuddyQuery query) throws KasperQueryException {

final String forBuddy = query.getForBuddy(); Collection<HelloMessageResult> ret = Lists.newArrayList();

if (store.has(forBuddy)) { ret = ((Map<KasperID, HelloMessageResult>) store.get(forBuddy).get()).values(); }

return QueryResponse.of(new HelloMessagesResult().<HelloMessagesResult>withList(ret)); }

}

Hands on Kasper componentshandling queries

You have to specify the attached domain and some optional documentation

The input query class to be processed

The ouput query result class to be returned

The base query handler class

The query to be processed

Return the query result

Build the query result

Hands on Kasper componentstwo kind of aggregates

Member RecruiterwasRecruitedByunidirectional relationconcept concept

@XKasperConcept( domain = Member.class ) public class Member extends Concept { ... }

@XKasperConcept( domain = Member.class ) public class Recruiter extends Member { ... }

@XKasperRelation( domain = Member.class ) public class Member_wasRecruitedBy_Recruiter extends Relation<Member, Recruiter> { ... }

Mandatory annotationfor domain sticking

Specify the relationedge concepts

relation verb is specified by default with name convention

Member

bi-directional relationconcept

concept

@XBidirectional( inverse_verb = "recruited" )@XKasperRelation( domain = Member.class, verb = "wasRecruitedBy" )public class Member_wasRecruitedBy_Recruiter extends Relation<Member, Recruiter> { ...}

recruitedwasRecruitedBy

bi-directional relation concept

Recruiter

MemberisConnectedToconcept

Member

Hands on Kasper componentsbi-directional relation

Hands on Kasper componentscomponent relation (unidirectional)

Status CreatorcreatedBy

concept conceptcomponent relation

(LinkedConcept<Member>)

@XKasperConcept( domain = Dashboard.class ) public class Creator extends Concept { /* Member avatar */ }

@XKasperConcept( domain = Dahsboard.class ) public class Status extends Concept {

private LinkedConcept<Creator> createdBy;

...

}

Hands on Kasper componentsevent listener

@XKasperEventListener( domain = HelloDomain.class, description = "Notice the world for each created Hello message")public class NoticeTheWorldAboutCreatedHelloListener extends CommandEventListener<HelloCreatedEvent> {

@Override public void handle(final HelloCreatedEvent event) { final String response = String.format( "Hi all, %s received an hello message", event.getForBuddy() );

this.getCommandGateway().get().sendCommand( new NoticeTheWorldCommand(response), /* forward the current context */ CurrentContext.value().get() ); }

}

Mandatory annotationfor domain sticking

Can be a CommandEventListener or a QueryEventListener

The event class to be listened

The handling method to be overriden

Call a command

Viadeo - The Kasper way

Software