Top Banner
www.emicnetworks.com © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005 Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet Principal architect - Emic Networks Chief architect – ObjectWeb consortium
78

© Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005 Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

Dec 30, 2015

Download

Documents

Ashlynn Greer
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Building Highly Available Database Applications for

Apache Derby

Emmanuel Cecchet

Principal architect - Emic Networks

Chief architect – ObjectWeb consortium

Page 2: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Internet

• Database tier should be

– scalable– highly available– without modifying the client application– database vendor independent– on commodity hardware

Motivations

JDBC

JDBCJDBC

Page 3: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Scaling the database tier – Master-slave replication

Internet

Web frontend

App. server

• Cons– failover time/data loss on master failure– read inconsistencies– scalability

Master

Page 4: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Internet

• Cons– atomic broadcast scalability– no client side load balancing– heavy modifications of the database engine

Scaling the database tier – Atomic broadcast

Atomicbroadcast

Page 5: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Scaling the database tier – SMP

Internet

Web frontend

App. server

Well-known database

vendor here

Database

Well-known hardware +database vendors here

• Cons

– Cost– Scalability limit

Page 6: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Scaling the database tier – Shared disks

Internet

Web frontend

App. server Database Disks

Another well-knowndatabase vendor

• Cons– still expensive hardware– availability

Page 7: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Outline

• RAIDb

• C-JDBC

• Derby and C-JDBC

• Scalability

• High availability

Page 8: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

RAIDb concept

• Redundant Array of Inexpensive Databases

• RAIDb controller– gives the view of a single database to the client– balance the load on the database backends

• RAIDb levels offers various tradeoff of performance and fault tolerance

Page 9: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

RAIDb levels

• RAIDb-0– partitioning– no duplication and no fault tolerance– at least 2 nodes

table 2 & 3 table ...

RAIDb controller

table n-1table 1 table n

SQL requests

Page 10: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

RAIDb levels

• RAIDb-1– mirroring– performance bounded by write broadcast– at least 2 nodes

Full DB Full DB

RAIDb controller

Full DBFull DB Full DB

SQL requests

Page 11: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

RAIDb levels

• RAIDb-2– partial replication– at least 2 copies of each table for fault tolerance– at least 3 nodes

table x table y

RAIDb controller

table x & yFull DB table z

SQL requests

Page 12: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Outline

• RAIDb

• C-JDBC

• Derby and C-JDBC

• Scalability

• High availability

Page 13: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

C-JDBC overview

• Middleware implementing RAIDb– 100% Java implementation– open source (LGPL)

• Two components– generic JDBC driver (C-JDBC driver)– C-JDBC Controller

• Read-one, Write all approach– provides eager (strong) consistency

• Supports heterogeneous databases

Page 14: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

architectural overview

JVM

Applicationserver

C-JDBC JDBC driver

C-JDBCcontroller

JVM

DB2 Univ.JDBC Driver

Derby with Network

server

Page 15: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Using C-JDBC as an open source driver for Derby

JVM

Applicationserver

C-JDBC JDBC driver

C-JDBC controller

JVM

Embedded Derby

Page 16: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Inside the C-JDBC Controller

C-JDBC Controller

Derby

C-JDBC driver

Derby

Virtual database 1

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Query result cache

Scheduler

Load balancer

Derby JDBC driver

Derby JDBC driver

Recovery Log

Authentication Manager

Derby

Database Backend

Connection Manager

Derby JDBC driver

Virtual database 2

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Query result cache

Scheduler

Load balancer

Recovery Log

Authentication Manager

Oracle JDBC driver

Oracle

JMX Server

Monitoring

JMX administration console

Derby

Derby JDBC driver

C-JDBC driver

Client application (Servlet, EJB, ...)

C-JDBC driver

HTTP RMI ...

Client application (Servlet, EJB, ...)

Client application (Servlet, EJB, ...)

Checkpointing service

Database dumps management

Configuration

Page 17: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Derby

C-JDBC driver

Java client program

(Servlet, EJB, ...)

Derby

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

Derby JDBC driver

Derby JDBC driver

Recovery Log

Authentication Manager

Derby

Database Backend

Connection Manager

Derby JDBC driver

Virtual Database

• gives the view of a single database

• establishes the mapping between the database name used by the application and the backend specific settings

• backends can be added and removed dynamically

• configured using an XML configuration file

Page 18: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Derby

C-JDBC driver

Java client program

(Servlet, EJB, ...)

Derby

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

Derby JDBC driver

Derby JDBC driver

Recovery Log

Authentication Manager

Derby

Database Backend

Connection Manager

Derby JDBC driver

Authentication Manager

• Matches real login/password used by the application with backend specific login/ password

• Administrator login to manage the virtual database

Page 19: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Derby

C-JDBC driver

Java client program

(Servlet, EJB, ...)

Derby

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

Derby JDBC driver

Derby JDBC driver

Recovery Log

Authentication Manager

Derby

Database Backend

Connection Manager

Derby JDBC driver

Scheduler

• Manages concurrency control

• Specific implementations for RAIDb 0, 1 and 2

• Pass-through

• Optimistic and pessimistic transaction level– uses the database schema that is

automatically fetched from backends

Page 20: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Derby

C-JDBC driver

Java client program

(Servlet, EJB, ...)

Derby

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

Derby JDBC driver

Derby JDBC driver

Recovery Log

Authentication Manager

Derby

Database Backend

Connection Manager

Derby JDBC driver

Request cache• 3 optional caches

– tunable sizes

• parsing cache

– parse request skeleton only once– INSERT INTO t VALUES (?,?,?)

• metadata cache

– column metadata– fields of a request

• result cache

– caches results from SQL requests– tunable consistency– fine grain invalidations– optimizations for findByPk requests

Page 21: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Derby

C-JDBC driver

Java client program

(Servlet, EJB, ...)

Derby

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

Derby JDBC driver

Derby JDBC driver

Recovery Log

Authentication Manager

Derby

Database Backend

Connection Manager

Derby JDBC driver

Load balancer 1/2

• RAIDb-0

– query directed to the backend having the needed tables

• RAIDb-1

– read executed by current thread– write executed in parallel by a dedicated

thread per backend– result returned if one, majority or all commit– if one node fails but others succeed, failing

node is disabled

• RAIDb-2

– same as RAIDb-1 except that writes are sent only to nodes owning the updated table

Page 22: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Derby

C-JDBC driver

Java client program

(Servlet, EJB, ...)

Derby

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

Derby JDBC driver

Derby JDBC driver

Recovery Log

Authentication Manager

Derby

Database Backend

Connection Manager

Derby JDBC driver

Load balancer 2/2

• Static load balancing policies– Round-Robin (RR)– Weighted Round-Robin (WRR)

• Least Pending Requests First (LPRF)– request sent to the node that has the

shortest pending request queue– efficient even if backends do not have

homogeneous performance

Page 23: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Derby

C-JDBC driver

Java client program

(Servlet, EJB, ...)

Derby

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

Derby JDBC driver

Derby JDBC driver

Recovery Log

Authentication Manager

Derby

Database Backend

Connection Manager

Derby JDBC driver

Connection Manager

• C-JDBC driver provides transparent connection pooling

• Connection pooling for a backend– no pooling– blocking pool– non-blocking pool– dynamic pool

• Connection pools defined on a per login basis– resource management per login– dedicated connections for admin

Page 24: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Derby

C-JDBC driver

Java client program

(Servlet, EJB, ...)

Derby

Virtual database

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Request Cache

Scheduler

Load balancer

Derby JDBC driver

Derby JDBC driver

Recovery Log

Authentication Manager

Derby

Database Backend

Connection Manager

Derby JDBC driver

Recovery Log

• Checkpoints are associated with database dumps

• Record all updates and transaction markers since a checkpoint

• Used to resynchronize a database from a checkpoint

• JDBCRecoveryLog– store log information in a database– can be re-injected in a C-JDBC cluster

for fault tolerance

Page 25: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

C-JDBC Controller

Derby

C-JDBC driver

Derby

Virtual database 1

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Query result cache

Scheduler

Load balancer

Derby JDBC driver

Derby JDBC driver

Recovery Log

Authentication Manager

Derby

Database Backend

Connection Manager

Derby JDBC driver

C-JDBC driver

Client application (Servlet, EJB, ...)

Client application (Servlet, EJB, ...)Functional overview (read)

connect myDBconnect login, passwordexecute SELECT * FROM t

Page 26: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

C-JDBC Controller

Derby

C-JDBC driver

Derby

Virtual database 1

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Query result cache

Scheduler

Load balancer

Derby JDBC driver

Derby JDBC driver

Recovery Log

Authentication Manager

Derby

Database Backend

Connection Manager

Derby JDBC driver

C-JDBC driver

Client application (Servlet, EJB, ...)

Client application (Servlet, EJB, ...)Functional overview (write)

execute INSERT INTO t …

Page 27: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

C-JDBC Controller

Derby

C-JDBC driver

Derby

Virtual database 1

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Query result cache

Scheduler

Load balancer

Derby JDBC driver

Derby JDBC driver

Recovery Log

Authentication Manager

Derby

Database Backend

Connection Manager

Derby JDBC driver

C-JDBC driver

Client application (Servlet, EJB, ...)

Client application (Servlet, EJB, ...)

Failures

• No 2 phase-commit

– parallel transactions– failed nodes are

automatically disabled

execute INSERT INTO t …

Page 28: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Outline

• RAIDb

• C-JDBC

• Derby and C-JDBC

• Scalability

• High availability

Page 29: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Internet

• Apache clustering

– L4 switch, RR-DNS, One-IP techniques, LVS, …

• Tomcat clustering

– mod_jk (T4), mod_proxy/mod_rewrite (T5), session replication

• Database clustering

– C-JDBC

Highly available web site

mod-jkRR-DNS

Open source driverParsing cacheResult cache

Metadata cache

embedded

Page 30: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Result cache• Cache contains a list of SQL→ResultSet

• Policy defined by queryPattern→Policy

• 3 policies

– EagerCaching: variable granularities for invalidations– RelaxedCaching: invalidations based on timeout– NoCaching: never cached

RUBiS bidding mix

with 450 clients

No cache Coherent

cache

Relaxed

cache

Throughput (rq/min) 3892 4184 4215

Avg response time 801 ms 284 ms 134 ms

Database CPU load 100% 85% 20%

C-JDBC CPU load - 15% 7%

Page 31: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Configuring C-JDBC as a Derby driver (1/3)

• copy c-jdbc-driver.jar in client application classpathClass.forName(“org.objectweb.cjdbc.driver.Driver”);

Connection c = DriverManager.getConnection(

“jdbc:cjdbc://host/db”, “login”, “password”);

• copy derby.jar in $CJDBC_HOME/drivers

C-JDBC ControllerSingleDB configuration

Client application

org.objectweb.cjdbc.driver.Driver

C-JDBC DriverEmbedded Derby

jdbc:cjdbc://host/db

jdbc:derby:path;create=true

Page 32: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Configuring C-JDBC as a Derby driver (2/3)<?xml version="1.0" encoding="UTF8"?>

<!DOCTYPE C-JDBC PUBLIC "-//ObjectWeb//DTD C-JDBC 1.1//EN" …>

<C-JDBC>

<VirtualDatabase name="xpetstore">

<AuthenticationManager>

<Admin> <User username="admin" password=""/> </Admin>

<VirtualUsers> <VirtualLogin vLogin="user" vPassword=“x"/> </VirtualUsers>

</AuthenticationManager>

<DatabaseBackend name="derby“

driver="org.apache.derby.jdbc.EmbeddedDriver“

url="jdbc:derby:c:/xpetstore;create=true"

connectionTestStatement="values 1">

<ConnectionManager vLogin="user" rLogin="APP" rPassword="APP">

<VariablePoolConnectionManager initPoolSize="1" maxPoolSize="50"/>

</ConnectionManager>

</DatabaseBackend>

Page 33: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Configuring C-JDBC as a Derby driver (3/3)<RequestManager>

<RequestScheduler>

<SingleDBScheduler level=“passThrough"/>

</RequestScheduler>

<RequestCache>

<MetadataCache/>

<ParsingCache/>

<ResultCache granularity="table"/>

</RequestCache>

<LoadBalancer>

<SingleDB/>

</LoadBalancer>

</RequestManager>

</VirtualDatabase>

</C-JDBC>

Page 34: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Internet

• Multiple databases– choosing RAIDb level– recovery log for

• adding nodes dynamically • recovering from failures

Highly available web site

Page 35: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Derby clustering with C-JDBC

C-JDBC Controller

RAIDb configuration

Client application

org.objectweb.cjdbc.driver.Driver

C-JDBC Driver

jdbc:cjdbc://host1/db

Client application

org.objectweb.cjdbc.driver.Driver

C-JDBC Driver

Derby

Network Server

org.objectweb.cjdbc.driver.Driver

C-JDBC Drivercom.ibm.db2.jcc.DB2Driver

DB2 Universal Driver for Derby

C-JDBC ControllerSingleDB configuration

Embedded Derby

jdbc:derby:path;create=true

jdbc:cjdbc://host3/dbjdbc:derby:net://host2:1527/db;create=true;retrieveMessagesFromServerOnGetMessage=true;

Page 36: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Configuring C-JDBC with Derby Network server

• Virtual database configuration file

<DatabaseBackend name=“derby1”

driver=“com.ibm.db2.jcc.DB2Driver”

url= “jdbc:derby:net://localhost:1527/xpetstore;create=true;retrieveMessagesFromServerOnGetMessage=true;”

connectionTestStatement="values 1"> <ConnectionManager …/>

</DatabaseBackend>

Page 37: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Configuring C-JDBC with Derby/C-JDBC

• Virtual database configuration file

<DatabaseBackend name=“derby2”

driver=“org.objectweb.cjdbc.driver.Driver”

url=“jdbc:cjdbc://host/xpetstore”

connectionTestStatement=“values 1"> <ConnectionManager …/>

</DatabaseBackend>

Page 38: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Configuring C-JDBC Clustering with Derby (1/2) <RequestManager>

<RequestScheduler>

<RAIDb-1Scheduler level=“passThrough"/>

</RequestScheduler>

<RequestCache>

<MetadataCache/>

<ParsingCache/>

<ResultCache granularity="table" />

</RequestCache>

<LoadBalancer>

<RAIDb-1>

<RAIDb-1-LeastPendingRequestFirst/>

</RAIDb-1>

</LoadBalancer>

Page 39: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Configuring C-JDBC Clustering with Derby (2/2)<RecoveryLog>

<JDBCRecoveryLog

driver="com.ibm.db2.jcc.DB2Driver“

url="jdbc:derby:net://localhost:1529/xpetstore;create=true;retrieveMessagesFromServerOnGetMessage=true;"

login="APP" password="APP">

<RecoveryLogTable tableName="RECOVERY"

idColumnType="BIGINT NOT NULL" sqlColumnName="sqlStmt“

sqlColumnType="VARCHAR(8192) NOT NULL“

extraStatementDefinition=",PRIMARY KEY (id)"/>

<CheckpointTable tableName="CHECKPOINT"/>

<BackendTable tableName="BACKENDTABLE"/>

<DumpaTable tableName=“DUMPTABLE”/>

</JDBCRecoveryLog>

</RecoveryLog>

</RequestManager>

</VirtualDatabase>

Page 40: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Internet

<VirtualDatabase name="myDB">

<Distribution/>…

Controller replication

Page 41: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Outline

• RAIDb

• C-JDBC

• Derby and C-JDBC

• Scalability

• High availability

Page 42: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

C-JDBC vertical scalability

• allows nested RAIDb levels

• allows tree architecture for scalable write broadcast

• necessary with large number of backends

• C-JDBC driver re-injected in C-JDBC controller

Page 43: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

DB 4DB 3

DB native JDBC driver

DB 5

C-JDBC driver

DB 1 DB 2

DB native JDBC driver

DB 6

DB native JDBC driver

DB7

C-JDBC controller Partial replication

C-JDBC controller Full replication

C-JDBC controller Full replication

C-JDBC controller Full replication

C-JDBC driverJVM

Client program C-JDBC

driver

JVM

Client program

C-JDBC driver

JVM

Client program

Vertical scalability

• Addresses JVM scalability issues

• Distributing large number of connections on many backends

Page 44: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

TPC-W benchmark (Amazon.com)

• Nearly linear speedups with the shopping mix

0

200

400

600

800

1000

1200

1400

1600

0 2 4 6

Number of nodes

Th

rou

gh

pu

t in

req

uests

per

min

ute

Single Database

Full replication

Partial replication

Page 45: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Outline

• RAIDb

• C-JDBC

• Derby and C-JDBC

• Scalability

• High availability

Page 46: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Derby PostgreSQL

C-JDBC Controller

Derby JDBC driver

PostgreSQL JDBC driver

C-JDBC driver

JVM

Java client program

C-JDBC driver

JVM

Java client program

C-JDBC driver

JVM

Java client program

C-JDBC driver

JVM

Java client program

C-JDBC Controller

JGroups

DerbyPostgreSQL

Derby JDBC driver

PostgreSQL JDBC driver

Controller replication

• Prevent the controller from being a single point of failure

• Group communication for controller synchronization

• C-JDBC driver supports multiple controllers with automatic failover

jdbc:c-jdbc://node1:25322,node2:12345/myDB

Page 47: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

C-JDBC Controller

Distributed Request Manager

C-JDBC Controller

Distributed Request Manager

Derby

C-JDBC driver

Derby

Virtual database 1

Database Backend

Connection Manager

Database Backend

Connection Manager

Derby JDBC driver

Derby JDBC driver

Virtual database 2

Database Backend

Connection Manager

Database Backend

Connection Manager

Request Manager

Query result cache

Scheduler

Load balancer

Authentication Manager

Oracle JDBC driver

OracleDerby

Derby JDBC driver

C-JDBC driver

Client application (Servlet, EJB, ...)

C-JDBC driver

Client application (Servlet, EJB, ...)

Client application (Servlet, EJB, ...)

Request Manager

Query result cache

Scheduler

Load balancer

Authentication Manager

jdbc:cjdbc://node1,node2/myDB

Total order reliable multicast

Controller replication

Page 48: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Mixing horizontal & vertical scalability

DB 6DB 5

DB native JDBC driver

DB 7

C-JDBC driver

DB 1 DB 2

DB native JDBC driver

DB 3

DB native JDBC driver

DB 4

C-JDBC controller Full replication

C-JDBC controller Full replication

C-JDBC controller Full replication

C-JDBC controller Full replication

C-JDBC driverJVM

Client program

C-JDBC driver

JVM

Client program

C-JDBC driver

JVM

Client program

C-JDBC driver

C-JDBC driver

Page 49: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Derby

C-JDBC Controller

Derby JDBC driver

C-JDBC driver

JVM

EJB Container JOnAS, WebLogic,

JBoss, WebSphere, ...

disabled

dump for initial

checkpoint

Recovery Log

disabled

Building initial checkpoint

• Dump initial Derby database using any tools (tar, zip, …)

• Initial checkpoint inserted in RecoveryLog

Page 50: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Derby

C-JDBC Controller

Derby JDBC driver

C-JDBC driver

JVM

EJB Container JOnAS, WebLogic,

JBoss, WebSphere, ...

enabled

dump for initial

checkpoint

Recovery Log

JDBC Recovery Log

enabled

Logging

• Backend is enabled

• All database updates are logged (SQL statement, user, transaction, …)

Page 51: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Derby

C-JDBC Controller

Derby JDBC driver

C-JDBC driver

JVM

EJB Container JOnAS, WebLogic,

JBoss, WebSphere, ...

dump for initial

checkpoint

Recovery Log

Derby Derbydisabled disabledenabled

enabled

JDBC Recovery Log

Adding new backends 1/3

• Add new backends while system online

• Restore dump corresponding to initial checkpoint

Page 52: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Derby

C-JDBC Controller

Derby JDBC driver

C-JDBC driver

JVM

EJB Container JOnAS, WebLogic,

JBoss, WebSphere, ...

dump for initial

checkpoint

Recovery Log

Derby Derbyenabled

enabled

JDBC Recovery Log

disableddisabled

Adding new backends 2/3

• Replay updates from the log

Page 53: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Derby

C-JDBC Controller

Derby JDBC driver

C-JDBC driver

JVM

EJB Container JOnAS, WebLogic,

JBoss, WebSphere, ...

dump for initial

checkpoint

Recovery Log

Derby Derbyenabled

enabled

JDBC Recovery Log

enabledenabled

Adding new backends 3/3

• Enable backends when done

Page 54: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

...

dump for last

checkpoint

Derby

C-JDBC Controller

Derby JDBC driver

C-JDBC driver

JVM

EJB Container JOnAS, WebLogic,

JBoss, WebSphere, ...

dump for last

checkpoint

Recovery Log

Derby Derbyenabled

enabled

JDBC Recovery Log

enableddisabled

dump for initial

checkpoint

Making new checkpoints (1/3)

• Disable one backend to have a coherent snapshot

• Mark the new checkpoint entry in the log

• Dump withtar/zip

Page 55: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Making new checkpoints (2/3)

• Replay missing updates from log

dump for

checkpoint

...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint

Derby

C-JDBC Controller

Derby JDBC driver

C-JDBC driver

JVM

EJB Container JOnAS, WebLogic,

JBoss, WebSphere, ...

Recovery Log

Derby Derbyenabled

enabled

JDBC Recovery Log

disabled enabled

Page 56: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Derby

C-JDBC Controller

Derby JDBC driver

C-JDBC driver

JVM

EJB Container JOnAS, WebLogic,

JBoss, WebSphere, ...

dump for

checkpoint

Recovery Log

Derby Derbyenabled

enabled

JDBC Recovery Log

enabledenabled

...

dump for last

checkpoint

dump for last

checkpoint

dump for initial

checkpoint

Making new checkpoints (3/3)

• Re-enable backend when done

Page 57: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Demo 2 – xPetstore/C-JDBC/Derby

backend1

backend2

JMX

Recovery log

RAIDb-1

xpetstore Servlet

C-JDBC administration console

embedded

embedded

Page 58: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

C-JDBC today

• Web site– ~200.000 hits/month– ~ 44.000 downloads

• Community– 27 committers both industrial & academics– [email protected]: ~300 subscribers, 200-300

msgs/month– translation in japanese, italian, german, chinese, turkish,

french

• RPM on JPackage.org

Page 59: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Current limitations

• JDBC only

• Distributed joins

• Out parameters for stored procedures

• Some JDBC 3.0 extensions

• XA support through XAPool only

• network partition/reconciliation not supported

Page 60: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Conclusion

• RAIDb– RAID-like scheme for databases

• C-JDBC– open source middleware for database replication– performance scalability– high availability

• Derby & C-JDBC– open source driver for Derby– high availability solution for Derby

Page 61: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Q&A_________

Thanks to all users and contributors ...

http://c-jdbc.objectweb.org

Page 62: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Bonus slides

Page 63: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

HORIZONTAL SCALABILITY

Page 64: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Horizontal scalability

• JGroups for controller synchronization

• Groups messages for writes only

DB1 DB2 DB3 DB4

Controller1

DB1 DB2 DB3 DB4

Controller1 Controller2

Client 1 Client... Client n Client 1 Client... Client n

Page 65: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Horizontal scalability• Centralized write approach issues

• Issues with transactions assigned to connections

DB1 DB2 DB3 DB4

Controller1 Controller2

commit

DB1 DB2 DB3 DB4

Controller1 Controller2

commit

commit commit

DB1 DB2 DB3 DB4

Controller2

? ? ? ?

DB1 DB2

Controller1 Controller2

DB1 DB2

Controller1 Controller2

T1(Read)

T1(R)

DB1 DB2

Controller1 Controller2

T1(Write)

T1(R)T1(W) T2(W)

Page 66: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Horizontal scalability

• General case for a write query

– 3 multicast + 2n unicastDistributedVirtualDatabase@controller2

DistributedRequestManager

DistributedVirtualDatabase@controller3

DistributedVirtualDatabase@controller1

DistributedRequestManager

DistributedRequestManager

execWriteRequest(AbstractWriteRequest)

Scheduler

Write query handler

Load Balancer

Scheduler

Load Balancer

Distributed scheduling handler

Wait for schedulers replies

Send request to all controllers

Send execute request order to all controllers

Wait for load balancer replies

Distributed load balancer handler

Scheduler

Load Balancer

Distributed scheduling handler

Distributed load balancer handler

Notify controllers of completion success or not

Distributed scheduling handler

Distributed load balancer handler

Notify scheduler of completion

Return result

return result

Notify scheduler of completion

Notify scheduler of completion

Page 67: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Horizontal scalability• Solution: No backend sharing

– 1 multicast + n unicast [+ 1 multicast]

DB1 DB2 DB3 DB4 DB5 DB6

Controller1 Controller2 Controller3

DistributedVirtualDatabase@controller2

DistributedVirtualDatabase@controller3

DistributedVirtualDatabase@controller1

DistributedRequestManager

DistributedRequestManager

execWriteRequest(AbstractWriteRequest)DistributedRequestManager

Scheduler

Write query handler

Load Balancer

Scheduler

Load Balancer

Distributed scheduling handler

Send request to all controllers

Wait for load balancer replies

Distributed load balancer handler

Scheduler

Load Balancer

Distributed scheduling handler

Distributed load balancer handler

Notify controller(s) that have failed if other have succeeded

Distributed scheduling handler

Distributed load balancer handler

Wait if all backends failed

Return result

return result

Wait if all backends failed

Wait if all backends failed

Page 68: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Horizontal scalability

• Issues with JGroups

– resources needed by a channel– instability of throughput with UDP– performance scalability

• TCP better than UDP but

– unable to disable reliability on top of TCP– unable to disable garbage collection– ordering implementation is sub-optimal

• Need for a new group communication layer optimized for cluster

Page 69: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Horizontal scalability• JGroups performance on UDP/FastEthernet

Page 70: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

USE CASES

Page 71: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Budget High Availability

• High availability infrastructure “on a budget”

• Typical eCommercesetup

• http://www.budget-ha.com

Page 72: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

OpenUSS: University Support System

• eLearning

• High availability

• Portability

– Linux, HP-UX, Windows

– InterBase, Firebird, PostgreSQL, HypersonicSQL

• http://openuss.sourceforge.net

Firebird

C-JDBC Controller Full replication

Firebird JDBC driver

Firebird

C-JDBC driver

JVM

C-JDBC driver

JVM

C-JDBC Controller Full replication

Firebird JDBC driver

C-JDBC driver

JVM

JOnAS J2EE server

C-JDBC driver

JVM

Enhydra server

OpenUSS OpenUSS OpenUSS OpenUSS

Enhydra Director

Apache

JOnAS J2EE server

Enhydra server

JOnAS J2EE server

Enhydra server

JOnAS J2EE server

Enhydra server

Enhydra Director

Apache

Enhydra Director

Apache

Page 73: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

Flood alert system• Disaster recovery

• Independent nodes synchronized with C-JDBC

• VPN for security issues

• http://floodalert.org

MySQL

C-JDBC Controller Full replication

MySQL JDBC driver

C-JDBC driver

JBoss J2EE server

Flood alert system

MySQL

C-JDBC Controller Full replication

MySQL JDBC driver

C-JDBC driver

JBoss J2EE server

Flood alert system

MySQL

C-JDBC Controller Full replication

MySQL JDBC driver

C-JDBC driver

JBoss J2EE server

Flood alert system

> 1000 miles

Page 74: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

J2EE benchmarking

• Large scaleJ2EE clusters

• http://jmob.objectweb.org

Internet

emulated users

Page 75: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

PERFORMANCE

Page 76: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

TPC-W• Browsing mix performance

Page 77: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

TPC-W• Shopping mix performance

Page 78: © Emic 2005 Emmanuel Cecchet | ApacheCon EU 2005  Building Highly Available Database Applications for Apache Derby Emmanuel Cecchet.

www.emicnetworks.com© Emic 2005 Emmanuel Cecchet | ApacheCon EU

2005

TPC-W• Ordering mix performance