Main Memory Databases Hadi Amiri, Abolfazle AleAhmad Department of ECE, University of Tehran.

Main Memory DatabasesMain Memory Databases

Hadi Amiri, Abolfazle AleAhmadHadi Amiri, Abolfazle AleAhmad

Department of ECE, University of TehranDepartment of ECE, University of Tehran

Top Main Memory DBsTop Main Memory DBsFastDBFastDBMonetDBMonetDBeXtremeDBeXtremeDBERDBERDBDataBlitzDataBlitzTimes TenTimes Ten

MonetDBMonetDB

MonetDB is an open source high-MonetDB is an open source high-performance database system performance database system developed at developed at CWICWI, (the Institute for , (the Institute for Mathematics andMathematics and Computer Science Computer Science Research of The Netherlands)Research of The Netherlands)

MonetDB has been successfully MonetDB has been successfully applied in high-performance applied in high-performance applications for data mining, OLAP, applications for data mining, OLAP, GIS, XML Query, text and multimedia GIS, XML Query, text and multimedia retrieval. (Real Time App)retrieval. (Real Time App)

DBMS ArchitectureDBMS Architecture

Architecture

MonetDB ArchitectureMonetDB Architecture

Architecture

MonetDB: architecture

Front-end/back-end:

• support multiple data models

• support multiple end-user languages

• support diverse application domains

perform all operations in main memory perform all operations in main memory a modern CPU-tuned vectorized query execution a modern CPU-tuned vectorized query execution

architecturearchitecture that often gives MonetDB a more than 10-fold raw that often gives MonetDB a more than 10-fold raw

speed advantage on the same algorithm over a speed advantage on the same algorithm over a typical interpreter-based RDBMStypical interpreter-based RDBMS

extensible algebra. extensible algebra. binary relation model binary relation model shared-memory parallelism shared-memory parallelism MonetDB is one of the first database systems to MonetDB is one of the first database systems to

focus its query optimization effort on exploiting focus its query optimization effort on exploiting CPU cachesCPU caches

MonetDB PropertiesMonetDB Properties

MonetDB has played a pivotal role in the MonetDB has played a pivotal role in the development of datamining applications development of datamining applications at DataDistilleries (now SPSS). at DataDistilleries (now SPSS).

Monet IdeasMonet Ideas

: دستورات از استفاده امکان اول : ايده دستورات از استفاده امکان اول ايدهBack-EndBack-End (( براي را جوهايي و پرس توانيد مي شما که معني اين براي به را جوهايي و پرس توانيد مي شما که معني اين واسطي Back-EndBack-End به از استفاده واسطي با از استفاده با

نام نام به کند. Mapi ClientMapi Client به مي کار آن با واسط اين که زباني کند. بنويسيد مي کار آن با واسط اين که زباني دارد MILMIL بنويسيد دارد نام (( نام : مي کاري هر دودويي جداول با دوم : ايده مي کاري هر دودويي جداول با دوم ايده

کرد کرد توان توان : دوباره اختراع به نياز عدم سوم : ايده دوباره اختراع به نياز عدم سوم ايده

عامل عامل سيستم سيستم : پرسو اجراي سازي بهينه چهارم : ايده پرسو اجراي سازي بهينه چهارم ايده

حافظه در حافظه جو در جو

Monet Data ModelMonet Data Model

Some simple valuesSome simple values Int,chr,str,oid,nil,intint(nil),chrchr(nil),Int,chr,str,oid,nil,intint(nil),chrchr(nil),

strstr(nil),oidoid(nil)strstr(nil),oidoid(nil) Binary Association Tables (BATs)Binary Association Tables (BATs)

BAT BAT ستون دو با است ستون جدولي دو با است که که Head Head وو Tail Tail جدولييا و فوق ساده مقادير يا شامل و فوق ساده مقادير باشد BAT BAT شامل باشد مي مي

Monet Data ModelMonet Data Model

BAT[oid,str]

BAT[oid,int]

BAT[oid, BAT[oid,int]]

Storing Relations in MonetDBStoring Relations in MonetDB

Relational MappingRelational Mapping

Object-Oriented MappingObject-Oriented Mapping

Monet PerformanceMonet Performance Compare some well-known open-source DBMSs with Monet on the Compare some well-known open-source DBMSs with Monet on the

task to build a binary table with the head representing an ordered task to build a binary table with the head representing an ordered sequence and the tail a permutation over the values in the head sequence and the tail a permutation over the values in the head

Response time is given in seconds while running on a dual Athlon 1400 with 1GB of RAM and 3.5GB of free space

-- DBtapestry Version=1.1 -- See http://monetdb.cwi.nl/DBtapestry/ -- (c) CWI -- DBtapestry Version=1.1 -- See http://monetdb.cwi.nl/DBtapestry/ -- (c) CWI 2004-2005 - rows=100K --columns=2 -- Produced Sat Feb 26 23:41:51 20052004-2005 - rows=100K --columns=2 -- Produced Sat Feb 26 23:41:51 2005

start transaction; start transaction; select now(); select now(); create table RKA( head int, tail int); create table RKA( head int, tail int); create table RKB( head int, tail int); create table RKB( head int, tail int); insert into RKA values(0,0); insert into RKA values(0,0); insert into RKA values(1,360); insert into RKA values(1,360); insert into RKA values(2,427); insert into RKA values(2,427); …… …… insert into RKB select head+0, tail+0 from RKA; insert into RKB select head+0, tail+0 from RKA; insert into RKB select head+8192, tail+8192 from RKA; insert into RKB select head+8192, tail+8192 from RKA; …… …… drop table RKA; drop table RKA; update RKB set tail=(tail*47) % 102400; update RKB set tail=(tail*47) % 102400; …………create table tapestry( attr0 int , attr1 int); create table tapestry( attr0 int , attr1 int); insert into tapestry select R0.head, R0.tail from RKB R0; insert into tapestry select R0.head, R0.tail from RKB R0; drop table RKB; drop table RKB; select now(); select now(); commit; commit; -- actions=1249 -- tuplesRead= 1.07M tuplesWritten= 2.07M -- volumeRead= -- actions=1249 -- tuplesRead= 1.07M tuplesWritten= 2.07M -- volumeRead=

8.59M volumeWritten=16.59M 8.59M volumeWritten=16.59M

Monet Interpreter LanguageMonet Interpreter Language

MIL ExtensibilityMIL Extensibility

What is MonetDB Good for?What is MonetDB Good for?

Query-intensive applicationQuery-intensive application very high performance demandingvery high performance demanding complex data modelscomplex data models complex query primitivescomplex query primitives

Transactions (OLTP)Transactions (OLTP)OLTPOLTP ( (OOnnlline ine TTransaction ransaction PProcessing) is a form of transaction processing conducted via computer rocessing) is a form of transaction processing conducted via computer network. Some applications of OLTP include electronic banking, order processing, employee time network. Some applications of OLTP include electronic banking, order processing, employee time

clock systems, e-commerce, and clock systems, e-commerce, and eTradingeTrading. .

OLAP, Data MiningOLAP, Data MiningOn Line Analytical ProcessingOn Line Analytical Processing. (. (OLAP)OLAP) It is an approach to quickly provide the answer to It is an approach to quickly provide the answer to

complex analytical queries. The typical applications of OLAP are in business reporting for sales, complex analytical queries. The typical applications of OLAP are in business reporting for sales, marketing, management reporting, business performance management, budgeting and marketing, management reporting, business performance management, budgeting and

forecasting, financial reporting and similar areas. forecasting, financial reporting and similar areas.

New Domains: GISNew Domains: GIS

New data types (point, polygon, New data types (point, polygon, etc..)etc..)

New search accelerators (R-Tree, New search accelerators (R-Tree, etc..)etc..)

New primitives New primitives boolean intersects(polygon,polygon)boolean intersects(polygon,polygon)

Complex topological structures Complex topological structures stored in DCELs that are decomposed over stored in DCELs that are decomposed over BATS queries are efficient due to MonetDB BATS queries are efficient due to MonetDB high join speedhigh join speed

New Domains: MultimediaNew Domains: Multimedia

New data types (url, image, etc..)New data types (url, image, etc..) new search accelerators (color new search accelerators (color

histograms)histograms) new primitives (similarity search)new primitives (similarity search) complex data structures: complex data structures:

bayesian inference networks (information bayesian inference networks (information retrieval) again decomposed in BATs and retrieval) again decomposed in BATs and efficient to queryefficient to query