Databases in BIG Data Applications...Organizing Big Data •Database: A database is an organized collection of data, generally stored and accessed electronically from a computer system.

Databases in BIG Data--From an application perspective

9/28/2019 1

Dimensions of Big Data

9/28/2019 2Picture from https://www.techentice.com/the-data-veracity-big-data/

Organizing Big Data

• Database: A database is an organized collection of data, generally stored and accessed electronically from a computer system.

• DBMS: The database management system (DBMS) is the software that interacts with end users, applications, and the database itself to capture and analyze the data.

• Data ecosystem: A data ecosystem is a collection of infrastructure, analytics, and applications used to capture and analyze data. Data ecosystems provide companies with data that they rely on to understand their customers and to make better pricing, operations, and marketing decisions.

9/28/2019 3Definitions from https://www.wikipedia.org/

Different Databases (ecosystem)

9/28/2019 4

MySQL (Relational Database)

9/28/2019 5Pictures from MangoDB https://www.youtube.com/watch?v=EE8ZTQxa0AM












MongoDB


MongoDB


Application

• Lots of data• Copies of messages, reverse indices of messages, per user data.

• Many incoming requests resulting in a lot of random reads and random writes.

9/28/2019 14

Cassandra


• Many incoming requests resulting in a lot of random reads and random writes.

9/28/2019 15

Cassandra

9/28/2019 16Pictures from https://www.guru99.com/cassandra-tutorial.html

Cassandra

9/28/2019 17Adopted from slides by By Perry Hoekstra, Jiaheng Lu, Avinash Lakshman, Prashant Malik, and Jimmy Lin

Cassandra

9/28/2019 18Adopted from slides by By Perry Hoekstra, Jiaheng Lu, Avinash Lakshman, Prashant Malik, and Jimmy Lin

Cassandra

Adopted from slides by By Perry Hoekstra, Jiaheng Lu, Avinash Lakshman, Prashant Malik, and Jimmy Lin

Cassandra


• Many incoming and time sensitive requests resulting in a lot of random reads and random writes.

9/28/2019 20

Some applications

• Lots of Lots of data• Copies of messages, reverse indices of messages, per user data.

• Many many incoming data and time in-sensitive requests resulting in a lot of random reads and random writes.

9/28/2019 21

Hadoop

• Lots of Lots of data• Copies of messages, reverse indices of messages, per user data.

• Many many incoming data and time in-sensitive requests resulting in a lot of random reads and random writes.

9/28/2019 22

Hadoop

9/28/2019 23Pictures from https://www.hdfstutorial.com/blog/hadoop-1-vs-hadoop-2-differences/

Hadoop

9/28/2019 24

Picture from https://blogs.msdn.microsoft.com/avkashchauhan/2012/02/24/master-slave-architecture-in-hadoop/

Hadoop

9/28/2019 25Picture from https://www.guru99.com/introduction-to-mapreduce.html

Hadoop

9/28/2019 26Pictures from https://www.hdfstutorial.com/blog/hadoop-1-vs-hadoop-2-differences/

Hadoop Ecosystem

9/28/2019 27Picture from http://blog.newtechways.com/2017/10/apache-hadoop-ecosystem.html

Spark

9/28/2019 28Picture from https://databricks.com/blog/2013/11/21/putting-spark-to-use.html

Spark Ecosystem

9/28/2019 29Picture from https://data-flair.training/blogs/apache-spark-ecosystem-components/

Comparison (Hadoop vs. SQL)

• Data Size

9/28/2019 30Picture from https://www.educba.com/hadoop-vs-sql/

• Scaling

9/28/2019 31


Picture from https://www.educba.com/hadoop-vs-sql/

• Query Speed

9/28/2019 32

Hadoop SQL


Picture from https://www.educba.com/sql-vs-hadoop/



• Query Speed

9/28/2019 34

Hadoop SQL



• Machine Learning Support

9/28/2019 35

Hadoop SQL



• Learning Curve

9/28/2019 36

Hadoop SQL



Which Database is suitable for…

9/28/2019 37

Recording exam grades for student in UCONN CSE Dept?


9/28/2019 38

Online AI/machine learning of all car trajectories?(Uber, Lyft)


9/28/2019 39

Record and analyze e-commerce customer behavior offline?(Amazon, Target etc.)


9/28/2019 40

Realtime gaming platforms? (WOW, PUBG, LOL, Dota2)


9/28/2019 41

Messaging system (IMessager, Wechat)

Databases in BIG Data Applications...Organizing Big Data •Database: A database is an organized collection of data, generally stored and accessed electronically from a computer system.

Documents