ParkMate - Real-time parking spot recommender Suhas, Insight Data Engineering - Sep 2015
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
ParkMate - Real-time parking spot recommenderSuhas, Insight Data Engineering - Sep 2015
Motivation
Making the task of finding a parking spot smooth and easy using real time parking sensor data
SF Parking Data
Total of 952 Parking Spots (15 garages and 937 street parking) data are ingested every 2 seconds.
Data throughput ~15 GB/day
Can be extended to handle huge loads for multiple cities
Cluster Setup- ec2 - m4.large - 4 instances
● Hadoop - 1 Namenode and 3 Worker nodes
● Spark - 1 Master and 3 Slaves
● Kafka - 4 brokers
● Cassandra - 4 Nodes
● Elasticsearch - 4 Nodes
● Zookeeper - 4 Nodes
Pipeline
SF ParkFirebase
Real time Ingestion
Storage for Batch Batch Processing
Stream Processing
Time series aggregate
Analytics Dashboard
Geo-Spatial Query
User GPS
Challenges
● Spark partitioning RDD for distributed computing.
● Writing data from Kafka to HDFS - Camus vs. custom script.
● Elasticsearch partial document update.
● Spark to Cassandra - PySpark-Cassandra driver.
About meSuhas - CS Grad @UIUC
www.thinkjs.ioBackground:
Full Stack Web Development
Passionate to learn Big-data technologies.
Future plan is to Contribute to Open Source.
Hobbies: Long drives
Thank You
Related Documents
![A Fuzzy Recommender System for eElections - unifr.ch Fuzzy Recommender System for eElections 63 2 Recommender Systems for eCommerce According to Yager [4], recommender systems used](https://static.cupdf.com/doc/110x72/5b08be647f8b9a93738cdc60/a-fuzzy-recommender-system-for-eelections-unifrch-fuzzy-recommender-system-for.jpg)
