Web Development To Big Data: A Journey Application pure Streaming System Streaming Computation Continuous Application Continuous Application Static Data Stream Output Sink Often up

Web Development To Big Data: A Journey

@samelamin

@samelamin

Sam I Am

@samelamin

Why I Changed Careers?

WTF Is Big Data

@samelamin

The 3 Vs

@samelamin

@samelamin

ETL??

@samelamin

Redshift Datawarehouse

@samelamin

State Of Play

@samelamin

Immutable Infrastructure

@samelamin

Slow Updates

@samelamin

Run Every 15 Mins

@samelamin

New Sources = Pain

@samelamin

Building A Data Pipelines

@samelamin

Make Data Queryable!

@samelamin

Architecture!

@samelamin

Handle Bad Data

@samelamin

Schema Updates!

@samelamin

Alerting!

@samelamin

Data Formats

@samelamin

Columnar Storage

@samelamin

Data Lake

@samelamin

Apache Spark

@samelamin

DAG Ting!

@samelamin

Data Serving Layer

@samelamin

Lambda Architecture

Data Streaming

@samelamin

@samelamin

Issues With Streaming

@samelamin

Structured Streaming AKA V2

@samelamin

Google BigQuery

@samelamin

Brag Alert!

https://github.com/samelamin/spark-bigquery

@samelamin

ML Powering Recommendations

@samelamin

DEMO!

@samelamin

Thank You!

@samelamin

Write to BQ DEMO!

@samelamin

Read From BQ DEMO!

@samelamin

Structured Streaming DEMO!