Tajinder Presentation6

crimeX Real time crime analysis and alert system

Tajinder Singh

Motivation

Motivation

• How criminals operate

• Dynamics between criminals and anti criminal squad

Demo

www.crimefighter.ninja

Pipeline

Crime data (real)

User data (real)

Crime data (batch)

Ingestion Batch Layer Serving Layer

Real Time

Data flow

• Seed: http://us-city.census.okfn.org/dataset/crime-stats

• Engineered Data (600 GB)

Data sources

Data flow

{ “crime_id”: “C786” , “crimetype”: “robbery”, “crime_rptd_ts”: “2015-05-06 11:34:43”,

“crime_occ_ts”: “2015-03-05 15:24:49”, “lat”: “34.5462”, “lon”: “-118.453”, ……etc….}

Crime data (batch)

Batch Processing

Data flow

Crime data (batch)

Batch Processing

{ “crime_id”: “C786” , “crimetype”: “robbery”, “crime_rptd_ts”: “2015-05-06 11:34:43”,

“crime_occ_ts”: “2015-03-05 15:24:49”, “lat”: “34.5462”, “lon”: “-118.453”, ……etc….}

Data flow

Crime data (batch)

Batch Processing

{ “crime_id”: “C786” , “crimetype”: “robbery”, “crime_rptd_ts”: “2015-05-06 11:34:43”,

“crime_occ_ts”: “2015-03-05 15:24:49”, “lat”: “34.5462”, “lon”: “-118.453”, ……etc….}

+ Python Script (Refining)

Data flow

Crime data (batch)

Batch Processing

{ “crime_id”: “C786” , “crimetype”: “robbery”, “crime_rptd_ts”: “2015-05-06 11:34:43”,

“crime_occ_ts”: “2015-03-05 15:24:49”, “lat”: “34.5462”, “lon”: “-118.453”,

“zip”:”90007”, “city”: “los angeles”, “state”:”california”, “country”:”usa”}

Index Type: crimes

Data flow

{ “crimetype”: “robbery”, “lat”: “34.5462”, “lon”: “-118.453”}

Real Time Processing

Crime data User data

{ “user_id”: “user453”, “username”: “Tajinder”, “lat”: “34.653356”, ”lon”: “-118.53243” }

Data flow

{ “crimetype”: “robbery”, “lat”: “34.5462”, “lon”: “-118.453”}

Real Time Processing

[ Processing ]

Crime data User data { “user_id”: “user453”, “username”: “Tajinder”, “lat”: “34.653356”, ”lon”: “-118.53243” }

Data flow Real Time Processing

{ “crimetype”: “robbery”, “crime_rptd_ts”: “2015-05-06 11:34:43”, “lat”: “34.5462”,

“lon”: “-118.453”, “zip”:”90007”, “city”: “los angeles”, “state”:”california”,

“country”:”usa”}

Crime data User data

{ “user_id”: “user453”, “username”: “Tajinder”, “lat”: “34.653356”, ”lon”: “-118.53243”,

”zip”:”90007”, “city”: “los angeles”, “state”:”california”, “country”:”usa” }

Index Type: crimes_realtime and user-subscribe-crime

Data flow use case 1 (batch)

Input [ “location”:”2611 portland street, los

angeles”]

Data flow use case 1 (batch)

Output Fields

Distance Covered (radius)

Total crimes analyzed

Average latency*

Crime Types

Average latency* : Average difference between crimes occurring timestamp & crimes reporting timestamp

Data flow use case 1 (batch)

Output Fields

Distance Covered (radius)

Total crimes analyzed

Average latency*

Crime Types

Average latency* : Average difference between crimes occurring timestamp & crimes reporting timestamp

[output]

Data flow use case 2 (real)

Real Time [ “crimetype”:”robbery”, “lat”:

”34.2353”, “lon”:”-113.42534”]

Data flow use case 2 (real)

Output Fields

Distance Covered (radius)

Total crimes analyzed

Average latency*

Crime Types

Alert nearby users

User Phone number

User Name

User latitude

User longitude

[output]

Challenge: Front-end display after 5 seconds per request

Reason:

• A lot of I/O operations (all crime documents were fetched to the UI)

• Business logic and query execution on front-end (flask)

Solution:

• Query execution on Elasticsearch cluster

• NO I/O operation

• Dynamic scripting enabled on ES cluster.

• Used Groovy scripts as opposed to Javascript, Python, MVEL (built-in),

expression (built-in) etc.

Challenge: Network Latency

Solution: Co-locate Storm and Elasticsearch cluster nodes to reduce network

latency

Performance Optimization

Challenges

Caveat: Vulnerable to outside attacks (Security vulnerability)

Reason:

• Enabled dynamic scripting

Solution:

• Don’t run Elasticsearch as root

• Provide read-only access to requisite directories

Performance Optimization

Challenges

about me

Tajinder Singh [University of Southern California]

5 yrs experience in web development