Understanding the state of your web application using Apache Kafka, Spark

A global consulting and IT solutions leader

Understanding the State of your Web Application

using Apache Kafka, Spark

What if webapps ….

… have feelings?

java.lang.NullPointerException at com.insideout.riley.anger.doAngry(Riley.java:123) at com.insideout.riley.joy.doHappy(Riley.java:456) at com.insideout.riley.sadness.doCry(Riley.java:789) at com.insideout.riley.disgust.doEwwwww(Riley.java:143) at com.insideout.riley.fear.doAaaaahhhh(Riley.java:888) at com.insideout.riley.anger.doAngry(Riley.java:123) at com.insideout.riley.joy.doHappy(Riley.java:456) at com.insideout.riley.sadness.doCry(Riley.java:789) at com.insideout.riley.disgust.doEwwwww(Riley.java:143) at com.insideout.riley.fear.doAaaaahhhh(Riley.java:888) at com.insideout.riley.anger.doAngry(Riley.java:123) at com.insideout.riley.joy.doHappy(Riley.java:456) at com.insideout.riley.sadness.doCry(Riley.java:789) at com.insideout.riley.disgust.doEwwwww(Riley.java:143) at com.insideout.riley.fear.doAaaaahhhh(Riley.java:888) at com.insideout.riley.anger.doAngry(Riley.java:123) at com.insideout.riley.joy.doHappy(Riley.java:456) at com.insideout.riley.sadness.doCry(Riley.java:789) at com.insideout.riley.disgust.doEwwwww(Riley.java:143) at com.insideout.riley.fear.doAaaaahhhh(Riley.java:888) at com.insideout.riley.anger.doAngry(Riley.java:123) at com.insideout.riley.joy.doHappy(Riley.java:456) at com.insideout.riley.sadness.doCry(Riley.java:789) at com.insideout.riley.disgust.doEwwwww(Riley.java:143) at com.insideout.riley.fear.doAaaaahhhh(Riley.java:888) at com.insideout.riley.doPuberty(Riley.java:58)

15/11/09 20:57:42 DEBUGInvalid character in customer name: ‘s@dness’

15/11/09 20:54:56 INFOSuccessfully saved customer record: customer.name=’Joy’

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Unknown Source) at java.lang.String.<init>(Unknown Source) at java.io.BufferedReader.readLine(Unknown Source) at java.io.BufferedReader.readLine(Unknown Source) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley .main(ScaryClown.java:1)

Log Data Tell You Everything/Anything/Nothing● Who tried to login to your app?

● Was the login attempt successful?

● Which pages the user visited?

● What did the user do?

● Did ze make any purchases?

● Did the customer encountered any problems while using your app?

● What were they doing at your website when ‘Tamang Panahon’ was ongoing at Philippine Arena?

More Users = More Logs = TMI !!!

How to make sense of it all???

Data Analyst!

All Your Data!

NOW!

Real-Time Insights Are More:● Relevant

● Actionable

LOG DATA

REAL-TIMEINSIGHTS

???

Logback Appender● Natural integration to your webapp

● Take advantage of logback framework (i.e. specific appenders per module, log level, etc.)

● Easier if you use structured data for your kafka messages (i.e. JSON, XML, etc.)

● Couple of OSS appenders available, but doesn’t seem to have an “official logback version”

Apache Kafka● Publish-Subscribe Messaging

● Fast, Scalable, Durable

● Distributed by Design

● Well Integrated to Hadoop Ecosystem

● … makes you yell ‘Hooray!’

Analytics Platform● Apache Spark Streaming (micro-batch)

● Apache Storm (real-time stream processing)

Summary● Application logging is important! Tells the story of your webapp.

● Not all useful data are stored in your database.

● Listening to your webapp will give you better insights to your business.

● Data is becoming the hottest commodity right now. The faster you are able to utilize it, the more edge you will have.

Questions?