Apache Zeppelin, the missing component for your BigData … · 2017-12-14 · Apache Zeppelin, the missing component for your BigData ecosystem DuyHai DOAN, Cassandra Technical Advocate

@doanduyhai

Apache Zeppelin, the missing component for your BigData ecosystem DuyHai DOAN, Cassandra Technical Advocate

@doanduyhai

Who Am I ?!Duy Hai DOAN Cassandra technical advocate •  talks, meetups, confs •  open-source devs (Achilles, …) •  OSS Cassandra point of contact

☞ [email protected] ☞ @doanduyhai

2

@doanduyhai

Datastax!•  Founded in April 2010

•  We contribute a lot to Apache Cassandra™

•  400+ customers (25 of the Fortune 100), 400+ employees

•  Headquarter in San Francisco Bay area

•  EU headquarter in London, offices in France and Germany

•  Datastax Enterprise = OSS Cassandra + extra features

3

What is Apache Zeppelin ?!

Presentation!Architecture!

!

@doanduyhai

Zeppelin Presentation!

5

@doanduyhai

Zeppelin Architecture!

Zeppelin Server

Zeppelin Engine

6

REST

Web

Sock

et

Spark Interpreter Group

Spark SparkSQL

Zeppelin Interpreter

Factory

Tajo Interpreter

Flink Interpreter

Cassandra Interpreter

JVM

JVM

JVM

JVM

JVM

@doanduyhai

What does Zeppelin provide ?!Front-end & display system for free Generic back-end with REST APIs & WebSocket Pluggable interpreters system Task scheduler (à la CRON)

7

Zeppelin UI Layout!

Notebook!Paragraph!

UI elements!

@doanduyhai

Demo https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation

Zeppelin Display System!!

Raw, Table, HTML!Available graphs!

View modes!Dynamic form!Iframe export!

@doanduyhai

Demo https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation

Interpreter system !!

Core interpreters!Third-parties interpreters!

Interpreters conf & usage!

@doanduyhai

Interpreter processing lifecycle!①  Receive input commands/data •  as raw text

•  from form data

②  Process the input commands/data by the external back-end ③  Format the response using Zeppelin display system ④  Send response back to the Zeppelin engine

13

@doanduyhai

Core interpreters !!•  Spark (Spark core, SparkSQL/DataFrame, PySpark) •  Spark core = default (or %spark)

•  SparkSQL = %sql

•  Shell (%sh)

•  Markdown (%md) !

•  AngularJS (%angular)

14

@doanduyhai

Third-parties interpreters!•  Hive •  Phoenix •  Tajo •  Flink •  Ignite •  Lens •  Cassandra •  Geode •  PostgreSQL •  Kylin

15

@doanduyhai

Interpreter conf & usage https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation

Writing An Interpreter !!

How To!Simple interpreter example (AsciiDoc)!

Complex interpreter example (Cassandra)!

@doanduyhai

Steps to write your own interpreter!

•  Create a class that extends Interpreter base class

•  Register it in a static block

•  Optionnally define default config params

18

static { Interpreter.register("MyInterpreterName", MyClassName.class.getName());

}

static { Interpreter.register("MyInterpreterName", MyClassName.class.getName(), new InterpreterPropertyBuilder() .add("property1", "default value", "Description of property1").build());

}

@doanduyhai

To register your interpreter as default !

•  Edit the enum ZeppelinConfiguration.ConfVars

•  Add your interpreter FQCN in the property ZEPPELIN_INTERPRETERS

19

@doanduyhai

To register your interpreter in config files!

•  Create conf/zeppelin-site.xml from conf/zeppelin-site.xml.template

•  Add your interpreter FQCN in the property zeppelin.interpreters

20

<property> <name>zeppelin.interpreters</name> <value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter, org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter, org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter, org.apache.zeppelin.hive.HiveInterpreter,com.me.MyNewInterpreter </value>

</property>

@doanduyhai

Simple AsciiDoc Interpreter!

21

Zeppelin Server

AsciiDoc Interpreter

JVM Zeppelin Engine

Raw Text Block

Raw Text Block

Converted To

HTML

HTML Output

① ②

③ ④

JVM

@doanduyhai

Simple interpreter (AsciiDoc) https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation

@doanduyhai

Cassandra Interpreter Architecture!

23

Cassandra Interpreter

JVM

Display Results as

HTML

① ②

⑤

Zeppelin Server

JVM

Raw Text Block

Raw Text Block

Cassandra Cassandra

Java Driver

③

Async CQL statements

④ Render HTML

⑥

@doanduyhai

Cassandra Interpreter Commands!

24

Native CQL statements SELECT * FROM …; INSERT INTO …; …

Schema commands DESCRIBE TABLE …; DESCRIBE KEYSPACE …; …

Prepared statements Commands

@prepare …; @bind …; @remove_prepared …;

Help command HELP;

Options Commands @consistency …; @retryPolicy …; @fetchSize …;

@doanduyhai

Complex interpreter (Cassandra) https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation

Zeppelin future!!

Roadmap!

@doanduyhai

Roadmap & future!•  More graph options (Map viz ZEPPELIN-157)

•  Helium project, packaging Zeppelin view, logic (code) & resource into Applications

•  Interpreters packaging re-design •  ship & compile core interpreters only

•  third-parties interpreters can be pulled from repository

•  which interpreter is core ? Who will maintain ? Community….

•  Integrate security (Apache Shiro, ZEPPELIN-53 )

27

@doanduyhai

Roadmap & future!•  Out of incubation state to become 1st class Apache project

28

@doanduyhai

Q & R

! " !

@doanduyhai

Thank You @doanduyhai

[email protected]

http://zeppelin.incubator.apache.org/