Apache Kafka & Kafka Connectをに使ったデータ連携パターン(改めETLの実装)

Apache Kafka & Kafka Connect

( ETL )

Future Architect ,IncKeigo Suda

2016/11/25D&S Data Night vol.04

12

!Kafka Connect ETL( )

( ) /

13

!Future Architect ,Inc ( )

!

! IoT

14

15

Kafka Connect

16

!Kafka ver 0.9!Kafka!

!

! /

Kafka Connect

17

Connectors

https://www.confluent.io/product/connectors/

Kafka Connect

21

22

!

!

!

Kafka Connect ( )

23

ETL Kakfa Connect( )

P T

// / /

AC

KM

I

M

KM

Kafka Connect

25http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started

Kakfa Connect

Kafka Connect

26

Kakfa Connect

http://docs.confluent.io/2.0.0/connect/userguide.html#getting-started

27

Stream & Partition(RDB )

http://www.slideshare.net/KaufmanNg/data-pipelines-with-kafka-connect


28

N:1

Stream & Partition(RDB )



29

Kakfa Connect


30

Worker & Connector


31

CONNECTERWORKER

STREAM

PARTITIONSTANDALONE MODE

DISTRIBUTED MODE

TASK

32

$ bin/connect-standalone.sh config/connect-standalone.properties connector1.properties

(Standalone mode)

!

!

! Worker! Connector

Worker Connector

33

name=local-file-source connector.class=FileStreamSource tasks.max=1 file=test.txt topic=connect-test

bootstrap.servers=localhost:9092 key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter ...

Worker

Connector

35

!

!

! ETL

36

!Kafka Connect Source Sink

!Sink/Source Sink

Kafka Connect

Source Sink

37

! Connector/Task OK! API OK

( )

( )

38

Connector( )

39

!!(source pull)

Task

40

! Task Connector

!

Deploy

name=sample-sink connector.class=SampleSinkConnector tasks.max=1 topic=connect-test

Connector

42

!Kafka

!

Kafka Connect

43

!

API

Kafka Connect

44

(kafka-connect-jdbc)

45

! ConfluentConfuluent Platform

Kafka Connect

46

!Kakfa ( )!Confluent….

Apache Kafka & Kafka Connectを に使ったデータ連携パターン(改めETLの実装)

Internet

Apache Kafka & Kafka Connectをに使ったデータ連携パターン(改めETLの実装)