‹#› © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices with Spring Cloud Stream, Task, and Data Flow Toshiaki Maki (@making) 2016-11-18 Spring Day 2016 #springday #jsug
Jan 06, 2017
‹#›© 2016 Pivotal Software, Inc. All rights reserved. ‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices with Spring Cloud Stream, Task, and Data Flow
Toshiaki Maki (@making) 2016-11-18 Spring Day 2016 #springday #jsug
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Who am I ?• Toshiaki Maki (@making) http://blog.ik.am
•Sr. Solutions Architect @Pivotal
•Spring Framework enthusiast
bit.ly/hajiboot2
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Input
Motivation
HTTP
TCP
S3
JDBC
Rabbit
JMS
Syslog
Output
HDFS
Cassandra
HAWQ
Greenplum
JDBC
S3
TCP
Gemfire
❓❔❓
• Real Time Analysis • Data Ingestion • ETL Process
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring XD (eXtreme Data)
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring XD (eXtreme Data)
🤔
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring XD Architecture
Container Containergpfdist
Cassandra
jms
http
ZooKeeper
Message Broker
XD Admin
stream1 = http | cassandrastream2 = jms | gpfdist
On Metal/VMs
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring XD Architecture
Container Containergpfdist
Cassandra
jms
http
ZooKeeper
Message Broker
XD Admin
stream1 = http | cassandrastream2 = jms | gpfdist
On Metal/VMs
😇
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring XD Architecture
Container Containergpfdist
Cassandra
jms
http
ZooKeeper
Message Broker
XD Admin
stream1 = http | cassandrastream2 = jms | gpfdist
On Metal/VMs
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Modern Platforms
Cloud Foundry YARN Kubernetes Mesos
Docker Swarm Nomad OpenShift
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Cloud Native Redesign
•From multiple modules embedded in a container to standalone executable applications •From our own runtime to delegating to existing modern platforms
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
Microservice for each data processing
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
Microservice for each data processing
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
Microservice for each data processing
bound with Message Brokers
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
Microservice for each data processing
bound with Message Brokers
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
Microservice for each data processing
bound with Message Brokers
on the modern platform
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
Microservice for each data processing
bound with Message Brokers
on the modern platform
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Data Microservices
$ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
Microservice for each data processing
bound with Message Brokers
on the modern platform
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow• Microservices-based Distributed Data Pipelines •Long Lived Stream Applications
•Short Lived Task Applications
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow• Microservices-based Distributed Data Pipelines •Long Lived Stream Applications
•Short Lived Task ApplicationsSpring Cloud Stream
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow• Microservices-based Distributed Data Pipelines •Long Lived Stream Applications
•Short Lived Task ApplicationsSpring Cloud Stream
Spring Cloud Task
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow• Microservices-based Distributed Data Pipelines •Long Lived Stream Applications
•Short Lived Task ApplicationsSpring Cloud Stream
Spring Cloud Task
Orchestration Layer
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
StructureSpring Cloud Data Flow
Spring Cloud Deployer (SPI)
Spring Cloud Stream Spring Cloud Task
Spring Integration Spring Boot Spring Batch
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
StructureSpring Cloud Data Flow
Spring Cloud Deployer (SPI)
Spring Cloud Stream Spring Cloud Task
Spring Integration Spring Boot Spring Batch
• Spring Cloud Deployer Local • Spring Cloud Deployer Cloud Foundry • Spring Cloud Deployer Yarn • Spring Cloud Deployer Kubernetes • Spring Cloud Deployer Mesos
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
SCDF Deployment Platform
Spring Cloud Data Flow Server
Deployer SPI
REST API
SCDF Shell SCDF Flo
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow Architecture
gpfdist
cassandra
jms
http
stream1 = http | cassandrastream2 = jms | gpfdist
Message Broker
Data Flow Server DB
Platform Runtime
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow Architecture
gpfdist
cassandra
jms
http
stream1 = http | cassandrastream2 = jms | gpfdist
Message Broker
Data Flow Server DB
Platform Runtime
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Stream
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Stream•Event-driven microservice framework
•Built on battle-tested components (Spring Boot / Spring Integration)
•Opinionated primitives for streaming applications
•Persistent Pub/Sub
•Consumer Groups
•Partitioning Support
•Pluggable messaging middleware bindings
source | processor | sink
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
JJUG CCC in 2 weeks
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Stream Applications
Twitter Stream
Cassandra
java -jar twittersource.jar --server.port=8080 --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings. output.destination=ingest
Source
Sink java -jar cassandrasink.jar --server.port=8081 --spring.cassandra.keyspace=tweet --spring.cloud.stream.bindings. input.destination=ingest
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Stream Applications
Twitter Stream
Cassandra
java -jar twittersource.jar --server.port=8080 --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings. output.destination=ingest
Source
Sink java -jar cassandrasink.jar --server.port=8081 --spring.cassandra.keyspace=tweet --spring.cloud.stream.bindings. input.destination=ingest
TwitterStream Cassandraingest
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Message Binders• @EnableBinding•Binder Implementations
•Production-Ready
•Rabbit MQ
•Apache Kafka
•Experimental
• JMS
•Google PubSub
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Programming Model (Sink)@SpringBootApplication @EnableBinding(Sink.class) public class DemoSinkApp { @StreamListener(Sink.INPUT) void receive(Message<String> message) { System.out.println("Received " + message); } public static void main(String[] args) { SpringApplication.run(DemoSinkApp.class, args); } }
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Sink
Properties (Sink)
spring.cloud.stream.bindings.input.destination=demo-strm
demo-strm in
put
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Programming Model (Source)@SpringBootApplication @RestController@EnableBinding(Source.class) public class DemoSourceApp { @Autowired Source source; @GetMapping void send(@RequestParam String text) { source.output() .send(MessageBuilder.withPayload(text).build()); } public static void main(String[] args) { SpringApplication.run(DemoSourceApp.class, args); } }
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Programming Model (Source)@SpringBootApplication @RestController@EnableBinding(Source.class) public class DemoSourceApp { @Bean @InboundChannelAdapter(channel = Source.OUTPUT, poller = @Poller(fixedDelay = "1000", maxMessagesPerPoll = "1")) MessageSource<String> source() { return () -> MessageBuilder.withPayload("Hi").build()); } public static void main(String[] args) { SpringApplication.run(DemoSourceApp.class, args); }}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Properties (Source)
spring.cloud.stream.bindings.output.destination=demo-strm
demo-strm
Source
outp
ut
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Binder (RabbitMQ)<dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-stream-rabbit</artifactId></dependency>
demo-strm
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Binder (Apache Kafka)<dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-stream-kafka</artifactId></dependency>
demo-strm
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Sink
Pipeline
demo-strm in
put
Source
outp
ut
source | sink
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Programming Model (Processor)@SpringBootApplication @EnableBinding(Processor.class) public class DemoProcessorApp { @StreamListener(Processor.INPUT) @SendTo(Processor.OUTPUT) void receive(String text) { return "[[" + text + "]]"; } public static void main(String[] args) { SpringApplication.run(DemoProcessorApp.class, args); } }
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Properties (Processor)
spring.cloud.stream.bindings.output.destination=my-source
spring.cloud.stream.bindings.input.destination=my-sourcespring.cloud.stream.bindings.output.destination=my-proc
spring.cloud.stream.bindings.input.destination=my-proc
Source
Processor
Sink
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Pipeline
my-source
Source
outp
utsource | processor | sink
Processor
outp
ut
inpu
t
my-proc
Sink
inpu
t
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Reactive API Support@SpringBootApplication @EnableBinding(Processor.class) public class DemoProcessorRxApp { @StreamListener @Output(Processor.OUTPUT) public Flux<String> receive(@Input(Processor.INPUT) Flux<String> stream) { return stream.map(text -> "[[" + text + "]]"); } public static void main(String[] args) { SpringApplication.run(DemoProcessorRxApp.class, args); } }
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Reactive API Support
@StreamListener @Output(Processor.OUTPUT) public Flux<AverageData> receive(@Input(Processor.INPUT) Flux<SensorData> stream) { return stream.window(Duration.ofSecond(20), Duration.ofSecond(10)) .flatMap(win -> win.groupBy(sensor -> sensor.id)) .flatMap(group -> calcAverage(group));}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Stream App StartersSource Processor Sink
file ftp gemfire gemfire-cq http jdbc jms load-generator loggregator mail mongodb rabbit
s3 sftp syslog tcp tcp-client time trigger triggertask twitterstream
bridge filter groovy-filter groovy-transform httpclient pommel scriptable-transform splitter tcp-client transform
aggregate-counter cassandra counter field-value-counter file ftp gemfire gpfdist hfs hdfs-dataset jdbc
log rabbit redis-pubsub router s3 sftp task-launcher-local task-lancher-yarn tcp throughput web socket
http://cloud.spring.io/spring-cloud-stream-app-starters/
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Stream Orchestration in SCDFingest = twitterstream | cassandra
dataflow:> stream create --name=ingest --definition="twitterstream | cassandra"dataflow:> stream deploy --name=ingest
|
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Import apps
dataflow:> app import --uri http://bit.ly/1-0-4-GA-stream-applications-rabbit-maven
rabbit mavenkafka docker
stream-applications-
http://cloud.spring.io/spring-cloud-stream-app-starters/
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Register your own apps
dataflow:> app register --name foo --type source --uri http://example.com/foo-source-1.0.jar
dataflow:> app register --name foo --type source --uri maven://com.example:foo-source:1.0dataflow:> app register --name foo --type source --uri docker:myapps/foo-source:1.0
dataflow:> app register --name foo --type source --uri file:///tmp/foo-source:1.0
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Instance Count
dataflow:> stream create --name=s1 --definition="http | work | hdfs"dataflow:> stream deploy --name=s1 --properties="app.http.count=2,app.work.count=3,app.hdfs.count=4"
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Instance Count
http
work
hdfs
http
work
work
hdfs
hdfs
hdfs
Load
Bal
ance
r
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Resource Management
dataflow:> stream deploy --name=s1 --properties="app.work.spring.cloud.deployer.cloudfoundry.memory=2048"
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Resource Management
http
http
work
work
work
hdfs
hdfs
hdfs
hdfs
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Flo http://projects.spring.io/spring-flo/
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Stream Core Features•Persistent Pub-Sub •Consumer Group •Partitioning Support
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average Top Ns1.http s1.ave
Message Broker
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average
HDFS
Top Ns1.http s1.ave
Message Broker
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average
HDFS
Top N
FaultDetection
s1.http s1.ave
Message Broker
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average
HDFS
Top N
FaultDetection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average
HDFS
Top N
FaultDetection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}{"id":1, "temperature":38}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average
HDFS
Top N
FaultDetection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Persistent Pub-Sub
HTTP Average
HDFS
Top N
FaultDetection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
{"id":1, "temperature":38} {"id":1, "temperature":38}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
FaultDetection
s1.http s1.ave
Message Broker
AverageAverage
HDFSHDFS
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
FaultDetection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
AverageAverage
HDFSHDFS
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
FaultDetection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}{"id":1, "temperature":38}
AverageAverage
HDFSHDFS
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
FaultDetection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
{"id":1, "temperature":38}
AverageAverage
HDFSHDFS
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
FaultDetection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
{"id":1, "temperature":38}
AverageAverage
HDFSHDFS
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
😩
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
FaultDetection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
{"id":1, "temperature":38}
AverageAverage
HDFSHDFS
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
😩
Consumer Group
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
HTTP Average
HDFS
Top N
FaultDetection
s1.http s1.ave
Message Broker
{"id":1, "temperature":38}
{"id":1, "temperature":38}
AverageAverage
HDFSHDFS
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
{"id":1, "temperature":38}
😩spring.cloud.stream.bindings.<channelName>.group=ave
Spring Cloud Stream
group is configured by default 😁
Spring Cloud Data Flow
Consumer Group
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Consumer Group
HTTP Average
HDFS
Top N
FaultDetection
s1.http s1.ave
Message Broker
AverageAverage
HDFSHDFS
group=ave
group=hdfs
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Consumer Group
HTTP Average
HDFS
Top N
FaultDetection
s1.http s1.ave
Message Broker
AverageAverage
HDFSHDFS
group=ave
group=hdfs
{"id":1, "temperature":38}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Consumer Group
HTTP Average
HDFS
Top N
FaultDetection
s1.http s1.ave
Message Broker
AverageAverage
HDFSHDFS
group=ave
group=hdfs
{"id":1, "temperature":38}{"id":1, "temperature":38}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Consumer Group
HTTP Average
HDFS
Top N
FaultDetection
s1.http s1.ave
Message Broker
AverageAverage
HDFSHDFS
group=ave
group=hdfs
{"id":1, "temperature":38}
{"id":1, "temperature":38}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Consumer Group
HTTP Average
HDFS
Top N
FaultDetection
s1.http s1.ave
Message Broker
AverageAverage
HDFSHDFS
group=ave
group=hdfs
{"id":1, "temperature":38}
{"id":1, "temperature":38} 🤓
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Consumer Group
HTTP Average
HDFS
Top N
FaultDetection
s1.http s1.ave
Message Broker
AverageAverage
HDFSHDFS
group=ave
group=hdfs
{"id":1, "temperature":38}
{"id":1, "temperature":38} 🤓consumer group subscriptions
are durable 😁
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}{"id":2, "temperature":41}
{"id":2, "temperature":42}{"id":1, "temperature":37}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}{"id":2, "temperature":41}{"id":2, "temperature":42}
{"id":1, "temperature":37}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}{"id":1, "temperature":37}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
😩
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
😩
Partitioning Support(Stateful Stream)
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}
{"id":1, "temperature":37}
😩stream deploy --name=xxxx --properties="app.http.producer.partitionKeyExpression=payload.id"
spring.cloud.stream.bindings.<channelName>.producer.partitionKeyExpression=payload.id
Spring Cloud Stream
Spring Cloud Data Flow
Partitioning Support(Stateful Stream)
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}{"id":2, "temperature":41}
{"id":2, "temperature":42}{"id":1, "temperature":37}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}{"id":2, "temperature":41}{"id":2, "temperature":42}
{"id":1, "temperature":37}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}
{"id":2, "temperature":42}{"id":1, "temperature":37}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}{"id":2, "temperature":42}
{"id":1, "temperature":37}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}{"id":2, "temperature":42}
{"id":1, "temperature":37}
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Partitioning Support(Stateful Stream)
HTTP
Average
s1.http
Average
{"id":1, "temperature":38}
{"id":2, "temperature":41}{"id":2, "temperature":42}
{"id":1, "temperature":37}
🤓
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Simple Real Time Analyticstweets = twitterstream | hdfs
analytics = :ingest.twitterstream > field-value-counter --fieldName=lang
HTTP
s1.http
HDFS COUNTER
Data Flow ServerREST API
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Demo (Twitter Stream!)
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Task
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Task•Spring Boot based framework for short lived task
•Spring Boot's CommandLineRunner or Spring Batch
•Result of each process persists in Task Repository
•Well Integrated with Spring Batch (Job Repository)
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Programming Model (CLR)@SpringBootApplication @EnableTaskpublic class DemoTaskApp { @Bean CommandLineRunner clr() { return args -> System.out.println("Task!"); } public static void main(String[] args) { SpringApplication.run(DemoTaskApp.class, args); } } spring.application.name=hello
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Task Execution ListTask Name ID Start Time End Time Exit Code
hello 3 Tue Nov 15 16:49:08 JST 2016
Tue Nov 15 16:49:08 JST 2016
0
hello 2 Tue Nov 15 16:48:54 JST 2016
Tue Nov 15 16:48:55 JST 2016
0
hello 1 Tue Nov 15 16:48:23 JST 2016
Tue Nov 15 16:48:23 JST 2016
0
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Programming Model (Spring Batch)@SpringBootApplication @EnableBatchProcessing@EnableTaskpublic class DemoBatchApp { @Autowired JobBuilderFactory jobBuilderFactory; @Autowired StepBuilderFactory stepBuilderFactory; @Bean Step step1() { /* ... */ } @Bean Step step2() { /* ... */ } @Bean Job job() { return jobBuilderFactory.get("job") .start(step1()).next(step2()).build()} public static void main(String[] args) { SpringApplication.run(DemoTaskApp.class, args); }}
https://github.com/making/cf-spring-batch-demo
spring.application.name=hello-batch
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Task Execution ListTask Name ID Start Time End Time Exit Code
hello-batch
5 Tue Nov 15 17:28:49 JST 2016
Tue Nov 15 17:28:49 JST 2016
0
hello-batch
4 Tue Nov 15 17:27:56 JST 2016
Tue Nov 15 17:27:57 JST 2016
0
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Job Execution ListID Task ID Start Time Step Execution Count Definition Status
2 5 Tue Nov 15 17:28:49 JST 2016
2 Created
1 4 Tue Nov 15 17:27:56 JST 2016
2 Created
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Step Execution ListID Step Name Job Exec Id Start Time End Time Status
1 step1 1 Tue Nov 15 17:28:49 JST 2016
Tue Nov 15 17:28:49 JST 2016
COMPLETED
2 step2 1 Tue Nov 15 17:27:56 JST 2016
Tue Nov 15 17:27:57 JST 2016
COMPLETED
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Task Orchestration in SCDF>task create hello --definition="timestamp --format=¥"yyyy¥"">task launch hello
timestamp
Data Flow Server DB Task NameStart TimeEnd TimeExit CodeExit MessageLast Updated TimeParameters
Message Broker
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Task Execution List
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Job Execution List
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Job Execution Details
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Spring Cloud Data Flow -> Taskhttp | tasklaunchrequest --uri=... | task-lancher
timestamp
Data Flow Server DB
Message Broker
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Demo (Scalable Tasks)
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Pivotal Cloud Foundry
PDF PDF
PDF PDF
S3
Spring Cloud Data FlowTask
TaskTask
Task
Spring Cloud Task Spring Batch
index
index
source sink
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Getting Started with SCDF on Cloud Foundry
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Install PCF Dev• https://docs.pivotal.io/pcf-dev
•Cloud Foundry on your laptop
• Included
•Redis / RabbitMQ / MySQL
•Spring Cloud Services
• Install with cf dev start
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
cf create-service p-mysql 512mb df-mysqlcf create-service p-rabbitmq standard df-rebbitmqcf create-service p-redis shared-vm df-rediscf push dataflow-server -p spring-cloud-dataflow-server-cloudfoundry-1.0.1.RELEASE.jar
Deploy SCDF Server on PCF Dev
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Tutorial
•📗 https://blog.ik.am/entries/396
•💎 https://github.com/making-demo-scdf/spring-cloud-dataflow-cookbook
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Upcoming Features•Some ‘porting’ from XD
• Batch Job DSL + Designer • Role based access
• Looking forward • Spring Cloud Sleuth • JavaDSL • In-place application version upgrades with Spinnaker • Application Groups • Polyglot • Expanded analytics with Redis and Python/R ecosystem • More provided apps/tasks
•SCDF Tile for Pivotal Cloud Foundry
‹#›© 2016 Pivotal Software, Inc. All rights reserved.
Thanks!•References
• http://www.slideshare.net/SpringCentral/data-microservices-in-the-cloud ✨
• https://cloud.spring.io/spring-cloud-dataflow/
• http://cloud.spring.io/spring-cloud-stream/
• http://cloud.spring.io/spring-cloud-stream-app-starters/
• https://github.com/spring-cloud/spring-cloud-task/
• http://cloud.spring.io/spring-cloud-dataflow-server-cloudfoundry/