Apache RocketMQ – Trillion Messaging in Practice 冯嘉(Von Gosling) ©2017 Alibaba Middleware Group
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Apache RocketMQ – Trillion Messaging in Practice
冯嘉(Von Gosling)
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
PROFILE
Von Gosling, Alibaba Architect , Master of Computer Science, Over 7 years of industrial software experience, especially indistributed tech., high performance webarchitecture and performance tuning.Possesses many patents in distributedsystem, recommender etc.
Open source fanatic. Focuses on distributedsystems, cloud computing and big data fields. Apache RocketMQ PPMC member &committer.
Currently in charge of Aliware MQ. Email:[email protected]
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
Part I
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
20162007 2010 2011 2012 2015
NotifyBorn from multicolored
stone project, designed for
Taobao B2C Trading
NapoliActiveMQ kernal, designed
for B2B Trading and other
scenarios
MetaQ v1.0Designed for ordered message,
especially for message
accumulation
Aliware MQ v1.0
Notify v3.0
MetaQ v2.0 v3.0
RocketMQ v3.0
OS
Apache
RocketMQ
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
MetaQ
RocketMQ
Notify Aliware MQ
Ordered messaging，Pull model
Commercial Distribution，Cloud MQ
Transactional messaging, Push model
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
1
2
3
4
5
Heterogeneous Application Integration
loosen the coupling between applications
Backbone for EDA or CEP
Data Replication
Streaming
10
33.2
194.2
650
1400
2012 double 11 day 2013 double 11 day 2014 double 11 day 2015 double 11 day 2016 double 11 day
Total number of Messages(Unit: Billion)
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
Part II
01 Architecture and Features
02 Monitoring and Administering
03 Performance Tuning
04 Apache RocketMQ 4.X
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
01 Architecture and Features
…… 合抱之木生于毫末；九层之台起于累土；千里之行始于足下……
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
ü Ordered and Non-Ordered messageü Broadcast messageü Scheduled messageü Transactional messageü Batched messageü One-way messageü Traceable messageü Retroactive messageü Message Filteringü Massive Accumulationü Backoff Strategyü Delivery QoSü Community features – console, docker, streaming etc.
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
Console
Scheduled Message
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
Transactional Message – Eventually Consistent
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
HA
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
Message Tracing Dashboard
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
Kafka Service
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
IoT - MQTT
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
Relay Service
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
02 Monitoring and Administering
千磨万击还坚韧，任尔东西南北风
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
$sh mqadminThe most commonly used mqadmin commands are:healthCheck health checkdeleteSubGroup Delete subscription group from broker.updateBrokerConfig Update broker's configgetBrokerConfig Get broker config by cluster or special brokerupdateTopicPerm Update topic permtopicRoute Examine topic route infotopicStatus Examine topic Status infoqueryMsgById Query Message by IdqueryMsgByKey Query Message by KeyconsumerProgress Query consumers's progress, speedconsumerStatus Query consumer's internal data structureclusterList List all of clustersresetOffsetByTime Reset consumer offset by timestamp(without client restart).…...
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
$sh mqadmin clusterlist
#Cluster Name #Broker Name #BID #Addr #Version #InTPS(LOAD) #OutTPS(LOAD) #Hour #SPACEshare share-01 0 10.10.10.10:10911 V4_0_0 3840.73(0,0ms) 7758.35(0,0ms) 42.31 0.8394share share-02 1 10.10.10.11:10911 V4_0_0 3764.42(1,0ms) 7734.55(1,0ms) 82.97 0.3245
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
$sh mqadmin consumerstatus -g CID_SHOPPING -s
001 10.10.10.4@29077 V4_0_0 1493971139224/10.10.10.4@29077#-1244857759#-163453988#14074607828995785002 10.10.10.2@28953 V4_0_0 1493971139224/10.10.10.2@28953#-1244857759#-163453988#14074505387423377003 10.10.10.1@11865 V4_0_0 1493971139224/10.10.10.1@11865#-1244857759#-163453988#14074508601484056004 10.10.10.3@23180 V4_0_0 1493971139224/10.10.10.3@23180#-1244857759#-163453988#14074400788700508
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
$sh mqadmin querymsgbykey -k Query_Key -t ORDER_TOPIC
#Message ID #QID #Offset707C8CC435FE3FD00E77180D7FCB0B24 0 1
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
$sh mqadmin querymsgbyuniquekey -i 707C8CC435FE3FD00E77180D7FCB0B24 -t ORDER_TOPICTopic: ORDER_TOPICTags: [ApacheCon]Keys: [Query_Key]Queue ID: 0Queue Offset: 1CommitLog Offset: 148053682794Reconsume Times: 0Born Timestamp: 2017-05-05 16:05:37,867Store Timestamp: 2017-05-05 16:05:37,880Born Host: 10.143.33.157:42775Store Host: 10.151.203.124:10911System Flag: 0Properties: {TRACE_ON=true, KEYS=Query_Key, MSG_REGION=cn-hangzhou, UNIQ_KEY=707C8CC435FE3FD00E77180D7FCB0B24, TAGS=ApacheCon}Message Body Path: /tmp/rocketmq/msgbodys/707C8CC435FE3FD00E77180D7FCB0B24
03 Performance Tuning
行路难，行路难，多歧路，今安在。长风破浪会有时，直挂云帆济沧海。
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
RocketMQ vs. Kafka
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
04 RocketMQ 4.X
咬定青山不放松，立根原在破岩中
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
E-Commerce
High ConcurrencyHigh Reliability
Finance
IoT
BigData
Massive Online DevicesHigh Throughput
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
Apache RocketMQ Community
ü Workshop, Meetupü Code Marathonü Community Merit
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
谢谢THANKS!
Alibaba Middleware
Recommended Latest Articles :1. From Alibaba to Apache: RocketMQ’s Past, Present, and Future. InfoQ2. Apache RocketMQ背后的设计思路与最佳实践. 云栖社区3. 专访RocketMQ联合创始人：项目思路、技术细节和未来规划. InfoQ4. 万亿级数据洪峰下的分布式消息引擎. 电子书节选章节5. 捐赠Apache背后那些鲜为人知的故事. 中间件博客节选
© 2 0 1 7 A l i b a b a Mi d d l e w ar e G r o u p
Related Documents
