Jeff Wolski - Explorations in Cooperative, Distributed Systems with Uber's Ringpop
Post on 16-Apr-2017
611 Views
Preview:
Transcript
WHAT IS RINGPOP?HIGH-LEVEL FACTS
It is a library written in Node.js and Go.
It is...
“...scalable, fault-tolerant application layer sharding.”
"...a library that brings cooperation and coordination to distributed applications.”
“...a hash ring.”
It is open source.
BUT WHAT IS UBER?REALTIME DISPATCH ENGINEERING
Mobile API, match-maker, trip orchestrator
Platform for marketplaces
Highly interactive, real-time
Data locality
Long-running transactions
NG
INX
HA
PR
OX
Y
NO
DE
.JS
TWE
MP
RO
XY
RE
DIS
NG
INX
HA
PR
OX
Y
NO
DE
.JS
TWE
MP
RO
XY
RE
DIS
NO
DE
.JS
NO
DE
.JS
RE
DIS
NO
DE
.JS
NO
DE
.JS
NO
DE
.JS
NO
DE
.JS
RE
DIS
NO
DE
.JS
NO
DE
.JS
ROUTING LAYER DISPATCH LAYER
LON
DO
NN
EW
YO
RK
ORIGINAL DISPATCH ARCHITECTUREUBER REAL-TIME DISPATCHING
0 1 2 3
NG
INX
HA
PR
OX
Y
NO
DE
.JS
TWE
MP
RO
XY
RE
DIS
NG
INX
HA
PR
OX
Y
NO
DE
.JS
TWE
MP
RO
XY
RE
DIS
NO
DE
.JS
NO
DE
.JS
RE
DIS
NO
DE
.JS
NO
DE
.JS
NO
DE
.JS
NO
DE
.JS
RE
DIS
NO
DE
.JS
NO
DE
.JS
ROUTING LAYER
ORIGINAL DISPATCH ARCHITECTUREUBER REAL-TIME DISPATCHING
0 1 2 3
DISPATCH LAYER
LON
DO
NN
EW
YO
RK
NG
INX
HA
PR
OX
Y
NO
DE
.JS
TWE
MP
RO
XY
RE
DIS
NG
INX
HA
PR
OX
Y
NO
DE
.JS
TWE
MP
RO
XY
RE
DIS
NO
DE
.JS
NO
DE
.JS
RE
DIS
NO
DE
.JS
NO
DE
.JS
NO
DE
.JS
NO
DE
.JS
RE
DIS
NO
DE
.JS
NO
DE
.JS
ROUTING LAYER
ORIGINAL DISPATCH ARCHITECTUREUBER REAL-TIME DISPATCHING
0 1 2 3
DISPATCH LAYER
LON
DO
NN
EW
YO
RK
NG
INX
HA
PR
OX
Y
NO
DE
.JS
TWE
MP
RO
XY
RE
DIS
NG
INX
HA
PR
OX
Y
NO
DE
.JS
TWE
MP
RO
XY
RE
DIS
NO
DE
.JS
NO
DE
.JS
RE
DIS
NO
DE
.JS
NO
DE
.JS
NO
DE
.JS
NO
DE
.JS
RE
DIS
NO
DE
.JS
NO
DE
.JS
ROUTING LAYER DISPATCH LAYER
LON
DO
NN
EW
YO
RK
ORIGINAL DISPATCH ARCHITECTUREUBER REAL-TIME DISPATCHING
0 1 2 3
NG
INX
HA
PR
OX
Y
NO
DE
.JS
TWE
MP
RO
XY
RE
DIS
NG
INX
HA
PR
OX
Y
NO
DE
.JS
TWE
MP
RO
XY
RE
DIS
NO
DE
.JS
NO
DE
.JS
RE
DIS
NO
DE
.JS
NO
DE
.JS
NO
DE
.JS
NO
DE
.JS
RE
DIS
NO
DE
.JS
NO
DE
.JS
ROUTING LAYER DISPATCH LAYER
LON
DO
NN
EW
YO
RK
ORIGINAL DISPATCH ARCHITECTUREUBER REAL-TIME DISPATCHING
0 1 2 3
EVOLUTION OF THINKING AT UBERUBER REAL-TIME DISPATCHING
Scaling our organization, product and
systems
100x scale
Availabilityover
consistency Self-healing
No “master” No monolith
Flexibility and safety
INST. A INST. B INST. C
DATABASE DATABASE
DISCRETE APPLICATION INSTANCESENTER RINGPOP: MEMBERSHIP PROTOCOL
FE1 FE2 FE3 FE4 FE5FRONT-END
APPLICATION
STORAGE
COOPERATIVE APPLICATION INSTANCESENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. A
INST. B
INST. C
DATABASE DATABASE
FE1 FE2 FE3 FE4 FE5FRONT-END
APPLICATION
STORAGE
SWIM GOSSIP PROTOCOLENTER RINGPOP: MEMBERSHIP PROTOCOL
PINGINST. A INST. B
INST. AINST. BINST. C
INST. AINST. BINST. C
INST. CINST. AINST. BINST. C
PING PING
Membership list
ENTER RINGPOP: MEMBERSHIP PROTOCOL
PINGINST. A INST. B
INST. AINST. BINST. C
INST. AINST. BINST. C
INST. CINST. AINST. BINST. C
FAILURE DETECTION
ENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. A INST. BINST. AINST. BINST. C
INST. AINST. BINST. C
INST. CINST. AINST. BINST. C
PING-REQ PING
AN INDIRECT PING
ENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. A INST. BINST. AINST. BINST. C
INST. AINST. BINST. C
INST. CINST. AINST. BINST. C
PING-REQ
DECLARE INST. B SUSPECT
ENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. A INST. BINST. AINST. BINST. C
INST. AINST. BINST. C
INST. CINST. AINST. BINST. C
PING
INST. B
PIGGYBACK MEMBERSHIP UPDATES
ENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. A INST. BINST. AINST. BINST. C
INST. AINST. BINST. C
INST. CINST. AINST. BINST. C
PING
INST. B
INFECTION-STYLE DISSEMINATION
ENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. A INST. BINST. AINST. BINST. C
INST. AINST. BINST. C
INST. CINST. AINST. BINST. C
PING
DETECTING A FAILURE
CREATING A CLUSTERENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. AINST. A
INST. C
INST. B
SENDS JOIN
SENDS JOIN
INST. B STARTS UPENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. AINST. A
INST. C
SENDS JOIN
SENDS JOININST. B INST. B
PIGGYBACK MEMBERSHIP ON JOINENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. AINST. A
INST. C
INST. B INST. B
SENDS JOIN
RESPONDS TO JOIN
INST. B
“A” APPLIES UPDATE FROM “B”ENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. AINST. A
INST. C
INST. B INST. BINST. B
SENDS JOIN
RESPONDS TO JOIN
INST. B
“A” PIGGYBACKS UPDATEENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. AINST. A
INST. C
INST. B INST. BINST. B
SENDS JOIN
SENDS PING
INST. A
“B” APPLIES “A” UPDATEENTER RINGPOP: MEMBERSHIP PROTOCOL
INST. AINST. A
INST. C
INST. B INST. BINST. B
SENDS JOIN
SENDS PING
INST. A
INST. A
ENTER RINGPOP: CONSISTENT HASHING
hash(“INST. A”);
hash(“INST. B”);
hash(“INST. C”);
HASH APPLICATION INSTANCES
ENTER RINGPOP: CONSISTENT HASHING
INST. A
INST. B
INST. C
ASSIGN OWNERSHIP “USER8”, “USER4”
hash(“USER4”);
hash(“USER8”);
ENTER RINGPOP: CONSISTENT HASHING
LOSING CAPACITY
hash(“USER1”);
hash(“USER5”);
hash(“USER4”);
hash(“USER8”);
INST. A
INST. B
INST. C
ADDING CAPACITYENTER RINGPOP: CONSISTENT HASHING
INST. D
INST. A
INST. B
INST. C
hash(“USER1”);
hash(“USER5”);
hash(“USER4”);
hash(“USER8”);
APPLICATION LAYER MIDDLEWAREENTER RINGPOP: REQUEST ROUTING
HTTP / THRIFT / ETC
ROUTING
HASH RING
BUSINESS LOGIC
MEMBERSHIP
INST. A
{RINGPOP } PROCESSAPPLICATION
FRONT-END
STORAGE
GEOSPATIAL INDEXAPPLICATIONS OF RINGPOP
REPLICA 2
REPLICA 1
OWNER
INST. D
DISPATCH UPDATE DRIVER LOCATION
UPDATE
UPDATE
APPLICATIONS OF RINGPOPRINGPOP IN PRODUCTION
Geospatial sharding
Work delegation
Server-side push / long-polling
Caching
Aggregation
Mailboxes
Database
Service Discovery and Routing
Convergence
Cross-pollination
Flappy nodes
Hard to forget
Slow start times
Anti-entropy
Tooling
Backwards compatibility
PRODUCTIONLESSONS LEARNED
top related