8/14/2019 Joost Network Architecture
1/29
mailto:[email protected]:[email protected]8/14/2019 Joost Network Architecture
2/29
8/14/2019 Joost Network Architecture
3/29
8/14/2019 Joost Network Architecture
4/29
Joost Network Architecture
Ground rules
No firewalls No hardware load-balancers
High availability (this is TV)
Lots of bandwidth (this is TV)
Ethernet only
Rapidly provisionable
Business requirements
Cost-effective
4 main service topologies ...
8/14/2019 Joost Network Architecture
5/29
Joost P2P Magic
Video Streaming
8/14/2019 Joost Network Architecture
6/29
Joost P2P Magic
Joost servers are the original seedersof all content
Joost servers also handle the long-
tail (which is still pretty long)
Joost servers top-up the DSLbandwidth gap
8/14/2019 Joost Network Architecture
7/29
Joost P2P Magic
A long-tail server cluster is 11 servers 1 control server
10 long-tail servers
2x Cisco 3560s (crypto image) HSRP
EBGP
802.1q
OSPF (for anycast services)
8/14/2019 Joost Network Architecture
8/29
Joost P2P Magic
Each cluster gets 1Gbit/sec of IP transit
/26 from transit provider PA space were not sensitive to re-numbering
avoids prefix-filtering and dampening
Each cluster is an island with noconnectivity back to Joost HQ apart froman IP tunnel for server lights-outmanagement
Preference is to add sites when scaling
Decreases average latencies
8/14/2019 Joost Network Architecture
9/29
Joost P2P Magic
8/14/2019 Joost Network Architecture
10/29
Joost P2P Magic
Pros Highly cost-effective for 1Gbit/sec of
usable, resilient, bandwidth
No multi-gig network scalability
issues, just add another cluster Rapidly provisionable, almost
anywhere
Cons May make future peerings pointless
3560s cant hold a full table, or doNetflow
8/14/2019 Joost Network Architecture
11/29
Joost P2P Magic
UDP-based (Port 33333)
Client will perform STUN, ICE inprogress.
TCP congestion control would kill anyvideo. Buffering usually significantlyincreases bandwidth usage.
Packets are generally 1k in size.
8/14/2019 Joost Network Architecture
12/29
Joost P2P Magic
Client first contacts super-node, whichhandles control traffic only, and directclients to peers. Peers are re-negotiated frequently.
Each video stream comes frommultiple peers, with FEC to handle livepeer loss.
1 hour of video
8/14/2019 Joost Network Architecture
13/29
8/14/2019 Joost Network Architecture
14/29
Joost P2P Magic
p2p code is prefix aware now, willprefer peers in same /24, /16 etc ...
adding real AS-level awareness is in
progress.
latency-based decisions aresomething to watch out for.
8/14/2019 Joost Network Architecture
15/29
Joost P2P Magic
30ms50ms
15msAB
C
What if asupernodeat C has tocoordinate
A and B?
This makesus highly
sensitive tolatency.
8/14/2019 Joost Network Architecture
16/29
HTTPS
Backend services (search ...)
8/14/2019 Joost Network Architecture
17/29
Joost Backend Services
Accessed via HTTPS
Provided using Apache Lucene, Hadoop,and many internally-developed services
Each IP is a wack-a-mole virtual-ip
Geographic fail-over provided by DNS
8/14/2019 Joost Network Architecture
18/29
HTTP, SMTP, DNS
General Internet Services
8/14/2019 Joost Network Architecture
19/29
Joost General Services
Joost.com website and e-mail
Content Owner Website (COW)
Provided by resilient servers withwackamole
Geographic fail-over provided by DNS
8/14/2019 Joost Network Architecture
20/29
Joost Internal Services
Recursive DNS
Syslog
Some authentication services
All provisioned via IP anycast, at each siteinclude long-tail cluster sites (the
addresses are internally reachable only).
8/14/2019 Joost Network Architecture
21/29
Super-secretTranscoding
DAV, SFTP, FASP,FEDEX
Content Ingestion
8/14/2019 Joost Network Architecture
22/29
Content Ingestion
Content has to get to Leiden
Currently investigating various networkmedia-delivery options for this
Transcoded in Leiden, and then sent toLuxembourg, and onward to all long-tailserver sites
Content-owner website for meta-data
8/14/2019 Joost Network Architecture
23/29
Leiden
AS5400
AS5400
All routing is viaLuxembourg
8/14/2019 Joost Network Architecture
24/29
Joost Benelux
Main Joost location is the Joost BeneluxNetwork
Hosts Joost Leiden office, Luxembourg
Datacentre, Primary Long-tail serversite, Primary back-end site
89.251.0.0/20 (deaggregates during
some outages)
AS42072
8/14/2019 Joost Network Architecture
25/29
8/14/2019 Joost Network Architecture
26/29
Joost Benelux
Routers are Cisco 7301 Can do a Gig with 1k packets, justabout
Full netflow support, anonymisedvia CryptoPan, for detailed analysisof p2p network performance
Using OSPF as IGP
8/14/2019 Joost Network Architecture
27/29
Joost Network Management
RT and JIRA for tickets
RANCID-SVN for config management
NAGIOS and syslog for monitoring
MRTG and Cricket for graphing
SSH only
8/14/2019 Joost Network Architecture
28/29
So, what can we do for ISPs?
Were willing to peer, but is there muchpoint? Only portions of the long-tail arepeerable.
In-ISP Long-tail servers?
Were L3 and up, not much we can do aboutthe last-mile
Any promising revenue share models?
8/14/2019 Joost Network Architecture
29/29
Any Questions?