Presenter: Christian Charette, John Tkaczewski Duration: 30 minutes May Webinar: 10Gbps transfers
Dec 29, 2015
Presenter Introduction
| © Copyright FileCatalyst, 20132
John Tkaczewski• Co-Founder and President of Unlimi-Tech
Software• Oversees creative and technical vision for
FileCatalyst Webmail and FileCatalyst Workflow
• Blog: http://www.filecatalyst.com/blog
Christian Charette• Director of Software Development• Project lead on 10 Gbps research
Agenda
• FileCatalyst Direct Overview • Quick refresher why TCP doesn’t work in high speed networks• 10Gbps test infrastructure
– Ie: what are we using to perform the demo?
• Watch the data fly!• Questions (please type them in as we go)
| © Copyright FileCatalyst, 20133
FileCatalyst Direct Overview
| © Copyright FileCatalyst, 20134
• Software architecture is Client-Server.• One side acts as the main file repository (FileCatalyst Server)• One or many clients connect to server, and push/pull data• Clients include:
– automated interfaces (API, CLI, Javascript, C++)– web components (applets)– feature rich applications (HotFolder, Express, mobile)
• Primarily written in Java, works on every major platform (Windows, Linux, Mac OSX, iOS, Android, and more)
• Transport protocol utilizes proprietary UDP, with error correction and congestion control built into the application layer (instead of relying on TCP network layer)
Putting 10 Gbps in context
| © Copyright FileCatalyst, 20135
• 10 Gbps transfers an average BlueRay disk (26GB) in under 30 seconds
• Cost of 10 Gbps dedicated lines is dropping – becoming cost effective to many companies (~10K/month US to UK)
• 1 Gbps fiber connections becoming available to residential consumers
TCP vs UDP for packet loss & latency
| © Copyright FileCatalyst, 20136
• Quick refresher: TCP can’t cut it. • Data provided as part of Photonics North 2012 presentation, FTP data uses FileZilla client
connecting to PureFTPD
TCP vs UDP for packet loss & latency
| © Copyright FileCatalyst, 20137
• TCP cannot handle packet loss -- even for networks with small latency.• High latency networks fall apart under real world conditions.
TCP vs UDP for packet loss & latency
| © Copyright FileCatalyst, 20138
• You cannot transfer files at 10 Gbps with TCP. It just doesn’t work.
10 Gbps transfers
| © Copyright FileCatalyst, 20139
• 10 Gbps research project started in May 2011.• Project completed 6 months later with Nov 2011 release of
FileCatalyst Direct v3.0, able to transfer at full 10gbps speed.• 1½ years of successful deployments of 2-10 Gbps in the field.• Tested in 2012 by independent researchers (ex-CEO of
competitor Tixel GmbH), validated as most stable transfer results between 5 managed transport solutions.Comparison of Contemporary Solutions for High Speed Data Transport on WAN 10 Gbit/s Connections, Dmitry Kachan, Eduard Siemens, Vyacheslav Shuvalov
What are we using to run this demo?
| © Copyright FileCatalyst, 201310
FileCatalyst HotFolderAres x.x.x.58
FileCatalyst ServerHermes x.x.x.59
What are we using to run these tests?
| © Copyright FileCatalyst, 201311
• Machines utilize standard components circa 2011.• Each machine costs approx. $5000 to put together.
– Intel Westmere 5600 series (used in demo testbed)– AMD Magny-Cours 6100 Opterons
FileCatalyst HotFolderAres x.x.x.58
Dual Intel Xeon 569048 GB of RAM(36 GB as RAMDRIVE ) Intel 10Gbps cards
Ubuntu 10.04.1 LTSJava 1.7 64-bitFileCatalyst v3.2
FileCatalyst ServerHermes x.x.x.59
Dual Intel Xeon 569048 GB of RAM(36 GB as RAMDRIVE ) Intel 10Gbps cards
Ubuntu 10.04.1 LTSJava 1.7 64-bitFileCatalyst v3.2
What are we using to run these tests?
| © Copyright FileCatalyst, 201312
• Machine has 48GB total, 36GB reserved for RAMFS (so 12GB for OS + FileCatalyst).• For the purpose of these tests, going to read/write to RAM drive. • Why? Want to remove disks as part of the equation for demo to simplify things
(everybody has different IO solutions in place). • If your disks can handle the speeds, our software can transfer it.
FileCatalyst HotFolderAres x.x.x.58
Dual Intel Xeon 569048 GB of RAM(36 GB as RAMDRIVE ) Intel 10Gbps cards
Ubuntu 10.04.1 LTSJava 1.7 64-bitFileCatalyst v3.2
FileCatalyst ServerHermes x.x.x.59
Dual Intel Xeon 569048 GB of RAM(36 GB as RAMDRIVE ) Intel 10Gbps cards
Ubuntu 10.04.1 LTSJava 1.7 64-bitFileCatalyst v3.2
What are we using to run these tests?
| © Copyright FileCatalyst, 201313
• In demo, to simulate WAN environment, a Linktropy box from Apposite Technologies is used– Control link speed, packet loss, and link latency
FileCatalyst HotFolderAres x.x.x.58
Dual Intel Xeon 569048 GB of RAM(36 GB as RAMDRIVE ) Intel 10Gbps cards
Ubuntu 10.04.1 LTSJava 1.7 64-bitFileCatalyst v3.2
FileCatalyst ServerHermes x.x.x.59
Dual Intel Xeon 569048 GB of RAM(36 GB as RAMDRIVE ) Intel 10Gbps cards
Ubuntu 10.04.1 LTSJava 1.7 64-bitFileCatalyst v3.2
WAN Emulator Apposite Technologies
So what are we going to see?
| © Copyright FileCatalyst, 201314
FileCatalyst HotFolderAres x.x.x.58
Dual Intel Xeon 569048 GB of RAM(36 GB as RAMDRIVE ) Intel 10Gbps cards
Ubuntu 10.04.1 LTSJava 1.7 64-bitFileCatalyst v3.2
FileCatalyst ServerHermes x.x.x.59
Dual Intel Xeon 569048 GB of RAM(36 GB as RAMDRIVE ) Intel 10Gbps cards
Ubuntu 10.04.1 LTSJava 1.7 64-bitFileCatalyst v3.2
WAN Emulator Apposite Technologies
So what are we going to see?
| © Copyright FileCatalyst, 201315
• (1) Going to test most basic scenario: – Transfer static file from point A to B– 30GB file… slightly larger than the average Blue Ray disk.
FileCatalyst HotFolderAres x.x.x.58
Dual Intel Xeon 569048 GB of RAM(36 GB as RAMDRIVE ) Intel 10Gbps cards
Ubuntu 10.04.1 LTSJava 1.7 64-bitFileCatalyst v3.2
FileCatalyst ServerHermes x.x.x.59
Dual Intel Xeon 569048 GB of RAM(36 GB as RAMDRIVE ) Intel 10Gbps cards
Ubuntu 10.04.1 LTSJava 1.7 64-bitFileCatalyst v3.2
WAN Emulator Apposite Technologies
So what are we going to see?
| © Copyright FileCatalyst, 201316
• (2) Going to test standard live media scenario: – Growth files (5) simulate live video feeds saved to system– Move files in real-time to destination as files are output
FileCatalyst HotFolderAres x.x.x.58
Dual Intel Xeon 569048 GB of RAM(36 GB as RAMDRIVE ) Intel 10Gbps cards
Ubuntu 10.04.1 LTSJava 1.7 64-bitFileCatalyst v3.2
FileCatalyst ServerHermes x.x.x.59
Dual Intel Xeon 569048 GB of RAM(36 GB as RAMDRIVE ) Intel 10Gbps cards
Ubuntu 10.04.1 LTSJava 1.7 64-bitFileCatalyst v3.2
WAN Emulator Apposite Technologies
What are we using to run these tests?
| © Copyright FileCatalyst, 201317
• Configurations used for single file transfer @ 10gbps:– Server & HotFolder each using 4GB of RAM– MTU 9000 (packet 8972 bytes)– Blocks: 10MB– Number of sender threads: 50– Sender sockets: 5– Receiver sockets: 3– RTT congestion control (because test scenario utilizes packet loss)– Congestion control set to 7 (more aggressive)
• Linktropy setup to simulate LA -> NY transfers– Latency of 60 ms– 0.1% packet loss
Upcoming Events:
- NYC Digital Asset Managers, (May 29th)
- Broadcast Asia, Singapore (June 18-21)
- IBC, Amsterdam (Sept 13-17)
- Interop New York (Oct 2-3)
- Broadcast India, Mumbai (Oct 9-11)
And of course don’t forget our monthly webinar....