Automating file transfers January 2015 webinar

Webinar: Things to consider when automating file transfers

January 15, 2015

News flash: Unlimi-Tech wins a Emmy

| © Copyright FileCatalyst, 20152

Introduction

• John Tkaczewski, President and Co-Founder

• Check out our blog: http://www.filecatalyst.com/blog

| © Copyright FileCatalyst, 20153

Overview

• FileCatalyst Introduction

• Challenges with Automated Workflows

• Demo HotFolder

• Final Remarks

• Questions

| © Copyright FileCatalyst, 20154

What is File Transfer Acceleration?

• File transfer acceleration technology allows large files, or “big data” to be sent quickly over large geographical distances without being impeded by network impairments such as latency and packet loss

• This is accomplished by transferring files using the UDP protocol instead of the TCP protocol, the protocol used for FTP

• The main difference between UDP and TCP is that UDP wastes no time communicating about the receipt of a block of data so that regardless of network latency, data transmission remains constant, thus overcoming the peak and valley effect that occurs with TCP based protocols on links with high latency

• Reliability and congestion control is added to UDP at the application layer thus it doesn’t sacrifice its other desirable properties

| © Copyright FileCatalyst, 20155

Speed Gains with Acceleration vs. FTP

| © Copyright FileCatalyst, 20156

Automation, there is nothing to it!

Setup Cron Job using SCP… and I’m done right?

…. Well Perhaps Not!

| © Copyright FileCatalyst, 20157

Challenges of Automated Workflows

• Volume of data

• System Notifications (message fatigue)

• System properties (system clock, mounts, network drives)

• Interpreting logs and finding errors

• Folder Synchronization

• Cross platform transfers, ex: Windows to Linux

• Transfer Speed

• Data Verification (Checksum)| © Copyright FileCatalyst, 20158

Automation tools within FileCatalyst

9

HotFolder

© Copyright FileCatalyst, 2015

Command Line

Automation tools within FileCatalyst (Cont’d)

Hotfolder features useful for data replication:• Scheduler for tasks and bandwidth• File System Events• Delta (RSYNC)• Data-set rules• Monitoring and notifications• ZIP chunking (Single Archive)• MD5 • File transfer cache• Multi-Client (multiple concurrent file transfers)

| © Copyright FileCatalyst, 201510

Demo

Platform specific gotchas

Common file attributes for all platforms:

File name, Date Created, Date Modified, Owner

| © Copyright FileCatalyst, 201512

Operating System File Attributes

MacOS Extended attributes (xattr), Resource Forks, Symlinks, Permissions (Read Write Execute), Access Control Levels (ACL’s)

Windows Permissions, Attributes: Read Only – Archive - Hidden – System, Security Attributes

Linux Permissions (group, user, other)

Data replication across operating systems

• Extended File Attributes are LOST when moving a file between operating systems.

• But it’s OK because extended attributes may not be always needed.

| © Copyright FileCatalyst, 201513

File system events

The file system events provides a way for an application to ask for notification when the contents of a directory hierarchy are modified.

For example, an application can use this to quickly detect when the user modifies a file within a folder using another application.

• Avoid scanning large folders to detect changes! Use file system events

• File system events only work on local drives (no SMB) Most Enterprise network drive technology does support file system events

• Transfer the initial file set without file system events, THEN turn it on.

| © Copyright FileCatalyst, 201514

Upcoming Events

- FileCatalyst User Forum, London, UK (around BVE)

- BVE 2015, Excel, London, UK (Feb 24-26), stand R40

- CABSAT 2015, Dubai World Trade Centre (March 10-12), stand 220

- NAB Show 2015, Las Vegas, Nevada (April 13-16), stand SL8505

And of course don’t forget our monthly webinars... And …

… our email newsletter “The Transfer Express”

| © Copyright FileCatalyst, 201515

Thank you!

Questions?