Big data in the energy sector

Big data in the energy sectorBy John Tkaczewski

Agenda

| © Copyright FileCatalyst, 20142

The Big Data in Oil & GasThe Big Data in Oil & Gasexploration exploration datadata

drilling drilling datadata

production production datadata

Data Supply ChainData Supply Chainsource to data centre for source to data centre for analysisanalysis

to to stakeholdersstakeholders

Case StudyCase StudyBig data movement from vessels Big data movement from vessels

to London data center to to London data center to global stakeholdersglobal stakeholders

© Copyright FileCatalyst, 2014

Data-driven Oilfields and the data we gather during production.


Digital Data gathered during exploration


Big data during exploration…• Logging While Drilling or Measurements While Drilling (LWD MWD)• Hole size and shape• structural analysis: Rock density and porosity• Acoustic scanning for rock faults• reservoir simulation• seismic analysis• fluid dynamics• Engineering and construction


... So we collected big data now what?

Need send it for further processing• Analysis• Simulation• Interpretation• High Performance Computing (HPC) Centers• Data Distribution to stake holders


Are we still shipping hard drives???


Or should we use the Internet!

- FTP, Email, File Sharing

- Cloud

- CIFS


• Last mile. Slower links between the operations and main data centers. Satellite, 3G/4G/LTE, Microwave – slower unreliable links

• High speed fat pipes between data centers. 100 Mbps fiber connections. The connections are fast but very far apart. (example Houston – London)

Sending big data – 2 fold problem


Traditional file transfers will remain slow, regardless of the size of the link.

For example:

Transferring a 10GB file from London to Houston can take several hours.

The slow file transfer in turn will affect production and exploration schedules.

Why traditional file transfers are so slow on the Internet?

• Latency (or delay on the network)

• Packet loss (percentage of data lost during transmission)

TCP Overview

What is TCP? Transmission Control Protocol• Works well for most internet traffic, email, web browsing

small ad-hoc transfers (HTTP, FTP, SFTP, SMTP etc)• Provides reliability, error checking, ordered packets in a

network stream• Congestion control built in• Internet could not survive without it


File Transfer Issues with TCP

• Flow control limits transmission window, causes “dead air” on the line when high latency present

• Very aggressive in response to network congestion

• Result is less than ideal performance on wireless, satellite, or long haul links

• TCP Can be tuned but still not ideal


TCP is a very serial process. Each packet of data must be received before a new packet is sent = Decreased transfer speed

Source FileDestination

File

Acknowledgments

Data Packet

File transfer acceleration via UDP

• Ideal for bulk file transfer• Predictable - Can send at a perfect rate• Not affected by latency or packet loss• Congestion Control implemented in application layer• Tunable congestion control aggression• Instantly detect link capacity


Source FileDestination

File

Acknowledgments

FileCatalyst completely saturates the pipe by sending multiple blocks of data = Increased transfer speed


*NA = North America

Bandwidth Improvement example

Accelerating File Transfers


• WAN optimization hardware

• Commercial off-the shelf (COTS) file transfer acceleration software

• Open source UDP based file transfer accelerations applications

File transfer acceleration options:


WAN optimizers

• Pros: • Almost Transparent• Optimizes all the network traffic including FTP• Hardware solution

• Cons: • Hardware solution• Cost• Lack of file transfer tools such as automation, monitoring

and QoS


File transfer COTS

• Pros: • Flexible software only deployment• Enterprise tools for transferring files are included• Web applications that include accelerated file transfer

• Cons: • Less expensive then hardware but not free• Limited to file transfers only• Some initial configuration required


Open Source file transfer applications

• Pros: • Free• Flexible software only deployment

• Cons: • Not adaptable to changing network conditions• no tools advanced tools (command line or API’s only)• Complex initial configuration required

Considerations for FTP Acceleration

• Plays fair with other traffic on network

• Top speed at 10 Gbps file transfer, single stream.

• Maintain high speed even with high latency and packet loss.

• High performance for many small files

• Security, reliability, automatic re-try and resume

• Automation tools, SDK, central management and configuration

• Rsync and Compression built-in



Case study: marine exploration

Project Requirements:

• Multiple ships delivering data in real time to London• London HPC center provides data analysis• Final data to be delivered to major markets in Houston and

Malaysia

Using FTP: • Data from ships was not deliverable• Final data deliveries taking hours or not arriving at all





Results:

• More control over file transfers and a dramatic speed improvement .

• 10 Gbps line was finally used to its full extent• All file transfers are a simple and straightforward experience,

logging all file transfers and exceptions• End user experience for both clients and back office processes

dramatically improved


Forecast for tomorrow

• Big data will only get bigger

• Global connectivity will get faster and more reliable

• Latency will remain as we can’t increase the speed of light

• Progressive tech will shorten the exploration cycle and ensure continuous success of oil and gas extraction

Questions?

Big data in the energy sector

Software

ideal copyright filecatalyst

packet of data

data centre

digital data

london data center

main data centers

datadriven oilfields

slow file transfer