How Big Data and Hadoop Integrated into BMC ControlM at CARFAX

© Copyright 4/7/2015 BMC Software, Inc1

Robert Stinnett (@robertstinnett)

CARFAX Automation AnalystOctober 14, 2014

We Came, We Saw, We Processed


How #BigData and #Hadoop integrated into @BMCControlM at CARFAXCARFAX helps millions of people buy, sell and service their used cars better.


Agenda

1. Workload Automation at CARFAX

2. Big Data and Hadoop Initiative

3. Reduce, Reuse, Recycle

4. Batches, Unite!


Support traditional batch,

data transfer and SLA/SLE

management across

various datacenters.

Everything under one

roof. Integrate with other

software packages to

create an Enterprise wide

workload management

system.

Capacity on demand.

React to the hyper-

growing business.

Workload Automation at CARFAX

1

Manage Integrate Scale


• 129,000+ processes a day

• 350 batch nodes

• 5 Different Datacenters

• 1 Unified Workload Management Platform


Data can be loaded and

live less than 30 minutes

after receipt

1 record to process today,

1 million records to

process tomorrow

Data is what makes

CARFAX who we are. We

can’t afford to “hope we

got it right”.

It’s All About The Data

#Fast #Dynamic #Reliable


Where Does it Come From?

34,000 Data Sources

13 billion records in our VHDB

Data comes in many formats, even pictures and PDFs!

CARFAX receives data in “any format, any time, any method”. We process more data in an hour than many businesses do in an entire month.


Our business is all about

data. We’ve been doing

Big Data long before it was

“cool”.

It had to integrate with

our currently business

processes. It is a vital part

of our data services and

will provide and consume

data from many other

applications.

We see Hadoop and other

Big Data initiatives

replacing many of our

legacy data processing

systems.

Big Data & Hadoop

2

#BigData #Integration #Future


Hadoop is still evolving and maturing. Thousands of pilot projects out there, but very few production installations.

We learned we were one of the pioneers when it came to integration.


Nobody wanted to

reinvent the wheel. We

didn’t need another

scheduling system.

Hadoop team wanted to

hit he ground running.

Reuse what they were

already familiar with.

Integrate with our existing

DevOps practices across

CARFAX. No silos!

Reuse, Reduce, Recycle

3

#DoMoreWithLess #TimetoMarket #DevOps


Agile Development +

Agile Operations

Increased usage of data center automation and configuration management tools

#DevOps


We’ve managed to free up

our personnel to do

awesome things, and let

the automation handle

the routine stuff.

Integration, more agile

operations, compliance

and remediation. These

are what we see on the

horizon.

This wasn’t a one person,

or one team project. It

was a whole company

initiative. It has been an

amazing journey, yet

we’ve only just begun.

Batches, Unite!

4

#Today #Tomorrow #AmazingJourney


Overall Integration Strategy

Basic Batch

File Transfer Under Control-M Control

Database Processing,Web Services

Bladelogic,Java

Informatica

Start “Run this script”SUMMARY Today “Manage our workloads” Future “Make it all just happen”

20102008 20132004 2012 Beyond

Hadoop,SAS, ServiceNOW,“Write Our Own”


#KeyTakeaways

1. Hadoop isn’t a silo, it’s part of other IT processes.

2. Significantly reduce your learning curve by using what you already have.

3. Integrations reduce management headaches.

4. Evolution from batch to workload


@robertstinnett

[email protected]

How Big Data and Hadoop Integrated into BMC ControlM at CARFAX

Technology

bmc software

data transfer

vhdb data

data sources

data services

carfax big data

devops copyright

big data hadoop2