Top Banner
Ironfan: Build a Hadoop Cluster in 20 Minutes Dhruv Bansal @dhruvbansal Nathaniel Eliot @temujin9 Saturday, May 19, 2012
27

Ironfan: Build a Hadoop Cluster in 20 minutes

May 15, 2015

Download

Business

temujin9

The slides from our talk at ChefConf 2012
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Ironfan: Build a Hadoop Cluster in 20 minutes

Ironfan: Build a Hadoop Cluster in 20 Minutes

Dhruv Bansal@dhruvbansal

Nathaniel Eliot@temujin9

Saturday, May 19, 2012

Page 2: Ironfan: Build a Hadoop Cluster in 20 minutes

Saturday, May 19, 2012

Page 3: Ironfan: Build a Hadoop Cluster in 20 minutes

Outline

Saturday, May 19, 2012

Page 4: Ironfan: Build a Hadoop Cluster in 20 minutes

Outline

• Why we love Chef• Why we don’t• Why we wrote Ironfan

Why Ironfan?

Saturday, May 19, 2012

Page 5: Ironfan: Build a Hadoop Cluster in 20 minutes

Outline

• Why we love Chef• Why we don’t• Why we wrote Ironfan

Why Ironfan?

• Build a Hadoop cluster• Run a Hadoop job• Put that cluster away

Ironfan in Action

Saturday, May 19, 2012

Page 6: Ironfan: Build a Hadoop Cluster in 20 minutes

Outline

• Why we love Chef• Why we don’t• Why we wrote Ironfan

Why Ironfan?

• Build a Hadoop cluster• Run a Hadoop job• Put that cluster away

Ironfan in Action

• Better environments• Cloud agnosticism• Continuous integration• Stacks

Ironfan roadmap

Saturday, May 19, 2012

Page 7: Ironfan: Build a Hadoop Cluster in 20 minutes

Why we love Chef

Infrastructure as code:

Version control

Shareable Recapitulable

Testable

Saturday, May 19, 2012

Page 8: Ironfan: Build a Hadoop Cluster in 20 minutes

MySQL Nginx SOLR

My Application

Modular, composable units:

Why we love Chef

Saturday, May 19, 2012

Page 9: Ironfan: Build a Hadoop Cluster in 20 minutes

Why we love Chef

Saturday, May 19, 2012

Page 10: Ironfan: Build a Hadoop Cluster in 20 minutes

Why we don’t love Chef

Anything is possible

Nothing is simple

And there’s much repetition (not DRY)

Saturday, May 19, 2012

Page 11: Ironfan: Build a Hadoop Cluster in 20 minutes

Why we don’t love Chef

Hard Coded Dynamic

Too much is hard-coded at development/upload time!

Write Upload

Development Machine

Compile Converge

Managed Server

Saturday, May 19, 2012

Page 12: Ironfan: Build a Hadoop Cluster in 20 minutes

Why we don’t love Chef

How do we make @server_ips dynamic?

Saturday, May 19, 2012

Page 13: Ironfan: Build a Hadoop Cluster in 20 minutes

We wrote Ironfan to

Enter Ironfan

Saturday, May 19, 2012

Page 14: Ironfan: Build a Hadoop Cluster in 20 minutes

We wrote Ironfan to

Enter Ironfan

• Simplify, unify, and standardize our usage of the Chef toolset

Saturday, May 19, 2012

Page 15: Ironfan: Build a Hadoop Cluster in 20 minutes

We wrote Ironfan to

Enter Ironfan

• Simplify, unify, and standardize our usage of the Chef toolset

• Build further abstractions on top of Chef

Saturday, May 19, 2012

Page 16: Ironfan: Build a Hadoop Cluster in 20 minutes

We wrote Ironfan to

Enter Ironfan

• Simplify, unify, and standardize our usage of the Chef toolset

• Build further abstractions on top of Chef

• Give us superpowers that Chef doesn’t have yet

Saturday, May 19, 2012

Page 17: Ironfan: Build a Hadoop Cluster in 20 minutes

We wrote Ironfan to

Enter Ironfan

• Simplify, unify, and standardize our usage of the Chef toolset

• Build further abstractions on top of Chef

• Give us superpowers that Chef doesn’t have yet

http://github.com/infochimps-labs/ironfan

Saturday, May 19, 2012

Page 18: Ironfan: Build a Hadoop Cluster in 20 minutes

What does Ironfan do?

Saturday, May 19, 2012

Page 19: Ironfan: Build a Hadoop Cluster in 20 minutes

What does Ironfan do?

ChefSaturday, May 19, 2012

Page 20: Ironfan: Build a Hadoop Cluster in 20 minutes

What does Ironfan do?

Simple helpers in the silverware cookbook abstract common Chef patterns and keep things DRY

Ironfan

ChefSaturday, May 19, 2012

Page 21: Ironfan: Build a Hadoop Cluster in 20 minutes

What does Ironfan do?

Saturday, May 19, 2012

Page 22: Ironfan: Build a Hadoop Cluster in 20 minutes

What does Ironfan do?

Dynamic service discovery:

Saturday, May 19, 2012

Page 23: Ironfan: Build a Hadoop Cluster in 20 minutes

What does Ironfan do?

Dynamic service discovery:

Saturday, May 19, 2012

Page 24: Ironfan: Build a Hadoop Cluster in 20 minutes

What does Ironfan do?

Saturday, May 19, 2012

Page 25: Ironfan: Build a Hadoop Cluster in 20 minutes

What does Ironfan do?

A simple DSL for defining clusters of machines

Saturday, May 19, 2012

Page 26: Ironfan: Build a Hadoop Cluster in 20 minutes

DEMO TIME!

Saturday, May 19, 2012

Page 27: Ironfan: Build a Hadoop Cluster in 20 minutes

Thanks for coming!

Learn more about what Ironfan and Infochimps can do for you.

[email protected]

Come work at a fast-growing, cutting-edge technology company based in Austin, TX.

[email protected]

http://github.com/infochimps-labs/ironfan

http://github.com/infochimps-labs/wukong

Saturday, May 19, 2012