BigTop vm and docker provisioner

BigTopVM and Docker

ProvisionerEvans Ye

2015.2.17

Apache Bigtop is a project for…

• Packaging• Packages Hadoop ecosystem

• Testing• integration/interoperability testing framework

• Virtualization• vagrant/docker based hadoop provisioning

• Deployment• puppet recipes for included components

What does vagrant-puppet do?

1) Create a set of VMs by vagrant box

2/17/2015

Vagrant Box(Centos, Ubuntu,…)

VirtualboxVMs

2) Install puppet agents

2/17/2015

3) Deploy Hadoop by bigtop puppet (masterless)

2/17/2015

puppet apply -d--modulepath="bigtop-deploy/puppet/modules:/etc/puppet/modules" \bigtop-deploy/puppet/manifests/site.pp

puppet apply…

puppet apply…

puppet apply…

One click auto deployment

2/17/2015

$ cd bigtop-deploy/vm/vagrant-puppet-vm$ vagrant up

And the docker-puppet?

Similar to vagrant-puppetbut spin up a cluster on top of Linux containers

2/17/2015

$ cd bigtop-deploy/vm/vagrant-puppet-docker$ ./docker-hadoop.sh --build-image --create 3

Some prerequisites (see each README.md)vagrant-puppet docker-puppet

2/17/2015 Confidential | Copyright 2013 TrendMicro Inc. 2

Use cases

• for BigTop• to do bigtop integration tests

• to test bigtop puppet recipes

• integrates packaging, deployment, and testing all into one package(demonstrate the BigTop life cycle)

• for Contributors & Users• run round trip test locally (from source code to a testing cluster)

• easy-to-get Hadoop cluster environment

2/17/2015 Confidential | Copyright 2013 TrendMicro Inc. 2

Execute BigTop smoke-tests

• vagrant-puppet• README.md

• vagrantconfig.yaml• set run_smoke_tests: true

• Vagrantfile

• docker-puppet• ./docker-hadoop.sh --smoke-tests

2/17/2015 Confidential | Copyright 2013 TrendMicro Inc. 2