DevOpsDays Amsterdam Cosmic workshop

@remibergsma @neubauerf

Workshop Build your own Cloud in record time with Cosmic

June 29th, 2016Amsterdam, NL

http://twitter.com/remibergsma

http://twitter.com/neubauerf

https://github.com/MissionCriticalCloud/cosmic


Cosmic Cloud delivers high quality Cloud Orchestration software to run Mission Critical workloads.

What is Cosmic?





What is Cosmic?



http://www.networkworld.com/article/3028065/hybrid-cloud/schuberg-philis-forks-cloudstack-what-happens-next.html

https://cupfighter.net/2016/01/welcome-mission-critical-cloud


cosmiccloud.io



http://cosmiccloud.io


What is Cosmic?

Lots of fast paced releases. 5.1.0.0 marks a milestone.



https://cupfighter.net/2016/06/cosmic-5-1-released


Cosmic projects

Cosmic Cloud Orchestrating SoftwarePackaging for Cosmic

SystemVM template Packer builderThe Bubble - easy integration testing

Toolkit for The Bubble




https://github.com/MissionCriticalCloud/packaging

https://github.com/MissionCriticalCloud/systemvm-packer

https://github.com/MissionCriticalCloud/bubble-cookbook

https://github.com/MissionCriticalCloud/bubble-toolkit



Cloudy building blocks

To build a Cloud we need: - primary storage - secondary storage - zone | pod | cluster - at least 1 hypervisor




Get a copy of the slides to copy/paste the commands

Getting the slides

Slides are here: http://www.slideshare.net/remibergsma



http://www.slideshare.net/remibergsma


Workshop environment

You will get a Bubble, hosted in the Schuberg Philis Mission Critical Cloud.

Remember: you can later create your very own Bubble with the

provided cookbooks and scripts.



https://www.schubergphilis.com/

https://github.com/MissionCriticalCloud/bubble-cookbook

https://github.com/MissionCriticalCloud/bubble-toolkit


Workshop environment

L0: KVM hypervisor in Schuberg Philis Cloud

L1: devops-cosmic0x hypervisor

L2: KVM hypervisors

systemvm

L3: uservm

L2: KVM hypervisors

systemvm

L3: uservm

Cosmic Mgt server

Services in box: VPN

NFS / WEB SSH

Ipsec VPN connection to 85.222.238.190

SSH to 192.168.22.1 over vpn

Management network: 192.168.22.0/24Public network : 192.168.23.0/24

192.168.22.1

All L3 VMs reachable via public network

All L2 VMs reachable via mgt network

Other VMs if needed




Bubbles available:devops-cosmic01 devops-cosmic06

devops-cosmic02 devops-cosmic07







Workshop VPN

Remote IP: 85.222.238.190 username: vpn@cosmicXX (01 to 12)password: <please ask> Preshared key: DevopsDays2016

Guide for MacOSX Guide for Windows



https://www.softether.org/4-docs/2-howto/9.L2TPIPsec_Setup_Guide_for_SoftEther_VPN_Server/5.Mac_OS_X_L2TP_Client_Setup

https://www.softether.org/4-docs/2-howto/9.L2TPIPsec_Setup_Guide_for_SoftEther_VPN_Server/4.Windows_L2TP_Client_Setup


Workshop Terminal / Xwindows

Windows: MobaXterm: Mac: Xquartz

Combined with Terminal.app



http://mobaxterm.mobatek.net/

https://www.xquartz.org


Workshop SSH config

Host cloudhostname 192.168.22.1ForwardX11 yesForwardX11Trusted yesForwardAgent yesStrictHostKeyChecking noUserKnownHostsFile=/dev/null

On OSX and Linux, add this to ~/.ssh/config:




Workshop Routes

sudo route add -net 192.168.23.0/24 gw 192.168.22.1

sudo route -n add -net 192.168.23.0./24 192.168.22.1

route add 192.168.23.0 mask 255.255.255.0 192.168.22.1

Linux:

Mac OSX:

Windows:




Connecting…

ssh 192.168.22.1

byobu-enable <enter> to confirm




Automation!




Automation!

We’ll show you some manual steps, that have been automated. For the fun of it, and to learn a thing or two, we’ll show you the steps ;-)




Prepare virtual infra

Press <F2> for a new tab, then type: virt-manager

$ /data/shared/deploy/kvm_local_deploy.py -m /data/shared/marvin/mct-zone1-kvm1-kvm2-v2.cfg --force




Prepare virtual infraNote: kvm1: Running pre_boot script: virt-customize -d kvm1 --firstboot /data/shared/deploy/default/firstboot/centos7-kvm-ovs.sh[ 0.0] Examining the guest ...Note: kvm2: Running pre_boot script: virt-customize -d kvm2 --firstboot /data/shared/deploy/default/firstboot/centos7-kvm-ovs.sh[ 0.0] Examining the guest ...Note: cs1: Running pre_boot script: virt-customize -d cs1 --firstboot /data/shared/deploy/default/firstboot/centos7-cloudstack-dev.sh[ 0.0] Examining the guest ...[ 10.0] Setting a random seed[ 10.0] Installing firstboot script: [1] /data/shared/deploy/default/firstboot/centos7-kvm-ovs.sh[ 10.0] Finishing offNote: kvm1: Running postboot script: /data/shared/deploy/default/postboot/post_detect_reboot.sh kvm1Note: kvm1: Waiting for the VM to boot...[ 11.0] Setting a random seed[ 11.0] Installing firstboot script: [1] /data/shared/deploy/default/firstboot/centos7-kvm-ovs.sh[ 10.0] Setting a random seed[ 10.0] Installing firstboot script: [1] /data/shared/deploy/default/firstboot/centos7-cloudstack-dev.sh[ 11.0] Finishing off[ 10.0] Finishing offNote: kvm2: Running postboot script: /data/shared/deploy/default/postboot/post_detect_reboot.sh kvm2Note: kvm2: Waiting for the VM to boot...Note: cs1: Running postboot script: /data/shared/deploy/default/postboot/post_detect_reboot.sh cs1Note: cs1: Waiting for the VM to boot...Note: kvm1: Installing and configuringNote: kvm1: This will take some time. You may send this to the background.Note: kvm2: Installing and configuringNote: kvm2: This will take some time. You may send this to the background.Note: cs1: Installing and configuringNote: cs1: This will take some time. You may send this to the background.Note: kvm1: RebootingNote: kvm2: RebootingNote: kvm1: Ready for duty!Note: kvm2: Ready for duty!Note: cs1: RebootingNote: cs1: Ready for duty!




[root@mct-devops1 deploy]# virsh listId Name State---------------------------------------------------- 3 cs1 running 4 kvm1 running 5 kvm2 running

Your cloud for today

KVM hypervisors

Cosmic management

Give it some time to finish the setup..




[root@devops-cosmic99 deploy]# ssh root@cs1 (password=password)

[root@cs1 ~]# cd /data/shared/helper_scripts/cosmic [root@cs1 ~]# ./prepare_cosmic_compile.sh

Prepare Mgt Server

This will setup all dependencies




[root@cs1 ~]# cd /data/git/$HOSTNAME/cosmic[root@cs1 ~]# cd /data/shared/helper_scripts/cosmic[root@cs1 ~]# ./build_run_deploy_test.sh -m /data/shared/marvin/mct-zone1-kvm1-kvm2-v2.cfg

Compile Cosmic mgt

Compile in done!

The UI should be up after a few minutes, available from your Mac: http://cs1.cloud.lan:8080/client/

[INFO] ------------------------------------------------------------------------[INFO] BUILD SUCCESS[INFO] ------------------------------------------------------------------------[INFO] Total time: 6:24.846s[INFO] Finished at: Tue Jun 28 17:15:15 CEST 2016[INFO] Final Memory: 92M/351M[INFO] ------------------------------------------------------------------------

Compile Cosmic, Build RPM packages for KVM, Start Mgt server, Deploy Cloud



cs1.cloud.lan:8080/client/


Deploy your CloudDeploy data center..[INFO] 2016-06-28 17:45:45,452 - marvin - Parsing Test data successful[INFO] 2016-06-28 17:45:45,719 - marvin - === Test Client Creation Successful ===[INFO] 2016-06-28 17:45:45,728 - marvin - === Deploy DC Started ===[INFO] 2016-06-28 17:45:46,191 - marvin - === Create Zone Successful ===[INFO] 2016-06-28 17:45:51,409 - marvin - === Creating Physical Network Name : mcct-pnet Id : 6d4920bd-a52a-49a4-91df-826f3d37ee85 Successful ===[INFO] 2016-06-28 17:46:16,913 - marvin - === Update Network Service Provider Successfull ===[INFO] 2016-06-28 17:46:27,086 - marvin - === Update Network Service Provider Successfull ===[INFO] 2016-06-28 17:46:37,270 - marvin - === Update Network Service Provider Successfull ===[INFO] 2016-06-28 17:47:46,764 - marvin - === Add Host Successful ===[INFO] 2016-06-28 17:48:50,170 - marvin - === Add Host Successful ===[INFO] 2016-06-28 17:49:50,717 - marvin - === Creating Storage Pool Successful ===[INFO] 2016-06-28 17:49:50,881 - marvin - === Add Image Store Successful ===[INFO] 2016-06-28 17:49:50,932 - marvin - === Enable Zone Successful ===[INFO] 2016-06-28 17:49:50,935 - marvin - === Data Center Settings are dumped to dc_entries_Jun_28_2016_17_49_50.obj ===[INFO] 2016-06-28 17:49:50,942 - marvin - === Deploy DC Successful ===Tue Jun 28 17:49:50 CEST 2016Checking template status..Tue Jun 28 17:56:06 2016 At least template 'tiny linux kvm' is not Readyhttp://localhost:8096/client/api?command=listTemplates&templatefilter=all&response=jsonTue Jun 28 17:56:21 2016 All templates are ready!TrueTue Jun 28 17:56:22 CEST 2016Not running tests (use -t flag to run them)FinishedTue Jun 28 17:56:22 CEST 2016

Fully automatic….




Cosmic UI

user: adminpass: passworddomain: <empty>




Check Cosmic UI




Check Cosmic UI

Wait until the systemVMs are ready. This may take ~5 minutes.




Check Cosmic UI




Starting your first VM




TemplatesThe only template available is ‘Tiny’, which is a very small template.

It is registered without a password service.

The default user/pass is root/password.




Template properties

Let’s edit the template properties and check the ‘password enabled’ checkbox.

Click to save.




Copy/paste to use later on (root passwd)

Random password




Console Proxy VM serves this console

Accessing the console




Global settings

We may want to bump this setting to 12

We’ll show you how to make that easy..




We started the Mgt server using Jetty.Cosmic is a Java app. So, let’s stop Jetty and start Tomcat.

That makes stop/start easier :-)

$ ssh cs1 $ killall -9 java $ ps aux | grep java $ cp /data/git/cs1/cosmic/cosmic-client/target/cloud-*.war ~tomcat/webapps/client.war $ systemctl start tomcat

We’ll drop the war we compiled before into Tomcat

It’s Java!

Make sure all is gone..




It’s Java!

[root@cs1 webapps]# systemctl status tomcat● tomcat.service - Apache Tomcat Web Application Container Loaded: loaded (/usr/lib/systemd/system/tomcat.service; disabled; vendor preset: disabled) Active: active (running) since Tue 2016-06-28 20:58:10 CEST; 11min ago

$ systemctl status tomcat

The UI / API will be back in a few minutes…

$ tail -n 100 /var/log/cosmic/management/management-server.log




Copy/paste to use later on (root passwd)

Random (longer) password




Metrics




Forwarding a port

We’ll set up port forwarding

Click Networking, select your network, then IP Addresses

We’ll open the port in the firewall




Forwarding a port

Fill-in this form to forward SSH

Result looks like this

Fill-in this form to open firewall for SSH

Result looks like this




SSH to your VM

Test from your laptop or from devops box




Using the API




Learn API calls using FireBug

API docs:https://cloudstack.apache.org/api/apidocs-4.4/TOC_Root_Admin.html

https://getfirebug.com/releases/lite/chrome/

Use this icon to start FireBug

Install using this page



https://cloudstack.apache.org/api/apidocs-4.4/TOC_Root_Admin.html

https://getfirebug.com/releases/lite/chrome/


Play with CloudMonkeyIt has been pre-setup on your cs1 VM

(local) 🐵 > set display table(local) 🐵 > list virtualmachines filter=name,state,hypervisor,templatename,instancename,count = 6virtualmachine:+------------+---------+--------------+----------------------+----------------+| hypervisor | state | instancename | name | templatename |+------------+---------+--------------+----------------------+----------------+| KVM | Running | i-2-10-VM | devops006 | tiny linux kvm || KVM | Running | i-2-8-VM | devops005 | tiny linux kvm || KVM | Running | i-2-7-VM | devops-with-password | tiny linux kvm || KVM | Running | i-2-6-VM | devops003 | tiny linux kvm || KVM | Running | i-2-5-VM | devops002 | tiny linux kvm || KVM | Running | i-2-3-VM | devops001 | tiny linux kvm |+------------+---------+--------------+----------------------+----------------+(local) 🐵 >

(local) 🐵 > list networks filter=name,networkdomain,vlan,cidrcount = 1network:+-------------+---------------+------------+------+| cidr | networkdomain | name | vlan |+-------------+---------------+------------+------+| 10.1.1.0/24 | cs2cloud | devopsdays | 115 |+-------------+---------------+------------+------+(local) 🐵 >




Deploy with CloudMonkey

Deploy another VM in the same network, using CloudMonkey.

Exercise:

TIP: You may want to deploy one via UI first and watch FireBug send API calls…

TIP: You can do this by sending just one API call.

<tab><tab>




CloudStackOps:

Operating CloudStackmade easy.

https://github.com/schubergphilis/cloudstackOps



https://github.com/schubergphilis/cloudstackOps


CloudStackOps[root@cs1 ~]# cd /data/shared/helper_scripts/cosmic[root@cs1 cloudstack]# ./setup_cloudstackOps.sh[root@cs1 cloudstack]# source /data/git/cs1/python_cloud/bin/activate[root@cs1 cloudstack]# cd /data/git/cs1/cloudstackOps[root@cs1 cloudstack]# python listVirtualMachines.py -o MCCT-KVM-1




More exercisesSetup multiple tenants

Create a new network from the API

Explore VPC

Add another cluster to Cosmic

Create HA service offering and apply it to Tiny VM

What happens when a hypervisor crashes?

…… Have fun!



Get in touch!

[email protected]

@remibergsma

[email protected]

@neubauerf

http://blog.remibergsma.com

mailto:[email protected]?subject=


mailto:[email protected]?subject=


DevOpsDays Amsterdam Cosmic workshop

Internet