Brezular's Technical Blog My worthless personal blog about networking Part4 – OPENVSWITCH – Playing with Bonding on Openvswitch DECEMBER 4, 2011 15 COMMENTS ( HTTP://BREZULAR.WORDPRESS.COM/2011/12 /04/PART4-OPENVSWITCH-PLAYING-WITH-BONDING-ON-OPENVSWITCH/#COMMENTS ) i 1 Vote B onding ( h,p://en.wikipedia.org/wiki/Channel_bonding ) is aggregation multiple links to single link in order to increase throughput and achieve redundancy in case one of links fails. At least, officials say it so it must be true. Now, let me ask the question. How can we configure bonding? From what I know about bonding, they are two possibilities to configure it. T he first option ( h,p://www.linuxfoundation.org/collaborate/workgroups/networking/bonding ) is to load bonding module to kernel and use ifenslave ( h,p://linux.die.net/man/8/ifenslave ) utility for its configuration. Configuration steps for Microcore are described here. h ,p://brezular.wordpress.com/2011/01/20/how-to-setup-linux-microcore-3-x-router-qemu-image- in-fedora-linux-part2/ ( h,p://brezular.wordpress.com/2011/01/20/how-to-setup-linux-microcore- 3-x-router-qemu-image-in-fedora-linux-part2/ ) If you want to have fun as I had with bonding configuration, check this lab for reference. h,p://brezular.wordpress.com/2011/03/10/etherchannelvrrp-dhcp-ospf-configuration-cisco-vya,a- microcore/ ( h,p://brezular.wordpress.com/2011/03/10/etherchannelvrrp-dhcp-ospf-configuration-cisco- vya,a-microcore/ ) As the tutorial should be dedicated to bonding on Openvswitch ( h,p://openvswitch.org/ ) I should focus on this topic. This is the second option how can we configure bonding without loading bonding module ( h,p://www.linuxfoundation.org/collaborate/workgroups/networking/bonding ) to Linux kernel. Let’s say we have two Microcore Linux ( h,p://distro.ibiblio.org/tinycorelinux/welcome.html ) boxes Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/ 1 de 48 02/10/2013 08:01 a.m.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Brezular's Technical Blog
My worthless personal blog about networking
Part4 – OPENVSWITCH – Playing with Bondingon Openvswitch
DECEMBER 4, 2011 15 COMMENTS (HTTP://BREZULAR.WORDPRESS.COM/2011/12/04/PART4-OPENVSWITCH-PLAYING-WITH-BONDING-ON-OPENVSWITCH/#COMMENTS)
i1 Vote
Bonding (h,p://en.wikipedia.org/wiki/Channel_bonding) is aggregation multiple links to single link inorder to increase throughput and achieve redundancy in case one of links fails. At least, officials say it so itmust be true. Now, let me ask the question. How can we configure bonding?
From what I know about bonding, they are two possibilities to configure it.
The first option (h,p://www.linuxfoundation.org/collaborate/workgroups/networking/bonding) is to loadbonding module to kernel and use ifenslave (h,p://linux.die.net/man/8/ifenslave) utility for itsconfiguration. Configuration steps for Microcore are described here.
As the tutorial should be dedicated to bonding on Openvswitch (h,p://openvswitch.org/) I should focuson this topic. This is the second option how can we configure bonding without loading bonding module(h,p://www.linuxfoundation.org/collaborate/workgroups/networking/bonding) to Linux kernel.
Let’s say we have two Microcore Linux (h,p://distro.ibiblio.org/tinycorelinux/welcome.html) boxes
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
1 de 48 02/10/2013 08:01 a.m.
connected with two links. As usual, our topology is created using GNS3 (h,p://www.gns3.net/) andQemu (h,p://en.wikipedia.org/wiki/QEMU) images of Microcore with installed Openvswitch 1.2.2 andQuagga (h,p://www.quagga.net/) 0.99.20 on the top of Microcore.
(h,p://brezular.files.wordpress.com/2011/12/topology.png)There is nothing in the world what couldprevent you to download my Openvswitch Microcore Qemu image available here.
As for Ethernet card I suggest you to use emulated i82557b Ethernet card. Hint – check GNS3 Qemuse,ing. According to Bill (h,p://en.wikipedia.org/wiki/Bill_Gates), 640 kB is enough for anyone but it isbe,er to assign at least 128 MB RAM for Microcore. Here is a snapshot of Qemu se,ing.
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
Part1 – Basic Microcore, Openvswitch and QuaggaConfiguration
Before we start to configure Openvswitch we should be aware of initial configuration requirement forOpenvswitch and Quagga. Frankly to say, Openvswitch is not Cisco IOS (h,p://en.wikipedia.org/wiki/Cisco_IOS) which you can configure right after device is booted. First you need(h,p://openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plain;f=INSTALL.Linux;hb=HEAD) tocreate database, configuration files, start services etc.
My Qemu Microcore image with Openvswitch and Quaggae is preconfigured for such things so feel freeto use it.
Unfortunately if we do it now, we cannot create bond0 interface, later in Part2. In this case a warningmessage informs us that eth0 and eth1 already exists on bridge br0. Therefore we skip a trunk definitionhere and we will configure the trunk later together with bonding configuration.
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
4 de 48 02/10/2013 08:01 a.m.
Switch2# exit
root@Switch2:~#
Save Microcore configuration.
root@Switch2:~# /usr/bin/filetool.sh -b
Ctrl+S
Part2 – Bonding
Now, it is time to start playing game called bonding. They are not any rules in our game but it is not badidea to read documentation first. Page number ten could answer many future question.
Now, start pinging IP address 192.168.10.2 from Switch1. Ping is supposed to be working. While pinging,start tcpdump to listen on interface ethernet1 of Switch2
root@Switch2:~# tcpdump -i eth1
tcpdump: WARNING: eth1: no IPv4 address assignedtcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on eth1, link-type EN10MB (Ethernet), capture size 68 bytes
11:47:07.214753 IP 192.168.10.2 > 192.168.10.1: ICMP echo reply, id 36360, seq 15, length 6411:47:08.215483 IP 192.168.10.2 > 192.168.10.1: ICMP echo reply, id 36360, seq 16, length 6411:47:09.215823 IP 192.168.10.2 > 192.168.10.1: ICMP echo reply, id 36360, seq 17, length 6411:47:10.216846 IP 192.168.10.2 > 192.168.10.1: ICMP echo reply, id 36360, seq 18, length 6411:47:11.217868 IP 192.168.10.2 > 192.168.10.1: ICMP echo reply, id 36360, seq 19, length 64
<output truncated>
15 packets captured15 packets received by filter0 packets dropped by kernel
Interface Ethernet1 sends icmp echo reply messages back to Switch1. As they are not any ICMP echorequests listed in the output, only the interface Ethernet0 receives these ICMP requests coming fromSwitch1.
My question is, what happens if we shutdown Ethernet0 on Switch2? Will be ICMP requests interruptedand possibly re-forwarded via interface Ethernet0 of Switch1? If yes, how long does it take?
root@Switch2:~# ifconfig eth0 down
Check the bond interface bond0 on Switch2 again. As soon as Switch2 detects failure of its interface,Openvswitch puts slave eth0 do disabled state
root@Switch2:~# ovs-appctl bond/show bond0
bond_mode: balance-slb
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
Switch1 cannot detect failure of far end interface Ethernet0. It continues to send ICMP requests toSwitch2 via its Ethernet0 interface. Stop ping command and check bond interface bond0 on Switch1.Obviously, Ethernet0 is still enabled in bundle
This behaviour we cannot tolerate. We need such as mechanism which helps to detect far end failure.Therefore LACP must be deployed in your configuration.
According to wiki (h,p://en.wikipedia.org/wiki/Link_Aggregation_Control_Protocol#Link_Aggregation_Control_Protocol), LACP works bysending frames (LACPDUs) down all links that have the protocol enabled. These are two mainadvantages comparing to static bonding.
1. Failover when a link fails when the peer will not see the link down. With static link aggregation the peerwould continue sending traffic down the link causing it to be lost.
2. The device can confirm that the configuration at the other end can handle link aggregation. With Staticlink aggregation a cabling or configuration mistake could go undetected and cause undesirable networkbehaviour.
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
7 de 48 02/10/2013 08:01 a.m.
Uhm, it looks like exactly what we want. So do not hesitate and let’s go configure bonding with LACP.
3. Switch1 and Switch2 configuration for bonding with LACP
Notice state of Ethernet0 interface. Remember, we shut-downed Ethernet0 on Switch2. Thanks to LACP,Ethernet0 is disabled in bundle on Switch1. It is awesome!
Note Even an interface Ethernet0 of Switch1 is disabled in bundle, it remains in up state
The last point we want to test is how much time is required for transition for interface in bundle todisabled state, if far end fails. Logically it should depend how often are LACP messages exchanged
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
8 de 48 02/10/2013 08:01 a.m.
between peers.
Start tcpdump listening on eth0 of Switch1. Then bring interface eth0 up on Switch2. Right after LACPmessages are exchanged via eth0 between Switch1 and Switch2, Ethernet0 interface is brought to enabledstate in bundle. See the output below
root@Switch1:~# tcpdump -i eth0 -v
tcpdump: WARNING: eth0: no IPv4 address assignedtcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 68 bytes
< DHCP requests>
14:00:14.555758 IP (tos 0×0, ,l 64, id 0, offset 0, flags [none], proto UDP (17), length 317) 0.0.0.0.bootpc >255.255.255.255.bootps: BOOTP/DHCP, Request [|bootp]
<LACP message coming from Switch1. Notice System 00:00:00:00:00:00. It means that Switch2 has notknown about partner’s system yet>
14:00:19.217584 LACPv1, length: 110Actor Information TLV (0×01), length: 20System 00:aa:00:85:34:00 (oui Unknown), System Priority 65534, Key 5, Port 6, Port Priority 65535State Flags [Activity, Aggregation, Default]Partner Information TLV (0×02), length: 20System 00:00:00:00:00:00 (oui Ethernet), System Priority 0, Key 0, Port 0, Port Priority 0State Flags [none]Collector Information TLV (0×03), length: 16packet exceeded snapshot
<LACP message sent to Switch2. As Switch1 knows partner ID 00:aa:00:85:34:00 – MAC address ofinterface eth0 of Switch2>
14:00:19.217914 LACPv1, length: 110Actor Information TLV (0×01), length: 20System 00:aa:00:d8:e6:00 (oui Unknown), System Priority 65534, Key 5, Port 6, Port Priority 65535State Flags [Activity, Aggregation, Synchronization, Collecting, Distributing]Partner Information TLV (0×02), length: 20System 00:aa:00:85:34:00 (oui Unknown), System Priority 65534, Key 5, Port 6, Port Priority 65535
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
9 de 48 02/10/2013 08:01 a.m.
State Flags [Activity, Aggregation, Default]Collector Information TLV (0×03), length: 16packet exceeded snapshot
<LACP message sent to Switch1. Now, Switch1 knows partner’s ID 00:aa:00:d8:e6:00 – MAC address ofinterface eth0 of Switch1>
14:00:19.220189 LACPv1, length: 110Actor Information TLV (0×01), length: 20System 00:aa:00:85:34:00 (oui Unknown), System Priority 65534, Key 5, Port 6, Port Priority 65535State Flags [Activity, Aggregation, Synchronization, Collecting, Distributing]Partner Information TLV (0×02), length: 20System 00:aa:00:d8:e6:00 (oui Unknown), System Priority 65534, Key 5, Port 6, Port Priority 65535State Flags [Activity, Aggregation, Synchronization, Collecting, Distributing]Collector Information TLV (0×03), length: 16packet exceeded snapshot
Explanation
00:aa:00:85:34:00 – MAC address of eth0 interface of Switch200:aa:00:d8:e6:00 – MAC address of eth0 interface of Switch1:
Nice! We can see structure of LACP messages but still do not know anything about time required fortransition. It is time to check it.
Start pinging 192.168.10.2 from Switch1 and with the help of tcpdump find an interface where packetsenter Switch2. In my case it is Ethernet1. Shutdown this interface and cheek the output of ping command.You see, ping is interrupted and it took approximately 50 seconds for ping to be recovered. It means thatfor 50 seconds interface Ethernet1 of Switch1 had been in enabled state. It is too much. It must be a wayto short this time to acceptable level.
Issue the command below on Switch1. LACP statistic can be found there.
root@Switch1:~# ovs-appctl lacp/show bond0
lacp: bond0status: active negotiatedsys_id: 00:aa:00:d8:e6:00sys_priority: 65534aggregation key: 5lacp_time: slow
slave: eth0: current a,achedport_id: 6port_priority: 65535
Focus on lacp-time parameter which is set to slow. According to documentation (h,p://openvswitch.org/ovs-vswitchd.conf.db.5.pdf) it is:
The LACP timing used on this Port. Possible values are fast, slow and a positive number of milliseconds.By default slow is used. When configured to be fast LACP heartbeats are requested at a rate of once persecond causing connectivity problems to be detected more quickly. In slow mode, heartbeats arerequested at a rate of once every 30 seconds.
That is exactly what we need. Changing lacp_time from slow to fast should help us to detect failure ofpeer’s interface more quickly.
5. Switch1 and Switch2 configuration for bonding with LACP and lacp time set to fast
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
11 de 48 02/10/2013 08:01 a.m.
Save configuration.
root@Switch1:~# /usr/bin/filetool.sh -b
root@Switch2:~# /usr/bin/filetool.sh -b
Ctrl+S
If you want to check interval of sending LACP messages, start tcpdump on Switch2 while pinging IPaddress of VLAN interface of Switch2 from Switch1. Now, LACP message are sent every 1 second
root@Switch2:~# tcpdump -i eth0
tcpdump: WARNING: eth0: no IPv4 address assignedtcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on eth0, link-type EN10MB (Ethernet), capture size 68 bytes21:42:25.525007 LACPv1, length: 11021:42:26.529846 LACPv1, length: 11021:42:27.532092 LACPv1, length: 11021:42:28.534532 LACPv1, length: 11021:42:29.543080 LACPv1, length: 11021:42:30.545620 LACPv1, length: 11021:42:31.547848 LACPv1, length: 11021:42:32.550717 LACPv1, length: 11021:42:33.551271 LACPv1, length: 11021:42:34.553382 LACPv1, length: 110
10 packets captured10 packets received by filter0 packets dropped by kernel
6. LACP bonding testing with enabled lacp_time=fast option
Start pinging 192.168.10.2 from Switch1 and find interface receiving ICMP echo requests on Switch2. Inmy case it is interface eth0. While pinging, shutdown this interface and check ping statistic
root@Switch1:~# ping 192.168.10.2
PING 192.168.10.2 (192.168.10.2): 56 data bytes64 bytes from 192.168.10.2: seq=0 ,l=64 time=1.810 ms64 bytes from 192.168.10.2: seq=1 ,l=64 time=0.940 ms64 bytes from 192.168.10.2: seq=2 ,l=64 time=1.610 ms64 bytes from 192.168.10.2: seq=3 ,l=64 time=0.755 ms
You see, three packet are lost. That is what we can call link redundancy if one of links fails.
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
12 de 48 02/10/2013 08:01 a.m.
Following commands helpful during troubleshooting.
Show MAC address table.
root@Switch1:~# ovs-appctl fdb/show br0
List ports.
root@Switch1:~# ovs-vsctl list port
END.
FILED UNDER OPENVSWITCH TAGGED WITH BONDING, CORE, LINUX TEAMING NIC,LLINUX TEAMING NETWORK, MICROCORE, TEAMING
Part3 – OPENVSWICH – Campus Model with Layer2Access, Built with Open-Source Applications
NOVEMBER 23, 2011 1 COMMENT (HTTP://BREZULAR.WORDPRESS.COM/2011/11/23/PART3-OPENVSWICH-CAMPUS-MODEL-WITH-LAYER2-ACCESS-BUILT-WITH-OPEN-SOURCE-APPLICATIONS/#COMMENTS)
i2 Votes
In part one (h,p://brezular.wordpress.com/2011/09/03/part1-openvswich-creating-and-submi,ing-openvswitch-extension-to-microcore-upstream/) we showed how to create Openvswitch(h,p://openvswitch.org/) extension and submit it to Microcore (h,p://distro.ibiblio.org/tinycorelinux/welcome.html) repository (h,p://distro.ibiblio.org/tinycorelinux/4.x/x86/tcz/). There were also presentedafter-install steps for Openvswitch, adapted for specific Microcore needs.
In part two (h,p://brezular.wordpress.com/2011/06/25/part2-openvswich-vlans-trunks-l3-vlan-interface-intervlan-routing-configuration-and-testing/) we did several tests in order to test feature such as vlans(h,p://en.wikipedia.org/wiki/Virtual_LAN), 8021q trunks (h,p://en.wikipedia.org/wiki/IEEE_802.1Q) andVLAN interfaces (h,p://www.itsyourip.com/cisco/how-to-create-vlan-interfaces-for-intervlan-routing-in-cisco-ios/) widely used in typical multilayer switches (h,p://en.wikipedia.org/wiki/Multilayer_switch).
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
In part three we are going to create Campus Model (h,p://www.mcmcse.com/cisco/guides/hierarchical_model.shtml) using Linux Microcore with installed Openvswitch extension. As we need toenable routing between Distribution and Core layer (h,p://en.wikipedia.org/wiki/Cisco%27s_3_Layered_Model) to follow reccomendations of design guide, we have to install Quagga(h,p://www.quagga.net/) open-source routing suite.
Keepalived (h,p://www.keepalived.org/) extension helps us to achieve default gateway’s redundancyusing VRRP (h,p://en.wikipedia.org/wiki/Virtual_Router_Redundancy_Protocol) protocol, in case offailure of default gateway. Other extensions such as iproute2 (h,p://en.wikipedia.org/wiki/Iproute2),tcpdump (h,p://en.wikipedia.org/wiki/Tcpdump) are not neccessary but useful and they are installed inour Qemu image (h,p://en.wikibooks.org/wiki/QEMU/Images).
Here is my own linux-microcore-3.0.3 Qemu image with installed Openvswitch 1.2.2, Quagga 0.99.20 andKeepalived 1.2.2 as extensions.
The image also contains tcpdump and iproute2 command. I recommend you to use my Qemu imageotherwise additional after install configuration for Quagga and Openvswitch is required. At least youshould put following commands to /opt/bootlocal.sh
Note: 8021q module does not have to be loaded to get trunk working with Openvswitch. It has to beloaded when creating sub-interfaces using vconfig (h,p://linux.about.com/library/cmd/blcmdl8_vconfig.htm) command is needed. On the other hand, choosing the right emulated networkcard in GNS3 Qemu se,ings is crucial to get trunk working. You cannot miss with i82557b ethernet card.
Note: Iproute2 extension is needed to show multiple equal-cost routes presented in Linux kernel. Thetopic is described more in detail here.
Core layer consists from two Multilayer Core switches – Core1 nad Core2. They are connected with point-to-point layer3 links. The full-mesh topology (h,p://en.wikipedia.org/wiki/Network_topology) betweenCore and Distribution layer provides path even in case of failure of two links. For example, if bothinterfaces eth4 and eth1 fail on switch Distrib1 there is still path to Core Layer via interface eth0.
Quagga routing daemon is running on all Core and Distribution switches. It offers routing capabilitiesusing OSPF routing protoco (h,p://en.wikipedia.org/wiki/Open_Shortest_Path_First)l.
Use vtysh (h,p://linux.die.net/man/1/vtysh) – Quagga shell to configure Core switches as following.
1. Command for accessing Quagga CLI
tc@box:~$ sudo /usr/local/bin/vtysh
Hello, this is Quagga (version 0.99.20).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
2. Configure hostname, interfaces and OSPF routing protocol for Core1
box# configure terminal
box(config)# hostname Core1
Core1(config)# interface eth0
Core1(config-if)# description Link to Core2
Core1(config-if)# ip address 10.10.10.5/30
Core1(config-if)# no shutdown
Core1(config-if)# exit
Core1(config)# interface eth2
Core1(config-if)# description Link to Distrib2
Core1(config-if)# ip address 10.10.10.18/30
Core1(config-if)# no shutdown
Core1(config-if)# exit
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
16 de 48 02/10/2013 08:01 a.m.
Core1(config)# interface eth1
Core1(config-if)# description Link to Distrib1
Core1(config-if)# ip address 10.10.10.10/30
Core1(config-if)# no shutdown
Core1(config-if)# exit
Core1(config)# router ospf
Core1(config-router)# network 10.10.10.4/30 area 0
Core1(config-router)# network 10.10.10.16/30 area 0
Core1(config-router)# network 10.10.10.8/30 area 0
Core1(config-router)# do write
Building Configuration…Configuration saved to /usr/local/etc/quagga/zebra.confConfiguration saved to /usr/local/etc/quagga/ripd.confConfiguration saved to /usr/local/etc/quagga/ripngd.confConfiguration saved to /usr/local/etc/quagga/ospfd.confConfiguration saved to /usr/local/etc/quagga/ospf6d.confConfiguration saved to /usr/local/etc/quagga/bgpd.confConfiguration saved to /usr/local/etc/quagga/isisd.conf[OK]
Now, exit from vtysh. As Quagga had not saved hostname “Core1″ to /usr/local/etc/quagga/zebra.conf,configure it from Microcore CLI..
Check if routes are properly propagated in the routing table.
Core2# show ip route ospf
Codes: K – kernel route, C – connected, S – static, R – RIP, O – OSPF,I – ISIS, B – BGP, > – selected route, * – FIB route
O 10.10.10.4/30 [110/10] is directly connected, eth0, 00:08:20O>* 10.10.10.8/30 [110/20] via 10.10.10.5, eth0, 00:08:10O 10.10.10.12/30 [110/10] is directly connected, eth1, 00:00:27O>* 10.10.10.16/30 [110/20] via 10.10.10.5, eth0, 00:08:10O 10.10.10.20/30 [110/10] is directly connected, eth2, 00:08:20
Part2 – Distribution Layer
Distribution layer consists from two Multilayer Distribution switches – Distrib1 and Distrib2. The main jobfor Distribution switches is routing between different vlan’s subnet that are terminated here. Any trafficfiltering rules should be configured on Distribution swiches.
Uplink interfaces connecting Distribution switches to Core switches are layer3 interfaces. They participatein OSPF messages forwarding. Downlink interfaces connecting Distribution switches to Access switchesare Layer2 interfaces. They are trunks capable of carrying traffic belongig to multiple VLANs.
The next configuration’s steps are shown for Distrib1 switch only. Continue and make similiarconfiguration for Distrib2 switch.
1. Configure Layer3 interfaces and OSPF routing protocol on Distrib1 switch
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
18 de 48 02/10/2013 08:01 a.m.
Distrib1(config-if)# no shutdown
Distrib1(config-if)# exit
Distrib1(config)# int eth2
Distrib1(config-if)# description Link to Core2
Distrib1(config-if)# ip address 10.10.10.21/30
Distrib1(config-if)# no shutdown
Distrib1(config-if)# exit
Distrib1(config)# int eth0
Distrib1(config-if)# description Link to Distrib2
Distrib1(config-if)# ip address 10.10.10.1/30
Distrib1(config-if)# no shutdown
Distrib1(config-if)# exit
Distrib1(config)# router ospf
Distrib1(config-router)# network 10.10.10.0/30 area 0
Distrib1(config-router)# network 10.10.10.20/30 area 0
Distrib1(config-router)# network 10.10.10.8/30 area 0
Distrib1(config-router)# exit
Distrib1(config)# do write
Exit from vtysh and save content of /usr/local/etc/quagga/ directory.
tc@box:~$ /usr/bin/filetool.sh -b
Ctrl + s
2. Configure Layer3 interfaces and OSPF routing protocol on Distrib2 switch
Configure Distrib1 switch according to the topology. Check if Distrib2 can see all three OSPF neighbours.
Distrib2# show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface RXmtL RqstL DBsmL10.10.10.21 1 Full/DR 38.389s 10.10.10.1 eth0:10.10.10.2 0 0 010.10.10.22 1 Full/DR 38.927s 10.10.10.14 eth1:10.10.10.13 0 0 010.10.10.18 1 Full/DR 33.153s 10.10.10.18 eth2:10.10.10.17 0 0 0
Check if routes are properly propagated in the routing table of Quagga.
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
19 de 48 02/10/2013 08:01 a.m.
Distrib2# show ip route ospf
Codes: K – kernel route, C – connected, S – static, R – RIP, O – OSPF,I – ISIS, B – BGP, > – selected route, * – FIB route
O 10.10.10.0/30 [110/10] is directly connected, eth0, 00:01:47O>* 10.10.10.4/30 [110/20] via 10.10.10.14, eth1, 00:01:17* via 10.10.10.18, eth2, 00:01:17O>* 10.10.10.8/30 [110/20] via 10.10.10.1, eth0, 00:01:17* via 10.10.10.18, eth2, 00:01:17O 10.10.10.12/30 [110/10] is directly connected, eth1, 00:01:37O 10.10.10.16/30 [110/10] is directly connected, eth2, 00:01:24O>* 10.10.10.20/30 [110/20] via 10.10.10.1, eth0, 00:01:31* via 10.10.10.14, eth1, 00:01:31
Exit from Quagga vtysh shell and check routing table of Linux Microcore.
tc@Distrib2:~$ ip route show
10.10.10.0/30 dev eth0 proto kernel scope link src 10.10.10.210.10.10.4/30 proto zebra metric 20nexthop via 10.10.10.14 dev eth1 weight 1nexthop via 10.10.10.18 dev eth2 weight 110.10.10.8/30 proto zebra metric 20nexthop via 10.10.10.1 dev eth0 weight 1nexthop via 10.10.10.18 dev eth2 weight 110.10.10.12/30 dev eth1 proto kernel scope link src 10.10.10.1310.10.10.16/30 dev eth2 proto kernel scope link src 10.10.10.1710.10.10.20/30 proto zebra metric 20nexthop via 10.10.10.1 dev eth0 weight 1nexthop via 10.10.10.14 dev eth1 weight 1127.0.0.1 dev lo scope link
They are two euqal-cost paths for each of network 10.10.10.4/30, 10.10.10.8/30, 10.10.10.20/30 presented inkernel routing table of Distrib2.
e) Configure IP addresses of VLAN interfaces and OSPF routing protocol on Distrib1 switch
We have two options here – either to use Linux kernel command – ifconfig (h,p://en.wikipedia.org/wiki/Ifconfig) or Quagga vtysh shell. I chose vtysh – no need to put commands to /opt/bootlocal.sh
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
21 de 48 02/10/2013 08:01 a.m.
tc@Distrib1:~$ sudo /usr/local/bin/vtysh
Distrib1# conf t
Distrib1(config)# interface vlan10
Distrib1(config-if)# ip address 192.168.10.2/24
Distrib1(config-if)# no shutdown
Distrib1(config-if)# interface vlan20
Distrib1(config-if)# ip address 192.168.20.2/24
Distrib1(config-if)# no shutdown
Distrib1(config-if)# interface vlan30
Distrib1(config-if)# ip address 192.168.30.2/24
Distrib1(config-if)# no shutdown
Distrib1(config-if)# interface vlan40
Distrib1(config-if)# ip address 192.168.40.2/24
Distrib1(config-if)# no shutdown
Distrib1(config-if)# router ospf
Distrib1(config-router)# network 192.168.10.0/24 area 0
Distrib1(config-router)# network 192.168.20.0/24 area 0
Distrib1(config-router)# network 192.168.30.0/24 area 0
Distrib1(config-router)# network 192.168.40.0/24 area 0
Distrib1(config)# do write
Exit from vtysh shell and save configuration.
tc@Distrib1:~$ /usr/bin/filetool.sh -b
Ctrl + s.
Do similar configuration for Distrib2 switch.
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
22 de 48 02/10/2013 08:01 a.m.
Part3 – Access Layer
Access Layer consists of two Layer2 switches – Access1 and Access2. VLANs are created here andswitchports are assigned to VLANS. As each VLAN is restricted to the local Access switch we call them –local VLANs. Primary task of Access layer is to provide switchports for end users and forward traffic. Since Layer2 Access switches do not route between different VLANs, traffic is sent to Distribution layervia 8021q trunks when routing is required. The design guides recommend a campus model with localVLANs when 20 percent of user’s traffic stays in Campus and 80 percent of traffic is forwarded outsidecampus.
1. Configure hostname, access ports for VLAN 10 and 20 and 8021q trunk ports on Access1 switch
Things look good now. We can ping a virtual interfaces on Distribution switches from host residing in thesame VLAN. But we still cannot ping any IP address residing in different subnet than host. It is becauseboth Distribution switches have not been configured to act as default gateway yet.
You might have noticed that all the hosts were configured with IP address of default gateway 192.168.x.1.This is a virtual IP address created by Keepalived extension.
Keepalived offers default gateway’s redundancy using VRRP protocol. In case of failure one of theDistribution switches, the other Distrib switch takes responsibility and continues to forward packets.Switch with higher priority is called Master and forwards packet. At least it is always one Master switch inone VRRP group. For example, in VRRP group 10 it is Distrib1 with priority 150. Switch with lowerpriority is called Backup and as it was said, it forwards packets only if Master switch fails. The Backupswitch in VRRP group 10 is Distrib2 with priority 100. The higher is priority, more likely switch becomesMaster switch.
Obviously, it must be communication between switches to let known each other about its existence. Bydefault every one second Advertisiment is sent from Master switch to each memember of VRRP group tomulticast IP address 224.0.0.18. After three missing Advertisiment plus screw time, Backup switch knownsthat Master is down and transition from Backup to Master state occurs. It forwards packets now.
Note:
Virtual IP address is tied with virtual MAC address – 00-00-5E-00-01-XX. The last byte of the address(XX) is the Virtual Router Identifier (VRID), which is different for each VRRP instance in the network.This address is used by only one physical router at a time, and it will reply with this MAC address whenan ARP request is sent for the virtual router’s IP address. If Master fails, the new Master will broadcastGratious ARP containing the virtual router MAC address for the associated IP address. If I understand itcorrectly, nothing has been change in host configuration but Access switches had changed their CAMtable – frames with destination MAC address 00-00-5E-00-01-XX will be send via new path to the newMASTER router.
Keepalived works slightly differently – it uses real MAC address of interface instead of Virtual MACaddress. Check ARP cache of hosts in VRRP testing section – it seems that the current Keepalived/VRRPdoes not support Virtual MAC addresses.
Distrib1 should become BACKUP router in VRRP group 20 and 30 and stays in MASTER state in VRRP10 and 40. Similarly, DOSTRIB2 is BACKUP router in VRRP group 10 and 40 and MASTER in VRRPgroup 20 and 30.
Make keepalived daeomon to be started after boot of Microcore.
Check if redundant routes are presented in kernel routing table of Core1. Non-redundant routes such as10.10.10.4/30, 10.10.10.8/30, 10.10.10.16/30 and 127.0.0.1 are directly connected networks. Two equal-costpath should exist to other network learned by OSPF routing protocol.
tc@Core1:~$ ip route show
10.10.10.0/30 proto zebra metric 20nexthop via 10.10.10.9 dev eth1 weight 1nexthop via 10.10.10.17 dev eth2 weight 110.10.10.4/30 dev eth0 proto kernel scope link src 10.10.10.510.10.10.8/30 dev eth1 proto kernel scope link src 10.10.10.1010.10.10.12/30 proto zebra metric 20nexthop via 10.10.10.6 dev eth0 weight 1nexthop via 10.10.10.17 dev eth2 weight 110.10.10.16/30 dev eth2 proto kernel scope link src 10.10.10.1810.10.10.20/30 proto zebra metric 20nexthop via 10.10.10.6 dev eth0 weight 1nexthop via 10.10.10.9 dev eth1 weight 1127.0.0.1 dev lo scope link192.168.10.0/24 proto zebra metric 20nexthop via 10.10.10.9 dev eth1 weight 1nexthop via 10.10.10.17 dev eth2 weight 1192.168.20.0/24 proto zebra metric 20nexthop via 10.10.10.9 dev eth1 weight 1nexthop via 10.10.10.17 dev eth2 weight 1192.168.30.0/24 proto zebra metric 20nexthop via 10.10.10.9 dev eth1 weight 1nexthop via 10.10.10.17 dev eth2 weight 1192.168.40.0/24 proto zebra metric 20nexthop via 10.10.10.9 dev eth1 weight 1nexthop via 10.10.10.17 dev eth2 weight 1
3. Check connectivity between Access and Core layer
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
30 de 48 02/10/2013 08:01 a.m.
Issue ping from host PC1 in VLAN10 to eth0 interface of Core2.
Issue ping from PC1 to IP address of VLAN10 interfaces – 192.168.10.2 and 192.168.10.3 on bothDistribution switches. Stop ping sequence and ping VRRP Virtual IP address 192.168.10.1.
Now check ARP cache of PC1.
tc@PC1:~$ arp
? (192.168.10.2) at 00:23:20:bc:57:67 [ether] on eth0? (192.168.10.1) at 00:23:20:bc:57:67 [ether] on eth0? (192.168.10.3) at 00:23:20:8c:1d:cf [ether] on eth
Virtual IP address 192.168.10.1 has assigned MAC address of VLAN10 interface – 00:23:20:bc:57:67 onDistrib1 switch. Packets destined outside VLAN10 will leave subnet 192.168.10.0/24 via vlan10 interfaceon Distrib1. We have expected it because Distrib1 is MASTER router for VRRP group 10.
Now start to ping IP address of eth0 interface of Core2 – 10.10.10.6/30 from PC1. We are going to killkeepalived proccess on Distrib1 switch to check if Disrib2 will transit to MASTER state for VRRP group10 and 40.
Keepalived: Terminating on signalKeepalived: Stopping Keepalived v1.2.2 (06/28,2011)Keepalived_vrrp: Terminating VRRP child process on signal
Check the output of console of Distrib2 switch. Information messages tell us about transition of Distrib2switch to MASTER state for VRRP group 10 and 40.
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
31 de 48 02/10/2013 08:01 a.m.
Keepalived_vrrp: VRRP_Instance(10) Transition to MASTER STATEKeepalived_vrrp: VRRP_Instance(40) Transition to MASTER STATEKeepalived_vrrp: VRRP_Instance(10) Entering MASTER STATEKeepalived_vrrp: VRRP_Instance(40) Entering MASTER STATE
Check ARP cache PC1 again. It is not big surprise that Virtual IP address 192.168.10.1 has assigned MACaddress 00:23:20:8c:1d:cf of VLAN10 interface of Distrib2 switch.
tc@PC1:~$ arp
? (192.168.10.2) at 00:23:20:bc:57:67 [ether] on eth0? (192.168.10.1) at 00:23:20:8c:1d:cf [ether] on eth0? (192.168.10.3) at 00:23:20:8c:1d:cf [ether] on eth0p
It seems that series of ping requests from PC1 to 10.10.10.6 has not been broken by transition.
Start keepalived daemon on Distrib1 switch again. Distrib1 is going immediately to state MASTER forVRRP group 10 and 40 and to BACKUP for VRRP group 20 and 30.
FILED UNDER OPENVSWITCH TAGGED WITH ACCESS LAYER, CAMPUS MODEL, CORELAYER, CORE LINUX, DISTRIBUTION LAYER, KEEPALIVED, MICROCORE, OPENVSWITCH,QEMU, QUAGGA
Part1 – OPENVSWICH – Creating and SubmittingOpenvswitch Extension To Tinycore Upstream
SEPTEMBER 3, 2011 8 COMMENTS (HTTP://BREZULAR.WORDPRESS.COM/2011/09/03/PART1-OPENVSWICH-CREATING-AND-SUBMITTING-OPENVSWITCH-EXTENSION-TO-MICROCORE-UPSTREAM/#COMMENTS)
i2 Votes
In May 2011, I read a request (h,p://www.gns3.net/phpBB/topic3549.html?sid=9da8d8dc809d413978e92277b9205d90) for installation Openvswitch(h,p://openvswitch.org/) on Qemu image. I started to play with Openvswitch and finally became fan ofthis project. I realized how powerful can be Openvswitch, offering many features listed here(h,p://openvswitch.org/). Spending my time playing with Microcore and Openvswitch after several days Iwas able to create Microcore Qemu image with 8021q (h,p://brezular.wordpress.com/2011/06/06/linux-microcore-kernel-compilation-for-802-1q-support/) support and install Openvswitch on the top of it.
In the tutorial I would like to show how to create Openvswitch extension (h,p://wiki.tinycorelinux.net/wiki:creating_extensions) and make it ready for submit to the Tinycore upstream. I chose Core Linux,because I am familiar with this minimal Linux distribution and the Core is incredibly small.
The following steps describe installation Openvswitch on Qemu image with pre-installed Microcore Linux.
I also created three labs using Openvswitch. I tested how can Openvswitch works with VLANs, 802.1qtrunk ports, if it was capable of creating L3 VLAN interfaces and Inter VLAN routing was working. Labsare available here (h,p://brezular.wordpress.com/2011/06/25/part2-openvswich-vlans-trunks-l3-vlan-
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
Assuming your Microcore Qemu image with console support (h,p://brezular.wordpress.com/2011/01/26/how-to-setup-linux-microcore-3-x-router-qemu-image-in-fedora-linux-part1/) has been created andsupports 8021q VLAN tagging (h,p://brezular.wordpress.com/2011/06/06/linux-microcore-kernel-compilation-for-802-1q-support/), start the image:
Note: The kernel source will be saved in /lib/modules/`uname -r`/build
It is recommended to check the list of necessary applications here (h,p://git.openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plain;f=INSTALL;hb=HEAD):
4. Openvswitch Installation
It is recommended to start here (h,p://wiki.tinycorelinux.net/wiki:creating_extensions) and continue here(h,p://git.openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plain;f=INSTALL;hb=HEAD) :-)After that you can type following:
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
34 de 48 02/10/2013 08:01 a.m.
cd /home/tc/openvswitch-1.11.0/
make DESTDIR=/tmp/openvswitch install
5. BACKUP and LOAD Module openvswitch.ko
a) BACKUP module openvswitch.ko
After building, a kernel module “openvswitch.ko” will be saved in a ./datapath/linux/ directory. We’vebuilt both userspace and kernel module as well. The performance is be,er when a kernel module is used.Create a directory in a structure of the openvswitch extension where the kernel module openvswitch.kowill be saved.
In order to initialize the configuration database using ovsdb-tool later, the file /home/tc/openvswitch-1.11.0/vswitchd/vswitch.ovsschema is needed. You need to copy it to /tmp/openvswitch/usr/local/etc/openvswitch/vswitchd/ to become part of the extension.
An info file describing its contents (.tcz.info) – this content is standardized. Check repository(h,p://distro.ibiblio.org/tinycorelinux/4.x/x86/tcz/) for examples.
Additional build instructions in a plain text file for future reference, mentioning such things as whichextensions are required to build the package and what compile flags were used.
g) Create the dependency list openvswitch-3.0.21-tinycore.tcz.dep
List of the extensions that have to be presented to run openvswitch extension correctly.
h) Install openssh.tcz an extension and copy all openvswitch-3.0.21-tinycore.tcz.* files and a sourcefile to the guest system
f) Enable IPv4 and IPV6 packets forwarding between interfaces
Although not directly connected with Openvswitch configuration we need to enable ipv4 and ipv6packets forwarding between interfaces for Microcore. This option is disabled in kernel by default.
g) Run commands you have entered into /opt/bootlocal.sh and make them persistent after restart ofthe Microcore
sudo /opt/bootlocal.sh
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
38 de 48 02/10/2013 08:01 a.m.
Optionally: It is recommended to delete the history of used commands.
echo "" > /home/tc/.ash_history
Save changes made for files and directories which are listed in /opt/.filetool.lst
/usr/bin/filetool.sh -b
10. Configuration example
Now you may use ovs-vsctl to set up bridges and other Open vSwitch features. For example, to create abridge named br0 and add ports eth0, eth1 and eth2 to it:
Before shutdown you always force Core to save configuration changes in the openvswitch database file – /usr/local/etc/openvswitch/conf.db. Use the command:
JUNE 25, 2011 17 COMMENTS (HTTP://BREZULAR.WORDPRESS.COM/2011/06/25/PART2-OPENVSWICH-VLANS-TRUNKS-L3-VLAN-INTERFACE-INTERVLAN-ROUTING-CONFIGURATION-AND-TESTING/#COMMENTS)
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
39 de 48 02/10/2013 08:01 a.m.
i1 Vote
In a previous tutorial we showed how to install Openvswitch on Qemu image with Microcore Linux. Atthe end of tutorial we created Openvswitch extension (h,p://brezular.wordpress.com/2011/06/25/part1-openvswich-creating-and-submi,ing-extension-to-microcore-upstream/) and submi,ed it to Microcoreupstream. Assuming that Openvswitch is configured and functional, we are ready to make three labswhich helps us to test features such as VLANs, trunks and interVLAN routing.
LAB 1 Access VLAN Configuration And Testing
In this lab we are going to test connectivity between PCs which reside in same VLAN. PC1 and PC2 areassigned to VLAN 10 and PC3 and PC4 reside in VLAN 20. According to theory, each VLAN should haveits own separate subnet. In our case, we are going to test not only connectivity between computers in thesame VLANs but also to test connectivity between PCs placed in different VLAN.
For this reason I chose one subnet 192.168.1.0/24 for all the computers so L3 connectivity exists betweene.g. PC1 (VLAN 10) and PC3 (VLAN20). Of course, the computers assigned to different VLANs areseparated on Layer 2 so we do not expect any connectivity between those computers.
We can see that ping is not working between PC1 and PC3. We confirm that there is not connectivitybetween computers placed in to different VLANs and openvswitch is working correctly.
LAB 2 VLAN Trunk Configuration And Testing
In previous LAB we proved, that openvswitch only forwards traffic between computers which reside inthe same VLAN. In this LAB we are going to add additional switch to the topology – openvswitch2 andconnect switches with link configured as 8021.1q trunk. The access VLAN 10, 20, 30 is added toopenvswitch2 and VLAN 30 is added to openvswitch. There is only VLAN 10 and 30 allowed on bothsides of the trunk.
Our goal is to show that only traffic from VLAN 10 and VLAN 30 is forwarded between switches.
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
Traffic from vlan 20 is not forwarded between switches. It is because VLAN 20 is not allowed on thetrunk. If there would be need to allow all the VLANs on the trunk, only parameter trunks (withoutspecific VLAN) should be presented.
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
44 de 48 02/10/2013 08:01 a.m.
And Testing
In this last LAB we are going to create L3 VLAN interface for VLAN 10, 20, 30. After that we will check ifMicrocore can forward traffic between different VLANs. We assume that particular ports were added to VLANs in previous LABs and only additional configuration will be shown in this LAB. I have alsochanged a topology to make it more simple and create the new IP plan for the devices.
2) PC1 to PC6 – IP address, netmask, default gateway – configuration
a) Delete old IP addresses from the computers
We should split IP address space in such way when each VLAN would have assigned separate subnet.First, for each computer delete a row with old IP address from /opt/bootlocal.sh which was previouslyassigned in LAB1 (use dd command for delete of particular row).
Thanks to enabled forwarding between interfaces in kernel, ping is working between different VLANS.Microcore with Openvswitch is acting as the Layer3 switch.
End.
FILED UNDER OPENVSWITCH TAGGED WITH 8021Q LINUX, CORE LINUX, GNS3, LINUXROUTER, LINUX SWITCH, MICROCORE, OPENVSWITCH, TRUNKS LINUX, VLAN LINUX
Blog at WordPress.com.
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/
47 de 48 02/10/2013 08:01 a.m.
The Enterprise Theme.
Follow
Follow “Brezular's Technical Blog”
Powered by WordPress.com
Openvswitch | Brezular's Technical Blog http://brezular.wordpress.com/category/openvswitch/