Thursday, September 29, 2016

The OVN Load Balancer

Untitled Document.md

Overview

Building upon my previous post I will explore the the load balancing feature of OVN. But before getting started, lets review the setup created in the last lab.

The lab network:

ovn lab

The OVN logical network:

ovn logical components

The OVN Load Balancer

The OVN load balancer is intended to provide very basic load balancing services to workloads within the OVN logical network space. Due to its simple feature set it is not designed to replace dedicated appliance-based load balancers which provide many more bells & whistles for advanced used cases.

The load balancer uses a hash-based algorithm to balance requests for a VIP to an associated pool of IP addresses within logical space. Since the hash is calculated using the headers of the client request the balancing should appear fairly random, with each individual client request getting stuck to a particular member of the load balancing pool for the duration of the connection. Load balancing in OVN may be applied to either a logical switch or a logical router. The choice of where to apply the feature depends on your specific requirements. There are caveats to each approach.

When applied to a logical router, the following considerations need to be kept in mind:

  1. Load balancing may only be applied to a “centralized” router (ie. a gateway router).
  2. Due to point #1, load balancing on a router is a non-distributed service.

When applied to a logical switch, the following considerations need to be kept in mind:

  1. Load balancing is “distributed” in that it is applied on potentially multiple OVS hosts.
  2. Load balancing on a logical switch is evaluted only on traffic ingress from a VIF. This means that it must be applied on the “client” logical switch rather than on the “server” logical switch.
  3. Due to point #2, you may need to apply the load balancing to many logical switches depending on the scale of your design.

Using Our Fake “VMs” as Web Servers

In order to demonstrate the load balancer we want to create a pair of web servers in our “dmz” which will serve up uniquely identifiable files. In order to keep things simple, we’ll use a single line python web server running in our vm1/vm2 namespaces. Lets kick things off by starting up our web servers.

From ubuntu2:

mkdir /tmp/www
echo "i am vm1" > /tmp/www/index.html
cd /tmp/www
ip netns exec vm1 python -m SimpleHTTPServer 8000

From ubuntu3:

mkdir /tmp/www
echo "i am vm2" > /tmp/www/index.html
cd /tmp/www
ip netns exec vm2 python -m SimpleHTTPServer 8000

The above commands will create a web server, listening on TCP 8000, which will serve up a file that can be used to identify the vm which served the file.

We’ll also want to be able to test connectivity to our web servers. For this we’ll use curl from the global namespace of our Ubuntu hosts. Be sure to install curl on them if its not already.

apt-get -y install curl

Configuring the Load Balancer Rules

As a first step we’ll need to define our load balancing rules; namely the VIP and the back-end server IP pool. All that is involved here is to create an entry in the OVN northbound DB and capture the resulting UUID. For our testing we’ll use the VIP 10.127.0.254 which resides within the “data” network in the lab. We’ll use the addresses of vm1/vm2 as our pool IPs.

From ubuntu1:

uuid=`ovn-nbctl create load_balancer vips:10.127.0.254="172.16.255.130,172.16.255.131"`
echo $uuid

The above command creates an entry in the load_balancer table of the northbound DB and stores the resulting UUID to the variable “uuid”. We’ll reference this variable in later commands.

The Gateway Router As a Load Balancer

Lets apply our load balancer profile to the OVN gateway router “edge1”.

From ubuntu1:

ovn-nbctl set logical_router edge1 load_balancer=$uuid

You can verify that this was applied by checking the database entry for edge1.

ovn-nbctl get logical_router edge1 load_balancer

Now, from the global namespace of any of the Ubuntu hosts we can attempt to connect to the VIP.

root@ubuntu1:~# curl 10.127.0.254:8000
i am vm2
root@ubuntu1:~# curl 10.127.0.254:8000
i am vm1
root@ubuntu1:~# curl 10.127.0.254:8000
i am vm2

I ran the above tests several times and the load balancing appeared to be quite random.

Lets see what happens if we disable one of our web servers. Try stopping the python process running in the vm1 namespace. Here is what I got:

root@ubuntu1:~# curl 10.127.0.254:8000
curl: (7) Failed to connect to 10.127.0.254 port 8000: Connection refused
root@ubuntu1:~# curl 10.127.0.254:8000
i am vm2

As you can see, the load balancer is not performing any sort of health checking. At present, the assumption is that health checks would be performed by an orchestration solution such as Kubernetes but it would be resonable to assume that this feature would be added at some future point.

Restart the python web server on vm1 before moving to the next test.

Load balancing works externally, but lets see what happens when we try to access the VIP from an internal VM. Try using curl from vm3 on ubuntu2:

root@ubuntu2:~# ip netns exec vm3 curl 10.127.0.254:8000
i am vm1
root@ubuntu2:~# ip netns exec vm3 curl 10.127.0.254:8000
i am vm2

Nice. This seems to work as well, but also raises an interesting point. Take a second look at the logical diagram for our OVN network and think about the traffic flow for the curl request from vm3. Also, look at the logs from the python web server. Mine are below:

10.127.0.130 - - [29/Sep/2016 09:53:44] "GET / HTTP/1.1" 200 -
10.127.0.129 - - [29/Sep/2016 09:57:42] "GET / HTTP/1.1" 200 -

Notice the client IP addresses in the logs. The first is from ubuntu1 per the previous round of tests. The second IP is edge1 itself and is from the request from vm3. Why is the request coming from edge1 rather than from vm3 directly? The answer is that the OVN developer who implemented load balancing took into account something known as “proxy mode” where the load balancer hides the client side IP under certain circumstances. Why is this necessary? Think about what would happen if the web server saw the real IP of vm3. The response from the server would route back directly to vm3, bypassing the load balancer on edge1. From the perspective of vm3 it would look like it made a request to the VIP but received a reply from the real IP of one of the web servers. This obviously would not work which is why proxy-mode functionality is important.

Lets remove the load balancer profile and move on to a second round of tests.

ovn-nbctl clear logical_router edge1 load_balancer
ovn-nbctl destroy load_balancer $uuid

Configuring the Load Balancer On a Logical Switch

Lets see what happens when we apply the load balancing rules to various logical switches within our setup. Since we’re moving load balancing away from the edge the first step we need is to create a new load balancer profile with an internal VIP. We’ll use 172.16.255.62 for this.

From ubuntu1:

uuid=`ovn-nbctl create load_balancer vips:172.16.255.62="172.16.255.130,172.16.255.131"`
echo $uuid

As a first test lets apply it to the “inside” logical switch.

From ubuntu1:

# apply and verify
ovn-nbctl set logical_switch inside load_balancer=$uuid
ovn-nbctl get logical_switch inside load_balancer

And test from vm3 (which resides on “inside”):

root@ubuntu2:~# ip netns exec vm3 curl 172.16.255.62:8000
i am vm1
root@ubuntu2:~# ip netns exec vm3 curl 172.16.255.62:8000
i am vm1
root@ubuntu2:~# ip netns exec vm3 curl 172.16.255.62:8000
i am vm2

This seems to work. Lets remove the load balancer from “inside” and apply it to “dmz”.

From ubuntu1:

ovn-nbctl clear logical_switch inside load_balancer
ovn-nbctl set logical_switch dmz load_balancer=$uuid
ovn-nbctl get logical_switch dmz load_balancer

And again test from vm3:

root@ubuntu2:~# ip netns exec vm3 curl 172.16.255.62:8000
^C

No good. It hangs. Lets try from vm1 (which resides on “dmz”):

root@ubuntu2:~# ip netns exec vm1 curl 172.16.255.62:8000
^C

Also no good. This highlights the requirement that load balancing be applied on the client’s logical switch rather than the server’s logical switch.

Be sure to clean up. From ubuntu1:

ovn-nbctl clear logical_switch dmz load_balancer
ovn-nbctl destroy load_balancer $uuid

Final Words

Basic load balancing is a very “nice to have” feature. Given that it is built directly into OVN means one less piece of software to deploy within your SDN. While the feature set is minimal, I actually think that it covers the needs of a very broad set of users. Given time, I also expect that certain limitations such as lack of health checking will be addressed.

In my next post I will look into network security using OVN.

Tuesday, September 27, 2016

The OVN Gateway Router

Untitled Document.md

Overview

Building upon my previous post I will now add an OVN gateway router into the lab setup. This gateway router will provide access to the lab network from within our overlay network.

The Lab

In order to demonstrate the gateway router we will need to add another physical network to our Ubuntu hosts. For my purposes I will add the network 10.127.0.128/25 via eth1 of each of my hosts. The final lab diagram is illustrated below.

ovn lab

Introducing the OVN L3 Gateway

An OVN Gateway serves as an onramp/offramp between the overlay network and the physical network. They come in two flavors: layer-2 which bridge an OVN logical switch into a VLAN, and layer-3 which provide a routed connection between an OVN router and the physical network. For the purposes of this lab we will focus on creating a layer-3 gateway which will serve as the demarcation point between our physical and logical networks.

Unlike a distributed logical router (DLR), an OVN gateway router is centralized on a single host (chassis) so that it may provide services which cannot yet be distributed (NAT, load balancing, etc…). As of this publication there is a restriction that gateway routers may only connect to other routers via a logical switch, whereas DLRs may connect to one other directly via a peer link. Work is in progress to remove this restriction.

It should be noted that it is possible to have multiple gateway routers tied into an environment, which means that it is possible to perform ingress ECMP routing into logical space. However, it is worth mentioning that OVN currently does not support egress ECMP between gateway routers. Again, this is being looked at as a future enhancement.

Make ubuntu1 an OVN Host

Rather than using a host which houses VMs, lets use ubuntu1 for our OVN gateway router host. Begin by installing the proper OVN packages for the host role:

dpkg -i ovn-host_2.6.0-1_amd64.deb

And then register with OVN Central (itself):

ovs-vsctl set open . external-ids:ovn-remote=tcp:127.0.0.1:6642
ovs-vsctl set open . external-ids:ovn-encap-type=geneve
ovs-vsctl set open . external-ids:ovn-encap-ip=10.127.0.2

And verify connectivity:

root@ubuntu1:~# netstat -antp | grep 127.0.0.1
tcp        0      0 127.0.0.1:6642          127.0.0.1:55566         ESTABLISHED 4999/ovsdb-server
tcp        0      0 127.0.0.1:55566         127.0.0.1:6642          ESTABLISHED 15212/ovn-controlle

Also, be sure to create the integration bridge if wasn’t created automatically by OVN:

ovs-vsctl add-br br-int -- set Bridge br-int fail-mode=secure

OVN Logical Design

Lets review the planned design before we start configuring things. The OVN logical network we are creating is illustrated below.

ovn logical components

As you can see we are adding the following new components:

  • OVN gateway router (edge1)
  • logical switch (transit) used to connect the edge1 and tenant1 routers
  • logical switch (outside) used to connect edge1 to the lab network

Adding the L3 Gateway

As mentioned earlier the gatway router will be bound to a specific chassis (ubuntu1 in our case). In order to accomplish this binding we will need to locate the chassis id for ubuntu1. Using the ovn-sbctl show command from ubuntu1 you should see output similar to this:

ovn-sbctl show
Chassis "833ae1bd-ced3-494a-a95b-f2dc54172b71"
        hostname: "ubuntu1"
        Encap geneve
                ip: "10.127.0.2"
                options: {csum="true"}
Chassis "239f2c28-90ff-468f-a701-655585c630bf"
        hostname: "ubuntu3"
        Encap geneve
                ip: "10.127.0.3"
                options: {csum="true"}
        Port_Binding "dmz-vm2"
        Port_Binding "inside-vm4"
Chassis "517d558e-158a-4cb2-8870-283e9d39685e"
        hostname: "ubuntu2"
        Encap geneve
                ip: "10.127.0.129"
                options: {csum="true"}
        Port_Binding "inside-vm3"
        Port_Binding "dmz-vm1"

Copy the Chassis UUID of the ubuntu1 host for use below.

Create the new logical router. Be sure to substitute {chassis_id} with a valid UUID. From ubuntu1:

# create router edge1
ovn-nbctl create Logical_Router name=edge1 options:chassis={chassis_uuid}

# create a new logical switch for connecting the edge1 and tenant1 routers
ovn-nbctl ls-add transit

# edge1 to the transit switch
ovn-nbctl lrp-add edge1 edge1-transit 02:ac:10:ff:00:01 172.16.255.1/30
ovn-nbctl lsp-add transit transit-edge1
ovn-nbctl lsp-set-type transit-edge1 router
ovn-nbctl lsp-set-addresses transit-edge1 02:ac:10:ff:00:01
ovn-nbctl lsp-set-options transit-edge1 router-port=edge1-transit

# tenant1 to the transit switch
ovn-nbctl lrp-add tenant1 tenant1-transit 02:ac:10:ff:00:02 172.16.255.2/30
ovn-nbctl lsp-add transit transit-tenant1
ovn-nbctl lsp-set-type transit-tenant1 router
ovn-nbctl lsp-set-addresses transit-tenant1 02:ac:10:ff:00:02
ovn-nbctl lsp-set-options transit-tenant1 router-port=tenant1-transit

# add static routes
ovn-nbctl lr-route-add edge1 "172.16.255.128/25" 172.16.255.2
ovn-nbctl lr-route-add tenant1 "0.0.0.0/0" 172.16.255.1

ovn-sbctl show

Notice the port bindings on the ubuntu1 host. You can now test connectivity to edge1 from vm1 on ubuntu2:

root@ubuntu2:~# ip netns exec vm1 ping 172.16.255.1
PING 172.16.255.1 (172.16.255.1) 56(84) bytes of data.
64 bytes from 172.16.255.1: icmp_seq=1 ttl=253 time=1.07 ms
64 bytes from 172.16.255.1: icmp_seq=2 ttl=253 time=1.13 ms
64 bytes from 172.16.255.1: icmp_seq=3 ttl=253 time=1.00 ms

Connecting to the “data” Network

We’re going to use the eth1 interface of ubuntu1 as our connection point between the edge1 router and the “data” network. In order to accomplish this we’ll need to set up OVN to used the eth1 interface directly through a dedicated OVS bridge. This type of connection is known as a “localnet” in OVN.

# create new port on router 'edge1'
ovn-nbctl lrp-add edge1 edge1-outside 02:0a:7f:00:01:29 10.127.0.129/25

# create new logical switch and connect it to edge1
ovn-nbctl ls-add outside
ovn-nbctl lsp-add outside outside-edge1
ovn-nbctl lsp-set-type outside-edge1 router
ovn-nbctl lsp-set-addresses outside-edge1 02:0a:7f:00:01:29
ovn-nbctl lsp-set-options outside-edge1 router-port=edge1-outside

# create a bridge for eth1
ovs-vsctl add-br br-eth1

# create bridge mapping for eth1. map network name "dataNet" to br-eth1
ovs-vsctl set Open_vSwitch . external-ids:ovn-bridge-mappings=dataNet:br-eth1

# create localnet port on 'outside'. set the network name to "dataNet"
ovn-nbctl lsp-add outside outside-localnet
ovn-nbctl lsp-set-addresses outside-localnet unknown
ovn-nbctl lsp-set-type outside-localnet localnet
ovn-nbctl lsp-set-options outside-localnet network_name=dataNet

# connect eth1 to br-eth1
ovs-vsctl add-port br-eth1 eth1

Test connectivity to edge1-outside from vm1

root@ubuntu2:~# ip netns exec vm1 ping 10.127.0.129
PING 10.127.0.129 (10.127.0.129) 56(84) bytes of data.
64 bytes from 10.127.0.129: icmp_seq=1 ttl=253 time=1.74 ms
64 bytes from 10.127.0.129: icmp_seq=2 ttl=253 time=0.781 ms
64 bytes from 10.127.0.129: icmp_seq=3 ttl=253 time=0.582 ms

Giving the Ubuntu Hosts Access to the “data” Network

Lets give the Ubuntu hosts a presence on the data network. For ubuntu2/ubuntu3 its simply a matter of setting an IP on their physical nics (eth1 in my setup). For ubuntu1 we’ll set an IP on the br-eth1 interface.

On ubuntu1:

ip addr add 10.127.0.130/24 dev br-eth1
ip link set br-eth1 up

On ubuntu2:

ip addr add 10.127.0.131/24 dev eth1
ip link set eth1 up

On ubuntu3:

ip addr add 10.127.0.132/24 dev eth1
ip link set eth1 up

Testing from ubuntu1 to edge1

root@ubuntu1:~# ping 10.127.0.129  
PING 10.127.0.129 (10.127.0.129) 56(84) bytes of data.
64 bytes from 10.127.0.129: icmp_seq=1 ttl=254 time=0.563 ms
64 bytes from 10.127.0.129: icmp_seq=2 ttl=254 time=0.290 ms
64 bytes from 10.127.0.129: icmp_seq=3 ttl=254 time=0.333 ms

Configuring NAT

Lets see what happens when we attempt to ping ubuntu1 from vm1:

root@ubuntu2:~# ip netns exec vm1 ping 10.127.0.130
PING 10.127.0.130 (10.127.0.130) 56(84) bytes of data.
^C
--- 10.127.0.130 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2016ms

Unsurprisingly, this does not work. Why not? Lets look at the output of a tcpdump from ubuntu1:

root@ubuntu1:~# tcpdump -i br-eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
14:41:53.057993 IP 172.16.255.130 > 10.127.0.130: ICMP echo request, id 19359, seq 1, length 64
14:41:54.065696 IP 172.16.255.130 > 10.127.0.130: ICMP echo request, id 19359, seq 2, length 64
14:41:55.073667 IP 172.16.255.130 > 10.127.0.130: ICMP echo request, id 19359, seq 3, length 64

We can see the requests coming in, however our responses are returning through a different interface (not seen in the tcpdump output). This is due to the fact that ubuntu1 has no route to 172.16.255.130 and is responding via its own default gateway. In order to get things working we will need to take 1 of 2 possible approaches:

  1. add static routes on the Ubuntu hosts
  2. set up NAT on the OVN gateway router

We’ll opt for option 2 because it is much less of a hassle than trying to manage static routes.

With OVN there are 3 types of NAT which may be configured:

  • DNAT – used to translate requests to an externally visible IP to an internal IP
  • SNAT – used to translate requests from one or more internal IPs to an externally visible IP
  • SNAT-DNAT – used to create a “static NAT” where an external IP is mapped to an internal IP, and vice versa

Since we don’t need (or want) the public network to be able to directly access our internal VMs, lets focus on allowing outbound SNAT from our VMs. In order to create NAT rules we’ll need to manipulate the OVN northbound database directly. The syntax is a bit strange but I’ll explain later. From ubuntu1:

# create snat rule which will nat to the edge1-outside interface
ovn-nbctl -- --id=@nat create nat type="snat" logical_ip=172.16.255.128/25 \
external_ip=10.127.0.129 -- add logical_router edge1 nat @nat

In brief, this command is creating an entry in the “nat” table of the northbound database, storing the resulting UUID within the ovsdb variable “@nat”, and then adding the UUID stored in @nat to the “nat” field of the “edge1” entry in the “logical_router” table of the northbound database. If you want to know the details of the northbound database then be sure to check out the man page for ovn-nb. The man page for ovn-nbctl also explains the command syntax used above.

Testing connectivity from vm1:

root@ubuntu2:~# ip netns exec vm1 ping 10.127.0.130
PING 10.127.0.130 (10.127.0.130) 56(84) bytes of data.
64 bytes from 10.127.0.130: icmp_seq=40 ttl=62 time=2.39 ms
64 bytes from 10.127.0.130: icmp_seq=41 ttl=62 time=1.61 ms
64 bytes from 10.127.0.130: icmp_seq=42 ttl=62 time=1.28 ms

As seen above, we can now ping the outside world from our internal VMs.

Final Words

Overlay networks are almost entirely useless unless you can connect them to the outside world. OVN gateways provide a means for making such a connection.

In the next post I will explore another important feature of OVN: the OVN load balancer.

Wednesday, September 21, 2016

An Introduction to OVN Routing

Untitled Document.md

Overview

Building upon my previous post I will now add basic layer-3 networking into the OVN setup. The end result will be a pair of logical switches connected by a logical router. As an added bonus, the router will be configured to serve IP addresses via the DHCP service which is built into OVN.

Re-Architecting the Logical Components

Since the setup is starting to become more complex we’re going to re-architect a bit. The new setup will be as follows:

  • 2 logical switches: “dmz” and “inside”
  • logical router “tenant1” connecting the two logical switches
  • IP network 172.16.255.128/26 for “dmz”
  • IP network 172.16.255.192/26 for “inside”
  • a pair of “VMs” on each logical switch

The proposed logical network is illustrated below.

ovn logical components

A Word On Routing

As part of our setup we will be creating an OVN router, also known as a “distributed logical router” (DLR). A DLR differs from a traditional router in that it is not an actual appliance but rather a logical construct (not unlike a logical switch). DLRs exist solely as a function in OVS: in other words each OVS instance is capable of simulating a layer-3 router hop locally prior to forwarding the traffic across the overlay network.

Creating the Logical Switches and Router

Define the new logical switches. From ubuntu1:

ovn-nbctl ls-add inside
ovn-nbctl ls-add dmz

Add the logical router with its associated router and switch ports:

# add the router
ovn-nbctl lr-add tenant1

# create router port for the connection to dmz
ovn-nbctl lrp-add tenant1 tenant1-dmz 02:ac:10:ff:01:29 172.16.255.129/26

# create the dmz switch port for connection to tenant1
ovn-nbctl lsp-add dmz dmz-tenant1
ovn-nbctl lsp-set-type dmz-tenant1 router
ovn-nbctl lsp-set-addresses dmz-tenant1 02:ac:10:ff:01:29
ovn-nbctl lsp-set-options dmz-tenant1 router-port=tenant1-dmz

# create router port for the connection to inside
ovn-nbctl lrp-add tenant1 tenant1-inside 02:ac:10:ff:01:93 172.16.255.193/26

# create the inside switch port for connection to tenant1
ovn-nbctl lsp-add inside inside-tenant1
ovn-nbctl lsp-set-type inside-tenant1 router
ovn-nbctl lsp-set-addresses inside-tenant1 02:ac:10:ff:01:93
ovn-nbctl lsp-set-options inside-tenant1 router-port=tenant1-inside

ovn-nbctl show

Adding DHCP

DHCP within OVN works a bit differently than most server solutions. The general idea is that the administrator will:

  1. define a set of DHCP options for use with a given subnet
  2. create logical switch ports which define both the mac address and the IP address expected to exist behind that port
  3. assign the DHCP options to that port.
  4. set port security to allow only the assigned addresses

Lets get started by configuring the logical ports for the (4) VMs we’ll be adding. From ubuntu1:

ovn-nbctl lsp-add dmz dmz-vm1
ovn-nbctl lsp-set-addresses dmz-vm1 "02:ac:10:ff:01:30 172.16.255.130"
ovn-nbctl lsp-set-port-security dmz-vm1 "02:ac:10:ff:01:30 172.16.255.130"

ovn-nbctl lsp-add dmz dmz-vm2
ovn-nbctl lsp-set-addresses dmz-vm2 "02:ac:10:ff:01:31 172.16.255.131"
ovn-nbctl lsp-set-port-security dmz-vm2 "02:ac:10:ff:01:31 172.16.255.131"

ovn-nbctl lsp-add inside inside-vm3
ovn-nbctl lsp-set-addresses inside-vm3 "02:ac:10:ff:01:94 172.16.255.194"
ovn-nbctl lsp-set-port-security inside-vm3 "02:ac:10:ff:01:94 172.16.255.194"

ovn-nbctl lsp-add inside inside-vm4
ovn-nbctl lsp-set-addresses inside-vm4 "02:ac:10:ff:01:95 172.16.255.195"
ovn-nbctl lsp-set-port-security inside-vm4 "02:ac:10:ff:01:95 172.16.255.195"

ovn-nbctl show

You may have noted that, unlike the previous lab, we are defining both mac and IP addresses as part of the logical switch definition. The IP address definition serves 2 purposes for us:

  1. It enables ARP suppression by allowing OVN to locally answer ARP requests for IP/mac combinations it knows about.
  2. It acts as a DHCP host assignment mechanism by issuing the defined IP address to any DHCP requests it sees from that port.

Next we need to define our DHCP options and assign them to our logical ports. The process here is going to be a bit different than we’ve seen before in that we will be directly interacting with the OVN NB database. The reason for this approach is that we need to capture the UUID of the DHCP_Options entry we create so that we can assign it to our switch ports. To do this we will capture the output of the ovn-nbctl command to a pair of bash variables.

dmzDhcp="$(ovn-nbctl create DHCP_Options cidr=172.16.255.128/26 \
options="\"server_id\"=\"172.16.255.129\" \"server_mac\"=\"02:ac:10:ff:01:29\" \
\"lease_time\"=\"3600\" \"router\"=\"172.16.255.129\"")" 
echo $dmzDhcp

insideDhcp="$(ovn-nbctl create DHCP_Options cidr=172.16.255.192/26 \
options="\"server_id\"=\"172.16.255.193\" \"server_mac\"=\"02:ac:10:ff:01:93\" \
\"lease_time\"=\"3600\" \"router\"=\"172.16.255.193\"")"
echo $insideDhcp

ovn-nbctl dhcp-options-list

See the man page for ovn-nb if you want to know more about the OVN NB database.

Now we’ll assign the DHCP_Options to our logical switch ports using the UUID stored within the variables.

ovn-nbctl lsp-set-dhcpv4-options dmz-vm1 $dmzDhcp
ovn-nbctl lsp-get-dhcpv4-options dmz-vm1

ovn-nbctl lsp-set-dhcpv4-options dmz-vm2 $dmzDhcp
ovn-nbctl lsp-get-dhcpv4-options dmz-vm2

ovn-nbctl lsp-set-dhcpv4-options inside-vm3 $insideDhcp
ovn-nbctl lsp-get-dhcpv4-options inside-vm3

ovn-nbctl lsp-set-dhcpv4-options inside-vm4 $insideDhcp
ovn-nbctl lsp-get-dhcpv4-options inside-vm4

Configuring the VMs

As in the last lab we will be using fake “VMs” using OVS internal ports and network namespaces. The difference now is that we will use DHCP for address assignments. Lets set up the VMs.

On ubuntu2:

ip netns add vm1
ovs-vsctl add-port br-int vm1 -- set interface vm1 type=internal
ip link set vm1 address 02:ac:10:ff:01:30
ip link set vm1 netns vm1
ovs-vsctl set Interface vm1 external_ids:iface-id=dmz-vm1
ip netns exec vm1 dhclient vm1
ip netns exec vm1 ip addr show vm1
ip netns exec vm1 ip route show

ip netns add vm3
ovs-vsctl add-port br-int vm3 -- set interface vm3 type=internal
ip link set vm3 address 02:ac:10:ff:01:94
ip link set vm3 netns vm3
ovs-vsctl set Interface vm3 external_ids:iface-id=inside-vm3
ip netns exec vm3 dhclient vm3
ip netns exec vm3 ip addr show vm3
ip netns exec vm3 ip route show

On ubuntu3:

ip netns add vm2
ovs-vsctl add-port br-int vm2 -- set interface vm2 type=internal
ip link set vm2 address 02:ac:10:ff:01:31
ip link set vm2 netns vm2
ovs-vsctl set Interface vm2 external_ids:iface-id=dmz-vm2
ip netns exec vm2 dhclient vm2
ip netns exec vm2 ip addr show vm2
ip netns exec vm2 ip route show

ip netns add vm4
ovs-vsctl add-port br-int vm4 -- set interface vm4 type=internal
ip link set vm4 address 02:ac:10:ff:01:95
ip link set vm4 netns vm4
ovs-vsctl set Interface vm4 external_ids:iface-id=inside-vm4
ip netns exec vm4 dhclient vm4
ip netns exec vm4 ip addr show vm4
ip netns exec vm4 ip route show

And testing connectivity from vm1 on ubuntu2:

# ping the default gateway on tenant1
root@ubuntu2:~# ip netns exec vm1 ping 172.16.255.129
PING 172.16.255.129 (172.16.255.129) 56(84) bytes of data.
64 bytes from 172.16.255.129: icmp_seq=1 ttl=254 time=0.689 ms
64 bytes from 172.16.255.129: icmp_seq=2 ttl=254 time=0.393 ms
64 bytes from 172.16.255.129: icmp_seq=3 ttl=254 time=0.483 ms

# ping vm2 through the overlay
root@ubuntu2:~# ip netns exec vm1  ping 172.16.255.131
PING 172.16.255.131 (172.16.255.131) 56(84) bytes of data.
64 bytes from 172.16.255.131: icmp_seq=1 ttl=64 time=2.16 ms
64 bytes from 172.16.255.131: icmp_seq=2 ttl=64 time=0.573 ms
64 bytes from 172.16.255.131: icmp_seq=3 ttl=64 time=0.446 ms

# ping vm3 through the router, via the local ovs bridge
root@ubuntu2:~# ip netns exec vm1  ping 172.16.255.194
PING 172.16.255.194 (172.16.255.194) 56(84) bytes of data.
64 bytes from 172.16.255.194: icmp_seq=1 ttl=63 time=1.37 ms
64 bytes from 172.16.255.194: icmp_seq=2 ttl=63 time=0.077 ms
64 bytes from 172.16.255.194: icmp_seq=3 ttl=63 time=0.076 ms

# ping vm4 through the router, across the overlay
root@ubuntu2:~# ip netns exec vm1  ping 172.16.255.195
PING 172.16.255.195 (172.16.255.195) 56(84) bytes of data.
64 bytes from 172.16.255.195: icmp_seq=1 ttl=63 time=1.79 ms
64 bytes from 172.16.255.195: icmp_seq=2 ttl=63 time=0.605 ms
64 bytes from 172.16.255.195: icmp_seq=3 ttl=63 time=0.503 ms

Final Words

OVN makes layer-3 overlay networking relatively easy and pain free. The fact that services such as DHCP are built directly into the system should help to reduce the number of external dependencies needed to build an effective SDN solution. In the next post I will discuss ways of connecting our (currently isolated) overlay network to the outside world.

Monday, September 19, 2016

A Primer on OVN

Untitled Document.md

Overview

OVN is a virtual networking platform developed by the fine folks over at openvswitch.org. The project was announced in early 2015 and has just recently released the first production ready version, version 2.6. In this posting I’ll walk through the basics of configuring a simple layer-2 overlay network between 3 hosts. But first, a brief overview of how the system functions.

OVN works on the premise of a distributed control plane where components are co-located on each node in the network. The roles within OVN are:

  • OVN Central – Currently a single host supports this role and this host acts as a central point of API integration by external resources such as a cloud management platform. The central control houses the OVN northbound database, which keeps track of high-level logical constructs such as logical switches/ports, and the OVN southbound database which determines how to map logical constructs in ovn-northdb to the physical world.
  • OVN Host – This role is distributed amongst all nodes which contain virtual networking end points such as VMs. The OVN Host contains a “chassis controller” which connects upstream to the ovn-southdb as its authoratative source of record for physical network information, and southbound to OVS for which it acts as an openflow controller.

The Lab

My lab is running as a nested setup on a single esxi machine. The OVN lab itself will consist of 3 Ubuntu 16.04 servers connected to a common management subnet (10.127.0.0/25). The hosts and their IP addresses are as follows:

  • ubuntu1 10.127.0.2 – will serve as OVN Central
  • ubuntu2 10.127.0.3 – will serve as an OVN Host
  • ubuntu3 10.127.0.4 – will serve as an OVN Host

The network setup for the lab is illustrated below.

ovn lab

For the sake of ease of testing I will simulate virtual machine workloads on the Ubuntu hosts by creating OVS internal interfaces and sandboxing them within a network namespace. The namespaces will ensure that our OVN overlay network is completely isolated from the lab network.

Building Open vSwitch 2.6

Open vSwitch version 2.6 was released on 2016/09/28 and may be downloaded from here. Since I am using a Ubuntu based system I found the instructions file INSTALL.Debian.md included with the download to work well. Below is a brief summary of the instructions. You should build ovs on either one of the three Ubuntu machines or on a dedicated build machine with an identical kernel version.

Update & install dependencies

apt-get update
apt-get -y install build-essential fakeroot

Install Build-Depends from debian/control file

apt-get -y install graphviz autoconf automake bzip2 debhelper dh-autoreconf libssl-dev libtool openssl
apt-get -y install procps python-all python-twisted-conch python-zopeinterface python-six

Check the working directory & build

cd openvswitch-2.6.0

# if everything is ok then this should return no output
dpkg-checkbuilddeps

`DEB_BUILD_OPTIONS='parallel=8 nocheck' fakeroot debian/rules binary`

The .deb files for ovs will be built and placed in the parent directory (ie. in …/). The next step is to build the kernel modules.

Install datapath sources

cd ..
apt-get -y install module-assistant
dpkg -i openvswitch-datapath-source_2.6.0-1_all.deb 

Build kernel modules using module-assistant

m-a prepare
m-a build openvswitch-datapath

Copy the resulting deb package. Note that your version may differ slightly depending on your specific kernel version.

cp /usr/src/openvswitch-datapath-module-*.deb ./

Transfer the following to all three Ubuntu hosts:

  • openvswitch-datapath-module-*.deb
  • openvswitch-common_2.6.0-1_amd64.deb
  • openvswitch-switch_2.6.0-1_amd64.deb
  • ovn-common_2.6.0-1_amd64.deb
  • ovn-central_2.6.0-1_amd64.deb
  • ovn-host_2.6.0-1_amd64.deb

Installing Open vSwitch

Install OVS/OVN + dependencies on ubuntu1:

apt-get update
apt-get -y install python-six python2.7
dpkg -i openvswitch-datapath-module-*.deb
dpkg -i openvswitch-common_2.6.0-1_amd64.deb openvswitch-switch_2.6.0-1_amd64.deb
dpkg -i ovn-common_2.6.0-1_amd64.deb ovn-central_2.6.0-1_amd64.deb ovn-host_2.6.0-1_amd64.deb

Install OVS/OVN + dependencies on ubuntu2 and ubuntu3:

apt-get update
apt-get -y install python-six python2.7
dpkg -i openvswitch-datapath-module-*.deb
dpkg -i openvswitch-common_2.6.0-1_amd64.deb openvswitch-switch_2.6.0-1_amd64.deb
dpkg -i ovn-common_2.6.0-1_amd64.deb ovn-host_2.6.0-1_amd64.deb

Once the packages are installed you should notice that ubuntu1 is now listening on the TCP ports associated with ovn-northd (6641) and the OVN southbound database (6642). Here is the output of a netstat on my system:

root@ubuntu1:~# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:6641            0.0.0.0:*               LISTEN      1798/ovsdb-server
tcp        0      0 0.0.0.0:6642            0.0.0.0:*               LISTEN      1806/ovsdb-server

If you do not see ovsdb-server listening on 6641/6642 then you may need to manually start the daemons via the init scripts (/etc/init.d/openvswitch-switch and /etc/init.d/ovn-central).

Creating the Integration Bridge

OVN will be responsible for managing a single bridge within OVS and anything that we wish to be connected to an OVN logical switch must be connected to this bridge. Although we can name this bridge anything we want, the standard convention is to name it “br-int”. We’ll want to verify that this bridge does not already exist and add it if needed.

On all ubuntu2/ubuntu3:

ovs-vsctl list-br

If you do not see “br-int” listed in the output then you’ll need to create the bridge manually. Remember, the integration bridge must exist on all hosts which will house VMs.

ovs-vsctl add-br br-int -- set Bridge br-int fail-mode=secure
ovs-vsctl list-br

The “fail-mode=secure” is a security feature which configures the bridge to drop traffic by default. This is important since we do not want tenants of the integration bridge to be able to communicate if our flows were somehow erased or if we plugged tenants into the bridge before the OVN controller was started.

Connecting the Chassis Controllers to the Central Controller

The next step in the setup is to connect the chassis controllers on ubuntu2/ubuntu3 to our central controller on ubuntu1.

On ubuntu2:

ovs-vsctl set open . external-ids:ovn-remote=tcp:10.127.0.2:6642
ovs-vsctl set open . external-ids:ovn-encap-type=geneve
ovs-vsctl set open . external-ids:ovn-encap-ip=10.127.0.3

On ubuntu3:

ovs-vsctl set open . external-ids:ovn-remote=tcp:10.127.0.2:6642
ovs-vsctl set open . external-ids:ovn-encap-type=geneve
ovs-vsctl set open . external-ids:ovn-encap-ip=10.127.0.4

These commands will trigger the OVN chassis controller on each host to open a connection to the OVN Central host on ubuntu1. We’ve specified that our overlay networks should use the geneve protocol to encapsulate our data plane traffic.

You may verify the connectivity with netstat. The output on my ubuntu3 machine is as follows:

root@ubuntu3:~# netstat -antp | grep 10.127.0.2
tcp        0      0 10.127.0.4:39256        10.127.0.2:6642         ESTABLISHED 3072/ovn-controller

Defining the Logical Network

Below is a diagram which illustrates the OVN logical components which will be created as part of this lab.

ovn logical components

Lets start by define the OVN logical switch “ls1” along with its associated logical ports. From ubuntu1:

# create the logical switch
ovn-nbctl ls-add ls1

# create logical port
ovn-nbctl lsp-add ls1 ls1-vm1
ovn-nbctl lsp-set-addresses ls1-vm1 02:ac:10:ff:00:11
ovn-nbctl lsp-set-port-security ls1-vm1 02:ac:10:ff:00:11

# create logical port
ovn-nbctl lsp-add ls1 ls1-vm2
ovn-nbctl lsp-set-addresses ls1-vm2 02:ac:10:ff:00:22
ovn-nbctl lsp-set-port-security ls1-vm2 02:ac:10:ff:00:22

ovn-nbctl show

In each command set we’ve created a uniquely named logical port. Normally these logical ports will be named with a UUID to ensure uniqueness but for our purposes we’ll use names which are more human friendly. We’re also defining which mac addresses we expect to be associated with each logical port and we take the further step of locking down the logical port using port security (allows only the macs we’ve listed to source from the logical port).

Adding “Fake” Virtual Machines

The next step is to create “virtual machines” which we’ll connect to our logical switch. As mentioned before, we’ll use OVS internal ports and network namespaces to simulate virtual machines. We’ll use the address space 172.16.255.0/24 for our logical switch.

On ubuntu2:

ip netns add vm1
ovs-vsctl add-port br-int vm1 -- set interface vm1 type=internal
ip link set vm1 netns vm1
ip netns exec vm1 ip link set vm1 address 02:ac:10:ff:00:11
ip netns exec vm1 ip addr add 172.16.255.11/24 dev vm1
ip netns exec vm1 ip link set vm1 up
ovs-vsctl set Interface vm1 external_ids:iface-id=ls1-vm1

ip netns exec vm1 ip addr show

On ubuntu3:

ip netns add vm2
ovs-vsctl add-port br-int vm2 -- set interface vm2 type=internal
ip link set vm2 netns vm2
ip netns exec vm2 ip link set vm2 address 02:ac:10:ff:00:22
ip netns exec vm2 ip addr add 172.16.255.22/24 dev vm2
ip netns exec vm2 ip link set vm2 up
ovs-vsctl set Interface vm2 external_ids:iface-id=ls1-vm2

ip netns exec vm2 ip addr show

If you’ve not worked much with namespaces or OVS then the above commands may be a bit confusing. In brief, we’re creating a network namespace named for our fake VM, adding an internal OVS port, adding the port to the namespace, setting up the IP interface from within the namespace (netns exec commands), and finally creating an OVS interface with an external_id set to the ID we defined for our logical port in OVN. Its this last bit which triggers OVS to alert OVN that a logical port has just come online. OVN acts upon this notification by pushing instructions down to the local chassis controller of the host machine which, in turn, pushes network flows down to OVS.

Note that I have explicitly set the mac addresses of these interfaces to match what we’ve defined in our OVN logical switch configuration. This is important. The logical network will not work if the mac addresses are not properly mapped. Keep in mind that I have actually reversed the workflow a bit here. Normally you would not change the mac address of the VM, but rather push an existing, known mac into OVN/OVS. My goal was to make the mac/IP easily visible for demonstration purposes, thus I manually set them.

Verifying and Testing Connectivity

From ubuntu1 we can verify the logical network configuration using ovn-sbctl. Here is the output from my system:

root@ubuntu1:~# ovn-sbctl show
Chassis "239f2c28-90ff-468f-a701-655585c630bf"
        hostname: "ubuntu3"
        Encap geneve
                ip: "10.127.0.4"
                options: {csum="true"}
        Port_Binding "ls1-vm2"
Chassis "517d558e-158a-4cb2-8870-283e9d39685e"
        hostname: "ubuntu2"
        Encap geneve
                ip: "10.127.0.3"
                options: {csum="true"}
        Port_Binding "ls1-vm1"

Note the port bindings associated to each of our OVN Host machines. In order to test connectivity we’ll simply launch a ping from the namespace of vm1. Here is the output from my machine:

root@ubuntu2:~# ip netns exec vm1 ping 172.16.255.22
PING 172.16.255.22 (172.16.255.22) 56(84) bytes of data.
64 bytes from 172.16.255.22: icmp_seq=1 ttl=64 time=1.60 ms
64 bytes from 172.16.255.22: icmp_seq=2 ttl=64 time=0.638 ms
64 bytes from 172.16.255.22: icmp_seq=3 ttl=64 time=0.344 ms

Adding a 3rd “VM” and Migrating It

Lets add a 3rd “VM” to our setup and then simulate migrating it between hosts. First, define its logical port using ovn-nbctl on ubuntu1:

ovn-nbctl lsp-add ls1 ls1-vm3
ovn-nbctl lsp-set-addresses ls1-vm3 02:ac:10:ff:00:33
ovn-nbctl lsp-set-port-security ls1-vm3 02:ac:10:ff:00:33

ovn-nbctl show

Next, create the interface for this VM on ubuntu2.

ip netns add vm3
ovs-vsctl add-port br-int vm3 -- set interface vm3 type=internal
ip link set vm3 netns vm3
ip netns exec vm3 ip link set vm3 address 02:ac:10:ff:00:33
ip netns exec vm3 ip addr add 172.16.255.33/24 dev vm3
ip netns exec vm3 ip link set vm3 up
ovs-vsctl set Interface vm3 external_ids:iface-id=ls1-vm3

ip netns exec vm3 ip addr show

Test connectivity from vm3.

root@ubuntu2:~# ip netns exec vm3 ping 172.16.255.22
PING 172.16.255.22 (172.16.255.22) 56(84) bytes of data.
64 bytes from 172.16.255.22: icmp_seq=1 ttl=64 time=2.04 ms
64 bytes from 172.16.255.22: icmp_seq=2 ttl=64 time=0.337 ms
64 bytes from 172.16.255.22: icmp_seq=3 ttl=64 time=0.536 ms

Note the OVN southbound DB configuration on ubuntu1. We see that ubuntu2 has 2 registered port bindings.

root@ubuntu1:~# ovn-sbctl show
Chassis "239f2c28-90ff-468f-a701-655585c630bf"
        hostname: "ubuntu3"
        Encap geneve
                ip: "10.127.0.4"
                options: {csum="true"}
        Port_Binding "ls1-vm2"
Chassis "517d558e-158a-4cb2-8870-283e9d39685e"
        hostname: "ubuntu2"
        Encap geneve
                ip: "10.127.0.3"
                options: {csum="true"}
        Port_Binding "ls1-vm3"
        Port_Binding "ls1-vm1"

In order to simulate a migration of vm3 we’ll delete the “vm3” namespace on ubuntu2, remove its port on br-int, and then recreate the setup on ubuntu3. From ubuntu2:

ip netns del vm3
ovs-vsctl --if-exists --with-iface del-port br-int vm3
ovs-vsctl list-ports br-int

From ubuntu3:

ip netns add vm3
ovs-vsctl add-port br-int vm3 -- set interface vm3 type=internal
ip link set vm3 netns vm3
ip netns exec vm3 ip link set vm3 address 02:ac:10:ff:00:33
ip netns exec vm3 ip addr add 172.16.255.33/24 dev vm3
ip netns exec vm3 ip link set vm3 up
ovs-vsctl set Interface vm3 external_ids:iface-id=ls1-vm3

And test connectivity:

root@ubuntu3:~# ip netns exec vm3 ping 172.16.255.11
PING 172.16.255.11 (172.16.255.11) 56(84) bytes of data.
64 bytes from 172.16.255.11: icmp_seq=1 ttl=64 time=1.44 ms
64 bytes from 172.16.255.11: icmp_seq=2 ttl=64 time=0.407 ms
64 bytes from 172.16.255.11: icmp_seq=3 ttl=64 time=0.395 ms

Again, note the OVN southbound DB configuration on ubuntu1. We see that the port binding has moved.

root@ubuntu1:~# ovn-sbctl show
Chassis "239f2c28-90ff-468f-a701-655585c630bf"
        hostname: "ubuntu3"
        Encap geneve
                ip: "10.127.0.4"
                options: {csum="true"}
        Port_Binding "ls1-vm2"
        Port_Binding "ls1-vm3"
Chassis "517d558e-158a-4cb2-8870-283e9d39685e"
        hostname: "ubuntu2"
        Encap geneve
                ip: "10.127.0.3"
                options: {csum="true"}
        Port_Binding "ls1-vm1"

Cleaning Up

Lets clean up the environment before quitting.

From ubuntu1:

# delete the logical switch and its ports
ovn-nbctl ls-del ls1

From ubuntu2:

# delete vm1
ip netns del vm1
ovs-vsctl --if-exists --with-iface del-port br-int vm1

From ubuntu3:

# delete vm2 and vm3
ip netns del vm2
ovs-vsctl --if-exists --with-iface del-port br-int vm2

ip netns del vm3
ovs-vsctl --if-exists --with-iface del-port br-int vm3

Final Words

As you can see, creating layer-2 overlay networks is relatively straight forward using OVN. In the next post I will discuss OVN layer-3 networking by introducing the OVN logical router.