Tuesday, October 4, 2016

OVN and Containers

Untitled Document.md

Overview

Following up on my previous post, the subject of this discussion is OVN integration with containers. By the end of this lab we will have created a container host “VM” which houses a pair of containers. These containers will be tied direcly into an OVN logical switch and will be reachable directly from all VMs within the logical network.

The OVN Container Networking Model

According to the man page for ovn-architecture, OVN’s container networking strategy of choice is to use a VLAN trunk connection to the conainer host VM and require that traffic from each container be isolated within a unique VLAN. This, of course, means that some level of coordination must take place between OVN and the container host to ensure that they are in sync regarding which VLAN tag is being used for a given container. It also places a certain level of responsibility on the container host to make sure that containers are properly isolated from one another internally.

Going into a bit more detail, the basic idea is that within OVN you will create a logical port representing the connection to the host VM. You will then define logical ports for your containers, mapping them to the “parent” VM logical port, and defining a VLAN tag to use. OVN will then configure OVS flows which will map VLAN tagged traffic from the parent VM’s logical port to the appropriate container logical port. The diagram below illustrates this design.

ovn containers

The Existing Setup

Take a moment to review the current setup prior to continuing.

The lab network:

ovn lab

The OVN logical network:

ovn logical components

Defining the Logical Network

For this lab we are going to create a new fake “VM”, vm5, which will host our fake “containers”. The new VM will plug into the existing DMZ switch alongside vm1 and vm2. We’re going to use DHCP for both the new VM and its containers.

Before creating the logical port for vm5 we need to locate the DHCP options that we created for the DMZ network during the previous lab. We’ll query the OVN northbound DB directly for this information. Here is the output on my system.

root@ubuntu1:~# ovn-nbctl list DHCP_Options
_uuid               : 7e32cec4-957e-46fa-a4cc-34218e1e17c8
cidr                : "172.16.255.192/26"
external_ids        : {}
options             : {lease_time="3600", router="172.16.255.193", server_id="172.16.255.193", server_mac="02:ac:10:ff:01:93"}

_uuid               : c0c29381-c945-4507-922a-cb87f76c4581
cidr                : "172.16.255.128/26"
external_ids        : {}
options             : {lease_time="3600", router="172.16.255.129", server_id="172.16.255.129", server_mac="02:ac:10:ff:01:29"}

We want the UUID for the “172.16.255.128/26” network (c0c29381-c945-4507-922a-cb87f76c4581 in my case). Capture this UUID for use in later commands.

Lets create the logical port for vm5. This should look pretty familiar. Be sure to replace {uuid} with the UUID from the DHCP options entry above. From ubuntu1:

ovn-nbctl lsp-add dmz dmz-vm5
ovn-nbctl lsp-set-addresses dmz-vm5 "02:ac:10:ff:01:32 172.16.255.132"
ovn-nbctl lsp-set-port-security dmz-vm5 "02:ac:10:ff:01:32 172.16.255.132"
ovn-nbctl lsp-set-dhcpv4-options dmz-vm5 {uuid}

Now we will create the logical ports for the containers which live on vm5. This process is nearly identical to creating a normal logical port but with a couple of additional settings. From ubuntu1:

# create the logical port for c51
ovn-nbctl lsp-add dmz dmz-c51
ovn-nbctl lsp-set-addresses dmz-c51 "02:ac:10:ff:01:33 172.16.255.133"
ovn-nbctl lsp-set-port-security dmz-c51 "02:ac:10:ff:01:33 172.16.255.133"
ovn-nbctl lsp-set-dhcpv4-options dmz-c51 {uuid}

# set the parent logical port and vlan tag for c51
ovn-nbctl set Logical_Switch_Port dmz-c51 parent_name=dmz-vm5
ovn-nbctl set Logical_Switch_Port dmz-c51 tag=51

# create the logical port for c52
ovn-nbctl lsp-add dmz dmz-c52
ovn-nbctl lsp-set-addresses dmz-c52 "02:ac:10:ff:01:34 172.16.255.134"
ovn-nbctl lsp-set-port-security dmz-c52 "02:ac:10:ff:01:34 172.16.255.134"
ovn-nbctl lsp-set-dhcpv4-options dmz-c52 {uuid}

# set the parent logical port and vlan tag for c52
ovn-nbctl set Logical_Switch_Port dmz-c52 parent_name=dmz-vm5
ovn-nbctl set Logical_Switch_Port dmz-c52 tag=52

So the only real difference here is that we’ve set a parent_name and tag for the container logical ports. You can validate these by looking at the database entries. For example, the output on my system:

root@ubuntu1:~# ovn-nbctl find Logical_Switch_Port name="dmz-c51"
_uuid               : ea604369-14a9-4e25-998f-ec99c2e7e47e
addresses           : ["02:ac:10:ff:01:31 172.16.255.133"]
dhcpv4_options      : c0c29381-c945-4507-922a-cb87f76c4581
dhcpv6_options      : []
dynamic_addresses   : []
enabled             : []
external_ids        : {}
name                : "dmz-c51"
options             : {}
parent_name         : "dmz-vm5"
port_security       : ["02:ac:10:ff:01:31 172.16.255.133"]
tag                 : 51
tag_request         : []
type                : ""
up                  : false

Configuring vm5

The first thing to remember about this lab is that we’re not using real VMs, but rather simulating them as ovs internal ports directly on the Ubuntu hosts. For vm1-vm4 we’ve been creating these internal ports directly on br-int but for vm5 our requirements are a bit different so we’ll be using a dedicated ovs bridge. This bridge, called br-vm5, will not be managed by OVN and will simulate how you might actually configure an ovs bridge internal to a real container host VM. This bridge will provide local networking to both the VM and its containers and will be configured to perform VLAN tagging. Below is a diagram illustrating how things will look when we’re done.

ovn container lab

My lab setup is simple in that I’m placing the containers all on the same logical switch. However, there is no requirement to do this since I can place the container logical port on any logical switch that I wish.

Lets get started. The first step is to create the setup for vm5. We’re going to do this on the ubuntu2 host. From ubuntu2:

# create the bridge for vm5
ovs-vsctl add-br br-vm5

# create patch port on br-vm5 to br-int
ovs-vsctl add-port br-vm5 brvm5-brint -- set Interface brvm5-brint type=patch options:peer=brint-brvm5

# create patch port on br-int to br-vm5. set external id to dmz-vm5 since this is our connection to vm5 
ovs-vsctl add-port br-int brint-brvm5 -- set Interface brint-brvm5 type=patch options:peer=brvm5-brint
ovs-vsctl set Interface brint-brvm5 external_ids:iface-id=dmz-vm5

# create vm5 within a namespace. vm5 traffic will be untagged
ovs-vsctl add-port br-vm5 vm5 -- set interface vm5 type=internal
ip link set vm5 address 02:ac:10:ff:01:32
ip netns add vm5
ip link set vm5 netns vm5
ip netns exec vm5 dhclient vm5

Verify connectivity from vm5 by pinging its default gateway.

root@ubuntu2:~# ip netns exec vm5 ping 172.16.255.129
PING 172.16.255.129 (172.16.255.129) 56(84) bytes of data.
64 bytes from 172.16.255.129: icmp_seq=1 ttl=254 time=0.797 ms
64 bytes from 172.16.255.129: icmp_seq=2 ttl=254 time=0.509 ms
64 bytes from 172.16.255.129: icmp_seq=3 ttl=254 time=0.404 ms

Configuring the vm5 “Containers”

Now that vm5 is up and working we can configure its fake “containers”. These are going to look almost exactly like our fake “vms” with the exception that we’re configuring them for vlan tagging.

# create c51 within a namespace. c51 traffic will be tagged with vlan 51
ip netns add c51
ovs-vsctl add-port br-vm5 c51 tag=51 -- set interface c51 type=internal
ip link set c51 address 02:ac:10:ff:01:33
ip link set c51 netns c51
ip netns exec vm5 dhclient c51

# create c52 within a namespace. c52 traffic will be tagged with vlan 52
ip netns add c52
ovs-vsctl add-port br-vm5 c52 tag=52 -- set interface c52 type=internal
ip link set c52 address 02:ac:10:ff:01:34
ip link set c52 netns c52
ip netns exec c52 dhclient c52

Verify connectivity.

root@ubuntu2:~# ip netns exec c51 ping 172.16.255.129
PING 172.16.255.129 (172.16.255.129) 56(84) bytes of data.
64 bytes from 172.16.255.129: icmp_seq=1 ttl=254 time=1.33 ms
64 bytes from 172.16.255.129: icmp_seq=2 ttl=254 time=0.420 ms
64 bytes from 172.16.255.129: icmp_seq=3 ttl=254 time=0.371 ms

root@ubuntu2:~# ip netns exec c52 ping 172.16.255.129
PING 172.16.255.129 (172.16.255.129) 56(84) bytes of data.
64 bytes from 172.16.255.129: icmp_seq=1 ttl=254 time=1.53 ms
64 bytes from 172.16.255.129: icmp_seq=2 ttl=254 time=0.533 ms
64 bytes from 172.16.255.129: icmp_seq=3 ttl=254 time=0.355 ms

Final Words

As per the ovn-architecture guide, if you run containers directly on a hypervisor or otherwise patch them directly into the integration bridge then they have the potential to completely bog down an OVN system depending on the scale of your setup. The “nested” network solution is nice in that it greatly reduces the number of VIFs on the integration bridge and thus minimizes the performance hit which containers would otherwise entail. Again, the point of this exercise was not to set up a real world container simulation, but rather to demonstrate the in-built container networking feature set of OVN.

Monday, October 3, 2016

OVN and ACLs

Untitled Document.md

Overview

Building upon my previous post I will now examine basic network security using OVN Access Control Lists.

OVN Access Control Lists and Address Sets

ACLs within OVN are implemented in the ACL table of the northbound DB and may be implemented using the acl commands of ovn-nbctl. At present, ACLs may only be applied to logical switches but it is conceivable that the ability to apply them to routers would be a future enhancement.

ACLs are applied and evaluated in one of two directions: ingress to a logical port from a workload (to-lport) and egress from a logical port to a workload (from-lport). Additionally, every ACL is assigned a priority which determines their order of evaluation. The highest priority is evaluated first and ACLs may be given identical priorities. However, in the case of two ACLs having an identical priority and both matching a given packet, only one will be evaluated. Exactly which one ultimately matches is indeterminite meaning you can’t really be sure which rule will be applied in a given situation. Moral of the story: try to use unique priorities in most cases.

The rules for matching in ACLs are based on the flow syntax from OVS and should look familiar to anyone with a programming background. The syntax is explained in the “Logical_Flow table” section of the man page for ovn-sb. Its worth a read. In particular you should pay attention to the section discussing the “!=” match rule. It is also worth highlighting the point that you can not create an ACL matching on a port with type=router.

In order to reduce the number of entries in the ACL table you may make use of address sets which define groups of identical type addresses. For example, a group of IPv4 addresses/networks, a group of mac addresses, or a group of IPv6 addresses may be placed within a named address set. Address sets can then be referenced by name (in the form of $name) within the match clause of an ACL.

Lets run through some samples.

# allow all ip traffic from port "ls1-vm1" on switch "ls1" and allowing related connections back in
ovn-nbctl acl-add ls1 from-lport 1000 "inport == \"ls1-vm1\" && ip" allow-related

# allow ssh to ls1-vm1
ovn-nbctl acl-add ls1 to-lport 999 "outport == \"ls1-vm1\" && tcp.dst == 22" allow-related

# block all IPv4/IPv6 traffic to ls1-vm1
ovn-nbctl acl-add ls1 to-lport 998 "outport == \"ls1-vm1\" && ip" drop

Note the use of “allow-related”. This is doing exactly what it seems like in that, under the covers, it is permitting related traffic through in the opposite direction (eg. responses, fragements, etc…). In the second rule I have used allow-related in order to allow responses to ssh back from the server.

Lets take a look at address sets. As mentioned earlier, address sets are groups of addresses of the same type. Address sets are created using the database commands of ovn-nbctl and are applied to ACLs using the name of the address set. Here are some examples:

ovn-nbctl create Address_Set name=wwwServers addresses=172.16.1.2,172.16.1.3
ovn-nbctl create Address_Set name=www6Servers addresses=\"fd00::1\",\"fd00::2\"
ovn-nbctl create Address_Set name=macs addresses=\"02:00:00:00:00:01\",\"02:00:00:00:00:02\"

Note the use of double quotes with address sets containing the “:” character. You’ll get an error if you don’t quote these.

Lab Testing

Lets experiment with ACLs in our lab environment. Here is a quick review of the setup.

The lab network:

ovn lab

The OVN logical network:

ovn logical components

As a first step we’ll open up external access to the servers in our DMZ tier by creating a static NAT rule for each of them.

From ubuntu1:

# create snat-dnat rule for vm1 & apply to edge1
ovn-nbctl -- --id=@nat create nat type="dnat_and_snat" logical_ip=172.16.255.130 \
external_ip=10.127.0.250 -- add logical_router edge1 nat @nat

# create snat-dnat rule for vm2 & apply to edge1
ovn-nbctl -- --id=@nat create nat type="dnat_and_snat" logical_ip=172.16.255.131 \
external_ip=10.127.0.251 -- add logical_router edge1 nat @nat

Test connectivity from ubuntu1.

root@ubuntu1:~# ping 10.127.0.250
PING 10.127.0.250 (10.127.0.250) 56(84) bytes of data.
64 bytes from 10.127.0.250: icmp_seq=1 ttl=62 time=2.57 ms
64 bytes from 10.127.0.250: icmp_seq=2 ttl=62 time=1.23 ms
64 bytes from 10.127.0.250: icmp_seq=3 ttl=62 time=0.388 ms

root@ubuntu1:~# ping 10.127.0.251
PING 10.127.0.251 (10.127.0.251) 56(84) bytes of data.
64 bytes from 10.127.0.251: icmp_seq=1 ttl=62 time=3.15 ms
64 bytes from 10.127.0.251: icmp_seq=2 ttl=62 time=1.52 ms
64 bytes from 10.127.0.251: icmp_seq=3 ttl=62 time=0.475 ms

Also, check that the VMs can connect to the outside world using the proper IPs.

root@ubuntu2:~# ip netns exec vm1 ping 10.127.0.130
PING 10.127.0.130 (10.127.0.130) 56(84) bytes of data.
64 bytes from 10.127.0.130: icmp_seq=1 ttl=62 time=3.05 ms

root@ubuntu3:~# ip netns exec vm2 ping 10.127.0.130
PING 10.127.0.130 (10.127.0.130) 56(84) bytes of data.
64 bytes from 10.127.0.130: icmp_seq=1 ttl=62 time=1.87 ms

root@ubuntu1:~# tcpdump -i br-eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
17:51:01.055258 IP 10.127.0.250 > 10.127.0.130: ICMP echo request, id 4565, seq 12, length 64
17:51:01.055320 IP 10.127.0.130 > 10.127.0.250: ICMP echo reply, id 4565, seq 12, length 64

17:51:56.378089 IP 10.127.0.251 > 10.127.0.130: ICMP echo request, id 4301, seq 6, length 64
17:51:56.378160 IP 10.127.0.130 > 10.127.0.251: ICMP echo reply, id 4301, seq 6, length 64

Excellent. We can see from the tcpdump on ubuntu1 that the VMs are using their proper NAT addresses. Lets apply some security policy. Firstly, we’ll completely lock down the DMZ.

# default drop
ovn-nbctl acl-add dmz to-lport 900 "outport == \"dmz-vm1\" && ip" drop
ovn-nbctl acl-add dmz to-lport 900 "outport == \"dmz-vm2\" && ip" drop

Lets do a quick access check from ubuntu1.

root@ubuntu1:~# ping 10.127.0.250
PING 10.127.0.250 (10.127.0.250) 56(84) bytes of data.
^C
--- 10.127.0.250 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1007ms

root@ubuntu1:~# ping 10.127.0.251
PING 10.127.0.251 (10.127.0.251) 56(84) bytes of data.
^C
--- 10.127.0.251 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1007ms

DMZ servers are now unreachable externally, however we’ve also managed to kill their outbound access.

root@ubuntu2:~# ip netns exec vm1 ping 10.127.0.130
PING 10.127.0.130 (10.127.0.130) 56(84) bytes of data.
^C
--- 10.127.0.130 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1008ms

root@ubuntu3:~# ip netns exec vm2 ping 10.127.0.130
PING 10.127.0.130 (10.127.0.130) 56(84) bytes of data.
^C
--- 10.127.0.130 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1008ms

Lets fix that.

# allow all ip trafficand allowing related connections back in
ovn-nbctl acl-add dmz from-lport 1000 "inport == \"dmz-vm1\" && ip" allow-related
ovn-nbctl acl-add dmz from-lport 1000 "inport == \"dmz-vm2\" && ip" allow-related

And verify.

root@ubuntu2:~# ip netns exec vm1 ping 10.127.0.130
PING 10.127.0.130 (10.127.0.130) 56(84) bytes of data.
64 bytes from 10.127.0.130: icmp_seq=1 ttl=62 time=4.16 ms
64 bytes from 10.127.0.130: icmp_seq=2 ttl=62 time=3.07 ms

root@ubuntu3:~# ip netns exec vm2 ping 10.127.0.130
PING 10.127.0.130 (10.127.0.130) 56(84) bytes of data.
64 bytes from 10.127.0.130: icmp_seq=1 ttl=62 time=3.59 ms
64 bytes from 10.127.0.130: icmp_seq=2 ttl=62 time=2.30 ms

Lets allow inbound https to the DMZ servers.

# allow tcp 443 in and related connections back out
ovn-nbctl acl-add dmz to-lport 1000 "outport == \"dmz-vm1\" && tcp.dst == 443" allow-related
ovn-nbctl acl-add dmz to-lport 1000 "outport == \"dmz-vm2\" && tcp.dst == 443" allow-related

Lets verify. For this we’ll need something listening on tcp 443. I like to use ncat, so the first step is to install it on all 3 Ubuntu hosts. It is actually part of the nmap package.

apt-get -y install nmap

Now we can start a process to listen on 443. The process will terminate at the end of the connection but you can use the -k flag to keep it going if you want.

From ubuntu2:

ip netns exec vm1 ncat -l -p 443

From ubuntu3:

ip netns exec vm2 ncat -l -p 443

And check connectivity from ubuntu1. If the connection succeeds then it will stay open until you terminate it. If not, then it should time out after 1 second.

root@ubuntu1:~# ncat -w 1 10.127.0.250 443
^C

root@ubuntu1:~# ncat -w 1 10.127.0.251 443
^C

That worked. Lets secure the “inside” servers as well. We’ll make it really tight, blocking all outbound access and only allowing access to tcp 3306 from the dmz. We’ll use an address set for allowing access from the DMZ. Note the use of the single quotes in the “acl-add” commands for allowing access to 3306. This is important. We’re actually referring to the address set by its literal name prefixed with a ‘$’ character. We don’t want bash to interpret this as a variable which is why we use the single quote.

# create an address set for the dmz servers. they fall within a common /31
ovn-nbctl create Address_Set name=dmz addresses=\"172.16.255.130/31\"

# allow from dmz on 3306
ovn-nbctl acl-add inside to-lport 1000 'outport == "inside-vm3" && ip4.src == $dmz && tcp.dst == 3306' allow-related
ovn-nbctl acl-add inside to-lport 1000 'outport == "inside-vm4" && ip4.src == $dmz && tcp.dst == 3306' allow-related

# default drop
ovn-nbctl acl-add inside to-lport 900 "outport == \"inside-vm3\" && ip" drop
ovn-nbctl acl-add inside to-lport 900 "outport == \"inside-vm4\" && ip" drop

Again, we’ll use ncat to listen on our VMs but this time well start it on vm3/vm4

From ubuntu2:

ip netns exec vm3 ncat -l -p 3306

From ubuntu3:

ip netns exec vm4 ncat -l -p 3306

Check connectivity from dmz to inside:

root@ubuntu2:~# ip netns exec vm1 ncat -w 1 172.16.255.195 3306
^C

root@ubuntu3:~# ip netns exec vm2 ncat -w 1 172.16.255.194 3306
    ^C

That seems to have worked. One final check. Lets make sure that vm3/vm4 are isolated from each other.

root@ubuntu2:~# ip netns exec vm3 ncat -w 1 172.16.255.195 3306
Ncat: Connection timed out.

root@ubuntu3:~# ip netns exec vm4 ncat -w 1 172.16.255.194 3306
Ncat: Connection timed out.

Clean up

Be sure to remove the ACLs and address sets prior to quitting. From ubuntu1:

ovn-nbctl acl-del dmz
ovn-nbctl acl-del inside
ovn-nbctl destroy Address_Set dmz

Final Words

Traditionally it has been that firewalling was performed on an in-path, layer-3 device such as a router or dedicated firewall appliance. In this regard it may seem odd that OVN applies policy on the logical switch, however in reality this approach is advantageous in that it creates an easy way to secure workloads in an east-west fashion by enforcing security at the logical port level. This approach to security has come to be known as “micro segmentation” due to the fact that it allows an administrator to apply security policy in a very fine grained manner. When you think about it, much of the heirarchy that exists in traditional network design (think web-tier/db-tier) is due to the fact that network security could previously only be done on some central appliance which sat between tiers. The micro segmentation approach actually allows you to flatten your designs to the point where you may end up having a single logical switch for everything with the “tiers” describing the security policy rather than describing the network layout.

In the next post I will cover the OVN model for enabling container networking.