Tuesday, August 9, 2016

Adding PXE support to Dnsmasq

Untitled Document.md

Overview

Expanding upon my previous post I will set up Dnsmasq to provide PXE services for my lab environment.

Installing syslinux

The first step is to copy the relevant syslinux files into my already existing /tftpboot directory. Since my primary use case with PXE is to boot esxi hypervisors, I’ll need syslinux version 3.86 (evidently later versions are not compatible with mboot.c32 included with esxi).

cd /tmp
wget https://www.kernel.org/pub/linux/utils/boot/syslinux/3.xx/syslinux-3.86.tar.gz
tar xzvf syslinux-3.86.tar.gz
cp /tmp/syslinux-3.86/core/pxelinux.0 /tftpboot/
cp /tmp/syslinux-3.86/memdisk/memdisk /tftpboot/
cp /tmp/syslinux-3.86/com32/menu/menu.c32 /tftpboot/
cd

Now I can create a basic PXE menu. To start, my menu will do nothing but boot local disk.

mkdir /tftpboot/pxelinux.cfg
cat > /tftpboot/pxelinux.cfg/default << EOF
DEFAULT menu.c32
PROMPT 0
TIMEOUT 300
ONTIMEOUT local

MENU TITLE PXE Server

LABEL local
MENU LABEL Boot local hard drive
LOCALBOOT 0

EOF

I can now append additional entries to the menu in order to support booting other operating systems.

Configuring an iPXE boot image

One use case I have in the lab is to perform an inital boot of a VM with the iPXE ISO image. I then use iPXE to bootstrap other images to the VM. For this, I’ll need to prepare the PXE server to serve up the iPXE image.

Grab the iPXE ISO image.

cd /tftpboot
wget http://boot.ipxe.org/ipxe.iso
cd

Add a PXE menu entry for ipxe.

cat >> /tftpboot/pxelinux.cfg/default << EOF
LABEL ipxe
MENU LABEL ipxe
linux memdisk
initrd ipxe.iso
append iso raw

EOF

Done. You should now be able to PXE boot an iPXE image… as odd as that sounds.

Configuring an esxi boot image

Another use case I have in the lab is the ability to kickstart physical hypervisors with esxi. For this, I’ll enable an esxi boot image with support for a kickstart file to automate the install.

Download the ISO image for esxi to /tmp of the CoreOS machine. For ease of scripting later on, store the esxi ISO file name in a variable. Adjust this per your exact version.

ESX_ISO=VMware-VMvisor-Installer-6.0.0-3620759.x86_64.iso

Mount the ISO and copy the files to /tftpboot, modifying the boot.cfg file to reflect the local directory structure.

mkdir /mnt/iso
mount /tmp/${ESX_ISO} /mnt/iso/
mkdir /tftpboot/esxi
rsync -a /mnt/iso/ /tftpboot/esxi
cat /mnt/iso/boot.cfg | sed -e "s#/##g" -e "3s#^#prefix=esxi\n#" > /tftpboot/esxi/boot.cfg
umount /mnt/iso

Add a PXE menu entry for esxi with support for a kickstart file. My environment listens on the interface mgmt0 port 8080, so change the MGMT_IF and HTTP_PORT variables below per your environment.

MGMT_IF=mgmt0
HTTP_PORT=8080
MGMT_IP=$(ip -o -4 addr show ${MGMT_IF} | awk '{print $4}' | cut -d/ -f1)

cat >> /tftpboot/pxelinux.cfg/default << EOF
LABEL esxi
MENU LABEL esxi
kernel esxi/mboot.c32
append -c esxi/boot.cfg ks=http://${MGMT_IP}:${HTTP_PORT}/esxi/kickstart.cfg
ipappend 2

EOF

Create the kickstart file. This assumes a working http server (as per this post).

mkdir /var/www/esxi
touch /var/www/esxi/kickstart.cfg

Edit /var/www/esxi/kickstart.cfg to look as below. Replace ‘aValidLicenseHere’ in the last line with an actual license, otherwise remove the line entirely. Also, replace the value of ‘rootpw’ with your own password.

accepteula
install --firstdisk --overwritevmfs
rootpw VMware1!
reboot

%include /tmp/networkconfig
%pre --interpreter=busybox

# extract network info from bootup
VMK_IF="vmk0"
VMK_NET=$(localcli network ip interface ipv4 get | grep "${VMK_IF}")
VMK_IP=$(echo "${VMK_NET}" | awk '{print $2}')
VMK_NETMASK=$(echo "${VMK_NET}" | awk '{print $3}')
GATEWAY=$(esxcfg-route | awk '{print $5}')
DNS=$(localcli network ip dns server list | grep 'DNS Servers' | awk '{print $3}'| cut -d, -f1)
HOSTNAME=$(nslookup "${VMK_IP}" | grep Address | grep "${VMK_IP}" | awk '{print $4}')

echo "network --bootproto=static --addvmportgroup=true --device=vmnic0 --ip=${VMK_IP} --netmask=${VMK_NETMASK} --gateway=${GATEWAY} --nameserver=${DNS} --hostname=${HOSTNAME}" > /tmp/networkconfig

%firstboot --interpreter=busybox

vim-cmd hostsvc/enable_ssh
vim-cmd hostsvc/start_ssh
vim-cmd hostsvc/enable_nest_shell
vim-cmd hostsvc/start_nest_shell
nestcli system settings advanced set -o /UserVars/SuppressShellWarning -i 1
vim-cmd vimsvc/license --set aValidLicenseHere

You should now be able to PXE boot esxi within the environment.

Thursday, August 4, 2016

CoreOS Local Mirror

Untitled Document.md

Overview

Expanding upon my previous post I will set up my CoreOS NAT gateway to provide a local etcd discovery service as well as to provide a local CoreOS mirror for use with installing additional nodes within my lab environment.

Enable etcd and configure the discovery service

In my orginal post I did not enable etcd. I will do so now.

Set up the systemd unit file. Adjust the value of ETC_ADV_CLIENT if needed.

ETC_ADV_CLIENT='http://10.127.0.1:2379'

cat > /etc/systemd/system/etcd2.service << EOF
[Unit]
Description=etcd2
Conflicts=etcd.service

[Service]
User=etcd
Type=notify
Environment=ETCD_DATA_DIR=/var/lib/etcd2
Environment=ETCD_NAME=%m
ExecStart=/usr/bin/etcd2 --listen-peer-urls 'http://0.0.0.0:2380' --listen-client-urls 'http://0.0.0.0:2379'  --advertise-client-urls '${ETC_ADV_CLIENT}'
Restart=always
RestartSec=10s
LimitNOFILE=40000
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target
EOF

Enable & start

systemctl enable etcd2.service
systemctl start etcd2.service

Allow etcd through iptables by adding the following to the *filter section of /var/lib/iptables/rules-save.

# allow etcd from mgmt network
-A INPUT -s 10.127.0.0/24 -p tcp --dport 2379:2380 -j ACCEPT

and apply the rules

iptables-restore /var/lib/iptables/rules-save

Using the local discovery service

In order to make use of the local discovery service you will simply create an entry within the well-known key prefix as documented here. You may then boot your cluster per the instructions. I make a slight syntactical change by using etcdctl rather than curl and setting a TTL on my registration key. Change the value of TTL_SECONDS to something sane for your use case. The key will automatically self-destruct at the end of the TTL period.

UUID=$(uuidgen)
TTL_SECONDS=300
etcdctl mkdir _etcd/registry/${UUID} --ttl ${TTL_SECONDS}
etcdctl set _etcd/registry/${UUID}/_config/size 3
echo $UUID

Create a web server app

In order to create a local mirror we need http services on our CoreOS machine. For this piece you can choose to use a pre-existing app container or you can follow this example and create your own using golang. If you choose the golang route then you’ll need a machine with a go compiler installed in order to create the binary. My example roughly follows the example given as part of the rkt documentation.

Create a file httpd.go

package main

import (
"log"
"net/http"
)

func main() {
log.Fatal(http.ListenAndServe("0.0.0.0:8080", http.FileServer(http.Dir("/var/www"))))
}

Compile with

CGO_ENABLED=0 go build -ldflags '-extldflags "-static"' httpd.go

Transfer the binary file httpd to the CoreOS machine (my machine is named natcore).

scp httpd natcore:/tmp

Create an app container

From the CoreOS machine we will create the directory structure for our ACI image and then use it to build the ACI. I am roughly following the example from the aci spec.

Create the directory structure for our app

mkdir /var/www
mkdir -p /tmp/go-httpd/rootfs/bin
mkdir -p /tmp/go-httpd/rootfs/var/www
mv /tmp/httpd /tmp/go-httpd/rootfs/bin

Create the manifest. Change the DOMAIN variable to your actual domain name.

DOMAIN=localdomain
cat > /tmp/go-httpd/manifest << EOF
{
    "acKind": "ImageManifest",
    "acVersion": "0.8.6",
    "name": "${DOMAIN}/go-httpd",
    "labels": [
        {"name": "version","value": "1.0.0"},
        {"name": "os", "value": "linux"},
        {"name": "arch", "value": "amd64"}
    ],
    "app": {
        "exec": [
            "/bin/httpd"
        ],
        "user": "0",
        "group": "0"
    }
}
EOF

Build the container

actool build /tmp/go-httpd/ /tmp/go-httpd.aci

Test the ACI image

rkt --insecure-options=image run --net=host /tmp/go-httpd.aci

From another shell verify that the app is listening on 8080

curl localhost:8080

Terminate the container with the escape sequence

^]^]^]

The ACI should now be visible to rkt.

rkt image list

Create the systemd unit file for go-httpd.service.

cat > /etc/systemd/system/go-httpd.service << EOF  
[Unit]
Description=go-httpd
ExecStartPre=/usr/bin/mkdir /var/www

[Service]
TimeoutStartSec=0
ExecStart=/usr/bin/rkt run --hostname=${HOSTNAME} --net=host \
--volume var-www,kind=host,source=/var/www \
${DOMAIN}/go-httpd:1.0.0 \
--mount volume=var-www,target=/var/www
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

Enable, start, and verify the go-httpd service.

systemctl enable go-httpd.service
systemctl start go-httpd.service
curl localhost:8080

Allow access through iptables by adding the following to the *filter section of /var/lib/iptables/rules-save.

# allow go-http access from mgmt network
-A INPUT -s 10.127.0.0/24 -p tcp --dport 8080 -j ACCEPT

and apply the rules

iptables-restore /var/lib/iptables/rules-save

Populate the local mirror

We’ll now copy the CoreOS production image files into /var/www. This will effectively create a local mirror which may then be used by the coreos-install script. For my example I am going to mirror CoreOS version 1010.6.0 but you should adjust the version according to your needs.

COS_VERSION=1010.6.0
mkdir -p /var/www/CoreOS/${COS_VERSION}
wget -r -l1 -np -nd "https://stable.release.core-os.net/amd64-usr/${COS_VERSION}/" -P /var/www/CoreOS/${COS_VERSION} -A "coreos_production_image*"
wget -r -l1 -np -nd "https://stable.release.core-os.net/amd64-usr/${COS_VERSION}/" -P /var/www/CoreOS/${COS_VERSION} -A "coreos_production_pxe*"

In order to make use of the local mirror when installing additional CoreOS machines, you will need to provide the -b option to coreos-install and specify the base URL for the local mirror. For example:

coreos-install -b http://10.127.0.1:8080/CoreOS -d /dev/sda -c cloud-config.yml

Monday, August 1, 2016

CoreOS + Dnsmasq

Untitled Document.md

Overview

Expanding upon my previous post I will add DHCP/TFTP services to my CoreOS NAT gateway using the dnsmasq image provided by quay.io.

Configure the Dnsmasq Container

All steps are performed from the console of the CoreOS machine.

Fetch the pre-made container from quay.io.

rkt fetch coreos.com/dnsmasq:v0.3.0

Set environment variables according to your setup. These will be used when generating config files for dnsmasq.

DNS_PRIMARY=8.8.8.8
DNS_DOMAIN=localdomain
PUB_IF=pub0
MGMT_IF=mgmt0
DATA_IF=data0
DMZ_IF=dmz0
MGMT_DHCP=10.127.0.128,10.127.0.254
DATA_DHCP=10.127.1.128,10.127.1.254
DMZ_DHCP=10.127.2.128,10.127.2.254

Create the systemd unit file for dnsmasq.service.

cat > /etc/systemd/system/dnsmasq.service << EOF  
[Unit]
Description=dnsmasq
ExecStartPre=/usr/bin/mkdir /etc/dnsmasq
ExecStartPre=/usr/bin/mkdir /tftpboot

[Service]
TimeoutStartSec=0
ExecStart=/usr/bin/rkt run --hostname=natcore --net=host \
--volume etc-dnsmasq,kind=host,source=/etc/dnsmasq \
--volume tftpboot,kind=host,source=/tftpboot \
coreos.com/dnsmasq:v0.3.0 \
--mount volume=etc-dnsmasq,target=/etc/dnsmasq \
--mount volume=tftpboot,target=/tftpboot \
-- -d -C /etc/dnsmasq/dnsmasq.conf -R -S ${DNS_PRIMARY}
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

Create the required directories for dnsmasq and create a symlink to /etc/hosts.

mkdir /tftpboot
mkdir /etc/dnsmasq
ln /etc/hosts /etc/dnsmasq/hosts

Create dnsmasq.conf

cat > /etc/dnsmasq/dnsmasq.conf << EOF
### GENERAL SETTINGS ###    
local=/${DNS_DOMAIN}/
domain=${DNS_DOMAIN}
expand-hosts
addn-hosts=/etc/dnsmasq/hosts

### TFTP SETTINGS ###    
dhcp-boot=pxelinux.0
enable-tftp
tftp-root=/tftpboot

### DHCP SETTINGS ###    
# ntp
dhcp-option=42,0.0.0.0    
# default gw set to mgmt network
dhcp-option=${MGMT_IF},3,10.127.0.1
# no services on public interface
no-dhcp-interface=${PUB_IF}

# ${MGMT_IF} dhcp
dhcp-range=${MGMT_IF},${MGMT_DHCP},12h

# ${DATA_IF} dhcp
dhcp-range=${DATA_IF},${DATA_DHCP},12h

# ${DMZ_IF} dhcp
dhcp-range=${DMZ_IF},${DMZ_DHCP},12h
EOF

Add some entries to /etc/hosts for natcore (optional).

cat >> /etc/hosts << EOF
10.127.0.1 mgmt0.natcore
10.127.1.1 data0.natcore
10.127.2.1 dmz0.natcore
EOF

Start the Container

Enable and start services.

systemctl enable /etc/systemd/system/dnsmasq.service
systemctl start dnsmasq.service

Verify the service.

systemctl status dnsmasq.service

Verify dnsmasq is listening on UDP 53/69.

netstat -lnup

If for some reason dnsmasq fails to start then you may view the logs using machinectl to find the machine ID of the container.

machinectl list

and using journalctl to view its logs (provide the id returned from machinectl)

journalctl -M {id}

Be sure to allow TFTP through iptables by adding the following to the *filter section of /var/lib/iptables/rules-save.

# allow tftp from mgmt network
-A INPUT -s 10.127.0.0/24 -p udp --dport 69 -j ACCEPT

and apply the rules

iptables-restore /var/lib/iptables/rules-save

As part of a later post I’ll walk through configuring PXE services using Dnsmasq.