Thursday, August 4, 2016

CoreOS Local Mirror



Expanding upon my previous post I will set up my CoreOS NAT gateway to provide a local etcd discovery service as well as to provide a local CoreOS mirror for use with installing additional nodes within my lab environment.

Enable etcd and configure the discovery service

In my orginal post I did not enable etcd. I will do so now.

Set up the systemd unit file. Adjust the value of ETC_ADV_CLIENT if needed.


cat > /etc/systemd/system/etcd2.service << EOF

ExecStart=/usr/bin/etcd2 --listen-peer-urls '' --listen-client-urls ''  --advertise-client-urls '${ETC_ADV_CLIENT}'


Enable & start

systemctl enable etcd2.service
systemctl start etcd2.service

Allow etcd through iptables by adding the following to the *filter section of /var/lib/iptables/rules-save.

# allow etcd from mgmt network
-A INPUT -s -p tcp --dport 2379:2380 -j ACCEPT

and apply the rules

iptables-restore /var/lib/iptables/rules-save

Using the local discovery service

In order to make use of the local discovery service you will simply create an entry within the well-known key prefix as documented here. You may then boot your cluster per the instructions. I make a slight syntactical change by using etcdctl rather than curl and setting a TTL on my registration key. Change the value of TTL_SECONDS to something sane for your use case. The key will automatically self-destruct at the end of the TTL period.

etcdctl mkdir _etcd/registry/${UUID} --ttl ${TTL_SECONDS}
etcdctl set _etcd/registry/${UUID}/_config/size 3
echo $UUID

Create a web server app

In order to create a local mirror we need http services on our CoreOS machine. For this piece you can choose to use a pre-existing app container or you can follow this example and create your own using golang. If you choose the golang route then you’ll need a machine with a go compiler installed in order to create the binary. My example roughly follows the example given as part of the rkt documentation.

Create a file httpd.go

package main

import (

func main() {
log.Fatal(http.ListenAndServe("", http.FileServer(http.Dir("/var/www"))))

Compile with

CGO_ENABLED=0 go build -ldflags '-extldflags "-static"' httpd.go

Transfer the binary file httpd to the CoreOS machine (my machine is named natcore).

scp httpd natcore:/tmp

Create an app container

From the CoreOS machine we will create the directory structure for our ACI image and then use it to build the ACI. I am roughly following the example from the aci spec.

Create the directory structure for our app

mkdir /var/www
mkdir -p /tmp/go-httpd/rootfs/bin
mkdir -p /tmp/go-httpd/rootfs/var/www
mv /tmp/httpd /tmp/go-httpd/rootfs/bin

Create the manifest. Change the DOMAIN variable to your actual domain name.

cat > /tmp/go-httpd/manifest << EOF
    "acKind": "ImageManifest",
    "acVersion": "0.8.6",
    "name": "${DOMAIN}/go-httpd",
    "labels": [
        {"name": "version","value": "1.0.0"},
        {"name": "os", "value": "linux"},
        {"name": "arch", "value": "amd64"}
    "app": {
        "exec": [
        "user": "0",
        "group": "0"

Build the container

actool build /tmp/go-httpd/ /tmp/go-httpd.aci

Test the ACI image

rkt --insecure-options=image run --net=host /tmp/go-httpd.aci

From another shell verify that the app is listening on 8080

curl localhost:8080

Terminate the container with the escape sequence


The ACI should now be visible to rkt.

rkt image list

Create the systemd unit file for go-httpd.service.

cat > /etc/systemd/system/go-httpd.service << EOF  
ExecStartPre=/usr/bin/mkdir /var/www

ExecStart=/usr/bin/rkt run --hostname=${HOSTNAME} --net=host \
--volume var-www,kind=host,source=/var/www \
${DOMAIN}/go-httpd:1.0.0 \
--mount volume=var-www,target=/var/www


Enable, start, and verify the go-httpd service.

systemctl enable go-httpd.service
systemctl start go-httpd.service
curl localhost:8080

Allow access through iptables by adding the following to the *filter section of /var/lib/iptables/rules-save.

# allow go-http access from mgmt network
-A INPUT -s -p tcp --dport 8080 -j ACCEPT

and apply the rules

iptables-restore /var/lib/iptables/rules-save

Populate the local mirror

We’ll now copy the CoreOS production image files into /var/www. This will effectively create a local mirror which may then be used by the coreos-install script. For my example I am going to mirror CoreOS version 1010.6.0 but you should adjust the version according to your needs.

mkdir -p /var/www/CoreOS/${COS_VERSION}
wget -r -l1 -np -nd "${COS_VERSION}/" -P /var/www/CoreOS/${COS_VERSION} -A "coreos_production_image*"
wget -r -l1 -np -nd "${COS_VERSION}/" -P /var/www/CoreOS/${COS_VERSION} -A "coreos_production_pxe*"

In order to make use of the local mirror when installing additional CoreOS machines, you will need to provide the -b option to coreos-install and specify the base URL for the local mirror. For example:

coreos-install -b -d /dev/sda -c cloud-config.yml