Revisit: Setting Up an Overlay Network on Docker without Docker Swarm

In 2016, I wrote a simple post explaining how to set up an Overlay Network across multiple hosts running Docker Engine. Unfortunately, a small number of people were unable to achieve the same results – sometimes due to the firewall, or incorrect settings.

In this walk-through, I go into the specific details on how to create a Docker Overlay Network across multiple hosts without using Docker Swarm.

Mục Lục

Background

With the latest version of Docker, Docker Swarm is the go-to option for multi-host networking. However, it is possible to achieve multi-host networking via an overlay network without swarm mode.

To do this, an external key-value store is required to maintain the state of the overlay network(s),

However, Docker documentation does warn that this method is not recommended for most Docker users, and may be deprecated in the future. To be fair, I wrote the original post before Docker Engine provided native support for Docker clusters (before version 1.12).

Do note that at the time of this post, Docker Engine(s) that use Swarm to create container clusters are not able to simultaneously support both methods.

Architecture

Hardware
- Operating System: CentOS 7 (Base Environment: Minimal Install, 64 bit)
- RAM: 2 Gibibytes
- HDD: 20 Gigabytes
- CPU: 1 core
Software
- Docker Engine Version 17.12.0-ce

Installation Steps

Step 1: Installing and Configuring the Hosts

I created 4 VMs/hosts with the above hardware and software specifications. Each VM has 2 network connections – a NAT (interface name “enp0s3”), and a Host-only Adapter (interface name “enp0s8”).

Upon successful installation, I used nmtui to configure the static IP addresses and their hostname:

Docker Host 1
- Host-only Adapter Static IP Address: 192.168.56.101/24
- Hostname: docker-host1
Docker Host 2
- Host-only Adapter Static IP Address: 192.168.56.102/24
- Hostname: docker-host2
Docker Host 3
- Host-only Adapter Static IP Address: 192.168.56.103/24
- Hostname: docker-host3
Docker Host 4
- Host-only Adapter Static IP Address: 192.168.56.104/24
- Hostname: docker-host4

The hostname has to be unique across the cluster nodes, as the Key-Value store uses it to differentiate the nodes from each other, and to route traffic between them.

Depending on your Host-only Network configuration, your IP address may be slightly different. Ensure that each Host can communicate with each other via the Host-only Network (e.g. Have each host to ping the other 3’s Host-only IP address)

Step 2: Installing Docker Engine (CE)

I followed the excellent instructions here: https://docs.docker.com/engine/installation/linux/docker-ce/centos/

I added my non-root user into the docker group, and configured Docker Engine to start automatically. (Refer to Docker’s Post Installation Steps for Linux – https://docs.docker.com/engine/installation/linux/linux-postinstall/)

Step 3: Setup a Key-Value Store

In this part, we will set up the Key-Value store used by the Docker daemons.

An overlay network requires a Key-Value store – it holds information about the network state which includes discovery, networks, endpoints, IP addresses, and more. Docker supports Consul, Etcd, and ZooKeeper. This post uses Consul.

I have designated Docker Host 1 to host the required Key-Value store for the multi-host overlay network. In Docker Host 1, we will start the Consul container:

$ docker run -d -p 8500:8500 -h consul --name consul progrium/consul -server -bootstrap

Recall that the -p flag is to bind the port of the container to the host. Meaning that Docker Host 1’s port 8500 will be connected to the consul container. Double check that the port binding of the consul container has been successful:

$ docker port consul
8500/tcp -> 0.0.0.0:8500

Lastly, use firewall-cmd to allow port 8500 via the TCP protocol

# firewall-cmd --permanent --add-port 8500/tcp
success
# firewall-cmd --reload
success
# firewall-cmd --list-ports
8500/tcp

Step 4: Configure Docker Daemon to use Key-Value Store for Clustering

In this part, we will configure the Docker daemons on the remaining Docker Hosts 2-4 to use the Key-Value store for clustering. In a nutshell, we will stop the Docker daemons, set additional options for the key-value store, and restart the Docker daemons to let them take effect.

First, lets stop the Docker daemons on Docker Hosts 2-4:

# systemctl stop docker.service

Next, we will use create/edit the daemon configuration file to add the following properties:

# vi /etc/docker/daemon.json
{
    "hosts": ["tcp://0.0.0.0:2375","unix:///var/run/docker.sock"],
    "cluster-store": "consul://(Docker Host 1 Host-only IP Address):8500",
    "cluster-store-opts": {},
    "cluster-advertise": "(Docker Host X Host-only Network Interface):2375"
}

Note:

More information on the Docker daemon configuration file can be found here: https://docs.docker.com/engine/reference/commandline/dockerd/#miscellaneous-options
Each unique cluster-store IP address and path is a separate store. This means that you can have one key-value store at (host IP):8500, and a separate one at (host IP):8500/mynetwork.

Specifically for this post’s setup:

# vi /etc/docker/daemon.json
{
    "hosts": ["tcp://0.0.0.0:2375","unix:///var/run/docker.sock"],
    "cluster-store": "consul://192.168.56.101:8500",
    "cluster-store-opts": {},
    "cluster-advertise": "enp0s8:2375"
}

(Re)Start the Docker daemon with the new options, and check that it was successful:

# systemctl start docker.service
$ docker info
...
Cluster Store: consul://192.168.56.101:8500
Cluster Advertise: 192.168.56.102:2375

Do note that the Cluster Advertise IP address will vary between Docker Hosts 2-4.

From Docker Documentation: On systems that use systemd to start the Docker daemon, -H is already set, so you cannot use the hosts key in daemon.json to add listening addresses. See https://docs.docker.com/engine/admin/systemd/#custom-docker-daemon-options for how to accomplish this task with a systemd drop-in file.

Step 5: Open Firewall Ports

Docker daemons participating in a cluster require the following ports to be open (https://docs.docker.com/engine/swarm/networking/#firewall-considerations):

2375/tcp for cluster advertise
7946/tcp for container network discovery
7946/udp for container network discovery
4789/udp for the container overlay network

In CentOS, use firewall-cmd to allow the following ports and protocols:

# firewall-cmd --permanent --add-port 2375/tcp
success
# firewall-cmd --permanent --add-port 4789/udp
success
# firewall-cmd --permanent --add-port 7946/udp
success
# firewall-cmd --permanent --add-port 7946/tcp
success
# firewall-cmd --reload
success
# firewall-cmd --list-ports
2375/tcp 4789/udp 7946/udp 7946/tcp

Step 6: Create An Overlay Network

In this part, we will create the Docker overlay network.

On any Docker Host connected to the Consul Key-Value store (i.e. Docker Host 2-4), create the overlay network by using the overlay driver. Define your own subnet and subnet mask.

$ docker network create -d overlay --subnet=(subnet IP range)/(subnet mask bits) (overlay network name)

where 1) overlay is the network driver to use, 2) subnet IP range is the range of IP addresses that containers will get, and the number after the slash is the number of subnet mask bits, and 3) overlay network name is a custom name of the network. Items 2 and 3 can be customized to your needs.

I used the following:

$ docker network create -d overlay --subnet=192.168.3.0/24 my-overlay

The network should be seen from all Docker Engine daemons that have been configured to use the Consul Key-Value store for network state (i.e. Docker Hosts 2 to 4). Use the following command to see the overlay network on any of the Docker Hosts connected to the Key-Value store:

$ docker network ls
NETWORK ID      NAME         DRIVER    SCOPE
56de73022aae    bridge       bridge    local
4d1a5a9d01bf    host         host      local
74b2b7c217c1    my-overlay   overlay   global  <-- This is our overlay network
b078ff914434    none         null      local

Step 7: Add Containers to Overlay Network

In this part, we will add containers to the overlay network, but on different Docker Host.

In each Docker Host 2 to 4, run a busybox container and add it to the overlay network:

$ docker run -itd --name containerX --net my-overlay busybox

where X is a unique differentiator for each container (e.g. a running number). For this tutorial, I created ContainerA for Docker Host 2, ContainerB for Docker Host 3, and ContainerC for Docker Host 4.

Check that all containers are added to the overlay network by running the following command on any Docker Host 2 to 4:

$ docker network inspect my-overlay
{
    ...,
    "Containers": {
        "79fdd581016dab475187cd33f722315143a33fe2726401eb4ddce95b4d2419e5": {
            "Name": "containerC",
            "EndpointID": "50486768f127914f6e9b9fe3971bd871d1f51e9ebbe8d390ec0180632b6bc19e",
            "MacAddress": "02:42:c0:a8:03:03",
            "IPv4Address": "192.168.3.4/24",
            "IPv6Address": ""
        },
        "ep-46ec68e80b0e80dc71bf358420c70560af1c2ef0d71cef2dbffccd90b34a8bd5": {
            "Name": "containerA",
            "EndpointID": "46ec68e80b0e80dc71bf358420c70560af1c2ef0d71cef2dbffccd90b34a8bd5",
            "MacAddress": "02:42:c0:a8:03:02",
            "IPv4Address": "192.168.3.2/24",
            "IPv6Address": ""
        },
        "ep-cdad5e433c6466892383b8b2811d6fa7ab4190a0f39530e8e72ac973115b2bd9": {
            "Name": "containerB",
            "EndpointID": "cdad5e433c6466892383b8b2811d6fa7ab4190a0f39530e8e72ac973115b2bd9",
            "MacAddress": "02:42:c0:a8:03:04",
            "IPv4Address": "192.168.3.3/24",
            "IPv6Address": ""
        }
     },
     ...
}

Step 8: Ping Containers Across the Overlay Network

In this part, we will test that communication across the overlay network is working.

Have the busybox container in Docker Host 2 to ping the busybox container in Docker Host 4 by passing in the unique container name:

$ docker exec containerA ping -w 5 containerC
PING containerC (192.168.3.4): 56 data bytes
64 bytes from 192.168.3.4: seq=0 ttl=64 time=0.493 ms
64 bytes from 192.168.3.4: seq=1 ttl=64 time=1.174 ms
64 bytes from 192.168.3.4: seq=2 ttl=64 time=1.035 ms
64 bytes from 192.168.3.4: seq=3 ttl=64 time=0.604 ms
64 bytes from 192.168.3.4: seq=4 ttl=64 time=0.626 ms

--- containerC ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.493/0.786/1.174 ms

Note that the Key-Value store helps to maintain the state of the overlay network, including containers that are added or removed.

Step 9: Multi-homed Containers Across Different Overlay Networks

Docker documentation states that: “You can create multiple networks. You can add containers to more than one network. Containers can only communicate within networks but not across networks. A container attached to two networks can communicate with member containers in either network.”

Let’s try this out.

Create another overlay network of your choice. I chose:

$ docker network create -d overlay --subnet=192.167.1.0/24 my-overlay-167

Next, add containerB from Docker Host 2 into the new overlay network

$ docker network connect my-overlay-167 containerB

In Docker Host 4, create a new busybox container (e.g. containerD) and add it to the new overlay network.

$ docker run -itd --name containerD --net my-overlay-167 busybox

Right now, we have two overlay networks with the following containers:

my-overlay
- containerA
- containerB (multi-homed)
- containerC
my-overlay-167
- containerB (multi-homed)
- containerD

Now to test the connection – containers within the same overlay network should be able to ping each other, but not to those in another overlay network. The exception is the multi-homed containerB, which can ping to all containers.

[On docker-host2]
$ docker exec containerA ping -w 2 containerB
PING containerB (192.168.3.3): 56 data bytes
64 bytes from 192.168.3.3: seq=0 ttl=64 time=0.500 ms
64 bytes from 192.168.3.3: seq=1 ttl=64 time=0.810 ms
--- containerB ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.500/0.655/0.810 ms

$ docker exec containerA ping -w 2 containerC
PING containerC (192.168.3.4): 56 data bytes
64 bytes from 192.168.3.4: seq=0 ttl=64 time=0.567 ms
64 bytes from 192.168.3.4: seq=1 ttl=64 time=1.341 ms
--- containerC ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.567/0.954/1.341 ms

$ docker exec containerA ping -w 2 containerD
ping: bad address 'containerD'


[On docker-host3]
$ docker exec containerB ping -w 2 containerA
PING containerA (192.168.3.2): 56 data bytes
64 bytes from 192.168.3.2: seq=0 ttl=64 time=0.501 ms
64 bytes from 192.168.3.2: seq=1 ttl=64 time=1.274 ms
--- containerA ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.501/0.887/1.274 ms

$ docker exec containerB ping -w 2 containerC
PING containerC (192.168.3.4): 56 data bytes
64 bytes from 192.168.3.4: seq=0 ttl=64 time=0.842 ms
64 bytes from 192.168.3.4: seq=1 ttl=64 time=1.056 ms
--- containerC ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.842/0.949/1.056 ms

$ docker exec containerB ping -w 2 containerD
PING containerD (192.167.1.3): 56 data bytes
64 bytes from 192.167.1.3: seq=0 ttl=64 time=1.720 ms
64 bytes from 192.167.1.3: seq=1 ttl=64 time=0.934 ms
--- containerD ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.934/1.327/1.720 ms


[On docker-host4]
$ docker exec containerC ping -w 2 containerA
PING containerA (192.168.3.2): 56 data bytes
64 bytes from 192.168.3.2: seq=0 ttl=64 time=0.486 ms
64 bytes from 192.168.3.2: seq=1 ttl=64 time=0.510 ms
--- containerA ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.486/0.498/0.510 ms

$ docker exec containerC ping -w 2 containerB
PING containerB (192.168.3.3): 56 data bytes
64 bytes from 192.168.3.3: seq=0 ttl=64 time=0.570 ms
64 bytes from 192.168.3.3: seq=1 ttl=64 time=0.814 ms
--- containerB ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.570/0.692/0.814 ms

$ docker exec containerC ping -w 2 containerD
ping: bad address 'containerD'

It is interesting to note that even though containerC and containerD are on the same Docker Host 4, they are unable to communicate with each other. Therefore, Docker networking can support multi-tenancy by separating containers/applications into different (overlay) networks.

And now you have finished setting up an overlay network without using Docker Swarm.