Container Networking: Overlay networks - Anirban Mukherjee

When you create an overlay network on one node(in a swarm), the other do not automatically discover the presence of the network:

Docker03:~ $ docker network ls
NETWORK ID   NAME             DRIVER    SCOPE
1b6d5a047710 bridge           bridge    local
1b091486939e docker_gwbridge  bridge    local
6a3a75471f3d host             host      local
cvh3xki82cci ingress          overlay   swarm
c755b8f4cb4e none             null      local

The Docker03 host machine will recognize and connect into this network only when it hosts a container that connects into this overlay network.

Docker03:~ $ docker network ls
NETWORK ID NAME DRIVER SCOPE
1b6d5a047710 bridge bridge local
1b091486939e docker_gwbridge bridge local
6a3a75471f3d host host local
cvh3xki82cci ingress overlay swarm
c755b8f4cb4e none null local

Docker03:~ $ docker container run -dit — network my-overlay-network — name container3 nginx:alpine
c21e1b6445fb112f0180e66a226cc16193e85e36d3ed0b0e6657a44c56bbae0a

Docker03:~ $ docker container ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c21e1b6445fb nginx:alpine “nginx -g ‘daemon …” 9 seconds ago Up 6 seconds 80/tcp container3

Docker03:~ $ docker network ls
NETWORK ID   NAME               DRIVER SCOPE
1b6d5a047710 bridge             bridge local
1b091486939e docker_gwbridge    bridge local
6a3a75471f3d host               host   local
cvh3xki82cci ingress            overlay swarm
7obl3n1z5vzk my-overlay-network overlay swarm
c755b8f4cb4e none               null   local

Until this point, we have created an overlay network in Docker01 host, a container on Docker03 host that connects into the overlay network which automatically created the overlay network on Docker03. But we havent yet connected a container on Docker01 to the overlay network.

So let us see how the overlay network looks like in Docker01 at this point:

Docker01:~ $ docker network inspect 7obl3n1z5vzk
[
 {
 “Name”: “my-overlay-network”,
 “Id”: “7obl3n1z5vzk9j6zx8xu3irsd”,
 “Created”: “0001–01–01T00:00:00Z”,
 “Scope”: “swarm”,
 “Driver”: “overlay”,
 “EnableIPv6”: false,
 “IPAM”: {
 “Driver”: “default”,
 “Options”: null,
 “Config”: []
 },
 “Internal”: false,
 “Attachable”: true,
 “Ingress”: false,
 “Containers”: null,
 “Options”: {
   “com.docker.network.driver.overlay.vxlanid_list”: “4097”
 },
 “Labels”: null
 }
]

Note the “Containers: {}” in the output. It has not recognised the container on Docker03 connected to this network. This is by design of overlay networks. Containers hosted on other hosts will show up as “Peers” and not as “Containers”. Only those locally-hosted containers connected to this network will show up in the “Containers” key.

Checking on Docker03 host, we can verify this:

Docker03:~ $ docker network inspect 7obl3n1z5vzk
[
 {
  “Name”: “my-overlay-network”,
  “Id”: “7obl3n1z5vzk9j6zx8xu3irsd”,
  “Created”: “2018–08–05T15:12:11.082630937Z”,
  “Scope”: “swarm”,
  “Driver”: “overlay”,
  “EnableIPv6”: false,
  “IPAM”: {
  “Driver”: “default”,
  “Options”: null,
  “Config”: [
    {
      “Subnet”: “10.0.0.0/24”,
      “Gateway”: “10.0.0.1”
    }
  ]
 },
 “Internal”: false,
 “Attachable”: true,
 “Ingress”: false,
 “Containers”: {
 “c21e1b6445fb112f0180e66a226cc16193e85e36d3ed0b0e6657a44c56bbae0a”:
 {
  “Name”: “container3”,
  “EndpointID”:
 “250b1877adfd1383b233f4fe8c6f909cc7f07125367239ff29f1094dcfa9b2f6”,
  “MacAddress”: “02:42:0a:00:00:02”,
  “IPv4Address”: “10.0.0.2/24”,
  “IPv6Address”: “”
 }
},
 “Options”: {
 “com.docker.network.driver.overlay.vxlanid_list”: “4097”
 },
 “Labels”: {},
 “Peers”: [
 {
  “Name”: “Docker03–54941c69b81a”,
  “IP”: “172.16.255.103”
 }
 ]
 }
]

Let us now go ahead and create a container on the host Docker01 and see what happens:

Docker01:~ $ docker container run -dit — name container1 — network my-overlay-network nginx:alpine

Let us see how the overlay network looks like now.

Docker01:~ $ docker network inspect my-overlay-network
[
 {
  “Name”: “my-overlay-network”,
  “Id”: “7obl3n1z5vzk9j6zx8xu3irsd”,
  “Created”: “2018–08–05T15:31:45.957714062Z”,
  “Scope”: “swarm”,
  “Driver”: “overlay”,
  “EnableIPv6”: false,
  “IPAM”: {
  “Driver”: “default”,
  “Options”: null,
  “Config”: [
    {
      “Subnet”: “10.0.0.0/24”,
      “Gateway”: “10.0.0.1”
    }
  ]
  },
  “Internal”: false,
  “Attachable”: true,
  “Ingress”: false,
  “Containers”: {
 “8e2f525943839a6f51882c89311a7c6c87e7bbae25aa4e0425ef7bb3b157cf4d”:
    {
      “Name”: “container1”,
      “EndpointID”: “b89636f23b47784411db31fd5f134328fba0d339e749ce787be67f3795ee66fc”,
      “MacAddress”: “02:42:0a:00:00:03”,
      “IPv4Address”: “10.0.0.3/24”,
      “IPv6Address”: “”
    }
  },
  “Options”: {
    “com.docker.network.driver.overlay.vxlanid_list”: “4097”
  },
  “Labels”: {},
  “Peers”: [
    {
      “Name”: “Docker03–54941c69b81a”,
      “IP”: “172.16.255.103”
    },
    {
      “Name”: “Docker01–60cd697a6982”,
      “IP”: “172.16.255.101”
    }
  ]
 }
]

Now that we have a container on host Docker01, the overlay network shows the locally hosted container under the “Containers” key and the others under the “Peers” in the inspection output.

Note: Check (using ifconfig) on the host machine, that it is transparent to this user-defined overlay network. This means, it does not have any adapter created on it for the overlay network.

Now, inspect the “docker_gwbridge” network. You will see 2 containers attached to it (parameters will be similar if run on other hosts).

Docker03:~ $ docker network inspect docker_gwbridge
[
  {
    “Name”: “docker_gwbridge”,
    “Id”:
“1b091486939e1ab652013e8416be32aa6d41d2f74bf8f145bcd24e31d3109c5d”,
    “Created”: “2018–08–05T14:51:25.57401118Z”,
    “Scope”: “local”,
    “Driver”: “bridge”,
    “EnableIPv6”: false,
    “IPAM”: {
      “Driver”: “default”,
      “Options”: null,
      “Config”: [
        {
          “Subnet”: “172.18.0.0/16”,
          “Gateway”: “172.18.0.1”
        }
      ]
     },
    “Internal”: false,
    “Attachable”: false,
    “Ingress”: false,
    “Containers”: {
“c21e1b6445fb112f0180e66a226cc16193e85e36d3ed0b0e6657a44c56bbae0a”:
    {
      “Name”: “gateway_c21e1b6445fb”,
      “EndpointID”:
 “8da6ce5b6f510a4cc09b1d653eca6eec8b13094c31b42b2ccb4f80400797873d”,
      “MacAddress”: “02:42:ac:12:00:03”,
      “IPv4Address”: “172.18.0.3/16”,
      “IPv6Address”: “”
    },
    “ingress-sbox”: {
      “Name”: “gateway_ingress-sbox”,
      “EndpointID”:
 “0d3e2cf0f6d91ac4afc040cd7134e06cfa1a70bd764b2aaa3de0d48640d2e759”,
      “MacAddress”: “02:42:ac:12:00:02”,
      “IPv4Address”: “172.18.0.2/16”,
      “IPv6Address”: “”
    }
  },
  “Options”: {
    “com.docker.network.bridge.enable_icc”: “false”,
    “com.docker.network.bridge.enable_ip_masquerade”: “true”,
    “com.docker.network.bridge.name”: “docker_gwbridge”
  },
  “Labels”: {}
  }
]

If you login into the container itself, you will see eth0 connected to the user-defined ‘my-overlay-network’. eth1 will be connected to this docker_gwbridge network with IP=172.18.0.3 (as is also seen in “Containers” key of the network inspection result). The docker_gwbridge has gateway set to 172.18.0.1 on each host.

Between the host nodes, traffic is transported using VxLAN using UDP port 4789. Using vxlan has some advantages:

Since it is IP traffic, it is easy to encrypt using IPSec.
An exact overhead of 50bytes.
No need for L2 connectivity between hosts. An IP (layer3) connectivity between hosts works as vxlan does the tunnelling on top.
It does not have the limit of 4096 IDs as in normal vlans.

A packet capture of the packets on the eth0 device on Docker03 shows how the packets look like on the physical network between the hosts: