Reddit – Dive into anything

I have a strange networking issue with my docker install on debian 11 Linux 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64 GNU/Linux withDocker version 20.10.8, build 3967b7d` .

Docker sporadically does not assign the virtual interface to the bridge docker0 on boot.

Someone has any clue how to fix this issue or debug further?

$ cat /etc/docker/daemon.json

{
  "data-root": "/data/docker/pool",
  "exec-opts": [
    "native.cgroupdriver=systemd"
  ],
  "log-driver": "journald",
  "ipv6": false,
  "storage-driver": "overlay2"
}

I have 2 containers running there with for example docker run -v /etc/promtail-config.yaml:/mnt/config/promtail-config.yaml -p 1514:1514 --name=promtail grafana/promtail:2.0.0.

$ docker network ls

NETWORK ID     NAME      DRIVER    SCOPE
73bb3bfa37f6   bridge    bridge    local
5bd5a8295b6e   host      host      local
6a91f3274136   none      null      local

$ docker network inspect bridge

[
    {
        "Name": "bridge",
        "Id": "73bb3bfa37f67ba2c78bc9089b36421c8d6099291f1327835236435d86e71c96",
        "Created": "2021-09-20T20:03:56.023096572+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "9ccfc7a1e7d2f1294eb3cb6962068268f98593520520a1b43629e50c9ef68ed5": {
                "Name": "promtail",
                "EndpointID": "fb4f5ea3357ced5907386dbcee2147a7d9320896c3f8b9f3618f4dd5f2d8c365",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            },
            "eaece68e689fba9297e8676cd9f7d0ee7caee72b84b7068e6979f52f29f3ceec": {
                "Name": "loki",
                "EndpointID": "f472fc727957c9eb57791c236f98721c93eaa4855368f291b62c5fdea07f23d3",
                "MacAddress": "02:42:ac:11:00:03",
                "IPv4Address": "172.17.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

Here you can see the interfaces with veth0bddc6f@if6not having docker0 set as master which causes the exported port not to be reachable from outside the host.

$ ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:73:e8:b7 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet 10.0.16.9/24 brd 10.0.16.255 scope global ens3
       valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:70:c3:4d:44 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
5: vethc6563bf@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
    link/ether fa:22:78:71:9f:60 brd ff:ff:ff:ff:ff:ff link-netnsid 0
7: veth0bddc6f@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 8a:5e:6b:0b:54:28 brd ff:ff:ff:ff:ff:ff link-netnsid 1

After restoring the bridge as master with ip link set veth0bddc6f master docker0 the port exported by the container is reachable from the network of the host.

In the logs with docker debug -D enabled I could not find any related errors to the interface setup.

$ journalctl -b | grep -e interface -e network

Sep 20 20:43:17 hostname kernel: Linux agpgart interface v0.103
Sep 20 20:43:17 hostname kernel: usbcore: registered new interface driver usbfs
Sep 20 20:43:17 hostname kernel: usbcore: registered new interface driver hub
Sep 20 20:43:17 hostname systemd-networkd[242]: /etc/systemd/network/99-dhcp-wildcard.network: DHCPv6 client is enabled but IPv6 link local addressing is disabled. Disabling DHCPv6 client.
Sep 20 20:43:17 hostname systemd-networkd[242]: Enumeration completed
Sep 20 20:43:17 hostname systemd-networkd[242]: ens3: Link UP
Sep 20 20:43:17 hostname systemd-udevd[248]: Using default interface naming scheme 'v247'.
Sep 20 20:43:17 hostname systemd-udevd[248]: ens3: Failed to rename network interface 2 from 'ens3' to 'enlan': Device or resource busy
Sep 20 20:43:17 hostname systemd-networkd[242]: ens3: Gained carrier
Sep 20 20:43:18 hostname consul[536]:     2021-09-20T20:43:18.668+0200 [INFO]  agent: Started DNS server: address=127.0.0.1:8600 network=tcp
Sep 20 20:43:18 hostname consul[536]:     2021-09-20T20:43:18.668+0200 [INFO]  agent: Started DNS server: address=127.0.0.1:8600 network=udp
Sep 20 20:43:18 hostname consul[536]:     2021-09-20T20:43:18.669+0200 [INFO]  agent: Starting server: address=127.0.0.1:8500 network=tcp protocol=http
Sep 20 20:43:19 hostname dockerd[550]: time="2021-09-20T20:43:19.014269569+02:00" level=debug msg="Did not find any interface with name docker0: Link not found"
Sep 20 20:43:19 hostname systemd-networkd[242]: rtnl: received neighbor for link '3' we don't know about, ignoring.
Sep 20 20:43:19 hostname dockerd[550]: time="2021-09-20T20:43:19.015120390+02:00" level=debug msg="Assigning address to bridge interface docker0: 172.17.0.1/16"
Sep 20 20:43:19 hostname systemd-networkd[242]: rtnl: received neighbor for link '3' we don't know about, ignoring.
Sep 20 20:43:19 hostname systemd-networkd[242]: rtnl: received neighbor for link '3' we don't know about, ignoring.
Sep 20 20:43:19 hostname systemd-networkd[242]: rtnl: received neighbor for link '3' we don't know about, ignoring.
Sep 20 20:43:19 hostname systemd-networkd[242]: docker0: Link UP
Sep 20 20:43:19 hostname systemd-udevd[247]: Using default interface naming scheme 'v247'.
Sep 20 20:43:19 hostname dockerd[550]: time="2021-09-20T20:43:19.078733311+02:00" level=debug msg="Allocating IPv4 pools for network bridge (74f44c408e772a8d912b2643fb8f7c199d9ca37aabeefccbe42e20cb0c58830a)"
Sep 20 20:43:19 hostname dockerd[550]: time="2021-09-20T20:43:19.078799071+02:00" level=debug msg="RequestAddress(LocalDefault/172.17.0.0/16, 172.17.0.1, map[RequestAddressType:com.docker.network.gateway])"
Sep 20 20:43:19 hostname dockerd[550]: time="2021-09-20T20:43:19.124483369+02:00" level=debug msg="releasing IPv4 pools from network bridge (74f44c408e772a8d912b2643fb8f7c199d9ca37aabeefccbe42e20cb0c58830a)"
Sep 20 20:43:19 hostname dockerd[550]: time="2021-09-20T20:43:19.127962970+02:00" level=debug msg="cleanupServiceDiscovery for network:74f44c408e772a8d912b2643fb8f7c199d9ca37aabeefccbe42e20cb0c58830a"
Sep 20 20:43:19 hostname dockerd[550]: time="2021-09-20T20:43:19.138796046+02:00" level=debug msg="Allocating IPv4 pools for network bridge (1b0557d635999ca8c484f898b5f21021064d765bac01dc367d979d96522fa1d5)"
Sep 20 20:43:19 hostname dockerd[550]: time="2021-09-20T20:43:19.138957703+02:00" level=debug msg="RequestAddress(LocalDefault/172.17.0.0/16, 172.17.0.1, map[RequestAddressType:com.docker.network.gateway])"
Sep 20 20:43:19 hostname dockerd[550]: time="2021-09-20T20:43:19.237962305+02:00" level=debug msg="Registering GET, /networks"
Sep 20 20:43:19 hostname dockerd[550]: time="2021-09-20T20:43:19.237989670+02:00" level=debug msg="Registering GET, /networks/"
Sep 20 20:43:19 hostname dockerd[550]: time="2021-09-20T20:43:19.238017620+02:00" level=debug msg="Registering GET, /networks/{id:.+}"
Sep 20 20:43:19 hostname dockerd[550]: time="2021-09-20T20:43:19.238057441+02:00" level=debug msg="Registering POST, /networks/create"
Sep 20 20:43:19 hostname dockerd[550]: time="2021-09-20T20:43:19.238082322+02:00" level=debug msg="Registering POST, /networks/{id:.*}/connect"
Sep 20 20:43:19 hostname dockerd[550]: time="2021-09-20T20:43:19.238112075+02:00" level=debug msg="Registering POST, /networks/{id:.*}/disconnect"
Sep 20 20:43:19 hostname dockerd[550]: time="2021-09-20T20:43:19.238150826+02:00" level=debug msg="Registering POST, /networks/prune"
Sep 20 20:43:19 hostname dockerd[550]: time="2021-09-20T20:43:19.238174506+02:00" level=debug msg="Registering DELETE, /networks/{id:.*}"
Sep 20 20:43:21 hostname dockerd[550]: time="2021-09-20T20:43:21.002133032+02:00" level=debug msg="Assigning addresses for endpoint promtail's interface on network bridge"
Sep 20 20:43:21 hostname systemd-udevd[245]: Using default interface naming scheme 'v247'.
Sep 20 20:43:21 hostname systemd-networkd[242]: veth4900952: Link UP
Sep 20 20:43:21 hostname systemd-networkd[242]: vethb11e613: Link UP
Sep 20 20:43:21 hostname dockerd[550]: time="2021-09-20T20:43:21.031484938+02:00" level=debug msg="Assigning addresses for endpoint promtail's interface on network bridge"
Sep 20 20:43:21 hostname dockerd[550]: time="2021-09-20T20:43:21.035240171+02:00" level=debug msg="Assigning addresses for endpoint loki's interface on network bridge"
Sep 20 20:43:21 hostname kernel: docker0: port 2(veth4190c0e) entered blocking state
Sep 20 20:43:21 hostname kernel: docker0: port 2(veth4190c0e) entered disabled state
Sep 20 20:43:21 hostname kernel: device veth4190c0e entered promiscuous mode
Sep 20 20:43:21 hostname kernel: docker0: port 2(veth4190c0e) entered blocking state
Sep 20 20:43:21 hostname kernel: docker0: port 2(veth4190c0e) entered forwarding state
Sep 20 20:43:21 hostname systemd-networkd[242]: vethb11e613: Gained carrier
Sep 20 20:43:21 hostname systemd-networkd[242]: veth4900952: Gained carrier
Sep 20 20:43:21 hostname systemd-networkd[242]: docker0: Gained carrier
Sep 20 20:43:21 hostname kernel: device veth4190c0e left promiscuous mode
Sep 20 20:43:21 hostname kernel: docker0: port 2(veth4190c0e) entered disabled state
Sep 20 20:43:21 hostname systemd-networkd[242]: veth4190c0e: Link UP
Sep 20 20:43:21 hostname dockerd[550]: time="2021-09-20T20:43:21.094458987+02:00" level=debug msg="Assigning addresses for endpoint loki's interface on network bridge"
Sep 20 20:43:21 hostname systemd-networkd[242]: veth0ff579e: Link UP
Sep 20 20:43:21 hostname systemd-networkd[242]: veth0ff579e: Gained carrier
Sep 20 20:43:21 hostname systemd-networkd[242]: veth4190c0e: Gained carrier
Sep 20 20:43:21 hostname systemd-networkd[242]: rtnl: received neighbor for link '7' we don't know about, ignoring.
Sep 20 20:43:21 hostname systemd-networkd[242]: rtnl: received neighbor for link '7' we don't know about, ignoring.
Sep 20 20:43:21 hostname systemd-networkd[242]: veth4900952: Link DOWN
Sep 20 20:43:21 hostname systemd-networkd[242]: veth4900952: Lost carrier
Sep 20 20:43:21 hostname systemd-networkd[242]: vethb11e613: Lost carrier
Sep 20 20:43:21 hostname systemd-networkd[242]: veth0ff579e: Link DOWN
Sep 20 20:43:21 hostname systemd-networkd[242]: veth0ff579e: Lost carrier
Sep 20 20:43:21 hostname systemd-networkd[242]: veth4190c0e: Lost carrier
Sep 20 20:43:21 hostname systemd-networkd[242]: vethb11e613: Gained carrier
Sep 20 20:43:21 hostname systemd-networkd[242]: veth4190c0e: Gained carrier