Reddit – Dive into anything
I’ve logged 22 hours to this issue and I just can’t seem to crack it.
I’m trying to create an overlay network in my swarm, it creates without issue and exists on both hosts but there’s no communication between the containers on separate nodes. I can, however, communicate between containers on the network if they’re on the same host.
Setup:
Host 1:
Our application server, also the manager. Is running Portainer.
LAN IP: 192.168.100.172
OS: Ubuntu 20.04
Docker version: 20.10.7
Host 2:
Our web server running Plesk, the worker.
LAN IP: 192.168.100.211
OS: Ubuntu 20.04
Docker: Version: 20.10.10; initially installed by Plesk.
Both are VMs on OpenStack.
docker-compose.yml:
version: '3.4'
services:
alpine:
image: alpine
entrypoint: /bin/sh
stdin_open: true
tty: true
deploy:
replicas: 2
networks:
- overnet
networks:
overnet:
driver: overlay
attachable: true
I have tried:
-
Ensuring ports 2377, 7946, 4789 are open to host 1 in the Plesk firewall
-
Ensuring ports 2377, 7946, 4789 are open to host 1 in the OpenStack firewall
-
Ensuring ports 2377, 7946, 4789 are open to host 2 in the OpenStack firewall
-
Ensuring the hosts are able to communicate by pinging each other.
-
Disabling ufw on host 1
-
Disabling the Plesk Firewall
-
Allowing all TCP & UDP traffic from host 1 in the OpenStack firewall
-
Allowing all TCP & UDP traffic from host 2 in the OpenStack firewall
-
Allowing all TCP & UDP traffic from host 1’s Public IP in the OpenStack firewall
-
Allowing all TCP & UDP traffic from host 2’s Public IP in the OpenStack firewall
-
Disabling ufw on host 2
-
root@host1:~#lsof -i:7946,2377,4789:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME dockerd 915 root 19u IPv6 5785127 0t0 TCP *:2377 (LISTEN) dockerd 915 root 24u IPv6 5785130 0t0 TCP *:7946 (LISTEN) dockerd 915 root 26u IPv6 5785131 0t0 UDP *:7946 dockerd 915 root 42u IPv6 6922305 0t0 TCP - host1:2377->192.168.100.211:41828 (ESTABLISHED)
-
root@host1:~#netstat -tuplen:
Proto Local Address Foreign Address State User Inode PID/Program name tcp 0.0.0.0:10050 0.0.0.0:* LISTEN 112 23379 768/zabbix_agentd tcp 127.0.0.53:53 0.0.0.0:* LISTEN 101 19366 682/systemd-resolve tcp 0.0.0.0:22 0.0.0.0:* LISTEN 0 28942 933/sshd: /usr/sbin tcp6 :::8123 :::* LISTEN 0 6920785 915/dockerd tcp6 :::25565 :::* LISTEN 0 26936079 915/dockerd tcp6 :::8000 :::* LISTEN 0 6921385 915/dockerd tcp6 :::10050 :::* LISTEN 112 23380 768/zabbix_agentd tcp6 :::9443 :::* LISTEN 0 6921403 915/dockerd tcp6 :::9000 :::* LISTEN 0 6921394 915/dockerd tcp6 :::2377 :::* LISTEN 0 5785127 915/dockerd tcp6 :::7946 :::* LISTEN 0 5785130 915/dockerd tcp6 :::8113 :::* LISTEN 0 6932625 915/dockerd tcp6 :::22 :::* LISTEN 0 28944 933/sshd: /usr/sbin udp 127.0.0.53:53 0.0.0.0:* 101 19365 682/systemd-resolve udp 192.168.100.172:68 0.0.0.0:* 100 23636 519/systemd-network udp 0.0.0.0:4789 0.0.0.0:* 0 5785266 - udp6 :::7946 :::* 0 5785131 915/dockerd
-
root@host2:~#lsof -i:7946,2377,4789:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME dockerd 2978182 root 40u IPv4 49243825 0t0 TCP 192.168.100.211:41828->192.168.100.172:2377 (ESTABLISHED) dockerd 2978182 root 46u IPv6 49261660 0t0 TCP *:7946 (LISTEN) dockerd 2978182 root 47u IPv6 49261661 0t0 UDP *:7946
-
root@host2:~#netstat -tuplen:
Proto Local Address Foreign Address State User Inode PID/Program name tcp 127.0.0.1:3030 0.0.0.0:* LISTEN 115 38571578 2706171/grafana-ser tcp 0.0.0.0:22 0.0.0.0:* LISTEN 0 21955 1018/sshd: /usr/sbi tcp 127.0.0.1:953 0.0.0.0:* LISTEN 114 9069975 584794/named tcp 0.0.0.0:25 0.0.0.0:* LISTEN 0 20127 3668/master tcp 127.0.0.1:12346 0.0.0.0:* LISTEN 0 20219 3668/master tcp 0.0.0.0:8443 0.0.0.0:* LISTEN 0 46269783 2963484/sw-cp-serve tcp 0.0.0.0:4190 0.0.0.0:* LISTEN 0 31994 954/dovecot tcp 127.0.0.1:12768 0.0.0.0:* LISTEN 113 19647868 1352965/psa-pc-remo tcp 0.0.0.0:49153 0.0.0.0:* LISTEN 0 46415637 2978557/docker-prox tcp 0.0.0.0:993 0.0.0.0:* LISTEN 0 32019 954/dovecot tcp 0.0.0.0:49154 0.0.0.0:* LISTEN 0 46426298 2978577/docker-prox tcp 0.0.0.0:10050 0.0.0.0:* LISTEN 118 28301 850/zabbix_agentd tcp 0.0.0.0:49155 0.0.0.0:* LISTEN 0 46417476 2978438/docker-prox tcp 0.0.0.0:995 0.0.0.0:* LISTEN 0 32004 954/dovecot tcp 0.0.0.0:49156 0.0.0.0:* LISTEN 0 46411365 2978460/docker-prox tcp 0.0.0.0:49157 0.0.0.0:* LISTEN 0 49280501 3204406/docker-prox tcp 0.0.0.0:49158 0.0.0.0:* LISTEN 0 46418478 2978486/docker-prox tcp 0.0.0.0:110 0.0.0.0:* LISTEN 0 32002 954/dovecot tcp 127.0.0.1:783 0.0.0.0:* LISTEN 0 48249434 3119145/perl tcp 0.0.0.0:143 0.0.0.0:* LISTEN 0 32017 954/dovecot tcp 0.0.0.0:8880 0.0.0.0:* LISTEN 0 46269784 2963484/sw-cp-serve tcp 0.0.0.0:465 0.0.0.0:* LISTEN 0 20228 3668/master tcp 172.18.0.1:53 0.0.0.0:* LISTEN 114 46331747 584794/named tcp 192.168.100.211:53 0.0.0.0:* LISTEN 114 45986087 584794/named tcp 172.17.0.1:53 0.0.0.0:* LISTEN 114 9069959 584794/named tcp 127.0.0.1:53 0.0.0.0:* LISTEN 114 9077846 584794/named tcp 127.0.0.53:53 0.0.0.0:* LISTEN 101 17025 709/systemd-resolve tcp6 :::22 :::* LISTEN 0 21957 1018/sshd: /usr/sbi tcp6 :::25 :::* LISTEN 0 20128 3668/master tcp6 :::8123 :::* LISTEN 0 49271347 2978182/dockerd tcp6 :::8443 :::* LISTEN 0 46269785 2963484/sw-cp-serve tcp6 :::25565 :::* LISTEN 0 49650435 2978182/dockerd tcp6 :::4190 :::* LISTEN 0 31995 954/dovecot tcp6 :::9983 :::* LISTEN 10000 34162331 2394464/loolwsd tcp6 :::8000 :::* LISTEN 0 49251241 2978182/dockerd tcp6 :::49153 :::* LISTEN 0 46419752 2978542/docker-prox tcp6 :::993 :::* LISTEN 0 32020 954/dovecot tcp6 :::49154 :::* LISTEN 0 46421932 2978564/docker-prox tcp6 :::10050 :::* LISTEN 118 28302 850/zabbix_agentd tcp6 :::9443 :::* LISTEN 0 49251259 2978182/dockerd tcp6 :::49155 :::* LISTEN 0 46424283 2978414/docker-prox tcp6 :::995 :::* LISTEN 0 32005 954/dovecot tcp6 :::49156 :::* LISTEN 0 46418441 2978445/docker-prox tcp6 :::49157 :::* LISTEN 0 49275723 3204392/docker-prox tcp6 :::49158 :::* LISTEN 0 46413407 2978469/docker-prox tcp6 :::2375 :::* LISTEN 0 46419672 2978182/dockerd tcp6 :::9000 :::* LISTEN 0 49251250 2978182/dockerd tcp6 :::7080 :::* LISTEN 0 45065023 2886204/apache2 tcp6 :::7081 :::* LISTEN 0 45065027 2886204/apache2 tcp6 :::7946 :::* LISTEN 0 49261660 2978182/dockerd tcp6 127.0.0.1:3306 :::* LISTEN 112 19422 908/mysqld tcp6 :::106 :::* LISTEN 0 30245 1081/xinetd tcp6 :::110 :::* LISTEN 0 32003 954/dovecot tcp6 ::1:783 :::* LISTEN 0 48249432 3119145/perl tcp6 :::143 :::* LISTEN 0 32018 954/dovecot tcp6 :::8880 :::* LISTEN 0 46269786 2963484/sw-cp-serve tcp6 :::8113 :::* LISTEN 0 49275190 2978182/dockerd tcp6 :::465 :::* LISTEN 0 20229 3668/master tcp6 ::1:53 :::* LISTEN 114 9077848 584794/named tcp6 :::21 :::* LISTEN 0 30244 1081/xinetd udp 172.18.0.1:53 0.0.0.0:* 114 46350405 584794/named udp 172.18.0.1:53 0.0.0.0:* 114 46350404 584794/named udp 192.168.100.211:53 0.0.0.0:* 114 45983142 584794/named udp 192.168.100.211:53 0.0.0.0:* 114 45983141 584794/named udp 172.17.0.1:53 0.0.0.0:* 114 9069958 584794/named udp 172.17.0.1:53 0.0.0.0:* 114 9069957 584794/named udp 127.0.0.1:53 0.0.0.0:* 114 9069954 584794/named udp 127.0.0.1:53 0.0.0.0:* 114 9069953 584794/named udp 127.0.0.53:53 0.0.0.0:* 101 17024 709/systemd-resolve udp 192.168.100.211:68 0.0.0.0:* 100 48887854 2935522/systemd-net udp 0.0.0.0:4789 0.0.0.0:* 0 49261799 - udp6 :::7946 :::* 0 49261661 2978182/dockerd udp6 ::1:53 :::* 114 9069960 584794/named udp6 ::1:53 :::* 114 9069961 584794/named
-
root@host1:~#nmap 192.168.100.211 -p 2377,7946:
... PORT STATE SERVICE 2377/tcp closed swarm 7946/tcp open unknown MAC Address: FA:16:3E:C8:15:84 (Unknown) ...
-
root@host1:~#nmap 192.168.100.211 -sU -p 4789:
... PORT STATE SERVICE 4789/udp open|filtered unknown MAC Address: FA:16:3E:C8:15:84 (Unknown) ...
-
root@host1:~#nc -zvw10 192.168.100.211 -u 4789:
Connection to 192.168.100.211 4789 port [udp/*] succeeded!
-
root@host2:~#nmap 192.168.100.172 -p 2377,7946:
... PORT STATE SERVICE 2377/tcp open swarm 7946/tcp open unknown MAC Address: FA:16:3E:C6:E8:A1 (Unknown) ..
-
root@host2:~#nc -zvw10 192.168.100.172 -u 4789:
Connection to 192.168.100.211 4789 port [udp/*] succeeded!
-
destroying and redeploying the swarm
-
ruling out issues with the images we use, by creating the above docker-compose.yml to try to ping the replicated alpine images; 100% packet loss in both directions.
-
root@host1:~#docker network inspect overtest_overnet -v:
[
{
"Name": "overtest_overnet",
"Id": "4sr72az5e83by2tz8tf6bcu13",
"Created": "2021-11-18T01:51:08.405389819Z",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.4.0/24",
"Gateway": "10.0.4.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"39f3c6b6bce0b851481fbe65295935cf40027882d47531aefd939b29ff6d41c0": {
"Name": "overtest_alpine.1.3faw89llptalropbsv4j7dnbq",
"EndpointID": "7ece6d473291a33842d9e6ce2e9d6e2c10ba1292f59cd08aa5905592fa0a4823",
"MacAddress": "02:42:0a:00:04:04",
"IPv4Address": "10.0.4.4/24",
"IPv6Address": ""
},
"lb-overtest_overnet": {
"Name": "overtest_overnet-endpoint",
"EndpointID": "a1e0af702b2ab0fe406695d44440c2a872c6ecb403a49972e3eb078cceb9cb8f",
"MacAddress": "02:42:0a:00:04:06",
"IPv4Address": "10.0.4.6/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4100"
},
"Labels": {
"com.docker.stack.namespace": "overtest"
},
"Peers": [
{
"Name": "0791c0d0a196",
"IP": "192.168.100.172"
},
{
"Name": "510e1cca004a",
"IP": "192.168.100.211"
}
],
"Services": {
"overtest_alpine": {
"VIP": "10.0.4.2",
"Ports": [],
"LocalLBIndex": 391,
"Tasks": [
{
"Name": "overtest_alpine.1.3faw89llptalropbsv4j7dnbq",
"EndpointID": "7ece6d473291a33842d9e6ce2e9d6e2c10ba1292f59cd08aa5905592fa0a4823",
"EndpointIP": "10.0.4.4",
"Info": {
"Host IP": "192.168.100.172"
}
},
{
"Name": "overtest_alpine.2.nqym48jt9j83x8fsnh47bow6i",
"EndpointID": "18a86f0d8d33e20a35e9a2439756c892222d903e3783192cdcdf40217b12acde",
"EndpointIP": "10.0.4.3",
"Info": {
"Host IP": "192.168.100.211"
}
}
]
}
}
}
]
-
root@host2:~#docker network inspect overtest_overnet -v:
[
{
"Name": "overtest_overnet",
"Id": "4sr72az5e83by2tz8tf6bcu13",
"Created": "2021-11-17T18:50:39.470320993-07:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.4.0/24",
"Gateway": "10.0.4.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"051f4fa29b4d09530aa627443e747c08f1adee6537ec8265da7066ec2b5e14d4": {
"Name": "overtest_alpine.2.nqym48jt9j83x8fsnh47bow6i",
"EndpointID": "18a86f0d8d33e20a35e9a2439756c892222d903e3783192cdcdf40217b12acde",
"MacAddress": "02:42:0a:00:04:03",
"IPv4Address": "10.0.4.3/24",
"IPv6Address": ""
},
"lb-overtest_overnet": {
"Name": "overtest_overnet-endpoint",
"EndpointID": "502205e155e921bbf2900279c378ea0f32145a9a081318dd31aa3cab3d69fbef",
"MacAddress": "02:42:0a:00:04:05",
"IPv4Address": "10.0.4.5/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4100"
},
"Labels": {
"com.docker.stack.namespace": "overtest"
},
"Peers": [
{
"Name": "510e1cca004a",
"IP": "192.168.100.211"
},
{
"Name": "0791c0d0a196",
"IP": "192.168.100.172"
}
],
"Services": {
"overtest_alpine": {
"VIP": "10.0.4.2",
"Ports": [],
"LocalLBIndex": 640,
"Tasks": [
{
"Name": "overtest_alpine.2.nqym48jt9j83x8fsnh47bow6i",
"EndpointID": "18a86f0d8d33e20a35e9a2439756c892222d903e3783192cdcdf40217b12acde",
"EndpointIP": "10.0.4.3",
"Info": {
"Host IP": "192.168.100.211"
}
},
{
"Name": "overtest_alpine.1.3faw89llptalropbsv4j7dnbq",
"EndpointID": "7ece6d473291a33842d9e6ce2e9d6e2c10ba1292f59cd08aa5905592fa0a4823",
"EndpointIP": "10.0.4.4",
"Info": {
"Host IP": "192.168.100.172"
}
}
]
}
}
}
]
-
page 2 of google
List is probably not complete, but this is as much as I and my bash history can remember.
Not sure what to do from here, any ideas?
![Toni Kroos là ai? [ sự thật về tiểu sử đầy đủ Toni Kroos ]](https://evbn.org/wp-content/uploads/New-Project-6635-1671934592.jpg)
