Submariner Gateways Not Establishing Connections Between Clusters #3158
Comments
|
Thanks for contacting @halekammoun ,
Submariner could not detect your CNI, what CNI are you using in your clusters? Submariner uses the detected CNI information to automatically retrieve service and pod CIDRs of the cluster. You can workaround service and pod CIDRs auto discovery by providing it in subctl join command, by adding the following params:
Could you reinstall using the updated join command and see if that helps ? |
|
Also please provide |
|
This seems to be duplicated by submariner-io/submariner-operator#3213. Closing that one has the discussion has started here. |
|
@yboaron @tpantelis kubectl config use-context karmada-cluster
#deploy the broker in the karmada cluster
sudo /home/hale/.local/bin/subctl deploy-broker --kubeconfig .kube/config --context karmada-cluster
# join member1 to the broker
sudo /home/hale/.local/bin/subctl join --kubeconfig .kube/config --context member1 broker-info.subm --natt=false \
--clustercidr 10.245.0.0/16 --servicecidr 10.100.0.0/16 --health-check=false
# join member2 to the broker
sudo /home/hale/.local/bin/subctl join --kubeconfig .kube/config --context member2 broker-info.subm --natt=false \
--clustercidr 10.246.0.0/16 --servicecidr 10.110.0.0/16 --health-check=falsethis is the output pf subctl show all hale@hale-VirtualBox:~$ subctl show all
Cluster "member2"
✓ Detecting broker(s)
✓ No brokers found
✓ Showing Connections
GATEWAY CLUSTER REMOTE IP NAT CABLE DRIVER SUBNETS STATUS RTT avg.
member1 member1 192.168.49.2 no libreswan 10.100.0.0/16, 10.245.0.0/16 connecting
✓ Showing Endpoints
CLUSTER ENDPOINT IP PUBLIC IP CABLE DRIVER TYPE
member2 192.168.58.2 41.227.8.45 libreswan local
member1 192.168.49.2 41.227.8.45 libreswan remote
✓ Showing Gateways
NODE HA STATUS SUMMARY
member2 active 0 connections out of 1 are established
✓ Showing Network details
Discovered network details via Submariner:
Network plugin: generic
Service CIDRs: [10.110.0.0/16]
Cluster CIDRs: [10.246.0.0/16]
✓ Showing versions
COMPONENT REPOSITORY CONFIGURED RUNNING ARCH
submariner-gateway quay.io/submariner 0.18.0 release-0.18-e3f3e56b57fe amd64
submariner-routeagent quay.io/submariner 0.18.0 release-0.18-e3f3e56b57fe amd64
submariner-metrics-proxy quay.io/submariner 0.18.0 release-0.18-011349c6f17e amd64
submariner-operator quay.io/submariner 0.18.0 release-0.18-68fefdd74105 amd64
submariner-lighthouse-agent quay.io/submariner 0.18.0 release-0.18-02b6a5b37266 amd64
submariner-lighthouse-coredns quay.io/submariner 0.18.0 release-0.18-02b6a5b37266 amd64
Cluster "karmada-cluster"
✓ Detecting broker(s)
NAMESPACE NAME COMPONENTS GLOBALNET GLOBALNET CIDR DEFAULT GLOBALNET SIZE DEFAULT DOMAINS
submariner-k8s-broker submariner-broker service-discovery, connectivity no 242.0.0.0/8 65536
✗ Showing Connections
✗ No connections found
✓ Showing Endpoints
CLUSTER ENDPOINT IP PUBLIC IP CABLE DRIVER TYPE
karmada-cluster 10.0.2.15 41.227.8.45 libreswan local
✓ Showing Gateways
NODE HA STATUS SUMMARY
karmada-cluster active There are no connections
✓ Showing Network details
Discovered network details via Submariner:
Network plugin: generic
Service CIDRs: [10.96.0.0/12]
Cluster CIDRs: [10.244.0.0/16]
✓ Showing versions
COMPONENT REPOSITORY CONFIGURED RUNNING ARCH
submariner-gateway quay.io/submariner 0.18.0 release-0.18-e3f3e56b57fe amd64
submariner-routeagent quay.io/submariner 0.18.0 release-0.18-e3f3e56b57fe amd64
submariner-metrics-proxy quay.io/submariner 0.18.0 release-0.18-011349c6f17e amd64
submariner-operator quay.io/submariner 0.18.0 release-0.18-68fefdd74105 amd64
submariner-lighthouse-agent quay.io/submariner 0.18.0 release-0.18-02b6a5b37266 amd64
submariner-lighthouse-coredns quay.io/submariner 0.18.0 release-0.18-02b6a5b37266 amd64
Cluster "member1"
✓ Detecting broker(s)
✓ No brokers found
✓ Showing Connections
GATEWAY CLUSTER REMOTE IP NAT CABLE DRIVER SUBNETS STATUS RTT avg.
member2 member2 192.168.58.2 no libreswan 10.110.0.0/16, 10.246.0.0/16 connecting
✓ Showing Endpoints
CLUSTER ENDPOINT IP PUBLIC IP CABLE DRIVER TYPE
member1 192.168.49.2 41.227.8.45 libreswan local
member2 192.168.58.2 41.227.8.45 libreswan remote
✓ Showing Gateways
NODE HA STATUS SUMMARY
member1 active 0 connections out of 1 are established
✓ Showing Network details
Discovered network details via Submariner:
Network plugin: generic
Service CIDRs: [10.100.0.0/16]
Cluster CIDRs: [10.245.0.0/16]
✓ Showing versions
COMPONENT REPOSITORY CONFIGURED RUNNING ARCH
submariner-gateway quay.io/submariner 0.18.0 release-0.18-e3f3e56b57fe amd64
submariner-routeagent quay.io/submariner 0.18.0 release-0.18-e3f3e56b57fe amd64
submariner-metrics-proxy quay.io/submariner 0.18.0 release-0.18-011349c6f17e amd64
submariner-operator quay.io/submariner 0.18.0 release-0.18-68fefdd74105 amd64
submariner-lighthouse-agent quay.io/submariner 0.18.0 release-0.18-02b6a5b37266 amd64
submariner-lighthouse-coredns quay.io/submariner 0.18.0 release-0.18-02b6a5b37266 amd64
subctl version: v0.18.0
and this is the output of subctl gather hale@hale-VirtualBox:~$ subctl gather
Cluster "member1"
Gathering information from cluster "member1"
✓ Gathering connectivity logs
✓ Found 1 pods matching label selector "app=submariner-gateway"
✓ Found 1 pods matching label selector "app=submariner-routeagent"
✓ Found 1 pods matching label selector "app=submariner-metrics-proxy"
✓ Found 0 pods matching label selector "app=submariner-globalnet"
✓ Found 0 pods matching label selector "app=submariner-addon"
✓ Gathering connectivity resources
✓ Gathering CNI data from 1 pods matching label selector "app=submariner-routeagent"
✓ Gathering CNI data from 1 pods matching label selector "app=submariner-gateway"
✓ Gathering cable driver data from 1 pods matching label selector "app=submariner-gateway"
✓ Found 2 endpoints in namespace "submariner-operator"
✓ Found 3 clusters in namespace "submariner-operator"
✓ Found 1 gateways in namespace "submariner-operator"
✓ Found 0 clusterglobalegressips in namespace ""
✓ Found 0 globalegressips in namespace ""
✓ Found 0 globalingressips in namespace ""
✓ Gathering service-discovery logs
✓ Found 3 pods matching label selector "component=submariner-lighthouse"
✓ Found 1 pods matching label selector "k8s-app=kube-dns"
✓ Gathering service-discovery resources
✓ Found 0 serviceexports in namespace ""
✓ Found 0 serviceimports in namespace ""
✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""
✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"
✓ Found 1 configmaps by field selector "metadata.name=coredns" in namespace "kube-system"
✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""
✓ Gathering broker logs
✓ Gathering broker resources
✓ Found 3 endpoints in namespace "submariner-k8s-broker"
✓ Found 3 clusters in namespace "submariner-k8s-broker"
✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace "submariner-k8s-broker"
✓ Found 0 serviceimports in namespace "submariner-k8s-broker"
✓ Gathering operator logs
✓ Found 1 pods matching label selector "name=submariner-operator"
✓ Gathering operator resources
✓ Found 1 submariners in namespace "submariner-operator"
✓ Found 1 servicediscoveries in namespace "submariner-operator"
✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"
✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"
✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"
✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"
✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"
✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"
✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"
Files are stored under directory "submariner-20240910171903/member1"
Cluster "member2"
Gathering information from cluster "member2"
✓ Gathering connectivity logs
✓ Found 1 pods matching label selector "app=submariner-gateway"
✓ Found 1 pods matching label selector "app=submariner-routeagent"
✓ Found 1 pods matching label selector "app=submariner-metrics-proxy"
✓ Found 0 pods matching label selector "app=submariner-globalnet"
✓ Found 0 pods matching label selector "app=submariner-addon"
✓ Gathering connectivity resources
✓ Gathering CNI data from 1 pods matching label selector "app=submariner-routeagent"
✓ Gathering CNI data from 1 pods matching label selector "app=submariner-gateway"
✓ Gathering cable driver data from 1 pods matching label selector "app=submariner-gateway"
✓ Found 2 endpoints in namespace "submariner-operator"
✓ Found 3 clusters in namespace "submariner-operator"
✓ Found 1 gateways in namespace "submariner-operator"
✓ Found 0 clusterglobalegressips in namespace ""
✓ Found 0 globalegressips in namespace ""
✓ Found 0 globalingressips in namespace ""
✓ Gathering service-discovery logs
✓ Found 3 pods matching label selector "component=submariner-lighthouse"
✓ Found 1 pods matching label selector "k8s-app=kube-dns"
✓ Gathering service-discovery resources
✓ Found 0 serviceexports in namespace ""
✓ Found 0 serviceimports in namespace ""
✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""
✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"
✓ Found 1 configmaps by field selector "metadata.name=coredns" in namespace "kube-system"
✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""
✓ Gathering broker logs
✓ Gathering broker resources
✓ Found 3 endpoints in namespace "submariner-k8s-broker"
✓ Found 3 clusters in namespace "submariner-k8s-broker"
✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace "submariner-k8s-broker"
✓ Found 0 serviceimports in namespace "submariner-k8s-broker"
✓ Gathering operator logs
✓ Found 1 pods matching label selector "name=submariner-operator"
✓ Gathering operator resources
✓ Found 1 submariners in namespace "submariner-operator"
✓ Found 1 servicediscoveries in namespace "submariner-operator"
✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"
✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"
✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"
✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"
✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"
✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"
✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"
Files are stored under directory "submariner-20240910171903/member2"
Cluster "karmada-cluster"
Gathering information from cluster "karmada-cluster"
✓ Gathering connectivity logs
✓ Found 1 pods matching label selector "app=submariner-gateway"
✓ Found 1 pods matching label selector "app=submariner-routeagent"
✓ Found 1 pods matching label selector "app=submariner-metrics-proxy"
✓ Found 0 pods matching label selector "app=submariner-globalnet"
✓ Found 0 pods matching label selector "app=submariner-addon"
✓ Gathering connectivity resources
✓ Gathering CNI data from 1 pods matching label selector "app=submariner-routeagent"
✓ Gathering CNI data from 1 pods matching label selector "app=submariner-gateway"
✓ Gathering cable driver data from 1 pods matching label selector "app=submariner-gateway"
✓ Found 1 endpoints in namespace "submariner-operator"
✓ Found 3 clusters in namespace "submariner-operator"
✓ Found 1 gateways in namespace "submariner-operator"
✓ Found 0 clusterglobalegressips in namespace ""
✓ Found 0 globalegressips in namespace ""
✓ Found 0 globalingressips in namespace ""
✓ Gathering service-discovery logs
✓ Found 3 pods matching label selector "component=submariner-lighthouse"
✓ Found 1 pods matching label selector "k8s-app=kube-dns"
✓ Gathering service-discovery resources
✓ Found 0 serviceexports in namespace ""
✓ Found 0 serviceimports in namespace ""
✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""
✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"
✓ Found 1 configmaps by field selector "metadata.name=coredns" in namespace "kube-system"
✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""
✓ Gathering broker logs
✓ Gathering broker resources
✓ Found 3 endpoints in namespace "submariner-k8s-broker"
✓ Found 3 clusters in namespace "submariner-k8s-broker"
✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace "submariner-k8s-broker"
✓ Found 0 serviceimports in namespace "submariner-k8s-broker"
✓ Gathering operator logs
✓ Found 1 pods matching label selector "name=submariner-operator"
✓ Gathering operator resources
✓ Found 1 submariners in namespace "submariner-operator"
✓ Found 1 servicediscoveries in namespace "submariner-operator"
✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"
✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"
✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"
✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"
✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"
✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"
✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"
Files are stored under directory "submariner-20240910171903/karmada-cluster"
|
|
@tpantelis @yboaron |
|
Sorry - I didn’t mean the actual console output from subctl gather but the files that it outputs which includes the pod logs. |
|
@tpantelis i'm sorry! here is the folder |
|
I can see from the logs that the IPsec tunnels are set up in the OS but they never go active:
Perhaps there's a firewall preventing traffic between them? I know certain port(s) have to be open but I'll let the dataplane experts chime in. @yboaron @aswinsuryan |
|
From GW pod logs it seems that NatDiscovery failed [1] on both clusters , You can use [2] subctl command to verify inter-cluster firewall prerequisites [1] [2]
|
|
yes i followed firewall prerequisites hale@hale-VirtualBox:~$ subctl diagnose firewall inter-cluster --context member1 --remotecontext member2
✓ Checking if tunnels can be setup on the gateway node of cluster "member1"
✓ Skipping this check as it's a single node cluster
✓ Tunnels can be established on the gateway node of cluster "member1"
hale@hale-VirtualBox:~$ subctl diagnose firewall inter-cluster --context member2 --remotecontext member1
✓ Checking if tunnels can be setup on the gateway node of cluster "member2"
✓ Skipping this check as it's a single node cluster
✓ Tunnels can be established on the gateway node of cluster "member2" |
|
A. B. C. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm experiencing an issue where the Submariner gateways between my clusters are not establishing connections as expected and gets stuck in the Connecting status (the ipsec tunnel is not established) .
When running subctl show gateways, I observed that the gateways in member1 and member2 clusters are active, but 0 connections out of 1 are established for each.
Below is a summary of the scenario and the troubleshooting steps I have taken so far.
i'm using 3 clusters with minikube profile (each cluster have isolated network) for karmada and 2 members, setting also different service and pod CIDRs to avoid overlapping .
this is the subctl gather output
this is the subctl diagnose all output
and this is how i deploy the broker then how i join members
I’m looking for recommendations or assistance on how to set up locally these clusters differently to avoid this gateway creation issue.
Thank you for your support!
The text was updated successfully, but these errors were encountered: