Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pods running on non-gateway nodes cannot communicate with each other. #3110

Open
lgy1027 opened this issue Aug 1, 2024 · 12 comments
Open

Pods running on non-gateway nodes cannot communicate with each other. #3110

lgy1027 opened this issue Aug 1, 2024 · 12 comments
Assignees
@yboaron
Labels
Calico datapath Datapath related issues or enhancements need-info support

Comments

@lgy1027
Copy link

lgy1027 commented Aug 1, 2024

Hello, I currently have two cluster container networks that need to communicate with each other. When using Subenmani to accommodate the tube, if the business container is deployed on a non-gateway node, there will be no response using curl. The following are my deployment steps.

K8s version: 1.25.3
subctl version: 0.18.0
cni: calico --- vxlan

broker deploy:

subctl deploy-broker

cluster1 join:

subctl join --kubeconfig ~/.kube/c1 broker-info.subm --clusterid c1

cluster2 join:

subctl join --kubeconfig ~/.kube/c2 broker-info.subm --clusterid c2

Deploy nginx on the non-gateway node in the cluster1 cluster and use subctl exportd to export
Deploy the nettool service in cluster2 to test access to the nginx service of cluster1
nslooku will return the resolved IP address. When I use curl, it will always get stuck without responding.
Of course, if the two services and the group are deployed on the node where the gateway is located, they can communicate with each other.
I don’t know where my problem lies. It’s normal to use subctl show all or subctl diagnose all. I also modified the mode of kube-proxy and vxlan of calico according to the official documents.
@lgy1027 lgy1027 added the support label Aug 1, 2024
@lgy1027
Copy link
Author

lgy1027 commented Aug 2, 2024

cluster1 ippool:
image
cluster2 ippool:
image

subctl show all

Cluster "kubernetes"
 ✓ Detecting broker(s)
NAMESPACE               NAME                COMPONENTS                        GLOBALNET   GLOBALNET CIDR   DEFAULT GLOBALNET SIZE   DEFAULT DOMAINS   
submariner-k8s-broker   submariner-broker   service-discovery, connectivity   no          242.0.0.0/8      65536                                      

 ✓ Showing Connections
GATEWAY            CLUSTER   REMOTE IP     NAT   CABLE DRIVER   SUBNETS                       STATUS      RTT avg.     
yigou-dev-102-64   dev       10.0.102.64   no    libreswan      10.96.0.0/16, 10.244.0.0/16   connected   711.052µs    

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP   PUBLIC IP        CABLE DRIVER   TYPE     
karmada   10.0.102.23   112.29.111.158   libreswan      local    
dev       10.0.102.64   112.29.111.158   libreswan      remote   

 ✓ Showing Gateways
NODE                    HA STATUS   SUMMARY                               
hero-cloud-dev-102-23   active      All connections (1) are established   

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  calico
        Service CIDRs:   [10.102.0.0/16]
        Cluster CIDRs:   [10.202.0.0/16]

 ✓ Showing versions 
COMPONENT                       REPOSITORY           CONFIGURED   RUNNING                     ARCH    
submariner-gateway              quay.io/submariner   0.18.0       release-0.18-e3f3e56b57fe   amd64   
submariner-routeagent           quay.io/submariner   0.18.0       release-0.18-e3f3e56b57fe   amd64   
submariner-metrics-proxy        quay.io/submariner   0.18.0       release-0.18-011349c6f17e   amd64   
submariner-operator             quay.io/submariner   0.18.0       release-0.18-68fefdd74105   amd64   
submariner-lighthouse-agent     quay.io/submariner   0.18.0       release-0.18-02b6a5b37266   amd64   
submariner-lighthouse-coredns   quay.io/submariner   0.18.0       release-0.18-02b6a5b37266   amd64

cluster2:

Cluster "kubernetes"
 ✓ Detecting broker(s)
 ✓ No brokers found

 ✓ Showing Connections
GATEWAY                 CLUSTER   REMOTE IP     NAT   CABLE DRIVER   SUBNETS                        STATUS      RTT avg.     
hero-cloud-dev-102-23   karmada   10.0.102.23   no    libreswan      10.102.0.0/16, 10.202.0.0/16   connected   526.891µs    

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP   PUBLIC IP        CABLE DRIVER   TYPE     
dev       10.0.102.64   112.29.111.158   libreswan      local    
karmada   10.0.102.23   112.29.111.158   libreswan      remote   

 ✓ Showing Gateways
NODE               HA STATUS   SUMMARY                               
yigou-dev-102-64   active      All connections (1) are established   

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  calico
        Service CIDRs:   [10.96.0.0/16]
        Cluster CIDRs:   [10.244.0.0/16]

 ✓ Showing versions 
COMPONENT                       REPOSITORY           CONFIGURED   RUNNING                     ARCH    
submariner-gateway              quay.io/submariner   0.18.0       release-0.18-e3f3e56b57fe   amd64   
submariner-routeagent           quay.io/submariner   0.18.0       release-0.18-e3f3e56b57fe   amd64   
submariner-metrics-proxy        quay.io/submariner   0.18.0       release-0.18-011349c6f17e   amd64   
submariner-operator             quay.io/submariner   0.18.0       release-0.18-68fefdd74105   amd64   
submariner-lighthouse-agent     quay.io/submariner   0.18.0       release-0.18-02b6a5b37266   amd64   
submariner-lighthouse-coredns   quay.io/submariner   0.18.0       release-0.18-02b6a5b37266   amd64

cluster1 subctl diagnose all:

[root@hero-cloud-dev-102-23 network]# subctl diagnose all
Cluster "kubernetes"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.25.14" is supported

 ✓ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
 ✓ Checking DaemonSet "submariner-gateway"
 ✓ Checking DaemonSet "submariner-routeagent"
 ✓ Checking DaemonSet "submariner-metrics-proxy"
 ✓ Checking Deployment "submariner-lighthouse-agent"
 ✓ Checking Deployment "submariner-lighthouse-coredns"
 ✓ Checking the status of all Submariner pods
 ✓ Checking that gateway metrics are accessible from non-gateway nodes 

 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("calico") is supported
 ✓ Calico CNI detected, checking if the Submariner IPPool pre-requisites are configured
 ✓ Checking gateway connections
 ✓ Checking Submariner support for the kube-proxy mode 
 ✓ The kube-proxy mode is supported
 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic 

 ✓ Checking that services have been exported properly

cluster2 diagnose all

[root@hero-cloud-dev-102-23 network]# subctl diagnose all kubeconfig ./subctl/a12397116471111680373621 
Cluster "kubernetes"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.25.14" is supported

 ✓ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
 ✓ Checking DaemonSet "submariner-gateway"
 ✓ Checking DaemonSet "submariner-routeagent"
 ✓ Checking DaemonSet "submariner-metrics-proxy"
 ✓ Checking Deployment "submariner-lighthouse-agent"
 ✓ Checking Deployment "submariner-lighthouse-coredns"
 ✓ Checking the status of all Submariner pods
 ✓ Checking that gateway metrics are accessible from non-gateway nodes 

 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("calico") is supported
 ✓ Calico CNI detected, checking if the Submariner IPPool pre-requisites are configured
 ✓ Checking gateway connections
 ✓ Checking Submariner support for the kube-proxy mode 
 ✓ The kube-proxy mode is supported
 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic 

 ✓ Checking that services have been exported properly

@yboaron yboaron added the Calico label Aug 4, 2024
@yboaron
Copy link
Contributor

yboaron commented Aug 5, 2024 •
edited
Loading

Thanks for reaching out @lgy1027 ,
This looks like datapath issue that needs further investigation,

A. Short background:
Only the egress of inter-cluster datapath is handled by Submariner while ingress is handled by the CNI (Calico in your case).

So, podA@non_gw_node@cluster1 communication with podB@non_gw_node@cluster2 consists of the following segments :

  1. Egress : podA@non_gw_node@cluster1 --> gw_node@cluster1 via vx-submariner
  2. Egress : gw_node@cluster1 -> gw_node@cluster2 via IPSec tunnel
  3. Ingress: Decrypt IPSec packet
  4. Ingress: CNI should forward the packet to podB IP

B. Tcpdumping the traffic on the relevant nodes on cluster1 and cluster2 can point us to the root cause.

C. Please share the content of Calico default-ipv4-ippool from both clusters
D. Also upload Submariner logs (using subctl gather ) from both clusters.

@yboaron yboaron self-assigned this Aug 5, 2024
@yboaron yboaron mentioned this issue Aug 5, 2024
Closed
@yboaron yboaron added the datapath Datapath related issues or enhancements label Aug 5, 2024
@lgy1027
Copy link
Author

lgy1027 commented Aug 6, 2024 

Recently, we spent some time trying to locate the problem. It is found that packet loss occurs on the FORWARD chain of the filter table. The default policy of the FORWARD chain is DROP. Because the request packet does not match any of the rules, the packet is discarded. After the default policy of the Forward chain is changed to ACCEPT or the rules that can be matched by data packets are added, the access is normal.

I looked at the above iss and found that by changing it to accept through this method, interoperability can be achieved, but I think the main problem should not be this.

cluster1 default-ipv4-ippool:

apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  creationTimestamp: "2024-07-31T08:52:01Z"
  name: default-ipv4-ippool
  resourceVersion: "246979"
  uid: 2df236cc-f21b-49f0-9cab-bbc1f4e4a785
spec:
  allowedUses:
  - Workload
  - Tunnel
  blockSize: 26
  cidr: 10.202.0.0/16
  ipipMode: Never
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: Always

cluster2

apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  creationTimestamp: "2024-07-26T03:50:41Z"
  name: default-ipv4-ippool
  resourceVersion: "155237418"
  uid: 6343a566-5ac4-4cfe-bb5c-15ab689916b8
spec:
  allowedUses:
  - Workload
  - Tunnel
  blockSize: 26
  cidr: 10.244.0.0/16
  ipipMode: Never
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: Always

subctl gather :
cluster1.zip
cluster2.zip

@yboaron
Copy link
Contributor

yboaron commented Aug 6, 2024 •
edited
Loading

Trying to understand the root cause for packet drop , so
podA@non_gw_node@cluster1 communication with podB@non_gw_node@cluster2 consists of the following segments :

  1. Egress : podA@non_gw_node@cluster1 --> gw_node@cluster1 via vx-submariner
  2. Egress : gw_node@cluster1 -> gw_node@cluster2 via IPSec tunnel
  3. Ingress: decrypt IPSec packet
  4. Ingress: Calico should forward the packet to podB IP
  5. Egress : podB@non_gw_node@cluster2 --> gw_node@cluster2 via vx-submariner
  6. Egress : gw_node@cluster2 -> gw_node@cluster1 via IPSec tunnel
  7. Ingress: decrypt IPSec packet
  8. Calico should forward the packet to podA IP

Recently, we spent some time trying to locate the problem. It is found that packet loss occurs on the FORWARD chain of the filter table. The default policy of the FORWARD chain is DROP. Because the request packet does not match any of the rules, the packet is discarded. After the default policy of the Forward chain is changed to ACCEPT or the rules that can be matched by data packets are added, the access is normal.

Can you elaborate ?
Is packet being dropped in ingress direction ? which step ?
Could it be that Calico drop the packet because source IP address doesn't belong to clusters CIDR (it's the IP address of the pod from the other cluster)?
Do you have networkpolicy defined in your clusters?

@lgy1027
Copy link
Author

lgy1027 commented Aug 15, 2024

Trying to understand the root cause for packet drop , so podA@non_gw_node@cluster1 communication with podB@non_gw_node@cluster2 consists of the following segments :

  1. Egress : podA@non_gw_node@cluster1 --> gw_node@cluster1 via vx-submariner
  2. Egress : gw_node@cluster1 -> gw_node@cluster2 via IPSec tunnel
  3. Ingress: decrypt IPSec packet
  4. Ingress: Calico should forward the packet to podB IP
  5. Egress : podB@non_gw_node@cluster2 --> gw_node@cluster2 via vx-submariner
  6. Egress : gw_node@cluster2 -> gw_node@cluster1 via IPSec tunnel
  7. Ingress: decrypt IPSec packet
  8. Calico should forward the packet to podA IP

Recently, we spent some time trying to locate the problem. It is found that packet loss occurs on the FORWARD chain of the filter table. The default policy of the FORWARD chain is DROP. Because the request packet does not match any of the rules, the packet is discarded. After the default policy of the Forward chain is changed to ACCEPT or the rules that can be matched by data packets are added, the access is normal.

Can you elaborate ? Is packet being dropped in ingress direction ? which step ? Could it be that Calico drop the packet because source IP address doesn't belong to clusters CIDR (it's the IP address of the pod from the other cluster)? Do you have networkpolicy defined in your clusters?

In fact, I am a newbie in the network, and I have some shortcomings in locating problems in various aspects. I previously captured packets and tested them. The pod container network of cluster2 can track the gateway node of cluster2, but there is no communication between the gateway and the gateway. I entered the container of the gateway. Query the routing table and find that the routing table of the object cluster shows unreachable 10.102.0.0/16 proto bird
unreachable 10.202.0.0/16 proto bird, is this normal?

@yboaron
Copy link
Contributor

yboaron commented Aug 15, 2024

@lgy1027 ,
Can you try restarting all Submariner RouteAgent pods (you can use [1] command) on both clusters, and update if that helps?

[1]
kubectl delete pods -n submariner-operator -l app=submariner-routeagent

@dfarrell07 dfarrell07 mentioned this issue Aug 20, 2024
Closed
@lgy1027
Copy link
Author

lgy1027 commented Aug 21, 2024

@lgy1027 , Can you try restarting all Submariner RouteAgent pods (you can use [1] command) on both clusters, and update if that helps?

[1] kubectl delete pods -n submariner-operator -l app=submariner-routeagent

I tried restarting the route container and gateway container, but the result was the same when executing inside the gateway container, as follows:

bash-5.2# ip route
default via 10.0.102.1 dev ens192 proto static metric 100 
10.0.102.0/24 dev ens192 proto kernel scope link src 10.0.102.64 metric 100 
unreachable 10.102.0.0/16 proto bird 
unreachable 10.202.0.0/16 proto bird

At the same time, I found that after a while, the container in cluster1 failed to access the container in cluster2, and it would always be stuck. The gateway log reported an error link public ip warning. I don’t know if it is related to this, but I did not set the public ip, of course. I restarted the gateways of the two clusters and it worked again.

@lgy1027
Copy link
Author

lgy1027 commented Aug 21, 2024 •
edited
Loading

@lgy1027 , Can you try restarting all Submariner RouteAgent pods (you can use [1] command) on both clusters, and update if that helps?

[1] kubectl delete pods -n submariner-operator -l app=submariner-routeagent

this gateway logs:
cluster1

+ trap 'exit 1' SIGTERM SIGINT
+ export CHARON_PID_FILE=/var/run/charon.pid
+ CHARON_PID_FILE=/var/run/charon.pid
+ rm -f /var/run/charon.pid
+ SUBMARINER_VERBOSITY=2
+ '[' false == true ']'
+ DEBUG=-v=2
++ cat /proc/sys/net/ipv4/conf/all/send_redirects
+ [[ 0 = 0 ]]
+ exec submariner-gateway -v=2 -alsologtostderr
submariner-gateway version: release-0.18-e3f3e56b57fe
2024-08-21T01:20:51.218Z INF ../versions/version.go:35 main                 Go Version: go1.22.4
2024-08-21T01:20:51.218Z INF ../versions/version.go:36 main                 Go Arch: amd64
2024-08-21T01:20:51.218Z INF ../versions/version.go:37 main                 Git Commit Hash: e3f3e56b57fe22371cdcf885315804f67663006e
2024-08-21T01:20:51.218Z INF ../versions/version.go:38 main                 Git Commit Date: 2024-06-24T18:49:44+03:00
2024-08-21T01:20:51.219Z INF ..o/submariner/main.go:94 main                 Parsed env variables: types.SubmarinerSpecification{ClusterCidr:[]string{"10.202.0.0/16"}, GlobalCidr:[]string{}, ServiceCidr:[]string{"10.102.0.0/16"}, Broker:"", CableDriver:"libreswan", ClusterID:"c1", Namespace:"submariner-operator", PublicIP:"", Token:"", Debug:false, NATEnabled:true, HealthCheckEnabled:true, Uninstall:false, HaltOnCertError:false, HealthCheckInterval:0x1, HealthCheckMaxPacketLossCount:0x5, MetricsPort:32780}
2024-08-21T01:20:51.219Z INF ..o/submariner/main.go:97 main                 Proxy env variables: HTTP_PROXY: , HTTPS_PROXY: , NO_PROXY: 
W0821 01:20:51.220062       1 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2024-08-21T01:20:51.234Z INF ../gateway/gateway.go:108 Gateway              Initializing the gateway engine
2024-08-21T01:20:51.234Z INF ../gateway/gateway.go:134 Gateway              Creating the cable engine
2024-08-21T01:20:51.234Z INF ../gateway/gateway.go:145 Gateway              AIR_GAPPED_DEPLOYMENT is set to false
2024-08-21T01:20:51.702Z DBG ..pkg/cni/cni_iface.go:70 CNI                  Interface "lo" has "127.0.0.1" address
2024-08-21T01:20:51.702Z DBG ..pkg/cni/cni_iface.go:70 CNI                  Interface "ens192" has "10.0.102.23" address
2024-08-21T01:20:51.702Z DBG ..pkg/cni/cni_iface.go:70 CNI                  Interface "docker0" has "172.17.0.1" address
2024-08-21T01:20:51.702Z DBG ..pkg/cni/cni_iface.go:70 CNI                  Interface "vxlan.calico" has "10.202.44.192" address
2024-08-21T01:20:51.702Z DBG ..pkg/cni/cni_iface.go:75 CNI                  Found CNI Interface "vxlan.calico" that has IP "10.202.44.192" from ClusterCIDR "10.202.0.0/16"
2024-08-21T01:20:51.702Z INF ../gateway/gateway.go:161 Gateway              Creating the datastore syncer
2024-08-21T01:20:51.704Z INF ../gateway/gateway.go:188 Gateway              Starting the gateway engine
2024-08-21T01:20:51.704Z DBG ..ery/natdiscovery.go:102 NAT                  NAT discovery server starting on port 4490
2024-08-21T01:20:51.720Z INF ../gateway/gateway.go:247 Gateway              Starting leader election
I0821 01:20:51.720556       1 leaderelection.go:250] attempting to acquire leader lease submariner-operator/submariner-gateway-lock...
I0821 01:20:51.727152       1 leaderelection.go:260] successfully acquired lease submariner-operator/submariner-gateway-lock
2024-08-21T01:20:51.727Z DBG ..ols/record/event.go:377 Gateway              Event(v1.ObjectReference{Kind:"Lease", Namespace:"submariner-operator", Name:"submariner-gateway-lock", UID:"feb4d5cc-aa03-4352-ada9-87330dec3637", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"7554264", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' hero-cloud-dev-102-23-submariner-gateway became leader
2024-08-21T01:20:51.727Z INF ../gateway/gateway.go:280 Gateway              Leadership acquired - starting controllers
2024-08-21T01:20:51.727Z INF ..reswan/libreswan.go:136 libreswan            Using NATT UDP port 4500
2024-08-21T01:20:51.727Z INF ..gine/cableengine.go:111 CableEngine          CableEngine started with driver "libreswan"
2024-08-21T01:20:51.727Z INF ..public_ip_watcher.go:58 Endpoint             Starting the public IP watcher.
2024-08-21T01:20:51.727Z INF ..r/datastoresyncer.go:70 DSSyncer             Starting the datastore syncer
2024-08-21T01:20:51.727Z INF ..ers/tunnel/tunnel.go:40 Tunnel               Starting the tunnel controller
2024-08-21T01:20:51.727Z INF ../gateway/gateway.go:349 Gateway              Updating Gateway pod HA status to "active"
2024-08-21T01:20:51.742Z INF ../gateway/gateway.go:370 Gateway              Successfully updated Gateway pod HA status to "active"
I0821 01:20:51.743058       1 reflector.go:359] Caches populated for submariner.io/v1, Kind=Endpoint from k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232
I0821 01:20:51.743426       1 reflector.go:359] Caches populated for submariner.io/v1, Kind=Endpoint from k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232
I0821 01:20:51.744889       1 reflector.go:359] Caches populated for submariner.io/v1, Kind=Cluster from k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232
2024-08-21T01:20:51.841Z INF ..er/healthchecker.go:114 HealthChecker        CableEngine HealthChecker started with PingInterval: 1, MaxPacketLossCount: 5
2024-08-21T01:20:51.841Z INF ..ery/natdiscovery.go:144 NAT                  Starting NAT discovery for endpoint "submariner-cable-c2-10-0-102-64"
2024-08-21T01:20:51.841Z INF ..thchecker/pinger.go:106 HealthChecker        Starting pinger for IP "10.244.187.128"
2024-08-21T01:20:51.841Z INF ..er/healthchecker.go:176 HealthChecker        CableEngine HealthChecker started pinger for CableName: "submariner-cable-c2-10-0-102-64" with HealthCheckIP "10.244.187.128"
2024-08-21T01:20:51.843Z INF ..thchecker/pinger.go:177 HealthChecker        Ping to remote endpoint IP "10.244.187.128" is successful
I0821 01:20:51.846437       1 reflector.go:359] Caches populated for submariner.io/v1, Kind=Endpoint from k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232
I0821 01:20:51.846436       1 reflector.go:359] Caches populated for submariner.io/v1, Kind=Cluster from k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232
2024-08-21T01:20:51.849Z INF .._update_federator.go:71 Federator            local -> broker: Updated Cluster "submariner-k8s-broker/c1" 
2024-08-21T01:20:51.902Z INF .._update_federator.go:71 Federator            broker -> local: Updated Cluster "submariner-operator/c2" 
2024-08-21T01:20:51.945Z INF ../datastoresyncer.go:208 DSSyncer             Ensuring we are the only endpoint active for this cluster
2024-08-21T01:20:51.945Z INF ../datastoresyncer.go:266 DSSyncer             Creating local submariner Cluster: {
  "id": "c1",
  "spec": {
    "cluster_id": "c1",
    "color_codes": [
      "blue"
    ],
    "service_cidr": [
      "10.102.0.0/16"
    ],
    "cluster_cidr": [
      "10.202.0.0/16"
    ],
    "global_cidr": []
  }
}
I0821 01:20:51.947979       1 reflector.go:359] Caches populated for submariner.io/v1, Kind=Endpoint from k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232
2024-08-21T01:20:51.950Z INF .._update_federator.go:71 Federator            broker -> local: Updated Cluster "submariner-operator/c1" 
2024-08-21T01:20:51.950Z INF ../datastoresyncer.go:281 DSSyncer             Creating local submariner Endpoint: {
  "metadata": {
    "name": "c1-submariner-cable-c1-10-0-102-23",
    "creationTimestamp": null
  },
  "spec": {
    "cluster_id": "c1",
    "cable_name": "submariner-cable-c1-10-0-102-23",
    "healthCheckIP": "10.202.44.192",
    "hostname": "hero-cloud-dev-102-23",
    "subnets": [
      "10.102.0.0/16",
      "10.202.0.0/16"
    ],
    "private_ip": "10.0.102.23",
    "public_ip": "112.29.111.158",
    "nat_enabled": true,
    "backend": "libreswan",
    "backend_config": {
      "natt-discovery-port": "4490",
      "preferred-server": "false",
      "udp-port": "4500"
    }
  }
}
2024-08-21T01:20:51.951Z INF .._update_federator.go:71 Federator            local -> broker: Updated Endpoint "submariner-k8s-broker/c1-submariner-cable-c1-10-0-102-23" 
2024-08-21T01:20:51.955Z INF .._update_federator.go:71 Federator            broker -> local: Updated Endpoint "submariner-operator/c1-submariner-cable-c1-10-0-102-23" 
2024-08-21T01:20:51.955Z INF ../datastoresyncer.go:100 DSSyncer             Datastore syncer started
2024-08-21T01:20:52.004Z INF .._update_federator.go:71 Federator            broker -> local: Updated Endpoint "submariner-operator/c2-submariner-cable-c2-10-0-102-64" 
2024-08-21T01:20:52.705Z DBG ..ery/request_send.go:117 NAT                  Sending request - REQUEST_NUMBER: 0x68b3c66c31ad4c90, SENDER: "submariner-cable-c1-10-0-102-23", RECEIVER: "submariner-cable-c2-10-0-102-64", USING_SRC: 10.0.102.23:4490, USING_DST: 10.0.102.64:4490
2024-08-21T01:20:52.705Z DBG ..ery/request_send.go:117 NAT                  Sending request - REQUEST_NUMBER: 0x68b3c66c31ad4c91, SENDER: "submariner-cable-c1-10-0-102-23", RECEIVER: "submariner-cable-c2-10-0-102-64", USING_SRC: 10.0.102.23:4490, USING_DST: 112.29.111.158:4490
2024-08-21T01:20:52.706Z DBG ..y/response_handle.go:31 NAT                  Received response from 10.0.102.64:4490 - REQUEST_NUMBER: 0x68b3c66c31ad4c90, RESPONSE: OK, SENDER: "submariner-cable-c2-10-0-102-64", RECEIVER: "submariner-cable-c1-10-0-102-23"
2024-08-21T01:20:52.706Z DBG ../remote_endpoint.go:184 NAT                  selected private IP "10.0.102.64" for endpoint "submariner-cable-c2-10-0-102-64"
2024-08-21T01:20:52.706Z INF ..gine/cableengine.go:219 CableEngine          Installing Endpoint cable "submariner-cable-c2-10-0-102-64"
2024-08-21T01:20:52.706Z INF ..reswan/libreswan.go:596 libreswan            Starting Pluto
2024-08-21T01:20:52.707Z DBG ..y/response_handle.go:31 NAT                  Received response from 112.29.111.158:4490 - REQUEST_NUMBER: 0x68b3c66c31ad4c91, RESPONSE: UNKNOWN_DST_CLUSTER, SENDER: "submariner-cable-cluster-a-10-0-102-111", RECEIVER: "submariner-cable-c1-10-0-102-23"
2024-08-21T01:20:52.709Z ERR ..iscovery/listener.go:82 NAT                  Error handling message from address 112.29.111.158:4490:
00000000  1a 7c 08 91 99 b5 8d c3  cd f1 d9 68 10 02 1a 34  |.|.........h...4|
00000010  0a 09 63 6c 75 73 74 65  72 2d 61 12 27 73 75 62  |..cluster-a.'sub|
00000020  6d 61 72 69 6e 65 72 2d  63 61 62 6c 65 2d 63 6c  |mariner-cable-cl|
00000030  75 73 74 65 72 2d 61 2d  31 30 2d 30 2d 31 30 32  |uster-a-10-0-102|
00000040  2d 31 31 31 22 25 0a 02  63 31 12 1f 73 75 62 6d  |-111"%..c1..subm|
00000050  61 72 69 6e 65 72 2d 63  61 62 6c 65 2d 63 31 2d  |ariner-cable-c1-|
00000060  31 30 2d 30 2d 31 30 32  2d 32 33 42 11 0a 0b 31  |10-0-102-23B...1|
00000070  39 32 2e 31 36 38 2e 37  2e 31 10 df 88 03        |92.168.7.1....|
 error="remote endpoint \"submariner-cable-cluster-a-10-0-102-111\" responded with \"UNKNOWN_DST_CLUSTER\" : &proto.SubmarinerNATDiscoveryResponse{state:impl.MessageState{NoUnkeyedLiterals:pragma.NoUnkeyedLiterals{}, DoNotCompare:pragma.DoNotCompare{}, DoNotCopy:pragma.DoNotCopy{}, atomicMessageInfo:(*impl.MessageInfo)(nil)}, sizeCache:0, unknownFields:[]uint8(nil), RequestNumber:0x68b3c66c31ad4c91, Response:2, Sender:(*proto.EndpointDetails)(0xc000488c80), Receiver:(*proto.EndpointDetails)(0xc000488ff0), SrcIpNatDetected:false, SrcPortNatDetected:false, DstIpNatDetected:false, ReceivedSrc:(*proto.IPPortPair)(0xc000168880)}"
Initializing NSS database

002 listening for IKE messages
002 Kernel supports NIC esp-hw-offload
002 adding UDP interface vx-submariner 240.0.102.23:500
002 adding UDP interface vx-submariner 240.0.102.23:4500
002 adding UDP interface vxlan.calico 10.202.44.192:500
002 adding UDP interface vxlan.calico 10.202.44.192:4500
002 adding UDP interface docker0 172.17.0.1:500
002 adding UDP interface docker0 172.17.0.1:4500
002 adding UDP interface ens192 10.0.102.23:500
002 adding UDP interface ens192 10.0.102.23:4500
002 adding UDP interface lo 127.0.0.1:500
002 adding UDP interface lo 127.0.0.1:4500
002 loading secrets from "/etc/ipsec.secrets"
002 loading secrets from "/etc/ipsec.d/submariner.secrets"
2024-08-21T01:20:53.423Z INF ..reswan/libreswan.go:384 libreswan            Creating connection(s) for {"metadata":{"name":"c2-submariner-cable-c2-10-0-102-64","namespace":"submariner-operator","uid":"bde7e753-8936-4c44-a88e-2c5d9c7350a0","resourceVersion":"5313411","generation":1,"creationTimestamp":"2024-08-15T01:40:13Z","labels":{"submariner-io/clusterID":"c2"}},"spec":{"cluster_id":"c2","cable_name":"submariner-cable-c2-10-0-102-64","healthCheckIP":"10.244.187.128","hostname":"yigou-dev-102-64","subnets":["10.96.0.0/16","10.244.0.0/16"],"private_ip":"10.0.102.64","public_ip":"112.29.111.158","nat_enabled":true,"backend":"libreswan","backend_config":{"natt-discovery-port":"4490","preferred-server":"false","udp-port":"4500"}}} in bi-directional mode
2024-08-21T01:20:53.423Z INF ..reswan/libreswan.go:448 libreswan            bidirectionalConnectToEndpoint: executing whack with args: [--psk --encrypt --name submariner-cable-c2-10-0-102-64-0-0 --id 10.0.102.23 --host 10.0.102.23 --client 10.102.0.0/16 --ikeport 4500 --to --id 10.0.102.64 --host 10.0.102.64 --client 10.96.0.0/16 --ikeport 4500 --dpdaction=hold --dpddelay 30]
002 "submariner-cable-c2-10-0-102-64-0-0": added IKEv2 connection
181 "submariner-cable-c2-10-0-102-64-0-0" #1: initiating IKEv2 connection
2024-08-21T01:20:53.468Z INF ..reswan/libreswan.go:448 libreswan            bidirectionalConnectToEndpoint: executing whack with args: [--psk --encrypt --name submariner-cable-c2-10-0-102-64-0-1 --id 10.0.102.23 --host 10.0.102.23 --client 10.102.0.0/16 --ikeport 4500 --to --id 10.0.102.64 --host 10.0.102.64 --client 10.244.0.0/16 --ikeport 4500 --dpdaction=hold --dpddelay 30]
002 "submariner-cable-c2-10-0-102-64-0-1": added IKEv2 connection
002 "submariner-cable-c2-10-0-102-64-0-1" #3: initiating Child SA using IKE SA #1
189 "submariner-cable-c2-10-0-102-64-0-1" #3: sent CREATE_CHILD_SA request for new IPsec SA
004 "submariner-cable-c2-10-0-102-64-0-1" #3: initiator established Child SA using #1; IPsec tunnel [10.102.0.0-10.102.255.255:0-65535 0] -> [10.244.0.0-10.244.255.255:0-65535 0] {ESP=>0xf81cf2a1 <0xdda7abc1 xfrm=AES_GCM_16_256-NONE DPD=active}
2024-08-21T01:20:53.526Z INF ..reswan/libreswan.go:448 libreswan            bidirectionalConnectToEndpoint: executing whack with args: [--psk --encrypt --name submariner-cable-c2-10-0-102-64-1-0 --id 10.0.102.23 --host 10.0.102.23 --client 10.202.0.0/16 --ikeport 4500 --to --id 10.0.102.64 --host 10.0.102.64 --client 10.96.0.0/16 --ikeport 4500 --dpdaction=hold --dpddelay 30]
002 "submariner-cable-c2-10-0-102-64-1-0": added IKEv2 connection
002 "submariner-cable-c2-10-0-102-64-1-0" #4: initiating Child SA using IKE SA #1
189 "submariner-cable-c2-10-0-102-64-1-0" #4: sent CREATE_CHILD_SA request for new IPsec SA
004 "submariner-cable-c2-10-0-102-64-1-0" #4: initiator established Child SA using #1; IPsec tunnel [10.202.0.0-10.202.255.255:0-65535 0] -> [10.96.0.0-10.96.255.255:0-65535 0] {ESP=>0x5ff63a4c <0xcfd84475 xfrm=AES_GCM_16_256-NONE DPD=active}
2024-08-21T01:20:53.553Z INF ..reswan/libreswan.go:448 libreswan            bidirectionalConnectToEndpoint: executing whack with args: [--psk --encrypt --name submariner-cable-c2-10-0-102-64-1-1 --id 10.0.102.23 --host 10.0.102.23 --client 10.202.0.0/16 --ikeport 4500 --to --id 10.0.102.64 --host 10.0.102.64 --client 10.244.0.0/16 --ikeport 4500 --dpdaction=hold --dpddelay 30]
002 "submariner-cable-c2-10-0-102-64-1-1": added IKEv2 connection
002 "submariner-cable-c2-10-0-102-64-1-1" #5: initiating Child SA using IKE SA #1
189 "submariner-cable-c2-10-0-102-64-1-1" #5: sent CREATE_CHILD_SA request for new IPsec SA
004 "submariner-cable-c2-10-0-102-64-1-1" #5: initiator established Child SA using #1; IPsec tunnel [10.202.0.0-10.202.255.255:0-65535 0] -> [10.244.0.0-10.244.255.255:0-65535 0] {ESP=>0x2a3dd195 <0x327d78af xfrm=AES_GCM_16_256-NONE DPD=active}
2024-08-21T01:20:53.579Z INF ..gine/cableengine.go:226 CableEngine          Successfully installed Endpoint cable "submariner-cable-c2-10-0-102-64" with remote IP 10.0.102.64
2024-08-21T01:21:09.451Z DBG ..ry/request_handle.go:54 NAT                  Received request from 10.0.102.64:4490 - REQUEST_NUMBER: 0x7c3dab443a12bf30, SENDER: "submariner-cable-c2-10-0-102-64", RECEIVER: "submariner-cable-c1-10-0-102-23"
2024-08-21T01:21:09.452Z DBG ..y/request_handle.go:119 NAT                  Sending response to 10.0.102.64:4490 - REQUEST_NUMBER: 0x7c3dab443a12bf30, RESPONSE: OK, SENDER: "submariner-cable-c1-10-0-102-23", RECEIVER: "submariner-cable-c2-10-0-102-64"

cluster2：

+ trap 'exit 1' SIGTERM SIGINT
+ export CHARON_PID_FILE=/var/run/charon.pid
+ CHARON_PID_FILE=/var/run/charon.pid
+ rm -f /var/run/charon.pid
+ SUBMARINER_VERBOSITY=2
+ '[' false == true ']'
+ DEBUG=-v=2
++ cat /proc/sys/net/ipv4/conf/all/send_redirects
+ [[ 0 = 0 ]]
+ exec submariner-gateway -v=2 -alsologtostderr
submariner-gateway version: release-0.18-e3f3e56b57fe
2024-08-21T01:21:07.719Z INF ../versions/version.go:35 main                 Go Version: go1.22.4
2024-08-21T01:21:07.719Z INF ../versions/version.go:36 main                 Go Arch: amd64
2024-08-21T01:21:07.719Z INF ../versions/version.go:37 main                 Git Commit Hash: e3f3e56b57fe22371cdcf885315804f67663006e
2024-08-21T01:21:07.719Z INF ../versions/version.go:38 main                 Git Commit Date: 2024-06-24T18:49:44+03:00
2024-08-21T01:21:07.720Z INF ..o/submariner/main.go:94 main                 Parsed env variables: types.SubmarinerSpecification{ClusterCidr:[]string{"10.244.0.0/16"}, GlobalCidr:[]string{}, ServiceCidr:[]string{"10.96.0.0/16"}, Broker:"", CableDriver:"libreswan", ClusterID:"c2", Namespace:"submariner-operator", PublicIP:"", Token:"", Debug:false, NATEnabled:true, HealthCheckEnabled:true, Uninstall:false, HaltOnCertError:false, HealthCheckInterval:0x1, HealthCheckMaxPacketLossCount:0x5, MetricsPort:32780}
2024-08-21T01:21:07.721Z INF ..o/submariner/main.go:97 main                 Proxy env variables: HTTP_PROXY: , HTTPS_PROXY: , NO_PROXY: 
W0821 01:21:07.721366       1 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2024-08-21T01:21:07.751Z INF ../gateway/gateway.go:108 Gateway              Initializing the gateway engine
2024-08-21T01:21:07.751Z INF ../gateway/gateway.go:134 Gateway              Creating the cable engine
2024-08-21T01:21:07.751Z INF ../gateway/gateway.go:145 Gateway              AIR_GAPPED_DEPLOYMENT is set to false
2024-08-21T01:21:08.452Z DBG ..pkg/cni/cni_iface.go:70 CNI                  Interface "lo" has "127.0.0.1" address
2024-08-21T01:21:08.452Z DBG ..pkg/cni/cni_iface.go:70 CNI                  Interface "ens192" has "10.0.102.64" address
2024-08-21T01:21:08.452Z DBG ..pkg/cni/cni_iface.go:70 CNI                  Interface "docker0" has "172.17.0.1" address
2024-08-21T01:21:08.452Z DBG ..pkg/cni/cni_iface.go:70 CNI                  Interface "vxlan.calico" has "10.244.187.128" address
2024-08-21T01:21:08.453Z DBG ..pkg/cni/cni_iface.go:75 CNI                  Found CNI Interface "vxlan.calico" that has IP "10.244.187.128" from ClusterCIDR "10.244.0.0/16"
2024-08-21T01:21:08.453Z INF ../gateway/gateway.go:161 Gateway              Creating the datastore syncer
2024-08-21T01:21:08.453Z INF ../gateway/gateway.go:188 Gateway              Starting the gateway engine
2024-08-21T01:21:08.453Z DBG ..ery/natdiscovery.go:102 NAT                  NAT discovery server starting on port 4490
2024-08-21T01:21:08.471Z INF ../gateway/gateway.go:247 Gateway              Starting leader election
I0821 01:21:08.471440       1 leaderelection.go:250] attempting to acquire leader lease submariner-operator/submariner-gateway-lock...
I0821 01:21:08.482354       1 leaderelection.go:260] successfully acquired lease submariner-operator/submariner-gateway-lock
2024-08-21T01:21:08.482Z DBG ..ols/record/event.go:377 Gateway              Event(v1.ObjectReference{Kind:"Lease", Namespace:"submariner-operator", Name:"submariner-gateway-lock", UID:"958a0848-5a70-48a9-ba3f-2cc6fac7f461", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"182027556", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' yigou-dev-102-64-submariner-gateway became leader
2024-08-21T01:21:08.482Z INF ../gateway/gateway.go:280 Gateway              Leadership acquired - starting controllers
2024-08-21T01:21:08.482Z INF ..reswan/libreswan.go:136 libreswan            Using NATT UDP port 4500
2024-08-21T01:21:08.482Z INF ..gine/cableengine.go:111 CableEngine          CableEngine started with driver "libreswan"
2024-08-21T01:21:08.482Z INF ..public_ip_watcher.go:58 Endpoint             Starting the public IP watcher.
2024-08-21T01:21:08.482Z INF ..r/datastoresyncer.go:70 DSSyncer             Starting the datastore syncer
2024-08-21T01:21:08.483Z INF ..ers/tunnel/tunnel.go:40 Tunnel               Starting the tunnel controller
2024-08-21T01:21:08.483Z INF ../gateway/gateway.go:349 Gateway              Updating Gateway pod HA status to "active"
2024-08-21T01:21:08.505Z INF ../gateway/gateway.go:370 Gateway              Successfully updated Gateway pod HA status to "active"
I0821 01:21:08.508266       1 reflector.go:359] Caches populated for submariner.io/v1, Kind=Cluster from k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232
I0821 01:21:08.509867       1 reflector.go:359] Caches populated for submariner.io/v1, Kind=Endpoint from k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232
I0821 01:21:08.511608       1 reflector.go:359] Caches populated for submariner.io/v1, Kind=Endpoint from k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232
I0821 01:21:08.601233       1 reflector.go:359] Caches populated for submariner.io/v1, Kind=Cluster from k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232
I0821 01:21:08.601343       1 reflector.go:359] Caches populated for submariner.io/v1, Kind=Endpoint from k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232
2024-08-21T01:21:08.605Z INF .._update_federator.go:71 Federator            local -> broker: Updated Cluster "submariner-k8s-broker/c2" 
2024-08-21T01:21:08.608Z INF ..ery/natdiscovery.go:144 NAT                  Starting NAT discovery for endpoint "submariner-cable-c1-10-0-102-23"
2024-08-21T01:21:08.611Z INF ..er/healthchecker.go:114 HealthChecker        CableEngine HealthChecker started with PingInterval: 1, MaxPacketLossCount: 5
2024-08-21T01:21:08.611Z INF ..thchecker/pinger.go:106 HealthChecker        Starting pinger for IP "10.202.44.192"
2024-08-21T01:21:08.611Z INF ..er/healthchecker.go:176 HealthChecker        CableEngine HealthChecker started pinger for CableName: "submariner-cable-c1-10-0-102-23" with HealthCheckIP "10.202.44.192"
2024-08-21T01:21:08.612Z INF ..thchecker/pinger.go:177 HealthChecker        Ping to remote endpoint IP "10.202.44.192" is successful
2024-08-21T01:21:08.662Z INF .._update_federator.go:71 Federator            broker -> local: Updated Cluster "submariner-operator/c1" 
2024-08-21T01:21:08.699Z INF ../datastoresyncer.go:208 DSSyncer             Ensuring we are the only endpoint active for this cluster
2024-08-21T01:21:08.700Z INF ../datastoresyncer.go:266 DSSyncer             Creating local submariner Cluster: {
  "id": "c2",
  "spec": {
    "cluster_id": "c2",
    "color_codes": [
      "blue"
    ],
    "service_cidr": [
      "10.96.0.0/16"
    ],
    "cluster_cidr": [
      "10.244.0.0/16"
    ],
    "global_cidr": []
  }
}
I0821 01:21:08.701872       1 reflector.go:359] Caches populated for submariner.io/v1, Kind=Endpoint from k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232
2024-08-21T01:21:08.706Z INF .._update_federator.go:71 Federator            local -> broker: Updated Endpoint "submariner-k8s-broker/c2-submariner-cable-c2-10-0-102-64" 
2024-08-21T01:21:08.707Z INF .._update_federator.go:71 Federator            broker -> local: Updated Cluster "submariner-operator/c2" 
2024-08-21T01:21:08.707Z INF ../datastoresyncer.go:281 DSSyncer             Creating local submariner Endpoint: {
  "metadata": {
    "name": "c2-submariner-cable-c2-10-0-102-64",
    "creationTimestamp": null
  },
  "spec": {
    "cluster_id": "c2",
    "cable_name": "submariner-cable-c2-10-0-102-64",
    "healthCheckIP": "10.244.187.128",
    "hostname": "yigou-dev-102-64",
    "subnets": [
      "10.96.0.0/16",
      "10.244.0.0/16"
    ],
    "private_ip": "10.0.102.64",
    "public_ip": "112.29.111.158",
    "nat_enabled": true,
    "backend": "libreswan",
    "backend_config": {
      "natt-discovery-port": "4490",
      "preferred-server": "false",
      "udp-port": "4500"
    }
  }
}
2024-08-21T01:21:08.717Z INF .._update_federator.go:71 Federator            broker -> local: Updated Endpoint "submariner-operator/c2-submariner-cable-c2-10-0-102-64" 
2024-08-21T01:21:08.717Z INF ../datastoresyncer.go:100 DSSyncer             Datastore syncer started
2024-08-21T01:21:08.762Z INF .._update_federator.go:71 Federator            broker -> local: Updated Endpoint "submariner-operator/c1-submariner-cable-c1-10-0-102-23" 
2024-08-21T01:21:09.455Z DBG ..ery/request_send.go:117 NAT                  Sending request - REQUEST_NUMBER: 0x7c3dab443a12bf30, SENDER: "submariner-cable-c2-10-0-102-64", RECEIVER: "submariner-cable-c1-10-0-102-23", USING_SRC: 10.0.102.64:4490, USING_DST: 10.0.102.23:4490
2024-08-21T01:21:09.455Z DBG ..ery/request_send.go:117 NAT                  Sending request - REQUEST_NUMBER: 0x7c3dab443a12bf31, SENDER: "submariner-cable-c2-10-0-102-64", RECEIVER: "submariner-cable-c1-10-0-102-23", USING_SRC: 10.0.102.64:4490, USING_DST: 112.29.111.158:4490
2024-08-21T01:21:09.456Z DBG ..y/response_handle.go:31 NAT                  Received response from 10.0.102.23:4490 - REQUEST_NUMBER: 0x7c3dab443a12bf30, RESPONSE: OK, SENDER: "submariner-cable-c1-10-0-102-23", RECEIVER: "submariner-cable-c2-10-0-102-64"
2024-08-21T01:21:09.456Z DBG ../remote_endpoint.go:184 NAT                  selected private IP "10.0.102.23" for endpoint "submariner-cable-c1-10-0-102-23"
2024-08-21T01:21:09.456Z INF ..gine/cableengine.go:219 CableEngine          Installing Endpoint cable "submariner-cable-c1-10-0-102-23"
2024-08-21T01:21:09.456Z INF ..reswan/libreswan.go:596 libreswan            Starting Pluto
2024-08-21T01:21:09.456Z DBG ..y/response_handle.go:31 NAT                  Received response from 112.29.111.158:4490 - REQUEST_NUMBER: 0x7c3dab443a12bf31, RESPONSE: UNKNOWN_DST_CLUSTER, SENDER: "submariner-cable-cluster-a-10-0-102-111", RECEIVER: "submariner-cable-c2-10-0-102-64"
2024-08-21T01:21:09.457Z ERR ..iscovery/listener.go:82 NAT                  Error handling message from address 112.29.111.158:4490:
00000000  1a 7c 08 b1 fe ca d0 c3  e8 ea 9e 7c 10 02 1a 34  |.|.........|...4|
00000010  0a 09 63 6c 75 73 74 65  72 2d 61 12 27 73 75 62  |..cluster-a.'sub|
00000020  6d 61 72 69 6e 65 72 2d  63 61 62 6c 65 2d 63 6c  |mariner-cable-cl|
00000030  75 73 74 65 72 2d 61 2d  31 30 2d 30 2d 31 30 32  |uster-a-10-0-102|
00000040  2d 31 31 31 22 25 0a 02  63 32 12 1f 73 75 62 6d  |-111"%..c2..subm|
00000050  61 72 69 6e 65 72 2d 63  61 62 6c 65 2d 63 32 2d  |ariner-cable-c2-|
00000060  31 30 2d 30 2d 31 30 32  2d 36 34 42 11 0a 0b 31  |10-0-102-64B...1|
00000070  39 32 2e 31 36 38 2e 37  2e 31 10 e0 88 03        |92.168.7.1....|
 error="remote endpoint \"submariner-cable-cluster-a-10-0-102-111\" responded with \"UNKNOWN_DST_CLUSTER\" : &proto.SubmarinerNATDiscoveryResponse{state:impl.MessageState{NoUnkeyedLiterals:pragma.NoUnkeyedLiterals{}, DoNotCompare:pragma.DoNotCompare{}, DoNotCopy:pragma.DoNotCopy{}, atomicMessageInfo:(*impl.MessageInfo)(nil)}, sizeCache:0, unknownFields:[]uint8(nil), RequestNumber:0x7c3dab443a12bf31, Response:2, Sender:(*proto.EndpointDetails)(0xc000b18a50), Receiver:(*proto.EndpointDetails)(0xc000b18aa0), SrcIpNatDetected:false, SrcPortNatDetected:false, DstIpNatDetected:false, ReceivedSrc:(*proto.IPPortPair)(0xc000f48200)}"
Initializing NSS database

002 listening for IKE messages
002 forgetting secrets
002 loading secrets from "/etc/ipsec.secrets"
002 loading secrets from "/etc/ipsec.d/submariner.secrets"
2024-08-21T01:21:11.578Z INF ..reswan/libreswan.go:384 libreswan            Creating connection(s) for {"metadata":{"name":"c1-submariner-cable-c1-10-0-102-23","namespace":"submariner-operator","uid":"1b558961-0e45-4766-8745-43ed334f56f2","resourceVersion":"173768085","generation":1,"creationTimestamp":"2024-08-15T01:40:13Z","labels":{"submariner-io/clusterID":"c1"}},"spec":{"cluster_id":"c1","cable_name":"submariner-cable-c1-10-0-102-23","healthCheckIP":"10.202.44.192","hostname":"hero-cloud-dev-102-23","subnets":["10.102.0.0/16","10.202.0.0/16"],"private_ip":"10.0.102.23","public_ip":"112.29.111.158","nat_enabled":true,"backend":"libreswan","backend_config":{"natt-discovery-port":"4490","preferred-server":"false","udp-port":"4500"}}} in bi-directional mode
2024-08-21T01:21:11.578Z INF ..reswan/libreswan.go:448 libreswan            bidirectionalConnectToEndpoint: executing whack with args: [--psk --encrypt --name submariner-cable-c1-10-0-102-23-0-0 --id 10.0.102.64 --host 10.0.102.64 --client 10.96.0.0/16 --ikeport 4500 --to --id 10.0.102.23 --host 10.0.102.23 --client 10.102.0.0/16 --ikeport 4500 --dpdaction=hold --dpddelay 30]
002 "submariner-cable-c1-10-0-102-23-0-0": added IKEv2 connection
181 "submariner-cable-c1-10-0-102-23-0-0" #1: initiating IKEv2 connection
2024-08-21T01:21:11.627Z INF ..reswan/libreswan.go:448 libreswan            bidirectionalConnectToEndpoint: executing whack with args: [--psk --encrypt --name submariner-cable-c1-10-0-102-23-0-1 --id 10.0.102.64 --host 10.0.102.64 --client 10.96.0.0/16 --ikeport 4500 --to --id 10.0.102.23 --host 10.0.102.23 --client 10.202.0.0/16 --ikeport 4500 --dpdaction=hold --dpddelay 30]
002 "submariner-cable-c1-10-0-102-23-0-1": added IKEv2 connection
002 "submariner-cable-c1-10-0-102-23-0-1" #3: initiating Child SA using IKE SA #1
189 "submariner-cable-c1-10-0-102-23-0-1" #3: sent CREATE_CHILD_SA request for new IPsec SA
004 "submariner-cable-c1-10-0-102-23-0-1" #3: initiator established Child SA using #1; IPsec tunnel [10.96.0.0-10.96.255.255:0-65535 0] -> [10.202.0.0-10.202.255.255:0-65535 0] {ESP=>0x05ad8384 <0x6cd0fe3c xfrm=AES_GCM_16_256-NONE DPD=active}
2024-08-21T01:21:11.692Z INF ..reswan/libreswan.go:448 libreswan            bidirectionalConnectToEndpoint: executing whack with args: [--psk --encrypt --name submariner-cable-c1-10-0-102-23-1-0 --id 10.0.102.64 --host 10.0.102.64 --client 10.244.0.0/16 --ikeport 4500 --to --id 10.0.102.23 --host 10.0.102.23 --client 10.102.0.0/16 --ikeport 4500 --dpdaction=hold --dpddelay 30]
002 "submariner-cable-c1-10-0-102-23-1-0": added IKEv2 connection
002 "submariner-cable-c1-10-0-102-23-1-0" #4: initiating Child SA using IKE SA #1
189 "submariner-cable-c1-10-0-102-23-1-0" #4: sent CREATE_CHILD_SA request for new IPsec SA
004 "submariner-cable-c1-10-0-102-23-1-0" #4: initiator established Child SA using #1; IPsec tunnel [10.244.0.0-10.244.255.255:0-65535 0] -> [10.102.0.0-10.102.255.255:0-65535 0] {ESP=>0xe76d5edc <0x8894fd4a xfrm=AES_GCM_16_256-NONE DPD=active}
2024-08-21T01:21:11.719Z INF ..reswan/libreswan.go:448 libreswan            bidirectionalConnectToEndpoint: executing whack with args: [--psk --encrypt --name submariner-cable-c1-10-0-102-23-1-1 --id 10.0.102.64 --host 10.0.102.64 --client 10.244.0.0/16 --ikeport 4500 --to --id 10.0.102.23 --host 10.0.102.23 --client 10.202.0.0/16 --ikeport 4500 --dpdaction=hold --dpddelay 30]
002 "submariner-cable-c1-10-0-102-23-1-1": added IKEv2 connection
002 "submariner-cable-c1-10-0-102-23-1-1" #5: initiating Child SA using IKE SA #1
189 "submariner-cable-c1-10-0-102-23-1-1" #5: sent CREATE_CHILD_SA request for new IPsec SA
004 "submariner-cable-c1-10-0-102-23-1-1" #5: initiator established Child SA using #1; IPsec tunnel [10.244.0.0-10.244.255.255:0-65535 0] -> [10.202.0.0-10.202.255.255:0-65535 0] {ESP=>0xb5f5bbef <0xf4bd3087 xfrm=AES_GCM_16_256-NONE DPD=active}
2024-08-21T01:21:11.748Z INF ..gine/cableengine.go:226 CableEngine          Successfully installed Endpoint cable "submariner-cable-c1-10-0-102-23" with remote IP 10.0.102.23

·

@yboaron