Gefyra v2 test feedback

[sbor23]

What happened?

$ pip install -U gefyra
[...]
Installing collected packages: gefyra
  Attempting uninstall: gefyra
    Found existing installation: gefyra 1.0.2
    Uninstalling gefyra-1.0.2:
      Successfully uninstalled gefyra-1.0.2
Successfully installed gefyra-2.0.0

$ gefyra up
Error: No module named 'alive_progress'

After installing alive_progress the command works.

There was an issue in my old kubeconfig, so the original part of the bug is not valid anymore:

However gefyra up tries to read the kubeconfig and automatically connect to somewhere. That it also seems to read/write from ~/.gefyra/default.yaml.

$ gefyra -d up
on 0: DEBUG gefyra.api.status status.py:93 - Reading API resources from Kubernetes
on 0: DEBUG gefyra.api.status status.py:100 - Reading gefyra namespace
on 0: DEBUG gefyra.api.status status.py:109 - Checking operator deployment
on 0: DEBUG gefyra.api.status status.py:126 - Checking Stowaway endpoint
on 0: DEBUG gefyra.api.status status.py:35 - Checking cargo container running
on 0: DEBUG gefyra.api.status status.py:49 - Checking gefyra network available
on 0: DEBUG gefyra.api.status status.py:63 - Probing wireguard connection
on 0: DEBUG gefyra.cargo cargo.py:18 - Probing connection to 192.168.99.1 (attempt 0/40))
[...]
on 0: DEBUG gefyra.cargo cargo.py:18 - Probing connection to 192.168.99.1 (attempt 39/40))
on 0: DEBUG gefyra.api.utils utils.py:69 - Operation time for 'status(...)' was 42162.5786ms
on 0: WARNING gefyra updown.py:31 - Gefyra is not installed, but operating properly. Aborting.
on 0: DEBUG gefyra.api.utils utils.py:69 - Operation time for 'get_client(...)' was 21.7891ms
on 2: DEBUG gefyra.api.utils utils.py:69 - Operation time for 'get_client(...)' was 23.3334ms
on 2: DEBUG gefyra.api.clients clients.py:111 - gefyra_server: 172.17.0.1:31820
on 2: DEBUG gefyra.api.utils utils.py:69 - Operation time for 'write_client_file(...)' was 35.5437ms
on 3: DEBUG gefyra updown.py:152 - Minikube profile None
on 3: DEBUG gefyra.cargo cargo.py:18 - Probing connection to 192.168.99.1 (attempt 0/2))
on 3: DEBUG gefyra.cargo cargo.py:18 - Probing connection to 192.168.99.1 (attempt 1/2))
on 3: DEBUG gefyra.api.utils utils.py:69 - Operation time for 'list_connections(...)' was 2117.9293ms
on 3: DEBUG gefyra.api.connect connect.py:48 - Restoring exinsting connection default
on 3: DEBUG gefyra.api.utils utils.py:69 - Operation time for 'get_client(...)' was 26.1622ms
on 3: INFO gefyra.local.networking networking.py:29 - Gefyra network already exists
on 3: DEBUG gefyra.local.networking networking.py:17 - Network {'Name': 'gefyra-network-default', 'Id': '8947bfd351d39d549c5ca930eb2fa4fe45819c37306280a5d19f4f260913c642', 'Created': '2023-09-19T08:38:20.537710097+02:00', 'Scope': 'local', 'Driver': 'bridge', 'EnableIPv6': False, 'IPAM': {'Driver': 'default', 'Options': None, 'Config': [{'Subnet': '172.18.0.0/16'}]}, 'Internal': False, 'Attachable': False, 'Ingress': False, 'ConfigFrom': {'Network': ''}, 'ConfigOnly': False, 'Containers': {'9ab547f50bfd0787be9008d5ec7ccb99ac8349e6bc9c543d1ed31e25886a8bf4': {'Name': 'gefyra-cargo-default', 'EndpointID': '52545e32296a8a824627717f8d93df35c38dd4ba93b085c4826279a22a98d218', 'MacAddress': '02:42:ac:12:00:95', 'IPv4Address': '172.18.0.149/16', 'IPv6Address': ''}}, 'Options': {'com.docker.network.driver.mtu': '1340'}, 'Labels': {'created_by.gefyra.dev': 'true'}}
on 3: DEBUG gefyra.types types.py:156 - Fetching object GefyraClient default
on 3: DEBUG gefyra.types types.py:156 - Fetching object GefyraClient default
on 3: DEBUG gefyra.types types.py:156 - Fetching object GefyraClient default
on 3: DEBUG gefyra.api.connect connect.py:136 - 172.17.0.1:31820
on 3: DEBUG gefyra.api.connect connect.py:202 - Checking wireguard connection
on 3: DEBUG gefyra.cargo cargo.py:18 - Probing connection to 192.168.99.1 (attempt 0/40))
[...]
on 3: DEBUG gefyra.cargo cargo.py:18 - Probing connection to 192.168.99.1 (attempt 39/40))
Connecting local network 'gefyra-network-default' to the cluster (up to 10 min) |██████████████████████████████⚠︎         | (!) 3/4 [75%] in 1:27.6 
Error: Gefyra could not successfully establish the connection to '172.17.0.1'.
If you have run 'gefyra up' with a remote cluster, a newly created route may not be working immediately.
Try running 'gefyra up' again after some time. Error: Gefyra could not successfully confirm the connection working.

Trying later doesn't work either. But that's not surprising, gefyra doesn't seem to find the correct cluster that runs on a private IP in the 10.xxx range.

I want to get rid of old gefyra v1 config, so I remove the ~/.gefyra folder.
Bad idea:

$ gefyra -d up
Error: KUBE_CONFIG_FILE ~/.gefyra/default.yaml not found.

However, bringing gefyra up still doesn't work. So I tried gefyra down:

$ gefyra -d down
on 0: DEBUG gefyra.cargo cargo.py:18 - Probing connection to 192.168.99.1 (attempt 0/2))
on 0: DEBUG gefyra.cargo cargo.py:18 - Probing connection to 192.168.99.1 (attempt 1/2))
on 0: DEBUG gefyra.api.utils utils.py:69 - Operation time for 'list_connections(...)' was 2100.3108ms
on 0: WARNING gefyra.local.bridge bridge.py:78 - Error getting GefyraBridges: (401)
      Reason: Unauthorized
      HTTP response headers: HTTPHeaderDict({'Audit-Id': '44c0432c-64c6-4686-97c5-04a7eee99218', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 19 Sep 2023 09:08:22 GMT', 'Content-Length': '129'})
      HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
on 0: WARNING gefyra.cli.connections connections.py:39 - Cannot detect it there are any Gefyra bridges running
on 0: DEBUG gefyra.cargo cargo.py:18 - Probing connection to 192.168.99.1 (attempt 0/2))
on 0: DEBUG gefyra.cargo cargo.py:18 - Probing connection to 192.168.99.1 (attempt 1/2))
on 0: DEBUG gefyra.api.utils utils.py:69 - Operation time for 'list_connections(...)' was 2108.0057ms
on 0: DEBUG gefyra.api.utils utils.py:69 - Operation time for 'list_containers(...)' was 2118.2415ms
on 0: INFO gefyra install.py:141 - Removing all Gefyra bridges
on 0: INFO gefyra install.py:146 - Removing remainder Gefyra clients
on 0: DEBUG gefyra install.py:150 - (401)
      Reason: Unauthorized
      HTTP response headers: HTTPHeaderDict({'Audit-Id': '68648953-9498-4242-be13-c5d63b96b41b', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 19 Sep 2023 09:08:24 GMT', 'Content-Length': '129'})
      HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
on 0: INFO gefyra install.py:151 - Removing Gefyra namespace
on 0: INFO gefyra install.py:156 - Removing Gefyra API extensions
on 0: INFO gefyra install.py:161 - Removing Gefyra RBAC resources
on 0: DEBUG gefyra.api.utils utils.py:69 - Operation time for 'uninstall(...)' was 169.4749ms
on 1: ERROR gefyra.local.clients clients.py:62 - A Kubernetes API Error occured. 
      Reason:Unauthorized 
      Body:{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
on 1: DEBUG gefyra.api.connect connect.py:270 - (401)
      Reason: Unauthorized
      HTTP response headers: HTTPHeaderDict({'Audit-Id': '3168b02c-552d-4755-90ff-df7c1a9e1b74', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 19 Sep 2023 09:08:25 GMT', 'Content-Length': '129'})
      HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
on 1: INFO gefyra.local.networking networking.py:92 - Removing Docker network gefyra-network-default
on 1: DEBUG gefyra.api.utils utils.py:69 - Operation time for 'remove_connection(...)' was 658.9367ms
Gefyra successfully removed |████████████████████████████████████████| 2/2 [100%] in 5.2s

but there are leftover resources in the gefyra namespace.

$ gefyra -d up 
on 0: DEBUG gefyra.api.status status.py:93 - Reading API resources from Kubernetes
on 0: DEBUG gefyra.api.status status.py:100 - Reading gefyra namespace
on 0: DEBUG gefyra.api.status status.py:109 - Checking operator deployment
on 0: DEBUG gefyra.api.status status.py:35 - Checking cargo container running
on 0: DEBUG gefyra.api.status status.py:49 - Checking gefyra network available
on 0: DEBUG gefyra.api.utils utils.py:69 - Operation time for 'status(...)' was 198.4815ms
on 0: DEBUG gefyra updown.py:34 - Preset None
on 0: DEBUG gefyra install.py:67 - Using options: GefyraInstallOptions(namespace='gefyra', version='2.0.0', service_type='NodePort', service_port=31820, service_labels={}, service_annotations={})
on 0: DEBUG gefyra install.py:83 - {'apiVersion': 'v1', 'kind': 'Namespace', 'metadata': {'name': 'gefyra'}}
on 0: DEBUG gefyra install.py:96 - Error from server (Conflict): {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"object is being deleted: namespaces \"gefyra\" already exists","reason":"AlreadyExists","details":{"name":"gefyra","kind":"namespaces"},"code":409}
on 0: DEBUG gefyra install.py:83 - {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'namespace': 'gefyra', 'name': 'gefyra-operator'}}
on 0: DEBUG gefyra install.py:96 - Error from server (Forbidden): {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"serviceaccounts \"gefyra-operator\" is forbidden: unable to create new content in namespace gefyra because it is being terminated","reason":"Forbidden","details":{"name":"gefyra-operator","kind":"serviceaccounts","causes":[{"reason":"NamespaceTerminating","message":"namespace gefyra is being terminated","field":"metadata.namespace"}]},"code":403}
on 0: ERROR gefyra install.py:98 - Error from server (Forbidden): {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"serviceaccounts \"gefyra-operator\" is forbidden: unable to create new content in namespace gefyra because it is being terminated","reason":"Forbidden","details":{"name":"gefyra-operator","kind":"serviceaccounts","causes":[{"reason":"NamespaceTerminating","message":"namespace gefyra is being terminated","field":"metadata.namespace"}]},"code":403}
Installing Gefyra to the cluster |⚠︎                                       | (!) 0/4 [0%] in 0.3s 
Error: Could not install Gefyra: Error from server (Forbidden): {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"serviceaccounts \"gefyra-operator\" is forbidden: unable to create new content in namespace gefyra because it is being terminated","reason":"Forbidden","details":{"name":"gefyra-operator","kind":"serviceaccounts","causes":[{"reason":"NamespaceTerminating","message":"namespace gefyra is being terminated","field":"metadata.namespace"}]},"code":403}

$ k describe gefyraclients.gefyra.dev -n gefyra
Name:         default
Namespace:    gefyra
Labels:       <none>
Annotations:  gefyra.dev/kopf-managed: yes
              gefyra.dev/last-handled-configuration:
                {"provider":"stowaway","providerConfig":{"Interface.Address":"192.168.99.2","Interface.DNS":"192.168.99.1","Interface.ListenPort":"51820",...
API Version:  gefyra.dev/v1
Kind:         gefyraclient
Metadata:
  Creation Timestamp:             2023-09-19T06:38:17Z
  Deletion Grace Period Seconds:  0
  Deletion Timestamp:             2023-09-19T09:01:44Z
  Finalizers:
    operator.gefyra.dev/kopf-finalizer
  Generation:  13
  Managed Fields:
    API Version:  gefyra.dev/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:provider:
      f:providerConfig:
        .:
        f:Interface.Address:
        f:Interface.DNS:
        f:Interface.ListenPort:
        f:Interface.PrivateKey:
        f:Peer.AllowedIPs:
        f:Peer.Endpoint:
        f:Peer.PresharedKey:
        f:Peer.PublicKey:
      f:providerParameter:
        .:
        f:subnet:
      f:serviceAccountData:
        .:
        f:ca.crt:
        f:namespace:
        f:token:
      f:serviceAccountName:
      f:state:
      f:stateTransitions:
        .:
        f:ACTIVE:
        f:CREATING:
        f:ENABLING:
        f:WAITING:
    Manager:      OpenAPI-Generator
    Operation:    Update
    Time:         2023-09-19T06:38:24Z
    API Version:  gefyra.dev/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:gefyra.dev/kopf-managed:
          f:gefyra.dev/last-handled-configuration:
        f:finalizers:
          .:
          v:"operator.gefyra.dev/kopf-finalizer":
      f:status:
        .:
        f:kopf:
          .:
          f:progress:
    Manager:         kopf
    Operation:       Update
    Time:            2023-09-19T06:38:24Z
  Resource Version:  310858710
  UID:               bfc0ff77-8038-4136-995d-922a4d0371f0
Provider:            stowaway
Provider Config:
  Interface.Address:     192.168.99.2
  Interface.DNS:         192.168.99.1
  Interface.ListenPort:  51820
  Interface.PrivateKey:  0EPUo1ZLBGw72OeLNfrNBpRyBuAQTu0Ctx37xDtG1EE=
  Peer.AllowedIPs:       0.0.0.0/0, ::/0
  Peer.Endpoint:         46.140.173.71:31820
  Peer.PresharedKey:     QE+SIjROm+v3A8uvHExNKkdj4HxeKDEXEuBMhHhtxxI=
  Peer.PublicKey:        0Pcd/mnYDACE000ByQEOD+D3Tw3QcDQU4FS9mvPe+wo=
Provider Parameter:
  Subnet:  172.18.0.0/16
Service Account Data:
  ca.crt:              LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeE1Ea3lNVEV6TXpJeU5Wb1hEVE14TURreE9URXpNekl5TlZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS2VsCm5jWGt1QThzcWRLQ2RVS2YzcDhYSmYyaUc1eHJ3SHg2NjlSaHdnQURyR3NabUNSSFFJakJ1UzViNE4zVmlJSHYKcDljV0M0SVFsUyt0cGlqbHdpNGJRR1RWUS9LOC9TTVhPbnBJMEFpZldydFVnMU5nYUR4YTlkQW1tNDhQTFZrVApid2lJZm5iTDNmNHNRbGF1YVF5Q3l6SmN0ZXRJcEtSUkE4MlRaWVZQY2pBbEptN3dJYzNZNXNxcDRLdllDWXprCkZveWh6WGk4N1hCK0JLbVhaYW5LUkdqZ2lLVXZZQlZFUjRuU0lpbTBnY2RUMElqL1pTWmhNMXVzOHFYZzN4b08KOE1YSWlMOFRwRWt2TFEvWGFqZEljU2xDa01rN0NBZEMwa3FFUk8xejNTaTEwcFo1Z0JLaXVzaC85dFdkcHZSSApwTDVzYkdIU2tXRVpKKzVsZFZjQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZMN3ROdTBGMTl3VWpFbmwxenZkYTJwY081czFNQTBHQ1NxR1NJYjMKRFFFQkN3VUFBNElCQVFCZVJMUy8wd2EwNURlbVNFeUE4Uk1KKzNVMWJwVGYrcEJ3VlQwNjZ0WlRwN1oyMFJiQwordExjMHdKbzYyQmMycGwrOVJvTUc2QjRucGZlRUVyb3F4R3ZnM1QxR25xKzd6K3N4M2dHYjcwb1VHTUxhSk1hCkt0VzBkVER1czNmdUNETmFIZmcyTXJ1ZnQyK1VHR2h0UTdoTXREeXVuT0Fid0FlUm1tMjhBWkVRTDROM0pPSy8KdFcrSVp6cmJhdFdVS0N2anZlZ1pzUHpNVGEzQkxHRHRYeUNnZ0J3M2huNjF4clRxcncrbmxuMktUQ09kTmlubwpoUzRDdXdVTHRYaHZicTFvVjUwQzFsaXVOS0RpOVNYeXlIMmNjbHF6OXZXb2dIb2ltTmtmc3R1eEI3ajJQd05QCkdGMzNjYTByRHFrUlh5TFA3bE80L3NLSHU0c0xMUWl2bmRiSwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
  Namespace:           Z2VmeXJh
  Token:               ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklsQmxlWFF5T1MxNWNXZzBOWEpEZFdJMVZrZGpUM28yVW5vNWRISm9lR2N4YzJ0bFRFbEhaRzlRUkdzaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUpuWldaNWNtRWlMQ0pyZFdKbGNtNWxkR1Z6TG1sdkwzTmxjblpwWTJWaFkyTnZkVzUwTDNObFkzSmxkQzV1WVcxbElqb2laMlZtZVhKaExXTnNhV1Z1ZEMxa1pXWmhkV3gwTFhSdmEyVnVJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpYSjJhV05sTFdGalkyOTFiblF1Ym1GdFpTSTZJbWRsWm5seVlTMWpiR2xsYm5RdFpHVm1ZWFZzZENJc0ltdDFZbVZ5Ym1WMFpYTXVhVzh2YzJWeWRtbGpaV0ZqWTI5MWJuUXZjMlZ5ZG1salpTMWhZMk52ZFc1MExuVnBaQ0k2SW1JeE1qQTRaR05tTFdaaU5EZ3ROR0ZoWmkwNE0yVTNMV1EyWm1Oak9HUmlNamcwWmlJc0luTjFZaUk2SW5ONWMzUmxiVHB6WlhKMmFXTmxZV05qYjNWdWREcG5aV1o1Y21FNloyVm1lWEpoTFdOc2FXVnVkQzFrWldaaGRXeDBJbjAuWnZDZG5Ia0J3R2kzWVBZcjBvcHFtZVNBTnpvcEJIc2ptMmlCZTUycVRaQ2dyQ3pnLU9ZMUpDbkxIazZ6Vkl0OEtETkNJWjE4ZjBVWlEwb01uWXN2Tl9DQXE2S0lDS0pJR1JQRHFtNGdpeEFYWFNubGtMN04wZlJzRTZGQ1djSE5NTTlfOElMcFJWdXU5ZFUwOFhhWTdzRnVaUXlHb3hEdlZ2YnlqZ1ZyWjJPaXVzUUN5cnRPOHBXSUliU3R5azk0S21jMlhnZ3JqZFE4YUlGaWoxZDJKX3lRY0ZMR2NlanFsUUVJMDBEcFl0c3lpMmdfMHNqeFdxSmMxOG9JVXNYOW4wRU9BdHUzYUxQYzV4cEdmUFB5S293ZzM1cnBfSmROVC1tNWN6TTJIbFB6cElvdDdBcy1GR25FN2hHdW1LYVR4dkFMeFlXVjdESS1CRlRzYmxvaUpB
Service Account Name:  gefyra-client-default
State:                 ACTIVE
State Transitions:
  ACTIVE:    2023-09-19T06:38:24.236967Z
  CREATING:  2023-09-19T06:38:19.890700Z
  ENABLING:  2023-09-19T06:38:23.831341Z
  WAITING:   2023-09-19T06:38:19.917812Z
Status:
  Kopf:
    Progress:
Events:  <none>

Forcefully deleting it didn't work: $ k delete gefyraclients.gefyra.dev default --force --grace-period=0

So I patch the finalizer: $ k describe gefyraclients.gefyra.dev -n gefyra

But still it still doesn't work, even after waiting 30+ min etc...

$ gefyra -d up
on 0: DEBUG gefyra.api.status status.py:93 - Reading API resources from Kubernetes
on 0: DEBUG gefyra.api.status status.py:100 - Reading gefyra namespace
on 0: DEBUG gefyra.api.status status.py:109 - Checking operator deployment
on 0: DEBUG gefyra.api.status status.py:126 - Checking Stowaway endpoint
on 0: DEBUG gefyra.api.status status.py:35 - Checking cargo container running
on 0: DEBUG gefyra.api.status status.py:49 - Checking gefyra network available
on 0: DEBUG gefyra.api.status status.py:63 - Probing wireguard connection
on 0: DEBUG gefyra.cargo cargo.py:18 - Probing connection to 192.168.99.1 (attempt 0/40))
[...]
on 0: DEBUG gefyra.cargo cargo.py:18 - Probing connection to 192.168.99.1 (attempt 39/40))
on 0: DEBUG gefyra.api.utils utils.py:69 - Operation time for 'status(...)' was 42318.9674ms
on 0: WARNING gefyra updown.py:31 - Gefyra is not installed, but operating properly. Aborting.
on 0: DEBUG gefyra.api.utils utils.py:69 - Operation time for 'get_client(...)' was 27.0364ms
on 2: DEBUG gefyra.api.utils utils.py:69 - Operation time for 'get_client(...)' was 24.2720ms
on 2: DEBUG gefyra.api.clients clients.py:111 - gefyra_server: 172.17.0.1:31820
on 2: DEBUG gefyra.api.utils utils.py:69 - Operation time for 'write_client_file(...)' was 35.2687ms
on 3: DEBUG gefyra updown.py:152 - Minikube profile None
on 3: DEBUG gefyra.cargo cargo.py:18 - Probing connection to 192.168.99.1 (attempt 0/2))
on 3: DEBUG gefyra.cargo cargo.py:18 - Probing connection to 192.168.99.1 (attempt 1/2))
on 3: DEBUG gefyra.api.utils utils.py:69 - Operation time for 'list_connections(...)' was 2118.7340ms
on 3: DEBUG gefyra.api.connect connect.py:48 - Restoring exinsting connection default
on 3: DEBUG gefyra.api.utils utils.py:69 - Operation time for 'get_client(...)' was 23.3051ms
on 3: INFO gefyra.local.networking networking.py:29 - Gefyra network already exists
on 3: DEBUG gefyra.local.networking networking.py:17 - Network {'Name': 'gefyra-network-default', 'Id': '26d8c1a4d3babc23e33bd17161b2843197f01a81e4e5a53409dfc019a8e4decc', 'Created': '2023-09-19T11:32:52.170833775+02:00', 'Scope': 'local', 'Driver': 'bridge', 'EnableIPv6': False, 'IPAM': {'Driver': 'default', 'Options': None, 'Config': [{'Subnet': '172.23.0.0/16'}]}, 'Internal': False, 'Attachable': False, 'Ingress': False, 'ConfigFrom': {'Network': ''}, 'ConfigOnly': False, 'Containers': {'5029f3d8b24aae33957c1bda056ca3e8e8a346d435a1d8c7fb1685153fbacffe': {'Name': 'gefyra-cargo-default', 'EndpointID': '0f18df111987524f24540c31a63d6005cc18c69c5e1e6c569a8db435e822fa50', 'MacAddress': '02:42:ac:17:00:95', 'IPv4Address': '172.23.0.149/16', 'IPv6Address': ''}}, 'Options': {'com.docker.network.driver.mtu': '1340'}, 'Labels': {'created_by.gefyra.dev': 'true'}}
on 3: DEBUG gefyra.types types.py:156 - Fetching object GefyraClient default
on 3: DEBUG gefyra.types types.py:156 - Fetching object GefyraClient default
on 3: DEBUG gefyra.types types.py:156 - Fetching object GefyraClient default
on 3: DEBUG gefyra.api.connect connect.py:136 - 172.17.0.1:31820
on 3: DEBUG gefyra.api.connect connect.py:202 - Checking wireguard connection
on 3: DEBUG gefyra.cargo cargo.py:18 - Probing connection to 192.168.99.1 (attempt 0/40))
[...]
on 3: DEBUG gefyra.cargo cargo.py:18 - Probing connection to 192.168.99.1 (attempt 39/40))
Connecting local network 'gefyra-network-default' to the cluster (up to 10 min) |██████████████████████████████⚠︎         | (!) 3/4 [75%] in 1:27.8 
Error: Gefyra could not successfully establish the connection to '172.17.0.1'.
If you have run 'gefyra up' with a remote cluster, a newly created route may not be working immediately.
Try running 'gefyra up' again after some time. Error: Gefyra could not successfully confirm the connection working.

$ cat ~/.kube/config
clusters:
- cluster:
    server: https://10.33.129.52:6443
	[...]

What did you expect to happen?

1. For pip install -U gefyra to work
2. For gefyra up to correct read my kube config

Please provide the output of gefyra check.

gefyra check doesn't exist anymore.

$ gefyra status
GefyraStatus(summary=<StatusSummary.INCOMPLETE: 'Gefyra is not running properly'>, cluster=GefyraClusterStatus(connected=True, operator=True, operator_image='quay.io/gefyra/operator:2.0.0', stowaway=True, stowaway_image='quay.io/gefyra/stowaway:2.0.0', namespace=True), client=GefyraClientStatus(version='2.0.0', cargo=True, cargo_image='quay.io/gefyra/cargo:2.0.0', network=True, connection=False, containers=0, bridges=0, kubeconfig='/home/xxx/.gefyra/default.yaml', context='default-context', cargo_endpoint='172.17.0.1:31820'))

How can we reproduce it (as minimally and precisely as possible)?

What Kubernetes setup are you working with?

$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:33:49Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.10", GitCommit:"e770bdbb87cccdc2daa790ecd69f40cf4df3cc9d", GitTreeState:"clean", BuildDate:"2023-05-17T14:06:35Z", GoVersion:"go1.19.9", Compiler:"gc", Platform:"linux/amd64"}

OS version

$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

$ uname -a
Linux l01sflalnxrds02 5.15.0-78-generic #85-Ubuntu SMP Fri Jul 7 15:25:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Anything else we need to know?

I think automatically detecting the kube config is a wrong assumption anyways, because we have multiple clusters running in different places. The kube config on this host is just not fully setup yet...

[SteinRobert]

Thanks for trying out Gefyra v2!

On the installation: pip install gefyra does not ship with the dependencies that are needed to use it as a cli. I open an issue in the docs to clear this up, it's not obvious from the current documentation.

gefyra up changed a lot under the hood. Obviously it has issues connecting to the Wireguard endpoint. I assume the cluster is running on your machine since you did not provide the --host parameter? What kind of k8s are you using then (minikube, k3d, colima...)? Did you follow any of the guides on local k8s setups (https://gefyra.dev/docs/local-environments/) ?

[sbor23]

@SteinRobert thanks for the quick reply and for checking for the pip install.

Regarding gefyra up: there is no more --host parameter:

$ gefyra up --help
Usage: gefyra up [OPTIONS]

  Install Gefyra on a cluster and directly connect to it

Options:
  --minikube TEXT  Connect Gefyra to a Minikube cluster (accepts minikube
                   profile name, default is 'minikube'))
  --preset TEXT    Set configs from a preset (available: aws,gke)
  --help           Show this message and exit.

Previously with v1 I used gefyra up --host 10.33.xxx and it worked. We don't use Colima, it's an on-premise deployment using kubespray mostly using default components and Calico for networking, CoreDNS for DNS. Maybe it's related to an old ticket I had: #214

[Schille]

@sbor23 we still have some homework on the documentation, I am sorry.

Is it a remote cluster? Please try the following with Gefyra 2

1. Install Gefyra to the cluster: gefyra install | kubectl -f - (https://gefyra.dev/docs/cli#install)
2. Create a GefyraClient for you: gefyra clients create --client-id sbor23 (https://gefyra.dev/docs/cli#clients-create)
3. Save your GefyraClient config: gefyra clients clients config sbor23 > sbor23.json (https://gefyra.dev/docs/cli#clients-config)
   In this step, you can override the automatically determined host parameter with --host=10.33.xxx
4. Connect to the cluster with: gefyra connections connect -f sbor23.json (https://gefyra.dev/docs/cli/#connections-connect)
   This automatically probes the connection (that is named default if --connection-name is unspecified).
   a) You are now able to stop this connection with gefyra connections disconnect [default]
   b) Reconnect with gefyra connections connect
5. You can remove Gefyra from your local machine with gefyra connections remove default
6. Remove Gefyra from the cluster with gefyra uninstall

You can repeat the steps 2/3/4 for every client/Dev that you want to onboard to your cluster.
I would appreciate it if you let us know if that works for you.

[tschale]

@sbor23 we still have some homework on the documentation, I am sorry.

Is it a remote cluster? Please try the following with Gefyra 2
[...]

You can repeat the steps 2/3/4 for every client/Dev that you want to onboard to your cluster. I would appreciate it if you let us know if that works for you.

This works for me on a remote cluster.

In this step, you can override the automatically determined host parameter with --host=[...] should've been need to override in my case :D

[sbor23]

Thanks for the pointers. Yes it's a remote cluster.

$ gefyra install | kubectl apply -f -
Error: KUBE_CONFIG_FILE /home/sbor23/.gefyra/default.yaml not found.
error: no objects passed to apply

After moving back the old ~/.gefyra folder:

$ gefyra install | kubectl apply -f -
Warning: resource namespaces/gefyra is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
namespace/gefyra configured
Warning: resource serviceaccounts/gefyra-operator is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
serviceaccount/gefyra-operator configured
Warning: resource clusterroles/gefyra:operator is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
clusterrole.rbac.authorization.k8s.io/gefyra:operator configured
Warning: resource clusterrolebindings/gefyra-operator is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
clusterrolebinding.rbac.authorization.k8s.io/gefyra-operator configured
Warning: resource services/gefyra-admission is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
service/gefyra-admission configured
Warning: resource validatingwebhookconfigurations/gefyra.dev is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
validatingwebhookconfiguration.admissionregistration.k8s.io/gefyra.dev configured
Warning: resource deployments/gefyra-operator is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
deployment.apps/gefyra-operator configured
Warning: resource deployments/gefyra-operator-webhook is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
deployment.apps/gefyra-operator-webhook configured
Warning: resource services/gefyra-stowaway-wireguard is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
service/gefyra-stowaway-wireguard configured

Recap:

1. ~/.gefyra folder is not properly initialized when using pip install
2. 1. should be gefyra install | kubectl apply -f -
3. needs the --host as @tschale noted, thanks! gefyra client config sbor23 -h 10.33.129.51 > sbor23.json

So far setting up the connection seems to work after a bit of back and forth

[sbor23]

After a couple of retries, recreating the deployment etc, we find that the containers cannot reach the services of other resources, although the DNS seems to work:

1. run gefyra run -d -i busybox -N busybox -n dataserver -c "sleep infinity"

2. docker exec -it busybox sh
   / # wget lab-dataserver-web:8000
   Connecting to lab-dataserver-web:8000 (10.233.11.109:8000)

3. no response

gefyra bridge doesn't change anything (but it shouldn't have an effect for outgoing requests anyways if I understand correctly).

It sounds like the same issue from #214

[SteinRobert]

@sbor23 seems like v2 was missing ip4 forwarding. This is now included in 2.0.1

[sbor23]

Can confirm it works now, thanks a lot!