minio web console auth issues when port-forwarding a local minikube k8s minio instance

[jinyius]

Steps to Reproduce (for bugs)

0. install minikube and minio inside of minikube
1. kubectl port-forward pod/myminio-blah-blah 9001
2. use browser to hit localhost:9001
3. try to authenticate and use the web console

Expected Behavior

web console loads fine and is usable

Current Behavior

web console loads, but acts oddly due to port-forwarding issues

• when the initial auth screen loads, the cli w/ the port-forward shows these errors:

Forwarding from 127.0.0.1:9001 -> 9001                                                                                                                                                                                                                                                                                         
Forwarding from [::1]:9001 -> 9001                                                                                                                                                                                                                                                                                               
Handling connection for 9001                                                                                                                                                                                                                                                                                                     
# ...                                                                                                                                                                                                                                                                                     
Handling connection for 9001                                                                                                                                                                                                                                                                                                     
E1229 14:24:40.046628   69982 portforward.go:379] error copying from remote stream to local connection: readfrom tcp6 [::1]:9001->[::1]:58450: write tcp6 [::1]:9001->[::1]:58450: write: broken pipe                                                                                                                            
Handling connection for 9001 

• later on, there are Timed out errors:

Connection to localhost port 9001 [tcp/etlservicemgr] succeeded!
Handling connection for 9001                                                                                                                                                                                                                                                                                                     
Connection to localhost port 9001 [tcp/etlservicemgr] succeeded!
E1229 14:23:34.674227   69851 portforward.go:345] error creating error stream for port 9001 -> 9001: Timeout occurred                                           
Handling connection for 9001                                                                                                                                                                                                                                                                                                     
E1229 14:23:34.710293   69851 portforward.go:345] error creating error stream for port 9001 -> 9001: Timeout occurred                                           
Handling connection for 9001                                                    
E1229 14:23:36.936280   69851 portforward.go:345] error creating error stream for port 9001 -> 9001: Timeout occurred                                                                                                                                                                                                            
Handling connection for 9001                                                                                                                                                                                                                                                                                                     
Connection to localhost port 9001 [tcp/etlservicemgr] succeeded!
E1229 14:23:38.616971   69851 portforward.go:345] error creating error stream for port 9001 -> 9001: Timeout occurred                                           
Handling connection for 9001 
Handling connection for 9001                                                                                                                                                                                                                                                                                                     
Handling connection for 9001                                                                                                                                                                                                                                                                                                     
E1229 14:23:41.950806   69851 portforward.go:345] error creating error stream for port 9001 -> 9001: Timeout occurred                                           
Handling connection for 9001                                                    
Connection to localhost port 9001 [tcp/etlservicemgr] succeeded!
E1229 14:23:46.966488   69851 portforward.go:345] error creating error stream for port 9001 -> 9001: Timeout occurred                                           
Handling connection for 900

• the behavior in the web ui is odd. authentication never succeeds.

Possible Solution

my coworker and i believe there's something going on w/ the minio web ui's around the login screen and authentication that causes the port-forwarding to break. we've gotten around this by brute-forcing the port forward to restart explicitly and manually during authentication. eventually, with enough port-forward restarts, it succeeds. once the browser session is authenticated, there are no port-forwarding issues.

workaround

1. while true; do kubectl port-forward pod/myminio-blah-blah 9001; done
2. open browser to localhost:9001
3. whenever there are timeouts or broken pipes in the cli w/ the port-forward, CTRL-C to restart the port-forward
4. interact w/ the web ui to login (revisiting 4) until fully authenticated and loaded
5. once loaded, port-forwarding is not brittle

Context

we use minio locally as a fake s3 for local flink development. we use eks in prod for flink, so we use minikube for a local k8s env and so we try to keep all flink development parts in the local minikube setup

Regression

yes. port-forwarding worked fine using the super-old helm chart: https://github.com/minio/charts

we noticed that this version doesn't have a split console and api port. it just has port 9000 that suffices for both endpoints. we had to version bump b/c this old version of minio has bugs that prevents proper s3 expectations.

Your Environment

• Version used (minio --version):

bash-4.4$ minio --version
minio version RELEASE.2022-12-12T19-27-27Z (commit-id=a469e6768df4d5d2cb340749fa58e4721a7dee96)
Runtime: go1.19.4 linux/amd64
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Copyright: 2015-2022 MinIO, Inc.

• Server setup and configuration:
  we tried using replicated and standalone setups for our minio install

• Operating System and version (uname -a):

Darwin myhost.mylan.whatever 22.2.0 Darwin Kernel Version 22.2.0: Fri Nov 11 02:08:47 PST 2022; root:xnu-8792.61.2~4/RELEASE_X86_64 x86_64

NOTE: we replicated this using both an m1 and intel mac. the m1 mac uses minikube over docker and the intel mac uses minikube over hyperkit.

• helm values overrides:

persistence:
  size: 10Gi

resources:
  requests:
    memory: 256Mi

buckets:
  - name: promoted-event-logs
    versioning: false
    purge: true

mode: standalone

replicas: 1

environment:
  MINIO_API_SELECT_PARQUET: "on"

[jinyius]

more context:

port-forwarding various kafka, redis and flink ports from the same minikube all work fine w/o issue in our setup. it's just this latest minio console. moreover, forwarding minio's api port (9000) and using mc locally in the host context against the api port *9000) works fine w/o any forward breaks.

[harshavardhana]

Forwarding won't work as MinIO Console UI requires websockets.

Moving this to relevant repo for more guidance..

[jinyius]

interesting. did the older version not use websockets?

[harshavardhana]

interesting. did the older version not use websockets?

yes, there have been some changes to the bucket browser that started using WebSockets for more efficient (responsive) UI.

So I don't know if port-forwarding handles these scenarios properly, as it's not meant to be used for production.

[jinyius]

it seems there's only issues when loading the login page based on more testing. for my setup, i've refined the workaround to just need one restart:

1. do the while true loop for the port forward in a cli session
2. just load up the login window (see the errors in the cli session)
3. back in the cli session, ctrl-c the port-forward to restart it
4. back in the browser, actually perform the login, ideally with one click (minimize interactions)

[jinyius]

So I don't know if port-forwarding handles these scenarios properly, as it's not meant to be used for production.

i agree that port-forwarding isn't for production use. the use case minio satisfies for us right now is stubbing s3 for local development which hopefully is a valid minio use case.

[clovis-dugue]

I also have websocket-related issues, on a console deployed via Minio Operator and Tenant

I see the following error in my pod: error upgrade connection:websocket: response does not implement http.Hijacker

While I understand port-forwarding and websockets together may not be meant for production use, I have done nothing but follow the documentation, which is essentially "pop up an operator, use the krew plugin to proxy your traffic, create a tenant and enjoy". The documentation should probably be updated to reflect the fact that issues should be expected when using port-forwarding

[ravindk89]

@clovis-dugue that's a fair ask. I missed this discussion as I typically monitor minio/minio and minio/operator, but forgot minio/console.

@harshavardhana am I right in thinking that moving forward, we'll need to use ingress to get this to work? that is, kubectl minio proxy likely wont work either as Operator Console is Console under the hood?

[ravindk89]

I'll note that this might be an issue related mostly to containerized K8s setups - asking internally, we haven't seen it come up yet in fully-deployed upstream Kubernetes. If that's the case, we might need to add a warning in about this behavior and the workaround. Documenting nginx/traefik ingress will take a little longer.

[pasztorl]

I've tested it with kubeadm/kubernetes on aws using aws nlb->haproxy-ingress->minio-operator->minio-console and websocket based communication not worked and got the "error upgrade connection.." message in the console log

[bkcsfi]

@ravindk89 I just encountered this issue on vanilla k8s on vSphere (capi based setup, not containerized).

I think an ingress will be needed to work around this issue.

[ravindk89]

I verified the issue on K3s as well, unfortunately.

We're going to update our docs with some guidance this week, mostly to drop port-forwarding as an option and recommend ingress.

Documenting ingress will take longer - we have resources for Nginx, but not Traefik right now. We appreciate everyone's patience while we push forward.

[ravindk89]

As a simple workaround, We deploy the Operator Console and Tenant Consoles w/ nodeports. You can use these to access the services directly.

For example:

kubectl get svc/console -n minio-operator -o json | jq -r '.spec.ports'                                                                                   ✔  default ⎈ 
[
  {
    "name": "http",
    "nodePort": 31055,
    "port": 9090,
    "protocol": "TCP",
    "targetPort": 9090
  },
  {
    "name": "https",
    "nodePort": 31388,
    "port": 9443,
    "protocol": "TCP",
    "targetPort": 9443
  }
]

kubectl get svc/miniodc-console -n miniodc -o json | jq -r '.spec.ports'                                                                           PIPE|3 ✘  default ⎈ 
[
  {
    "name": "https-console",
    "nodePort": 30496,
    "port": 9443,
    "protocol": "TCP",
    "targetPort": 9443
  }
]

In both cases, you can connect to a worker node via the nodePorts and access the underlying service. I'm not sure what node actually matters. I was able to access it regardless, so this is likely what the docs will reference in the short term.

w.r.t. the Operator login, you can retrieve the JWT by reading the console-sa-secret

kubectl get secret/console-sa-secret -n minio-operator -o json | jq -r '.data.token' | base64 -d

[dvaldivia]

the root cause is the net/http package doing something when we start serving a video asset for the background, I'm still looking into this

[dvaldivia]

This seems to have to do with kubectl port-forward alone when serving a video asset, strangely if I stream files from the object browser it works, https://github.com/dvaldivia/gofileserve has a way to replicate this issue in isolation

[harshavardhana]

Looks like an upstream issue at this point

kubernetes/kubectl#1368

[kannappanr]

@jinyius once you enable port forwarding, can you try this on another terminal?

while true ; do nc -vz 127.0.0.1 9001 ; sleep 10 ; done

I tried this on kind. Not sure if this will work in minikube

[Darrekt]

Not directly addressing the issue, but a good alternative workaround to the port forwarding loop it is to simply expose the console deployment as a NodePort instead.

[mrwormhole]

do you know which version is safe to use? I observed the same at the weekend in k3s @jinyius

[stepanovdg]

Have the same issue on k3s cluster with ingress.
First time trying minio, just installed operator ans was sure I have something with ca-secret.
But it appears issue related to with proxy.

In case of ingress:

Error is misleading.
in browser ui red banner:
Post "http://localhost:9000/": dial tcp [::1]:9000: connect: connection refused
in browser console:
post https://{fqdn}/api/v1/login 401
in console logs :

ErrorWithContext:Get "http://localhost:9000/minio/admin/v3/accountinfo": dial tcp [::1]:9000: connect: connection refused
%!(EXTRA *errors.errorString=invalid session)
ErrorWithContext:Post "http://localhost:9000/": dial tcp [::1]:9000: connect: connection refused
%!(EXTRA *errors.errorString=invalid Login)

If using proxy either kubectl minio proxy or usual kubectl proxy -> page just hangs and timeouts - no red banner. In console logs errors are the same.

[jarlefosen]

I downgraded to Helm Chart version 4.1.0 and it seems to have resolve the connection issue for me.
Using Macbook M1 Pro with k3d running in standalone mode.

I struggled to connect with version 5.0.10 and 5.0.5. Got through the login page by re-starting port-forwarding after login screen was fully loaded and before I hit the login button as a workaround. But I chose to downgrade in the end to avoid the problems all together.

Helm values with chart version 4.1.0

mode: standalone
replicas: 1
persistence:
  enabled: true
  size: 5Gi
resources:
  requests:
    memory: 200Mi
users:
- accessKey: testuser
  secretKey: supersecret
  policy: consoleAdmin

[mrwormhole]

@jarlefosen I couldn't find helm chart 4.1.0 because latest seem to be 4.0.15 https://github.com/minio/minio/blob/RELEASE.2022-10-24T18-35-07Z/helm/minio/Chart.yaml

however, changing image to

image:
  repository: quay.io/minio/minio
  tag: RELEASE.2022-09-17T00-09-45Z
  pullPolicy: IfNotPresent

solved my issues with port forwarding so thank you very much, clearly something is wrong after this date, this websockets issue is gonna bite more people due to software supply chain issue of gorilla/websockets being no longer maintained

[dnskr]

The issue has been resolved (see #2799 and minio/minio#17123) and released in RELEASE.2023-05-04T21-44-30Z.
Set MINIO_BROWSER_LOGIN_ANIMATION env variable value to off, i.e. the following in the chart:

environment:
  MINIO_BROWSER_LOGIN_ANIMATION: off

[ValentinVoigt]

I'm really glad I'm already subscribed to this issue, because I would never make a connection between a broken login page due to network errors and disabling an animation. 😂 Thanks for looking into it!

[leeren]

Even adding that environment variable to my deployment, I still encounter connection issues. Are we sure that was the culprit?

[cesnietor]

Closing since this would be solved on kubernetes/kubectl#1368. Waiting for resolution of it. We provided a workaround which is disabling the animation via an environment variable.

[callingmedic911]

For some reason off was interpreted as false when using as is from the suggested workaround. YAML being YAML I guess. Just wrapped in quotes to fix this:

environment:
  MINIO_BROWSER_LOGIN_ANIMATION: 'off'

[kakaru1331]

@callingmedic911 Holy 👍