From 94d1faa9c173be304a628e043364d54d5504e149 Mon Sep 17 00:00:00 2001 From: Ravind Kumar Date: Tue, 7 Feb 2023 11:17:24 -0500 Subject: [PATCH] Port-Forwarding cleanups (#722) As per https://github.com/minio/console/issues/2539 , the websocket behavior integrated as part of Console 0.22.1 (https://github.com/minio/console/pull/2419) seems to break port-forwarding behavior. There's no easy fix for this. NodePorts are a workaround, but slightly kludgy. Ingress is the better long-term solution, but requires more work. This is a stopgap: - For Operator, point users towards NodePorts if port-forwarding fails - For Tenant Console, simply drop port-forwarding entirely and point only at Ingress/LB Out of scope but in progress is Ingress guidance for Nginx and Traefik so we can close the loop on this. --- source/includes/common-installation.rst | 13 +-- ...common-install-operator-kubectl-plugin.rst | 20 +---- .../common-k8s-connect-operator-console.rst | 82 +++++++++++++++++++ source/includes/k8s/common-minio-kes.rst | 43 +++------- .../includes/k8s/steps-deploy-tenant-cli.rst | 66 ++++++++------- .../openshift/deploy-minio-on-openshift.rst | 15 +--- .../openshift/steps-deploy-minio-tenant.rst | 22 ----- .../deploy-minio-single-node-single-drive.rst | 2 +- .../deploy-minio-tenant.rst | 77 +---------------- .../upgrade-minio-operator.rst | 7 +- 10 files changed, 136 insertions(+), 211 deletions(-) create mode 100644 source/includes/common/common-k8s-connect-operator-console.rst diff --git a/source/includes/common-installation.rst b/source/includes/common-installation.rst index 7dfaad8c..06599d36 100644 --- a/source/includes/common-installation.rst +++ b/source/includes/common-installation.rst @@ -168,17 +168,8 @@ Console. .. start-local-jbod-single-node-desc MinIO strongly recommends direct-attached :abbr:`JBOD (Just a Bunch of Disks)` -arrays with XFS-formatted disks for best performance. - -- Direct-Attached Storage (DAS) has significant performance and consistency - advantages over networked storage (NAS, SAN, NFS). - -- Deployments using non-XFS filesystems (ext4, btrfs, zfs) tend to have - lower performance while exhibiting unexpected or undesired behavior. - -- RAID or similar technologies do not provide additional resilience or - availability benefits when used with distributed MinIO deployments, and - typically reduce system performance. +arrays with XFS-formatted disks for best performance. +Using any other type of backing storage (SAN/NAS, ext4, RAID, LVM) typically results in a reduction in performance, reliability, predictability, and consistency. Ensure all server drives for which you intend MinIO to use are of the same type (NVMe, SSD, or HDD) with identical capacity (e.g. ``12`` TB). MinIO does not distinguish drive types and does not benefit from mixed storage types. diff --git a/source/includes/common/common-install-operator-kubectl-plugin.rst b/source/includes/common/common-install-operator-kubectl-plugin.rst index c6154b72..b4814689 100644 --- a/source/includes/common/common-install-operator-kubectl-plugin.rst +++ b/source/includes/common/common-install-operator-kubectl-plugin.rst @@ -67,23 +67,7 @@ The output resembles the following: 4) Open the Operator Console ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Run the :mc:`kubectl minio proxy` command to temporarily forward traffic from -the :ref:`MinIO Operator Console ` service to your -local machine: +.. include:: /includes/common/common-k8s-connect-operator-console.rst -.. code-block:: shell - :class: copyable - - kubectl minio proxy - -The command output includes a JWT token you must use to log into the -Operator Console. - -.. image:: /images/k8s/operator-dashboard.png - :align: center - :width: 70% - :class: no-scaled-link - :alt: MinIO Operator Console + -You can deploy a new :ref:`MinIO Tenant ` from -the Operator Dashboard. diff --git a/source/includes/common/common-k8s-connect-operator-console.rst b/source/includes/common/common-k8s-connect-operator-console.rst new file mode 100644 index 00000000..9a37004d --- /dev/null +++ b/source/includes/common/common-k8s-connect-operator-console.rst @@ -0,0 +1,82 @@ +.. tab-set:: + + .. tab-item:: Port Forwarding + + .. note:: + + Starting with Operator 4.5.8, the MinIO Operator Console implements websockets for improved performance. + This protocol may exhibit unpredictable behavior with Kubernetes port-forwarding, especially during the login sequence. + + Users who encounter timeouts when using port forwarding should select from the remaining tabs to view alternatives for connecting to the Operator Console. + + Run the :mc:`kubectl minio proxy` command to temporarily forward traffic from + the :ref:`MinIO Operator Console ` service to your + local machine: + + .. cond:: k8s and not openshift + + .. code-block:: shell + :class: copyable + + kubectl minio proxy + + .. cond:: openshift + + .. code-block:: shell + :class: copyable + + oc minio proxy + + The command output includes a JWT token you must use to log into the + Operator Console. + + .. image:: /images/k8s/operator-dashboard.png + :align: center + :width: 70% + :class: no-scaled-link + :alt: MinIO Operator Console + + You can deploy a new :ref:`MinIO Tenant ` from + the Operator Dashboard. + + .. tab-item:: Node Ports + + Use the following command to identify the :kube-docs:`NodePorts ` configured for the Operator Console. + If your local host does not have the ``jq`` utility installed, you can run the first command and locate the ``spec.ports`` section of the output. + + .. code-block:: shell + :class: copyable + + kubectl get svc/console -n minio-operator -o json | jq -r '.spec.ports' + + The output resembles the following: + + .. code-block:: json + + [ + { + "name": "http", + "nodePort": 31055, + "port": 9090, + "protocol": "TCP", + "targetPort": 9090 + }, + { + "name": "https", + "nodePort": 31388, + "port": 9443, + "protocol": "TCP", + "targetPort": 9443 + } + ] + + Use the ``http`` or ``https`` port depending on whether you deployed the Operator with Console TLS enabled via :mc-cmd:`kubectl minio init --console-tls`. + + Append the ``nodePort`` value to the externally-accessible IP address of a worker node in your Kubernetes cluster. + + Use the following command to retrieve the JWT token necessary for logging into the Operator Console: + + .. code-block:: shell + :class: copyable + + kubectl get secret/console-sa-secret -n minio-operator -o json | jq -r '.data.token' | base64 -d \ No newline at end of file diff --git a/source/includes/k8s/common-minio-kes.rst b/source/includes/k8s/common-minio-kes.rst index acdcb1e6..a90b8896 100644 --- a/source/includes/k8s/common-minio-kes.rst +++ b/source/includes/k8s/common-minio-kes.rst @@ -19,17 +19,9 @@ You can use either the MinIO Tenant Console or the MinIO :mc:`mc` CLI to enable .. tab-item:: MinIO Tenant Console - You can manually :ref:`port forward ` the MinIO Tenant Console service to your local host machine for simplified access: - - .. code-block:: shell - :class: copyable - - # Replace 'minio-tenant' with the name of the MinIO Tenant - # Replace '-n minio' with the namespace of the MinIO Tenant - - kubectl port-forward svc/minio-tenant-console 9443:9443 -n minio - - Open the MinIO Console by navigating to http://127.0.0.1:9443 in your preferred browser and logging in with the root credentials for the deployment. + Connect to the :ref:`MinIO Tenant Console service ` and log in. + For clients internal to the Kubernetes cluster, you can specify the :kube-docs:`service DNS name `. + For clients external to the Kubernetes cluster, specify the hostname of the service exposed by Ingress, Load Balancer, or similar Kubernetes network control component. Once logged in, create a new Bucket and name it to your preference. Select the Gear :octicon:`gear` icon to open the management view. @@ -44,35 +36,19 @@ You can use either the MinIO Tenant Console or the MinIO :mc:`mc` CLI to enable .. tab-item:: MinIO CLI - You can manually :ref:`port forward ` the ``minio`` service for temporary access via the local host. - - Run this command in a separate Terminal or Shell: + Use the :ref:`MinIO API Service ` to create a new :ref:`alias ` for the MinIO deployment. + You can then use the :mc:`mc encrypt set` command to enable SSE-KMS encryption for a bucket: .. code-block:: shell :class: copyable - # Replace '-n minio' with the namespace of the MinIO deployment - # If you deployed the Tenant without TLS you may need to change the port range - - # You can validate the ports in use by running - # kubectl get svc/minio -n minio - - kubectl port forward svc/minio 443:443 -n minio - - The following commands in a new Terminal or Shell window: - - - Create a new :ref:`alias ` for the MinIO deployment - - Create a new bucket for storing encrypted data - - Enable SSE-KMS encryption on that bucket - - .. code-block:: shell - :class: copyable - - mc alias set k8s https://127.0.0.1:443 ROOTUSER ROOTPASSWORD + mc alias set k8s https://minio.minio-tenant-1.svc.cluster-domain.example:443 ROOTUSER ROOTPASSWORD mc mb k8s/encryptedbucket mc encrypt set SSE-KMS encrypted-bucket-key k8s/encryptedbucket + For clients external to the Kubernetes cluster, specify the hostname of the service exposed by Ingress, Load Balancer, or similar Kubernetes network control component. + Write a file to the bucket using :mc:`mc cp` or any S3-compatible SDK with a ``PutObject`` function. You can then run :mc:`mc stat` on the file to confirm the associated encryption metadata. @@ -90,7 +66,8 @@ MinIO requires that the |EK| for a given bucket or object exist on the root KMS You can use the :mc-cmd:`mc admin kms key create` command against the MinIO Tenant. You must ensure your local host can access the MinIO Tenant pods and services before using :mc:`mc` to manage the Tenant. -You can manually :ref:`port forward ` the ``minio`` service for temporary access via the local host. +For hosts internal to the Kubernetes cluster, you can use the :kube-docs:`service DNS name `. +For hosts external to the Kubernetes cluster, specify the hostname of the service exposed by Ingress, Load Balancer, or similar Kubernetes network control component. Run this command in a separate Terminal or Shell: diff --git a/source/includes/k8s/steps-deploy-tenant-cli.rst b/source/includes/k8s/steps-deploy-tenant-cli.rst index 1e95cf8c..1d5f60a5 100644 --- a/source/includes/k8s/steps-deploy-tenant-cli.rst +++ b/source/includes/k8s/steps-deploy-tenant-cli.rst @@ -18,8 +18,6 @@ To deploy a tenant from the command line, complete the following steps: :ref:`create-tenant-cli-access-tenant-console` -:ref:`create-tenant-cli-forward-ports` - .. _create-tenant-cli-determine-settings-required-options: 1) Determine Values for Required Settings @@ -234,47 +232,55 @@ In addition to access credentials, the output shows the service name and service 5) Access the Tenant's MinIO Console ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -To access the :ref:`MinIO Console ` for the tenant, forward the tenant's port. +The MinIO Operator creates services for the MinIO Tenant. -- If necessary, run ``kubectl get svc -n `` to retrieve the tenant's port number. -- Run the following to forward the tenant's port and access it from a browser: +.. cond:: openshift - .. code-block:: shell - :class: copyable + Use the ``oc get svc -n TENANT-PROJECT`` command to review the deployed services: - kubectl port-forward svc/-console -n : + .. code-block:: shell + :class: copyable - - Replace ```` with the name of your tenant. - - Replace ```` with the namespace the tenant exists in. - - Replace ```` with the port number to use on your local machine to access the tenant's MinIO Console. - - Replace ```` with the port number the MinIO Operator assigned to the tenant. + oc get svc -n minio-tenant-1 -- Go to ``https://127.0.0.1:`` to Access the tenant's MinIO Console. +.. cond:: k8s and not openshift - Replace ```` with the port number you used when forwarding the tenant's port. + Use the ``kubectl get svc -n NAMESPACE`` command to review the deployed services: -- Login with the username and password shown in the tenant creation output and recorded in step 4 above. + .. code-block:: shell + :class: copyable -.. _create-tenant-cli-forward-ports: + kubectl get svc -n minio-tenant-1 -6) Forward Ports -~~~~~~~~~~~~~~~~ +.. code-block:: shell -You can temporarily expose each service using the ``kubectl port-forward`` utility. -Run the following examples to forward traffic from the local host running ``kubectl`` to the services running inside the Kubernetes cluster. + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + minio LoadBalancer 10.97.114.60 443:30979/TCP 2d3h + minio-tenant-1-console LoadBalancer 10.106.103.247 9443:32095/TCP 2d3h + minio-tenant-1-hl ClusterIP None 9000/TCP 2d3h + minio-tenant-1-log-hl-svc ClusterIP None 5432/TCP 2d3h + minio-tenant-1-log-search-api ClusterIP 10.103.5.235 8080/TCP 2d3h + minio-tenant-1-prometheus-hl-svc ClusterIP None 9090/TCP 7h39m -.. tab-set:: +- The ``minio`` service corresponds to the MinIO Tenant service. + Applications should use this service for performing operations against the MinIO Tenant. + +- The ``*-console`` service corresponds to the :minio-git:`MinIO Console `. + Administrators should use this service for accessing the MinIO Console and performing administrative operations on the MinIO Tenant. - .. tab-item:: MinIO Tenant +The remaining services support Tenant operations and are not intended for consumption by users or administrators. + +By default each service is visible only within the Kubernetes cluster. +Applications deployed inside the cluster can access the services using the ``CLUSTER-IP``. - .. code-block:: shell - :class: copyable +Applications external to the Kubernetes cluster can access the services using the ``EXTERNAL-IP``. +This value is only populated for Kubernetes clusters configured for Ingress or a similar network access service. +Kubernetes provides multiple options for configuring external access to services. - kubectl port-forward service/minio 443:443 +.. cond:: k8s and not openshift - .. tab-item:: MinIO Console - - .. code-block:: shell - :class: copyable + See the Kubernetes documentation on :kube-docs:`Publishing Services (ServiceTypes) ` and :kube-docs:`Ingress ` for more complete information on configuring external access to services. + +.. cond:: openshift - kubectl port-forward service/minio-tenant-1-console 9443:9443 \ No newline at end of file + See the OpenShift documentation on :openshift-docs:`Route or Ingress ` for more complete information on configuring external access to services. \ No newline at end of file diff --git a/source/includes/openshift/deploy-minio-on-openshift.rst b/source/includes/openshift/deploy-minio-on-openshift.rst index 78452c78..4b793d02 100644 --- a/source/includes/openshift/deploy-minio-on-openshift.rst +++ b/source/includes/openshift/deploy-minio-on-openshift.rst @@ -153,20 +153,7 @@ You can find the MinIO Operator Interface from the :guilabel:`Operators` left-ha The MinIO Operator includes the Operator Console, a browser-based management interface for managed MinIO tenants. -You can temporarily port-forward the Operator Console to your local host machine using the ``oc minio port-forward`` command: - -.. code-block:: shell - :class: copyable - - oc minio port-forward - -The command returns a JWT token and a URL you can open in your browser. - -.. image:: /images/k8s/operator-dashboard.png - :align: center - :width: 70% - :class: no-scaled-link - :alt: Operator Dashboard +.. include:: /includes/common/common-k8s-connect-operator-console.rst You can create a permanent routing rule by creating a :openshift-docs:`Route or Ingress ` to allow access from external clients, such as your local computer browser. diff --git a/source/includes/openshift/steps-deploy-minio-tenant.rst b/source/includes/openshift/steps-deploy-minio-tenant.rst index 1ce1dcf1..6aa2bf77 100644 --- a/source/includes/openshift/steps-deploy-minio-tenant.rst +++ b/source/includes/openshift/steps-deploy-minio-tenant.rst @@ -93,25 +93,3 @@ See the Kubernetes documentation on :kube-docs:`Publishing Services (ServiceTypes) ` and :kube-docs:`Ingress ` for more complete information on configuring external access to services. - -4) Forward Ports -~~~~~~~~~~~~~~~~ - -You can temporarily expose each service using the ``oc port-forward`` utility. -Run the following examples to forward traffic from the local host running ``oc`` to the services running inside the Kubernetes cluster. - -.. tab-set:: - - .. tab-item:: MinIO Tenant - - .. code-block:: shell - :class: copyable - - oc port-forward service/minio 443:443 - - .. tab-item:: MinIO Console - - .. code-block:: shell - :class: copyable - - oc port-forward service/minio-tenant-1-console 9443:9443 diff --git a/source/operations/install-deploy-manage/deploy-minio-single-node-single-drive.rst b/source/operations/install-deploy-manage/deploy-minio-single-node-single-drive.rst index 93d851d4..45947bad 100644 --- a/source/operations/install-deploy-manage/deploy-minio-single-node-single-drive.rst +++ b/source/operations/install-deploy-manage/deploy-minio-single-node-single-drive.rst @@ -77,7 +77,7 @@ The following procedure deploys MinIO consisting of a single MinIO server and a :class: note MinIO's strict **read-after-write** and **list-after-write** consistency - model requires local drive filesystems (``xfs``, ``ext4``, etc.). + model requires local drive filesystems. MinIO cannot provide consistency guarantees if the underlying storage volumes are NFS or a similar network-attached storage volume. diff --git a/source/operations/install-deploy-manage/deploy-minio-tenant.rst b/source/operations/install-deploy-manage/deploy-minio-tenant.rst index 5c977e4b..e9856417 100644 --- a/source/operations/install-deploy-manage/deploy-minio-tenant.rst +++ b/source/operations/install-deploy-manage/deploy-minio-tenant.rst @@ -223,40 +223,12 @@ To deploy a tenant from the MinIO Operator Console, complete the following steps :ref:`create-tenant-connect-tenant` -:ref:`create-tenant-operator-forward-ports` - .. _create-tenant-access-minio-operator-console: 1) Access the MinIO Operator Console ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. cond:: k8s and not openshift - - Use the :mc-cmd:`kubectl minio proxy` command to temporarily forward traffic between the local host machine and the MinIO Operator Console: - - .. code-block:: shell - :class: copyable - - kubectl minio proxy - -.. cond:: openshift - - Use the :mc-cmd:`oc minio proxy ` command to temporarily forward traffic between the local host machine and the MinIO Operator Console: - - .. code-block:: shell - :class: copyable - - oc minio proxy - -The command returns output similar to the following: - -.. code-block:: shell - - Starting port forward of the Console UI. - - To connect open a browser and go to http://localhost:9090 - - Current JWT to login: TOKEN +.. include:: /includes/common/common-k8s-connect-operator-console.rst Open your browser to the specified URL and enter the JWT Token into the login page. You should see the :guilabel:`Tenants` page: @@ -763,53 +735,6 @@ Kubernetes provides multiple options for configuring external access to services See the OpenShift documentation on :openshift-docs:`Route or Ingress ` for more complete information on configuring external access to services. -.. _create-tenant-operator-forward-ports: - -13) Forward Ports -~~~~~~~~~~~~~~~~~ - -.. cond:: k8s and not openshift - - You can temporarily expose each service using the ``kubectl port-forward`` utility. - Run the following examples to forward traffic from the local host running ``kubectl`` to the services running inside the Kubernetes cluster. - - .. tab-set:: - - .. tab-item:: MinIO Tenant - - .. code-block:: shell - :class: copyable - - kubectl port-forward service/minio 443:443 - - .. tab-item:: MinIO Console - - .. code-block:: shell - :class: copyable - - kubectl port-forward service/minio-tenant-1-console 9443:9443 - -.. cond:: openshift - - You can temporarily expose each service using the ``oc port-forward`` utility. - Run the following examples to forward traffic from the local host running ``oc`` to the services running inside the Kubernetes cluster. - - .. tab-set:: - - .. tab-item:: MinIO Tenant - - .. code-block:: shell - :class: copyable - - oc port-forward service/minio 443:443 - - .. tab-item:: MinIO Console - - .. code-block:: shell - :class: copyable - - oc port-forward service/minio-tenant-1-console 9443:9443 - .. cond:: openshift .. include:: /includes/openshift/steps-deploy-minio-tenant.rst diff --git a/source/operations/install-deploy-manage/upgrade-minio-operator.rst b/source/operations/install-deploy-manage/upgrade-minio-operator.rst index 26320d23..b58a9b50 100644 --- a/source/operations/install-deploy-manage/upgrade-minio-operator.rst +++ b/source/operations/install-deploy-manage/upgrade-minio-operator.rst @@ -108,12 +108,7 @@ This procedure upgrades the MinIO Operator from any 4.2.3 or later release to |o You can check the Operator version by reviewing the object specification for an Operator Pod using a previous step. - Use :mc-cmd:`kubectl minio proxy` to access the Operator Console and confirm the status of managed Tenants: - - .. code-block:: shell - :class: copyable - - kubectl minio proxy + .. include:: /includes/common/common-k8s-connect-operator-console.rst .. _minio-k8s-upgrade-minio-operator-4.2.2-procedure: