From 96d50a955df3561f73dca4604748cb8cf41e990a Mon Sep 17 00:00:00 2001 From: Lenny Chen <55669665+lennessyy@users.noreply.github.com> Date: Fri, 13 Sep 2024 11:26:38 -0700 Subject: [PATCH] docs: add new vsphere known issue (#3896) * docs: first draft * Apply suggestions from code review Co-authored-by: Karl Cardenas <29551334+karl-cardenas-coding@users.noreply.github.com> * docs: add clarifying details * docs: move old content to deprecated * docs: clarify known issue not automatically resolved * docs: add warning to more obvious places * docs: add link to warning and xlinks * Apply suggestions from code review Co-authored-by: Karl Cardenas <29551334+karl-cardenas-coding@users.noreply.github.com> * ci: auto-formatting prettier issues --------- Co-authored-by: Lenny Chen Co-authored-by: Karl Cardenas <29551334+karl-cardenas-coding@users.noreply.github.com> Co-authored-by: lennessyy (cherry picked from commit b46a973cd05ddbfabf9061c2e9867984aa27cc74) --- .../troubleshooting/enterprise-install.md | 126 +++++++++++++++++ .../upgrade/upgrade-notes.md | 9 ++ .../upgrade/upgrade-vmware/airgap.md | 9 +- .../upgrade/upgrade-vmware/non-airgap.md | 10 +- .../release-notes/known-issues.md | 3 +- .../troubleshooting/enterprise-install.md | 79 +++++++++++ .../troubleshooting/palette-upgrade.md | 128 ------------------ .../vertex/upgrade/upgrade-notes.md | 7 + .../vertex/upgrade/upgrade-vmware/airgap.md | 10 +- .../upgrade/upgrade-vmware/non-airgap.md | 9 +- 10 files changed, 247 insertions(+), 143 deletions(-) create mode 100644 docs/deprecated/troubleshooting/enterprise-install.md diff --git a/docs/deprecated/troubleshooting/enterprise-install.md b/docs/deprecated/troubleshooting/enterprise-install.md new file mode 100644 index 0000000000..c98c7354b2 --- /dev/null +++ b/docs/deprecated/troubleshooting/enterprise-install.md @@ -0,0 +1,126 @@ +## Volume Attachment Errors Volume in VMware Environment + +If you deployed Palette in a VMware vSphere environment and are experiencing volume attachment errors for the MongoDB +pods during the upgrade process, it may be due to duplicate resources in the cluster causing resource creation errors. +Palette versions between 4.0.0 and 4.3.0 are affected by a known issue where cluster resources are not receiving unique +IDs. Use the following steps to correctly identify the issue and resolve it. + +### Debug Steps + +1. Open up a terminal session in an environment that has network access to the Kubernetes cluster. + +2. Configure kubectl CLI to connect to the self-hosted Palette or VerteX's Kubernetes cluster. Refer to the + [Access Cluster with CLI](../clusters/cluster-management/palette-webctl.md) for additional guidance. +3. Verify the MongoDB pods are not starting correctly by issuing the following command. + + ```shell + kubectl get pods --namespace=hubble-system --selector='app=spectro,role=mongo' + ``` + + ```shell {4} hideClipboard + NAME READY STATUS RESTARTS AGE + mongo-0 2/2 Running 0 17h + mongo-1 2/2 Running 0 17h + mongo-2 0/2 ContainerCreating 0 16m + ``` + +4. Inspect the pod that is not starting correctly. Use the following command to describe the pod. Replace `mongo-2` + with the name of the pod that is not starting. + + ```shell + kubectl describe pod mongo-2 --namespace=hubble-system + ``` + +5. Review the event output for any errors. If an error related to the volume attachment is present, proceed to the next + step. + + ```shell hideClipboard + Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning FailedAttachVolume 106s (x16 over 18m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-94cbb8f5-9145-4b18-9bf9-ee027b64d0c7" : volume attachment is being deleted + Warning FailedMount 21s (x4 over 16m) kubelet Unable to attach or mount volumes: unmounted volumes=[mongo-data], unattached volumes=[spectromongokey kube-api-access-sz5lz mongo-data spectromongoinit spectromongopost]: timed out waiting for the condition + ``` + +6. The remaining steps may need to be performed on all MongoDB pods and their associated Persistent Volume (PV), and + Persistent Volume Claim (PVC). Do each step sequentially for each MongoDB pod that is encountering the volume + attachment error. + + :::warning + + Only do the steps for one MongoDB pod at a time to prevent data loss. Wait for the pod to come up correctly before + proceeding to the next pod. + + ::: + +7. Delete the PVC associated with the MongoDB pod. Replace `mongo-2` with the name of the pod that is not starting. + + ```shell + kubectl delete pvc mongo-data-mongo-2 --namespace=hubble-system + ``` + +8. Delete the PV associated with the MongoDB pod. Use the following command to list all PVs and find the PV associated + with the MongoDB pod you started with. In this example, the PV associated with `mongo-2` is + `pvc-94cbb8f5-9145-4b18-9bf9-ee027b64d0c7`. Make a note of this name. + + ```shell + kubectl get pv | grep 'mongo-data-mongo-2' + ``` + + ```shell hideClipboard + pvc-94cbb8f5-9145-4b18-9bf9-ee027b64d0c7 20Gi RWO Delete Bound hubble-system/mongo-data-mongo-2 spectro-storage-class 18h + ``` + +9. Using the PV name from the previous step, delete the PV. + + ```shell + kubectl delete pv pvc-94cbb8f5-9145-4b18-9bf9-ee027b64d0c7 + ``` + + :::tip + + The kubectl command may hang after issuing the delete command, press `Ctrl+C` to exit the command and proceed to the + next step. + + ::: + +10. Delete the MongoDB pod that was not starting correctly. Replace `mongo-2` with the name of the pod that is not + starting. + + ```shell + kubectl delete pod mongo-2 --namespace=hubble-system + ``` + +11. Wait for the pod to come up correctly. Use the following command to verify the pod is up and available. + + ```shell + kubectl get pods --namespace=hubble-system --selector='app=spectro,role=mongo' + ``` + + ```shell {4} hideClipboard + NAME READY STATUS RESTARTS AGE + mongo-0 2/2 Running 0 18h + mongo-1 2/2 Running 0 18h + mongo-2 2/2 Running 0 68s + ``` + + :::warning + + Once the pod is in the **Running** status, wait for at least five minutes for the replication to complete before + proceeding with the other pods. + + ::: + + Palette will proceed with the upgrade and attempt to upgrade the remaining MongoDB pods. Repeat the steps for each + of the MongoDB pods that are not starting correctly due to the volume attachment error. + + The upgrade process will continue once all MongoDB pods are up and available. Verify the new nodes deployed + successfully by checking the status of the nodes. Log in to the + [system console](../enterprise-version/system-management/system-management.md#access-the-system-console), navigate + to left **Main Menu** and select **Enterprise Cluster**. The **Nodes** tab will display the status of the nodes in + the cluster. + + ![A view of three nodes in a healthy status](/troubleshootig_palette-upgrade_nodes-healthy.webp) + + If you continue to encounter issues, contact our support team by emailing + [support@spectrocloud.com](mailto:support@spectrocloud.com) so that we can provide you with further guidance. diff --git a/docs/docs-content/enterprise-version/upgrade/upgrade-notes.md b/docs/docs-content/enterprise-version/upgrade/upgrade-notes.md index 308e24e090..2eae950a9e 100644 --- a/docs/docs-content/enterprise-version/upgrade/upgrade-notes.md +++ b/docs/docs-content/enterprise-version/upgrade/upgrade-notes.md @@ -52,6 +52,15 @@ Palette 4.0 includes the following major enhancements that require user interven ### Upgrade with VMware +:::warning + +A known issue impacts all self-hosted Palette instances older then 4.4.14. Before upgrading a Palette instance with +version older than 4.4.14, ensure that you execute a utility script to make all your cluster IDs unique in your +Persistent Volume Claim (PVC) metadata. For more information, refer to the +[Troubleshooting Guide](../../troubleshooting/enterprise-install.md#non-unique-vsphere-cns-mapping). + +::: + From the Palette system console, click the **Update version** button. Palette will be temporarily unavailable while system services update. diff --git a/docs/docs-content/enterprise-version/upgrade/upgrade-vmware/airgap.md b/docs/docs-content/enterprise-version/upgrade/upgrade-vmware/airgap.md index 2bacc5fb0a..79c932df76 100644 --- a/docs/docs-content/enterprise-version/upgrade/upgrade-vmware/airgap.md +++ b/docs/docs-content/enterprise-version/upgrade/upgrade-vmware/airgap.md @@ -9,13 +9,16 @@ keywords: ["self-hosted", "enterprise"] --- This guide takes you through the process of upgrading a self-hosted airgap Palette instance installed on VMware vSphere. - -:::warning - Before upgrading Palette to a new major version, you must first update it to the latest patch version of the latest minor version available. Refer to the [Supported Upgrade Paths](../upgrade.md#supported-upgrade-paths) section for details. +:::warning + +If you are upgrading from a Palette version that is older than 4.4.14, ensure that you have executed the utility script +to make the CNS mapping unique for the associated PVC. For more information, refer to the +[Troubleshooting guide](../../../troubleshooting/enterprise-install.md#non-unique-vsphere-cns-mapping). + ::: If your setup includes a PCG, you must also diff --git a/docs/docs-content/enterprise-version/upgrade/upgrade-vmware/non-airgap.md b/docs/docs-content/enterprise-version/upgrade/upgrade-vmware/non-airgap.md index 7aab23c688..4229176347 100644 --- a/docs/docs-content/enterprise-version/upgrade/upgrade-vmware/non-airgap.md +++ b/docs/docs-content/enterprise-version/upgrade/upgrade-vmware/non-airgap.md @@ -8,13 +8,15 @@ tags: ["palette", "self-hosted", "vmware", "non-airgap", "upgrade"] keywords: ["self-hosted", "enterprise"] --- -This guide takes you through the process of upgrading a self-hosted Palette instance installed on VMware vSphere. +This guide takes you through the process of upgrading a self-hosted Palette instance installed on VMware vSphere. Before +upgrading Palette to a new major version, you must first update it to the latest patch version of the latest minor +version available. Refer to the [Supported Upgrade Paths](../upgrade.md#supported-upgrade-paths) section for details. :::warning -Before upgrading Palette to a new major version, you must first update it to the latest patch version of the latest -minor version available. Refer to the [Supported Upgrade Paths](../upgrade.md#supported-upgrade-paths) section for -details. +If you are upgrading from a Palette version that is older than 4.4.14, ensure that you have executed the utility script +to make the CNS mapping unique for the associated PVC. For more information, refer to the +[Troubleshooting guide](../../../troubleshooting/enterprise-install.md#non-unique-vsphere-cns-mapping). ::: diff --git a/docs/docs-content/release-notes/known-issues.md b/docs/docs-content/release-notes/known-issues.md index 83b134353b..25b7a7d81f 100644 --- a/docs/docs-content/release-notes/known-issues.md +++ b/docs/docs-content/release-notes/known-issues.md @@ -16,6 +16,7 @@ The following table lists all known issues that are currently active and affecti | Description | Workaround | Publish Date | Product Component | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ---------------------------- | +| Persistent Volume Claims (PVCs) metadata do not use a unique identifier for self-hosted Palette clusters. This causes incorrect Cloud Native Storage (CNS) mappings in vSphere, potentially leading to issues during node operations and cluster upgrades. | Refer to the [Troubleshooting section](../troubleshooting/enterprise-install.md#non-unique-vsphere-cns-mapping) for guidance. | Sep 13, 2024 | Self-hosted | | Third-party binaries downloaded and used by the Palette CLI may become stale and incompatible with the CLI. | Refer to the [Incompatible Stale Palette CLI Binaries](../troubleshooting/automation.md#scenario---incompatible-stale-palette-cli-binaries) troubleshooting guide for workaround guidance. | September 11, 2024 | CLI | | An issue with Edge hosts using [Trusted Boot](../clusters/edge/trusted-boot/trusted-boot.md) and encrypted drives occurs when TRIM is not enabled. As a result, Solid-State Drive and Nonvolatile Memory Express drives experience degraded performance and potentially cause cluster failures. This [issue](https://github.com/kairos-io/kairos/issues/2693) stems from [Kairos](https://kairos.io/) not passing through the `--allow-discards` flag to the `systemd-cryptsetup attach` command. | Check out the [Degreated Performance on Disk Drives](../troubleshooting/edge.md#scenario---degreated-performance-on-disk-drives) troubleshooting guide for guidance on workaround. | September 4, 2024 | Edge | | The AWS CSI pack has a [Pod Disruption Budget](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) (PDB) that allows for a maximum of one unavailable pod. This behavior causes an issue for single-node clusters as well as clusters with a single control plane node and a single worker node where the control plane lacks worker capability. [Operating System (OS) patch](../clusters/cluster-management/os-patching.md) updates may attempt to evict the CSI controller without success, resulting in the node remaining in the un-schedulable state. | If OS patching is enabled, allow the control plane nodes to have worker capability. For single-node clusters, turn off the OS patching feature. | September 4, 2024 | Cluster, Packs | @@ -48,7 +49,6 @@ The following table lists all known issues that are currently active and affecti | K3s version 1.27.7 has been marked as _Deprecated_. This version has a known issue that causes clusters to crash. | Upgrade to a newer version of K3s to avoid the issue, such as versions 1.26.12, 1.28.5, and 1.27.11. You can learn more about the issue in the [K3s GitHub issue](https://github.com/k3s-io/k3s/issues/9047) page. | April 14, 2024 | Packs, Clusters | | When deploying a multi-node AWS EKS cluster with the Container Network Interface (CNI) , the cluster deployment fails. | A workaround is to use the AWS VPC CNI in the interim while the issue is resolved. | April 14, 2024 | Packs, Clusters | | If a Kubernetes cluster deployed onto VMware is deleted, and later re-created with the same name, the cluster creation process fails. The issue is caused by existing resources remaining inside the PCG, or the System PCG, that are not cleaned up during the cluster deletion process. | Refer to the [VMware Resources Remain After Cluster Deletion](../troubleshooting/pcg.md#scenario---vmware-resources-remain-after-cluster-deletion) troubleshooting guide for resolution steps. | April 14, 2024 | Clusters | -| In a VMware environment, self-hosted Palette instances do not receive a unique cluster ID when deployed, which can cause issues during a node repave event, such as a Kubernetes version upgrade. Specifically, Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) will experience start problems due to the lack of a unique cluster ID. | To resolve this issue, refer to the [Volume Attachment Errors Volume in VMware Environment](../troubleshooting/palette-upgrade.md#volume-attachment-errors-volume-in-vmware-environment) troubleshooting guide. | April 14, 2024 | Self-Hosted | | Day-2 operations related to infrastructure changes, such as modifying the node size and count, when using MicroK8s are not taking effect. | No workaround is available. | April 14, 2024 | Packs, Clusters | | If a cluster that uses the Rook-Ceph pack experiences network issues, it's possible for the file mount to become and remain unavailable even after the network is restored. | This a known issue disclosed in the [Rook GitHub repository](https://github.com/rook/rook/issues/13818). To resolve this issue, refer to pack documentation. | April 14, 2024 | Packs, Edge | | Edge clusters on Edge hosts with ARM64 processors may experience instability issues that cause cluster failures. | ARM64 support is limited to a specific set of Edge devices. Currently, Nvidia Jetson devices are supported. | April 14, 2024 | Edge | @@ -61,6 +61,7 @@ for information on the fix version and the date the issue was resolved. | Description | Publish Date | Product Component | Fix Version | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | ------------------- | ----------- | +| In a VMware environment, self-hosted Palette instances do not receive a unique cluster ID when deployed, which can cause issues during a node repave event, such as a Kubernetes version upgrade. Specifically, Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) will experience start problems due to the lack of a unique cluster ID. | April 14, 2024 | Self-Hosted | 4.4.14 | | The Palette agent fails to pull Helm charts from private registries when initializing Edge clusters. | July 25, 2024 | Edge | 4.4.12 | | The VerteX enterprise cluster is unable to complete backup operations. | June 6, 2024 | VerteX | 4.4.11 | | An issue prevents RKE2 and Palette eXtended Kubernetes (PXK) on version 1.29.4 from operating correctly with Canonical MAAS. | July 20, 2024 | Packs, Clusters | 4.4.11 | diff --git a/docs/docs-content/troubleshooting/enterprise-install.md b/docs/docs-content/troubleshooting/enterprise-install.md index 3a743b5d54..b6b736e8f6 100644 --- a/docs/docs-content/troubleshooting/enterprise-install.md +++ b/docs/docs-content/troubleshooting/enterprise-install.md @@ -77,3 +77,82 @@ following steps to restart the management pod. ```shell hideClipboard pod "mgmt-f7f97f4fd-lds69" deleted ``` + +## Non-unique vSphere CNS Mapping + +In Palette and VerteX releases 4.4.8 and earlier, Persistent Volume Claims (PVCs) metadata do not use a unique +identifier for self-hosted Palette clusters. This causes incorrect Cloud Native Storage (CNS) mappings in vSphere, +potentially leading to issues during node operations and upgrades. + +This issue is resolved in Palette and VerteX releases starting with 4.4.14. However, upgrading to 4.4.14 will not +automatically resolve this issue. If you have self-hosted instances of Palette in your vSphere environment older than +4.4.14, you should execute the following utility script manually to make the CNS mapping unique for the associated PVC. + +### Debug Steps + +1. Ensure your machine has network access to your self-hosted Palette instance with `kubectl`. Alternatively, establish + an SSH connection to a machine where you can access your self-hosted Palette instance with `kubectl`. + +2. Log in to your self-hosted Palette instance System Console. + +3. In the **Main Menu**, click **Enterprise Cluster**. + +4. In the cluster details page, scroll down to the **Kubernetes Config File** field and download the kubeconfig file. + +5. Issue the following command to download the utility script. + + ```bash + curl --output csi-helper https://software.spectrocloud.com/tools/csi-helper/csi-helper + ``` + +6. Adjust the permission of the script. + + ```bash + chmod +x csi-helper + ``` + +7. Issue the following command to execute the utility script. Replace the placeholder with the path to your kubeconfig + file. + + ```bash + ./csi-helper --kubeconfig= + ``` + +8. Issue the following command to verify that the script has updated the cluster ID. + + ```bash + kubectl describe configmap vsphere-cloud-config --namespace=kube-syste + ``` + + If the update is successful, the cluster ID in the ConfigMap will have a unique ID assigned instead of + `spectro-mgmt/spectro-mgmt-cluster`. + + ```hideClipboard {12} + Name: vsphere-cloud-config + Namespace: kube-system + Labels: component=cloud-controller-manager + vsphere-cpi-infra=config + Annotations: cluster.spectrocloud.com/last-applied-hash: 17721994478134573986 + + Data + ==== + vsphere.conf: + ---- + [Global] + cluster-id = "896d25b9-bfac-414f-bb6f-52fd469d3a6c/spectro-mgmt-cluster" + + [VirtualCenter "vcenter.spectrocloud.dev"] + insecure-flag = "true" + user = "example@vsphere.local" + password = "************" + + [Labels] + zone = "k8s-zone" + region = "k8s-region" + + + BinaryData + ==== + + Events: + ``` diff --git a/docs/docs-content/troubleshooting/palette-upgrade.md b/docs/docs-content/troubleshooting/palette-upgrade.md index fcf71f653e..1f1adeb3ff 100644 --- a/docs/docs-content/troubleshooting/palette-upgrade.md +++ b/docs/docs-content/troubleshooting/palette-upgrade.md @@ -41,134 +41,6 @@ Error: UPGRADE FAILED: failed to create resource: admission webhook "validate.ng 4. Restart the upgrade process. -## Volume Attachment Errors Volume in VMware Environment - -If you deployed Palette in a VMware vSphere environment and are experiencing volume attachment errors for the MongoDB -pods during the upgrade process, it may be due to duplicate resources in the cluster causing resource creation errors. -Palette versions between 4.0.0 and 4.3.0 are affected by a known issue where cluster resources are not receiving unique -IDs. Use the following steps to correctly identify the issue and resolve it. - -### Debug Steps - -1. Open up a terminal session in an environment that has network access to the Kubernetes cluster. - -2. Configure kubectl CLI to connect to the self-hosted Palette or VerteX's Kubernetes cluster. Refer to the - [Access Cluster with CLI](../clusters/cluster-management/palette-webctl.md) for additional guidance. - -3. Verify the MongoDB pods are not starting correctly by issuing the following command. - - ```shell - kubectl get pods --namespace=hubble-system --selector='app=spectro,role=mongo' - ``` - - ```shell {4} hideClipboard - NAME READY STATUS RESTARTS AGE - mongo-0 2/2 Running 0 17h - mongo-1 2/2 Running 0 17h - mongo-2 0/2 ContainerCreating 0 16m - ``` - -4. Inspect the pod that is not starting correctly. Use the following command to describe the pod. Replace `mongo-2` with - the name of the pod that is not starting. - - ```shell - kubectl describe pod mongo-2 --namespace=hubble-system - ``` - -5. Review the event output for any errors. If an error related to the volume attachment is present, proceed to the next - step. - - ```shell hideClipboard - Events: - Type Reason Age From Message - ---- ------ ---- ---- ------- - Warning FailedAttachVolume 106s (x16 over 18m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-94cbb8f5-9145-4b18-9bf9-ee027b64d0c7" : volume attachment is being deleted - Warning FailedMount 21s (x4 over 16m) kubelet Unable to attach or mount volumes: unmounted volumes=[mongo-data], unattached volumes=[spectromongokey kube-api-access-sz5lz mongo-data spectromongoinit spectromongopost]: timed out waiting for the condition - ``` - -6. The remaining steps may need to be performed on all MongoDB pods and their associated Persistent Volume (PV), and - Persistent Volume Claim (PVC). Do each step sequentially for each MongoDB pod that is encountering the volume - attachment error. - - :::warning - - Only do the steps for one MongoDB pod at a time to prevent data loss. Wait for the pod to come up correctly before - proceeding to the next pod. - - ::: - -7. Delete the PVC associated with the MongoDB pod. Replace `mongo-2` with the name of the pod that is not starting. - - ```shell - kubectl delete pvc mongo-data-mongo-2 --namespace=hubble-system - ``` - -8. Delete the PV associated with the MongoDB pod. Use the following command to list all PVs and find the PV associated - with the MongoDB pod you started with. In this example, the PV associated with `mongo-2` is - `pvc-94cbb8f5-9145-4b18-9bf9-ee027b64d0c7`. Make a note of this name. - - ```shell - kubectl get pv | grep 'mongo-data-mongo-2' - ``` - - ```shell hideClipboard - pvc-94cbb8f5-9145-4b18-9bf9-ee027b64d0c7 20Gi RWO Delete Bound hubble-system/mongo-data-mongo-2 spectro-storage-class 18h - ``` - -9. Using the PV name from the previous step, delete the PV. - - ```shell - kubectl delete pv pvc-94cbb8f5-9145-4b18-9bf9-ee027b64d0c7 - ``` - - :::tip - - The kubectl command may hang after issuing the delete command, press `Ctrl+C` to exit the command and proceed to the - next step. - - ::: - -10. Delete the MongoDB pod that was not starting correctly. Replace `mongo-2` with the name of the pod that is not - starting. - - ```shell - kubectl delete pod mongo-2 --namespace=hubble-system - ``` - -11. Wait for the pod to come up correctly. Use the following command to verify the pod is up and available. - - ```shell - kubectl get pods --namespace=hubble-system --selector='app=spectro,role=mongo' - ``` - - ```shell {4} hideClipboard - NAME READY STATUS RESTARTS AGE - mongo-0 2/2 Running 0 18h - mongo-1 2/2 Running 0 18h - mongo-2 2/2 Running 0 68s - ``` - - :::warning - - Once the pod is in the **Running** status, wait for at least five minutes for the replication to complete before - proceeding with the other pods. - - ::: - - Palette will proceed with the upgrade and attempt to upgrade the remaining MongoDB pods. Repeat the steps for each - of the MongoDB pods that are not starting correctly due to the volume attachment error. - -The upgrade process will continue once all MongoDB pods are up and available. Verify the new nodes deployed successfully -by checking the status of the nodes. Log in to the -[system console](../enterprise-version/system-management/system-management.md#access-the-system-console), navigate to -left **Main Menu** and select **Enterprise Cluster**. The **Nodes** tab will display the status of the nodes in the -cluster. - -![A view of three nodes in a healthy status](/troubleshootig_palette-upgrade_nodes-healthy.webp) - -If you continue to encounter issues, contact our support team by emailing -[support@spectrocloud.com](mailto:support@spectrocloud.com) so that we can provide you with further guidance. - ## Mongo DNS Configmap Value is Incorrect In VMware vSphere VerteX installations, if you encounter an error during the upgrade process where the MongoDB DNS diff --git a/docs/docs-content/vertex/upgrade/upgrade-notes.md b/docs/docs-content/vertex/upgrade/upgrade-notes.md index b1bc1882ca..4807e45a9f 100644 --- a/docs/docs-content/vertex/upgrade/upgrade-notes.md +++ b/docs/docs-content/vertex/upgrade/upgrade-notes.md @@ -21,3 +21,10 @@ Addressing this Mongo DNS issue will prevent system pods from experiencing _Cras After the upgrade, if Enterprise Cluster backups are stuck, refer to the [Enterprise Backup Stuck](../../troubleshooting/enterprise-install.md#scenario---enterprise-backup-stuck) troubleshooting guide for resolution steps. + +## Upgrade with VMware + +A known issue impacts all self-hosted Palette instances older then 4.4.14. Before upgrading an Palette instance with +version older than 4.4.14, ensure that you execute a utility script to make all your cluster IDs unique in your +Persistent Volume Claim (PVC) metadata. For more information, refer to the +[Troubleshooting Guide](../../troubleshooting/enterprise-install.md#non-unique-vsphere-cns-mapping). diff --git a/docs/docs-content/vertex/upgrade/upgrade-vmware/airgap.md b/docs/docs-content/vertex/upgrade/upgrade-vmware/airgap.md index 18898c5e14..dd402a056b 100644 --- a/docs/docs-content/vertex/upgrade/upgrade-vmware/airgap.md +++ b/docs/docs-content/vertex/upgrade/upgrade-vmware/airgap.md @@ -9,13 +9,15 @@ keywords: ["self-hosted", "vertex"] --- This guide takes you through the process of upgrading a self-hosted airgap Palette VerteX instance installed on VMware -vSphere. +vSphere. Before upgrading Palette VerteX to a new major version, you must first update it to the latest patch version of +the latest minor version available. Refer to the [Supported Upgrade Paths](../upgrade.md#supported-upgrade-paths) +section for details. :::warning -Before upgrading Palette VerteX to a new major version, you must first update it to the latest patch version of the -latest minor version available. Refer to the [Supported Upgrade Paths](../upgrade.md#supported-upgrade-paths) section -for details. +If you are upgrading from a Palette VerteX version that is older than 4.4.14, ensure that you have executed the utility +script to make the CNS mapping unique for the associated PVC. For more information, refer to the +[Troubleshooting guide](../../../troubleshooting/enterprise-install.md#non-unique-vsphere-cns-mapping). ::: diff --git a/docs/docs-content/vertex/upgrade/upgrade-vmware/non-airgap.md b/docs/docs-content/vertex/upgrade/upgrade-vmware/non-airgap.md index 78a26464a3..4c9d117c7b 100644 --- a/docs/docs-content/vertex/upgrade/upgrade-vmware/non-airgap.md +++ b/docs/docs-content/vertex/upgrade/upgrade-vmware/non-airgap.md @@ -9,13 +9,16 @@ keywords: ["self-hosted", "vertex"] --- This guide takes you through the process of upgrading a self-hosted Palette VerteX instance installed on VMware vSphere. - -:::warning - Before upgrading Palette VerteX to a new major version, you must first update it to the latest patch version of the latest minor version available. Refer to the [Supported Upgrade Paths](../upgrade.md#supported-upgrade-paths) section for details. +:::warning + +If you are upgrading from a Palette VerteX version that is older than 4.4.14, ensure that you have executed the utility +script to make the CNS mapping unique for the associated PVC. For more information, refer to the +[Troubleshooting guide](../../../troubleshooting/enterprise-install.md#non-unique-vsphere-cns-mapping). + ::: If your setup includes a PCG, you must also