Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: PVCs in state: "Lost" #141

Closed
a-mcf opened this issue Dec 27, 2021 · 6 comments
Closed

Fix: PVCs in state: "Lost" #141

a-mcf opened this issue Dec 27, 2021 · 6 comments

Comments

@a-mcf
Copy link
Owner

a-mcf commented Dec 27, 2021

Change the storage class of NFS per the example here:
https://levelup.gitconnected.com/how-to-use-nfs-in-kubernetes-cluster-storage-class-ed1179a83817

Also, do some researching after this explosion occured after importing node1 into terraform and draining the node:
https://stackoverflow.com/questions/51358856/kubernetes-cant-delete-persistentvolumeclaim-pvc
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#storage-object-in-use-protection

@a-mcf
Copy link
Owner Author

a-mcf commented Dec 27, 2021

This could be useful for a longhorn migration:
https://itnext.io/debugging-kubernetes-pvcs-a150f5efbe95

@a-mcf
Copy link
Owner Author

a-mcf commented Dec 27, 2021

@a-mcf
Copy link
Owner Author

a-mcf commented Dec 28, 2021

I've correlated several timestamps and am now convinced that the issue is with flux. I opened an issue here:
fluxcd/flux2#2250

In the interim, I'm going to attempt to work around this by adding the uid to all of the PVs. Here is a list generated with PowerShell:

name                                                                                         uid
----                                                                                         ---
nfs-nextcloud-data-pvc                                                                       ce483ebd-5ce8-49c2-a398-47476a048904
nfs-nextcloud-mariadb-pvc                                                                    024e2ea7-0c5f-465c-924b-7e2568dc69b9
nfs-paperless-consume-pvc                                                                    ee378caa-03d4-4b48-be0e-d71280964ee0
nfs-paperless-data-pvc                                                                       7f259b6a-1792-4758-8973-6f84d13dd9db
nfs-photoprism-originals-pvc                                                                 89f3cb31-d72f-4728-a15b-d8c84a75c56c
nfs-photoprism-storage-pvc                                                                   042093b8-3e34-44fe-a471-85c229d4dda1
nfs-movies-pvc                                                                               58b94b45-626a-4cef-bc35-90836741db2d
nfs-home-assistant-config-pvc                                                                1efc6827-407a-4c83-b0de-b0dc6e853a8a
nfs-plex-pvc                                                                                 521a52c3-b3ab-4ef8-a987-8328b85f2f1e
nfs-tv-pvc                                                                                   2ffc1852-6df8-493a-b1be-0df293611ae7
nfs-paperless-export-pvc                                                                     cd9a9982-add1-47b2-9386-fbfe93f4677e
nfs-paperless-media-pvc                                                                      c8890daf-5015-45ff-92df-16478efe1f0e
nfs-photoprism-import-pvc                                                                    a4bca61a-acbb-49e0-9157-5cc2e80e9bed
nfs-music-pvc                                                                                935ea96e-1328-4c1a-bee3-053265d78f20
nfs-loki-pvc                                                                                 201fbf4f-f360-47c8-8e54-72a9f137e6e5
prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 589ae38d-b8e7-4ef6-b7f7-3dbcb9b07a5a
nfs-homer-pvc                                                                                8b2c1dd8-d696-4546-88be-5244960309fd

@a-mcf
Copy link
Owner Author

a-mcf commented Dec 28, 2021

All PVCs are now showing as bound. I'll keep this open so that I can:

  1. Follow up on PV.spec.ClaimRef.uid removed from PersistentVolume after reconciliation. fluxcd/flux2#2250
  2. Remove the hack (the uid:) for all of the PVs.

@a-mcf a-mcf changed the title Change storage class of NFS PVCs to nfs, or nfs-direct Fix: PVCs in state: "Lost" Dec 28, 2021
@a-mcf a-mcf closed this as completed Jan 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant