[Proposal] Improve Local Storage Management

[vishh]

A note to reviewers: A detailed design proposal will be posted once the overall design in this proposal has been accepted by the community. So kindly hold on to specific design questions or suggestions that are not relevant to the overall high level design for local storage.

cc @kubernetes/sig-storage-proposals @kubernetes/sig-node-proposals @kubernetes/sig-apps-proposals @kubernetes/sig-scheduling-proposals

cc @msau42

For kubernetes/enhancements#121

TODO:

• Handle scheduling of runtime primary partition capacity
• Propose design choices for mitigating or avoiding Disk IO issues for system daemons, container logs and writable layer. EmptyDir IO isolation is out of scope because local PV can be used instead.

[kow3ns]

Will this be global for all PVCs for StatefulSet going forward? Also, we will be depending on reasonable collection timeouts to ensure that users have time to collect data from Volumes after deletion (assuming they have a need to do so)?

[msau42]

By default, the PVC will need to be deleted by the user to retain similar behavior as today. We are looking into an "inline" PVC feature that can automatically delete the PVCs when the StatefulSet gets destroyed. I'll update this to clarify that.

Regarding the retention policy, the PV can be changed to use the "Retain" policy if users need to collect data after deletion.

[kow3ns]

So we are depending on priority and preemption to be implemented prior to this?

[davidopp]

It seems like what's described in this section also relies on some variant of #7562 / #30044 being implemented, as today there is no notion of a local PV (beyond the experimental HostPath volume type which doesn't do what's needed here).

[msau42]

Yes, this proposal is also covering this new local PV type.

[msau42]

@kow3ns regarding priority and preemption, it is not a strict requirement for this feature, but will make the workflow smoother. There are also plans to implement this soon.

[F21]

Can you elaborate on what support for "random access storage devices only" means? Does this mean using RAM as storage?

[kow3ns]

Good question. I took this to mean DASD (i.e. this will not work for Tape drives or other sequential access storage media). Is this not the case?

[msau42]

Yes, it means not supporting tape.

[davidopp]

This could be written more concisely as "Provide usage isolation for non-shared partitions" which would also make it more parallel with the Goal "Provide storage usage isolation for shared partitions"

[davidopp]

s/during which/after which/

[davidopp]

s/IO/IOPs/

[davidopp]

s/foo/fooc/

[msau42]

foo is correct because it's referring to the pod, and not the container.

[lpabon]

Quota feature assumes an appropriate supporting file system is being used. A large part of the distributed storage systems require raw (no file system) storage. How would that be managed? Would a raw partition be crated by a logical manager?

[msau42]

We don't plan to support raw partitions as a primary partition. Secondary partitions can have block level support though.

[davidopp]

Add a link to the LimitRange user guide that explains this.
(BTW did you mean "burst" rather than "bust"?)

[davidopp]

s/intent/intend/
BTW I'm not clear about the connection between prohibiting a minimum storage requirement in LimitRange and overcommit. Won't the scheduler prohibit overcommit regardless of what storage requirement you give for the EmptyDir (regardless of whether it's set manually or via LimitRange)?

[derekwaynecarr]

i assume this means we will just not allow a request a limit for volumes.

[davidopp]

There is currently no notion of tainting PVs, only nodes. Can you say more about what semantics you are expecting for tainting a PV?

[msau42]

We would like to evict pods that are using tainted PVs, unbind the PVC, and reschedule the pod so that it can bind to a different PV. I think everything after the eviction could be handled by a separate controller.

[davidopp]

It seems like what's described in this section also relies on some variant of #7562 / #30044 being implemented, as today there is no notion of a local PV (beyond the experimental HostPath volume type which doesn't do what's needed here).

[davidopp]

So there is always a 1:1 correspondence between PV and partition on a secondary device?

[msau42]

Yes, this will let the cluster administrator decide how they want to provision local storage. They can have one partition per disk for IOPS isolation, or if sharing is ok, then create multiple partitions on a device.

[lpabon]

I'm guessing this is based on the technology of the underlying filesystem. If not, then I think this depends a lot on some type of logical volume manager. If not only two things can happen: 1. a secondary partition is the entire disk, 2: A lot of disk fragmentation. I think more information on how number 1 is done may shed more light on this model

[msau42]

Yes, it is up to the administrator to partition and create the filesystem first. And how that is done will depend on the partitioning tools (parted, mdadm, lvm, etc) available and which filesystems the administrator decides to use. From Kubernetes point of view, we will not handle creating partitions or filesystems.

[jsafrane]

it is up to the administrator to partition and create the filesystem first

That's very inconvenient for an admin. Also, when such PV gets Released, who/how removes the data there and puts it back to Available? We'd like to deprecate recycler as soon as possible.

IMO, some sort of simple dynamic provisioning would be very helpful and it's super simple with LVM. It should be pluggable though to work on all other platforms.

[msau42]

The current thought for the PV deletion workflow is to set the PV to Released phase, delete all the files (similar to how EmptyDir cleans up), delete the PV object, and then the addon daemonset will detect that the partition and then create the PV for it again.

So from an admin's point of view, the partitioning and fs setup is just a one time step whenever new disks are added. And for the use case that we are targeting, which requires whole disks for IOPs guarantees, the setup is simple: one partition across the whole disk, and create the filesystem on that partition.

As for LVM, I agree it is a simpler user model, but we cannot get IOPs guarantees from it, which is what customers we've talked to want the most. I don't think this design will prevent supporting an LVM-based model in the future though. I can imagine there can be a "storageType = lvm" option as part of the PV spec, and a dynamic provisioner can be written to use that to carve out LVs from a single VG. The scheduling changes that we have to make to support local PVs can still apply to a lvm-based volume. We're just not prioritizing it right now based on user requirements.

[krmayankk]

i agree with @jsafrane that we should have some default out of the box local disk PV provisioner and for default cases, we dont have to do a addon or some such thing. 90% use cases might be just simple use of local disks

[msau42]

Based on feedback we have gotten from customers and workloads team, it's the opposite. Most of the use cases require dedicated disks. We have not seen many requests for dynamic provisioning of shared disks. If you see some real use cases where an app wants to use persistent local storage (and all its semantics), but doesn't need performance guarantees, then I would be interested in hearing about them as well.

I do want to make sure that nothing in this proposal would prevent LVM and dynamic provisioning from being supported in the future. And that it will be able to take advantage of the scheduling and failure handling features we will be adding.

In terms of admin work, my hope is that the default addon will require a similar amount of admin intervention as the LVM model (configure the disk once in the beginning, the system takes care of the rest).

[kow3ns]

This should occur only when the PV backing the PVC is permanently unavailable. If a controller creates a new PVC and relaunches the Pod with that PVC, it will never be able to reuse the data on the old PV anyway. To simplify this for controller developers, when some policy is applied to indicate that K8s should "give up" on recovering a PV, can we just delete the PV, and set the status of the PVC to pending? This would reduce the complexity of the interaction with DaemonSet, StatefulSet, and any other controllers and local persistent storage.

[msau42]

This situation could also occur if the node has failed or can no longer fulfill other requested resources, for example, if other pods got scheduled and took up the cpu or memory needed.

The main concern with deleting the PV and keeping the PVC, is that it may not follow the retention policy. The user may want to recover data from the PV, but won't have the pod->PVC->PV binding anymore. As another alternative, we could remove the PVC->PV binding, and if the PV policy is retain, also add an annotation with the old pod, PVC information so the user can figure out which PV had their data.

[enisoc]

I like the idea of keeping the PVC and just removing the PVC->PV binding. If we expect the StatefulSet controller to modify the Pod to use a new PVC, that essentially means only the StatefulSet controller can perform the task of unblocking its unschedulable Pods. That in turn means that every controller needs to separately implement this behavior. For example, what if I have "stateless" Deployment Pods that want this behavior for their large caches on local PV?

If unblocking can be done without modifying the Pods to use a different PVC, then it leaves the door open to write a generic "local PV unbinding" controller that implements this behavior once for everyone who requests it via some annotation or field.

[msau42]

The generic PVC unbinding controller can monitor for this error condition, unbind the PVC, clean up the PV according to the reclaim policy, and then evict and reschedule the pods to force them to obtain a new PV.

[kow3ns]

How does this interact with kubectl logs? Right now we are aggregating and rolling stdout and stderr? Are you proposing that we use local storage instead of, or in addition to, the current K8s logging infra?

[msau42]

It should have no impact to kubectl logs. It's only changing the log rotation mechanism to be on a per container basis instead of on a node basis.

[kow3ns]

If you are targeting DBs and DFSs, and if a "taint" is really pertaining to a problem with the underlying storage media, I don't think anything in your target set will tolerate a taint. @davidopp shouldn't this be expressed by the controller in terms of declarative tolerations against node taints in any case. That is, don't I have to explicitly declare the taint as tolerated?

[kow3ns]

Instead of having every controller/operator watch for the appearance of taints on a node and delete Pods should we consider the following approach?

1. DBs and DFS should include a health check that causes the Pod to fail when the contained application can't write to storage media (Most, if not all, storage applications will fail on errors returned from fysnc/sync).
2. When the application monitoring the storage device decides that the mounted PVs are unrecoverable, it should delete the PVs and mark the Bound PVCs as pending. The policy deciding when to do this can be applied here. Note that, this is no scarier than having the controller make the decision to delete the PVC. In either case, once the Pod is disassociated from its local volume and launched with another, it can never be safely re-associated with the prior volume. Both cases also need a good story around snapshops and backup. I think that, as the device monitoring application is a node local agent, it can make a better decision about when to "give up" trying bind a Pod to a local mount.
3. As the volumes are deleted, we need not be concerned with the PVCs being fulfilled by this node unless it has volumes mounted on another, functional device.
4. When controllers/operators recreate the Pods, their existing PVCs must be Bound to volumes provided by another node.

If we take an approach that is closer to this we don't have to duplicate the watch logic in every controller/operator.

[msau42]

If I understand correctly, are you suggesting to leave it up to the application to handle local storage failures, since each application may have its own unique requirements and policies?

[kow3ns]

Sorry if I was not clear. I am saying the opposite. The "application monitoring the storage device" referred to above is based on the design statement that " Health is monitored by an external entity like the “Node Problem Detector” which is expected to place appropriate taints."
Rather than have every controller/operator attempt to heuristically guess when it should delete a PVC, might it not be better to have the "Node Problem Detector", kublet, or another local agent make a decision that the volume is no longer usable due to device failure, and to set the associated PVC back to pending. Perhaps using your suggestion above to retrain the volume for data recovery purposes. I can't think of a distributed storage application that will want to re-balance or re-replicate its data due to a temporary network partition or intermittent node failure. The only time, IMO, that we'd want a controller/operator to move a Pod with local PVs to a new node is if the storage device failed, or if the MTTR is so high that it might as well have. In the former case it might be best if a node local agent made the decision that the storage device is failed. In the latter case, we should at least consider having a global controller with a policy that can unbind local PVs from PVCs, rather than having every controller/operator have to implement its own policy.

[enisoc]

I don't think the controller (e.g. StatefulSet) should be responsible for deleting Pods. I think @kow3ns is also saying that if I'm reading him correctly.

My understanding is that regular Node taints are noticed and enforced by kubelet, which may evict the Pod if it doesn't tolerate the taint. Wouldn't it make sense for kubelet to also evict the Pod if it does not tolerate a taint on one of its local PVs?

If recreated with the same PVC, the Pod would remain unschedulable due to the taint on the PV. At this point, the problem is reduced to being the same as (7) above. In this way, both (7) and (8) can be handled without necessarily requiring any changes to StatefulSet or other controllers (if a generic controller can be implemented as suggested above).

[msau42]

Currently taints are only at the node level, but I think it could worth looking into expanding, as it already has a flexible interface for specifying per pod the tolerations and forgiveness for each taint. This workflow could also work for the case when the node fails or becomes unavailable. @davidopp

Then, when the pod gets evicted due to the taint, it reduces the problem to (7), as mentioned above.

[msau42]

It is also possible to implement this without taints, and instead add an error state to the PV, and have a controller monitor for the error state and evict pods that way. But using taints may be nice as a future enhancement to unify the API.

[kow3ns]

intent=intend

[derekwaynecarr]

to clarify, each empty dir backed volume will pick up this default capacity? so if a user had multiple empty dirs for some reason, each would get 1Gi?

[msau42]

Yes, it's the limit per emptydir. You bring up a good point though, that the "type: Pod" implies that it's for the whole pod. We can change it to "type: EmptyDir"

[derekwaynecarr]

do you worry users will get confused with this field as only being meaningful when the medium is disk and not memory?

[msau42]

It can be used for memory-backed emptydir too.

[enisoc]

I think there's a missing - here.

[enisoc]

Missing -.

[enisoc]

Is there a reason for doing it this way rather than:

limits:
  storage:
    logs: 500Mi
    overlay: 1Gi

?

[msau42]

It's a limitation in the LimitRange design. It doesn't support nesting of limits.

[enisoc]

Wouldn't the convention be volumeType?

[enisoc]

I like the idea of keeping the PVC and just removing the PVC->PV binding. If we expect the StatefulSet controller to modify the Pod to use a new PVC, that essentially means only the StatefulSet controller can perform the task of unblocking its unschedulable Pods. That in turn means that every controller needs to separately implement this behavior. For example, what if I have "stateless" Deployment Pods that want this behavior for their large caches on local PV?

If unblocking can be done without modifying the Pods to use a different PVC, then it leaves the door open to write a generic "local PV unbinding" controller that implements this behavior once for everyone who requests it via some annotation or field.

[enisoc]

I don't think the controller (e.g. StatefulSet) should be responsible for deleting Pods. I think @kow3ns is also saying that if I'm reading him correctly.

My understanding is that regular Node taints are noticed and enforced by kubelet, which may evict the Pod if it doesn't tolerate the taint. Wouldn't it make sense for kubelet to also evict the Pod if it does not tolerate a taint on one of its local PVs?

If recreated with the same PVC, the Pod would remain unschedulable due to the taint on the PV. At this point, the problem is reduced to being the same as (7) above. In this way, both (7) and (8) can be handled without necessarily requiring any changes to StatefulSet or other controllers (if a generic controller can be implemented as suggested above).

[enisoc]

Should this be storageType? Also, having both volumeType and storageType seems confusing. Not sure what else these could be called though.

[msau42]

Would storageLevel be better?

[jsafrane]

Were other options considered? Especially LVM would allow us to add / remove devices on hosts and RAID 0/1/5 per volume with very little overhead. With partitions, you must do all this manually.

[msau42]

In the case of persistent local storage, most of the use cases we have heard about prioritize performance and being able to use dedicated disks.

In addition, LVM is only available on Linux, so it could be difficult to use as a generic solution.

[lpabon]

I'm assuming a Primary and Secondary partitions are logical objects which can be implemented a multiple of ways. Do you mind elaborating on possible implementations?

[msau42]

Is this the kind of information you were looking for?

• Using an entire disk (this is the primary use case for persistent local storage)
• Adding multiple disks into a RAID volume
• Using LVM to carve out multiple logical partitions (if you don't need IOPs guarantees)

[lpabon]

Will performance include a QoS IOPS requirement for distributed storage systems?

[msau42]

The PVs will have to be created by the admin/addon that utilizes the entire disk to guarantee IOPs for performance use cases.

[dwlehman]

Not sure I understand correctly, but why do PVs have to be full-disk? Why not a properly aligned partition?

[msau42]

We're not requiring that the PV has to use the whole disk (the volume is created on a partition), but if you need IOPS guarantees, then it should be a dedicated disk. Especially for rotational disks, the IO will still end up being on a shared path at the device layer.

SSDs may offer high enough IOPS that you can share them.

[vishh]

As @msau42 mentioned, the API that kubernetes would consume is a logical partition. It can map to any storage duration (RAID, JBOD, etc.). We recommend not sharing spinning disks unless either the storage configuration or IOPS requirements permits sharing them.

[lpabon]

When they are exposed as PVs, are they created and available as a pool? Do you mind elaborating a little more on Secondary Partitions? Who creates them, how are they managed, what sizes, etc.

[msau42]

Yes, the workflow mentions this a little bit. I will also add it here.

There will be an addon daemonset that can discover all the secondary partitions on the local node and create PV objects for them. The capacity of the PV will be the size of the entire partition. We can provide a default daemonset that will look for the partitions under a known directory, and it is also possible to write your own addons for your own environment.

So the PVs will be statically created and not dynamically provisioned, but the addon can be long running and create new PVs as disks are added to the node.

[lpabon]

I'm assuming a Primary and Secondary partitions are logical objects which can be implemented a multiple of ways. Do you mind elaborating on possible implementations?

[lpabon]

How are the guarantees provided by the system? The FS?, logical volume?

[msau42]

For primary partitions, the node's local storage capacity will be exposed so that the scheduler can take into account a pod's storage limits and what nodes can satisfy that limit.

Then kubelet will monitor the storage usage of the emptydir volumes and containers so that they stay within their limits. If quota is supported, then it will use that.

[lpabon]

Quota feature assumes an appropriate supporting file system is being used. A large part of the distributed storage systems require raw (no file system) storage. How would that be managed? Would a raw partition be crated by a logical manager?

[lpabon]

Assuming the FS supports such a feature.

[msau42]

Yes, I will mention otherwise kubelet can only enforce soft limits.

[lpabon]

I know number 8 showed there will be a Health monitor, but how would it detect that the primary partition is unhealthy on number 9? What does it mean to be unhealthy?

[msau42]

Now that I think about it more, health monitoring is dependent on the environment and configuration, so an external monitor may be needed for both primary and secondary.

It can monitor at various layers depending on how the partitions are configured:
disk layer: look at SMART data
raid layer: look for complete raid failure (non-recoverable)

[chengyli]

How to do disk health monitoring if the Node is a VM and disk is a virtual disk? The smartctl or raid tools may not return correct data.

[msau42]

That's a good point. Because the partition configuration is very dependent on the environment, I think we cannot do any monitoring ourselves. Instead, we can define a method for external monitors to report errors, and also define how kubernetes will react to those errors.

[krmayankk]

Does our proposal/design require this health monitor. lets say in the default configuration, when there is no external health monitor, what is the behavior ?

[msau42]

The health monitor is not required. In that case, it will behave the same way that it does today, which is undefined.

[lpabon]

This part is a little bit confusing "His cluster administrator being aware....". Does that mean that this solution would require the administrator to take action or things may be incorrectly allocated?

[msau42]

In order to solve today's local storage isolation problem, pods should specify limits for their local storage usage. In the absence of that, the administrator has the option to specify defaults for the namespace. If neither of those two occur, then you just have the same issue today.

[lpabon]

The concern I have here is that it requires a lot of interaction with an administrator and the user. If I am "Bob", I'm just going to keep asking for more storage (1, then 2, then .. ). That would move the Pod from node to node satisfying the storage size request. I'm guessing... How different is this from the current model?

[msau42]

Yes there is a little bit of a trial and error going on here for Bob. But as an application developer, you will have to do this in order to size your apps appropriately. One goal that we're trying to achieve here is provide pods better isolation from other pods running on that node through storage isolation.

[lpabon]

I'm guessing this is based on the technology of the underlying filesystem. If not, then I think this depends a lot on some type of logical volume manager. If not only two things can happen: 1. a secondary partition is the entire disk, 2: A lot of disk fragmentation. I think more information on how number 1 is done may shed more light on this model

[lpabon]

Just to clear my confusion, these all are created by hand?

[msau42]

They could be created by hand, or if you put the partitions in a known directory, then the addon daemonset can discover the partitions and automatically create the PVs.

[lpabon]

@msau42 FYI, I appreciate the quick turnaround! 👏

[rootfs]

@msau42 here is a related use case for local storage handling from @fabiand
kubernetes/kubernetes#38606

[stp-ip]

*created

[stp-ip]

Logs should usually not accumulate as they should be collected to a central location.
--> no need for separation

Overlay FS data can be used, but for heavy use or increased storage needs we do recommend and provide emptyDirs and the new local PVs.
--> no need for separation

As emptyDirs might be used for caches and heavy IO it might makes sense to let this be separated from the planned root PV.

Complicating the Kubelet for logs and overlay doesn't seem to make sense. We should definitely think about the usage pattern of emptyDirs after local PVs are available.
Would we recommend local PV usage for heavy IO caches instead of emptyDirs? If yes, then we might leave emptyDirs inside the root PV and let the user know that for anything serious he might need to migrate away from emptyDirs.

Definitely needs to be clearly documented what use-cases each one solves.

[msau42]

Yes, since we don't plan to provide IOPS isolation for emptydir, then local PV should be used instead for those use cases. One question we have is are there use cases that need ephemeral IOPS guarantees that cannot be adapted to use local PVs? Do we need to look into an "inline" local PV feature where the PV gets created and destroyed with the pod?

[stp-ip]

Depends on the automation of creating and using local PVs.
EmptyDirs work great without having to involve the cluster admin.
Local PVs most likely need cluster admin intervention. Maybe not always, but it's not 100% automated.

The path I see as reasonable would be:

• Leave emptyDir as "best-effort" scratch pad.
• Recommend local PVs for guaranteed IOPS.
• First iteration having to use manual cluster admin action
• Iterate on automating local PVs to bring them closer to emptyDir and PDs aka provide local PVs via dynamic provisioning

This would lead to no huge complexity additions in the kubelet as root, emptyDir, log and overlay FS are kept on the primary partition in the first iteration.

As additional note:
Persistent Volume as name seems confusing especially, when we recommend it as IOPS guaranteed scratch pad. (Maybe: Local Disk?)

[msau42]

Good plan! LocalDisk as the actual volume plugin name sounds good.

[stp-ip]

For clarification. We would have:

• PersistentDisk (networked "unfailable" disk)
• emptyDir (shared temporary volume without guarantees)
• LocalDisk (local volume with guarantees, which might have some persistence)
• hostPath (local volume for testing)

• all the provider specific stuff, flexVolume, gitRepo and k8s API backed volumes.

[msau42]

We are planning to recommend using LocalDisk only through the PV/PVC interfaces for the following reasons:

• In failure scenarios, like the node failing, you may want to give up on the local disk and find a new one to use. You can do that by unbinding the PVC from the PV, instead of having to change the volume in the pod spec
• If you use local disk directly, it would be very similar to HostPath volumes, and have all its problems, where you have to specify the path, and understand the storage layout of the node, and understand that that particular volume can satisfy the pod's capacity needs. The PV interface hide those details.
• The PV interface gives a way to pool all the local volumes across the entire cluster and easily query for them, and find ones that will fit a pod's requirements.

[stp-ip]

Thanks for the clarification. Always great to have that documented.

So recommendation:
PD + LD using PVC
emptyDir + hostPath used directly

Small addition:
The notion it projects using PV/PVCs about LD being persistent could create some confusion.

[msau42]

I will update this doc to clarify that thanks! I agree the PV name could be misleading since the local disk can only offer semi-persistence, and has different semantics then normal PVs. I can add a section about the different semantics. Also, because of the different behavior, and its very targeted use cases, I want to make sure in the API layer, the user explicitly selects this option, and that they cannot use a local PV "accidentally".

[thockin]

/lgtm
/approve

[vishh]

This proposal has gotten amazing feedback. However it has reached a point where it's too large to continue discussing further. I'm merging this PR based on @thockin's approval. As we implement the features in this proposal, some aspects of this proposal can (and possibly will) change. Expect this proposal to evolve over the next year as local storage features get added to kubernetes.
If you have any further feedback for this proposal, kindly open up a separate patch updating the proposal directly.

If someone feels strongly about any pending comments, I'm happy to revert the merge and continue discussing if necessary.

[sky-big]

at present, local PVs have above question，PV, PVC bounded before schedulering，when bounded completed，scheduler select the node with PV node affinity，but now the node CPU, Mem not enough and so on，so the pod all the time schedule failed，so above question have plan to solve?

[msau42]

Yes, that is the limitation in the first phase. We hope to solve it in the next release, but no concrete ideas yet, we're just prototyping at this point. At a high level, the PVC binding needs to be delayed until a pod is scheduled, so that it can take into account all the other scheduling requirements of the pod.

[sky-big]

It's cool if solve local volume PV PVC delay bound,now my project team worry about the question,so not dare use local volume plugin because pod schedule fail all the time easily.

[msau42]

Yes, it will not work well right now in general purpose situations. But if you use the critical pod feature, or you run your workload that needs local storage first, then it that may work better. Still, the PVs may not get spread very well because the PV controller doesn't know that all these PVCs are replicas in the same workload. You may be able to work around the issue by labeling the PVs per workload.

[sky-big]

OK,Thanks. I pay close attention to V1.8(Scheduler predicate for prebound local PVCs#43640) .