Make resourceVersion parameter semantics consistent across all storage.Interface implementations

[jpbetz]

[This is part of effort to improve Apiserver Resource Version Semantics (as previously circulated with sig-apimachinery) and fix the "stale read" issue]

Make the minimum resourceVersion semantics consistent across all storage.Interface implementations by:

• enforcing "minimum RV" constraints in the etcd3 storage implementation per the storage.Interface documentation
• modifying the etcd3 List() implementation to use "minimum RV" semantics instead of "exact RV" semantics when RV>0

Important: the watch cache is enabled by default on the kube-apiserver for all but a select few types (e.g. Events, CRDs) and so while API clients do not get to decide when they make requests what semantics they get, they typically will get the watch cache semantics.

Before:

	Watch Cache (RV=0 & RV>0)	etcd3 RV=0	etcd RV>0
Get	"min RV"	quorum read, RV ignored	quorum read, RV ignored
List	"min RV"	quorum read, RV ignored	"exact RV"
GetToList	"min RV"	quorum read, RV ignored	quorum read, RV ignored
Watch	"min RV"	"min RV"	"min RV"

Note that "quorum read, RV ignored" behaves like "min RV" for all RVs that were retrieved from some previous operation on the store because quorum reads are guaranteed to be processed with the latest RV. So it appears from these semantics that "min RV" semantics are pervasive across both implementations and what we should converge to.

After:

	Watch Cache (RV=0 & RV>0)	etcd3 (RV=0 & RV>0)
Get	"min RV"	"min RV"
List	"min RV"	"min RV"
GetToList	"min RV"	"min RV"
Watch	"min RV"	"min RV"

The change from "exact RV" to "min RV" for the List operation is the most notable change here. But given that the watch cache is enabled by default for all but a select few types, and that the semantics change here would only be visible if providing a non-zero RV, we don't expect this change to be particularly risky/visible. And we believe it to be lower risk than leaving the semantic inconsistent across implementations, esp. given that clients don't necessarily have any way of knowing if the watch cache is enabled and what semantics to expect.

As part of this change we're introducing a ResourceVersionTooLargeError type to the storage interface so that both the watch cache and etcd3 return the same error type if unable to serve a requested resource version because it's too high. Previously the watch cache would return a timeout error in the case because it attempts to wait for 3 seconds to see if the "too high" resource version might appear if it waits a bit. I've done a quick review and it looks like this change is low risk, but if reviewers know of reasons why this might be breaking, please let me know.

The resource version option, when passed to a list call, is now consistently interpreted as the minimum allowed resource version.  Previously when listing resources that had the watch cache disabled clients could retrieve a snapshot at that exact resource version.  If the client requests a resource version newer than the current state, a TimeoutError is returned suggesting the client retry in a few seconds.  This behavior is now consistent for both single item retrieval and list calls, and for when the watch cache is enabled or disabled.

cc @cheftako @lavalamp @wojtek-t @liggitt @smarterclayton

[jpbetz]

/test pull-kubernetes-integration

[jpbetz]

/test pull-kubernetes-integration

[jpbetz]

/test pull-kubernetes-kubemark-e2e-gce-big

[wojtek-t]

This looks reasonable to me - I just left one comment.

But I would like at least someone else to take a look into that too.
@liggitt @smarterclayton

[jpbetz]

This is ready for another review pass.

[jpbetz]

friendly nudge

[wojtek-t]

Just two minor nits - other than that LGTM.
Though I would also wait for @liggitt to make a final look too.

[jpbetz]

/test pull-kubernetes-dependencies

[jpbetz]

Friendly nudge

[smarterclayton]

But given that the watch cache is enabled by default for all but a select few types, and that the semantics change here would only be visible if providing a non-zero RV, we don't expect this change to be particularly risky/visible

I'm really hoping this is true. Is watch cache enabled for CRDs by default?

Edit: looks like not - so are we really sure no one is relying on this?

[liggitt]

I'm really hoping this is true. Is watch cache enabled for CRDs by default?

Edit: looks like not - so are we really sure no one is relying on this?

Yes, it is. I know because we found several watch-cache-related flakes testing custom resources.

#46967, #64796, #78713

[jpbetz]

I'm really hoping this is true. Is watch cache enabled for CRDs by default?
Edit: looks like not - so are we really sure no one is relying on this?

There is no in-tree code relying on it. We can't entirely eliminate the possibility there is out-of-tree code relying on it. The conditions they'd have to meet are quite specific: (1) they've disabled the watch cache, or are using events, and (2) they're performing a List(... ResourceVersion: SomeSpecificRevision) request and (3) they cannot tolerate receiving a more recent revision than the one they requested.

[jpbetz]

/retest

[smarterclayton]

So if we had to bring back exact semantics? Say we were wrong, we break a large chunk of people, they need a fix. How do we make them whole?

[jpbetz]

So if we had to bring back exact semantics? Say we were wrong, we break a large chunk of people, they need a fix. How do we make them whole?

Worst case it that it's widespread (i.e. we really misread the situation), in which case we could revert the part of this change that applies to lists. We'd then have to regroup and figure out how to deprecate the old behavior and migrate.

Slightly less improbable is that there are a limited set of use cases, but they're important and cannot easily migrate on to "minimum RV" semantics. In this case I suppose we could introduce a separate param to GetOptions to support exact RV semantics. I'd really like to avoid doing that, but it would be an option..

[smarterclayton]

I updated the release note, please double check it for me to see if you agree.

[jpbetz]

I updated the release note, please double check it for me to see if you agree.

Release note looks great. Thanks!

[smarterclayton]

/lgtm
/approve

I hope I don't regret this.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jpbetz, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• staging/src/k8s.io/apiserver/pkg/storage/OWNERS [smarterclayton]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.