spike: e2e smoke test over a CAPI ephemeral cluster

[enxebre]

As a dev I'd like to have some smoke tests to run on PR to validate I'm not breaking e2e expectations.
/area provider/cluster-api

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[detiber]

/remove-lifecycle stale

[ncdc]

I'm interested in seeing this happen - can we brainstorm & enumerate things that would be needed for this to be realized?

• What needs to happen to be able to test this against different cloud providers (AWS, Azure, etc.)?
• What behaviors should we test? How "large" do we want to go when testing scale up? (I assume this will likely be influenced by quota limits in our cloud accounts)
• How long (wall clock) is a reasonable amount of time for a smoke test to take, since the ask is to make it PR-blocking?
• Do we want to consider separate periodic jobs that are broader / take longer to run?

[enxebre]

That's great @ncdc, we are planning on doing efforts to make this happen as well.

WRT your points above, I think we need to answer what would it take for us to run this suite https://github.com/kubernetes/kubernetes/tree/master/test/e2e/autoscaling against the capi provider in GCP/AWS/Azure?
TODO: enumerate the technical impediments.
TODO: enumerate cloud infra costs impediments.

If we can overcome the TODOs above we could run periodic jobs for the CAPI provider against those clouds as in https://testgrid.k8s.io/sig-autoscaling-cluster-autoscaler#Summary&show-stale-tests=. That suite seems take ~4h https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-autoscaling/1291216872313720836

Since the test suite above takes ~4h we'll need a different smoke test suite for PRs.
Ideally we'd run this against any CAPI supported provider e.g AWS/Azure/GCP and also against a fast ephemeral CAPI environment e.g libvirt/docker/kubemark.
To accelerate the job we could use intentionally reduced timeouts for things like scale-down-delay-after-add, scale-down-delay-after-delete, scale-down-delay-after-failure, scale-down-unneeded-time... This is fine, it's a smoke test and real values would be use in the periodic jobs.

If there's hard impediments to run against the clouds on PRs we could start by only running the job against libvirt/docker/kubemark.
In terms of what tests we want to run I'd start simple as we have nothing today:
Get the plumbing in place.
Run a trivial smoke test suite e.g expose machineSets to autoscaler, run targeted workload watch it scale out, delete workload, watch it scale down.
Then we go from there.

[detiber]

I don't necessarily think that cost is an impediment wrt cloud infra, but there are likely several other cloud infra related impediments:

• availability of accounts to use for specific accounts (ideally boskos managed)
  • we'd likely need additional aws accounts added to boskos for this purpose to avoid contention with the CAPA jobs for the existing pool of aws accouunts
  • Azure, vsphere, openstack, digitalocean, packet, and possibly other sig-sponsored providers do not have any boskos managed accounts available today
• quota limits on available accounts
  • I don't expect this is likely an issue for boskos managed gce accounts, since the current autoscaler tests are consuming them
  • The aws accounts that have previously been made available to boskos have the default account limits, so we'll need to ensure that any accounts used for this purpose have appropriate quota limits increased prior to use
  • Do we know what the max instance count we are looking at for the test suite? That should be sufficient for guessing initial quota requirements?

We could potentially look at alternatives to boskos-managed accounts, but I think I personally would prefer that route, since we can have a janitor process ensure cleanup of resources if tests fail or are otherwise abandoned before cleaning up after themselves and wouldn't require a separate process to try and mark/sweep resources.

All that said, while it would be nice to have testing against actual cloud providers, I'm not sure it's necessarily needed. Since the interaction is through Cluster API core resources, I think we'd get sufficient signal using any provider. CAPD is probably a good initial target, since it can run in prow, but it could potentially run into resource limits at higher scale and be susceptible to noisy neighbor flakiness.

@ncdc @vincepri we've talked about potentially having a "mock" or "fake" provider in the past, this might be a good use case for that. envtest from controller-runtime might make a decent base for building such a provider.

[elmiko]

WRT your points above, I think we need to answer what would it take for us to run this suite https://github.com/kubernetes/kubernetes/tree/master/test/e2e/autoscaling against the capi provider in GCP/AWS/Azure?
TODO: enumerate the technical impediments.
TODO: enumerate cloud infra costs impediments.

i've spent several weeks going over these tests and imo i think we need to refactor them to be more provider agnostic. currently they have a heavy reliance on gcp/gke specific infrastructure. i would like to see us take the core of these tests and make the provider implementation more generic.

i think if we can convert those tests to use a more generic provider, with a clean abstracted interface, we will make it easier for provider-specific teams to add their tests (i am talking about autoscaler in-tree providers here). for the CAPI specific tests then, we would just need to create our provider implementation and then we could leverage all the clouds covered by CAPI.

so far, i have started breaking down the e2e tests in a separate repo. my intent is to create a library which can be consumed by providers to create testing suites. my thinking is that if the tests can be contained into something that is easily usable by the larger community, it will be easier to migrate those changes if the need arise (eg if the community wants to have them centralized here or in k/k). sadly, i don't have much to share yet i am still working around some issues with configuring the external package and breaking some hard dependencies on k/k.

[elmiko]

@ncdc @vincepri we've talked about potentially having a "mock" or "fake" provider in the past, this might be a good use case for that. envtest from controller-runtime might make a decent base for building such a provider.

i think this is an interesting idea. it shouldn't be too difficult to build a mock which could present the CAPI objects back to the autoscaler. i do wonder about the node and pod detection pieces though.

[ncdc]

cc @benmoss

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[elmiko]

this issue is going to take time to properly address. we have had some good discussions at the sig autoscaler and sig testing meetings and i believe we have a plan to move forward. i am still working on a KEP to describe the overall action, and @benmoss has proposed #3651 , which adds the framework we will use for e2e testing.
there are still some details to finalize in terms of how and where the tests would run and what type of default configuration would be used.
/remove-lifecycle rotten

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[elmiko]

i am still interested in seeing this happen, the work has been slow going.
/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[elmiko]

i have been continuing to hack on this locally. recently there has been some interesting movement in the cluster-api project around using kubemark for testing in e2e (see kubernetes-sigs/cluster-api#4566).

my current progress is around building a capi cluster with the hollow nodes and then attempting to run e2e tests for the autoscaler on top of that. i think the kubemark approach will bear more fruit than the basic capd approach mainly due to the way that the node resources (cpu, mem, etc) are propagated. in capd, the way docker handles these limits does not apply into the node, wheres it appears that kubemark is doing this properly. having these resources be accurately represented is key to testing the autoscaler in this topology.

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[elmiko]

i am still keenly interested in seeing this move forward. i have had several configuration issues that are blocking me from getting the kubemark cluster (mentioned in my previous comment) working. regardless, i continue to push on this. i have a feeling that once we merge the kubemark testing changes into capi, then we can start to setup automation to try out the autoscaler.

/remove-lifecycle stale

[elmiko]

i am working towards making this a reality but it is taking some time to get there. we now have a published kubemark provider for cluster-api and i am adding a feature which will allow us to specify the node resources. once that patch is in the kubemark provider, we should be able to start creating more meaningful tests with the autoscaler and cluster-api.

no estimate on time frame for landing all of this, but i am working towards pushing several pieces forward. will update as progress is made.

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[elmiko]

slow progress is continuing on this. we have released an updated version of the kubemark provider with support for custom resource definitions. This update also supports scale from zero for the provider. The next steps are to update the clusterapi provider for the autoscaler to support the new features.

still no estimate for completion, but the next steps will involve transforming the test work done by @benmoss to run on top of the capi provider with kubemark enabled.

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[elmiko]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[elmiko]

i am still working towards this, although it does have a glacial pace ;)

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[elmiko]

/remove-lifecycle rotten

i'm not sure if we are making progress towards having this smoke test in the autoscaler, although the topic has come up at sig meetings and i believe there might be some support for doing it. i would like to share some work that is happening on the capi kubemark provider, this PR kubernetes-sigs/cluster-api-provider-kubemark#69 proposes a test suite for the kubemark provider that uses the autoscaler for testing. i think it will be important work for creating a repeatable pattern that could be used with tests in this repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.