Launch Templates for Managed Machine Pools

[richardcase]

Launch Templates for Managed Machine Pools

Table of Contents

• Glossary
• Summary
• Motivation
  • Goals
  • Non-Goals/Future Work
• Proposal
  • User Stories
    • Story 1
    • Story 2
    • Story 3
    • Story 4
    • Story 5
  • Requirements
    • Functional Requirements
    • Non-Functional Requirements
  • Implementation Details/Notes/Constraints
  • Security Model
  • Risks and Mitigations
• Alternatives
  • New AWSLaunchTemplate CRD & Controller
    • Benefits
    • Downsides
    • Decision
• Upgrade Strategy
• Additional Details
  • Test Plan
  • Graduation Criteria

Glossary

• CAPA - Cluster API Provider AWS.
• Launch Template - a configuration template that is used to configure an AWS EC2 instance when its created.
• ASG - an auto-scale group that represents a pool of EC2 instances that can scale up & down automatically.

Summary

Currently, with CAPA we have 2 varieties of Machine Pools implemented called AWSMachinePool and AWSManagedMachinePool. Each variety has a differing level of support for launch templates.

The AWSMachinePool is used to create an ASG who's EC2 instances are used as worker nodes for the Kubernetes cluster. The specification for AWSMachinePool exposes settings that are ultimately used to create a EC2 launch template (and version of it thereafter) via the AWSLaunchTemplate field and struct:

// AWSLaunchTemplate specifies the launch template and version to use when an instance is launched.
// +kubebuilder:validation:Required
AWSLaunchTemplate AWSLaunchTemplate `json:"awsLaunchTemplate"`

(source)

The AWSManagedMachinePool is used to create a EKS managed node group which results in an AWS managed ASG being created that utilises AWS managed EC2 instances. In the spec for AWSManagedMachinePool we expose details of the pool to create but we don't support using a launch template, and we don't automatically create launch templates (like we do for AWSMachinePool). There have been a number of requests from users of CAPA that have wanted to use AWSManagedMachinePool but we don't expose required functionality that only comes with using launch templates.

This proposal outlines changes to CAPA that will introduce new capabilities to utilise launch templates for AWSManagedMachinePool and brings its functionality in line with AWSMachinePool.

Motivation

We are increasingly hearing requests from users of CAPA that a particular feature / configuration option isn't exposed by CAPAs implementation of managed machine pools (i.e. AWSManagedMachinePool) and on investigation the feature is available via a launch template (nitro enclaves or placement as an example). It some instances, users of CAPA have had to use unmanaged machine pools (i.e. AWSMachinePool) instead.

The motivation is to improve consistency between the2 varieties of machine pools and expose to the user features of launch templates.

Note: it may not be completely consistent in the initial implementation as we may need to deprecate some API definitions over time but the plan will be to be eventually consistent ;)

Goals

• Consistent API to declare a use launch templates for AWSMachinePool and AWSManagedMachinePool
• Single point of reconciliation of launch templates
• Guide to the deprecation of certain API elements in AWSManagedMachinePool

Non-Goals/Future Work

• Add non-existent controller unit tests for AWSMachinePool and AWSManagedMachinePool

Proposal

At a high level, the plan is to:

1. Add a new AWSLaunchTemplate field to AWSManagedMachinePoolSpec that uses the existing AWSLaunchTemplate struct. For example:

// AWSLaunchTemplate specifies the launch template and version to use when an instance is launched. This field 
// will become mandatory in the future and its recommended you use this over fields AMIType,AMIVersion,InstanceType,DiskSize,InstanceProfile.
// +optional
AWSLaunchTemplate AWSLaunchTemplate `json:"awsLaunchTemplate"`

2. Update the comments on the below fields of AWSManagedMachinePoolSpec to indicate that the fields is deprecated and that AWSlaunchTemplate should be used.
   • AMIVersion
   • AMIType
   • DiskSize
   • InstanceType
3. Add new LaunchTemplateID and LaunchTemplateVersion fields to AWSManagedMachinePoolStatus to store details of the launch template and version used.
4. Add a new LaunchTemplateVersion field to AWSMachinePoolStatus to store the version of the launch template used.
5. Refactor the code from the AWSMachinePool controller that reconciles AWSLaunchTemplate into a common location so that it can be shared.
6. Update the controller for AWSManagedMachinePool to use the AWSLaunchTemplate reconciliation logic.
7. Add checks in the AWSManagedMachinePool create/update validation webhooks that stops users specifying AWSLaunchTemplate if fields AMIType,AMIVersion,InstanceType,DiskSize,InstanceProfile are set
8. Add warning logs to the AWSManagedMachinePool create/update validation webhooks if fields AMIType,AMIVersion,InstanceType,DiskSize,InstanceProfile stating that these fields will be deprecated in the future and that AWSLaunchTemplate should be used instead

An area that is undecided upon is should we auto convert the AMIType,AMIVersion,InstanceType,DiskSize,InstanceProfile fields if specified into a AWSLaunchTemplate. We should investigate this as part of implementation.

10. Update the cluster templates that use AWSManagedMachinePool so that they use AWSLaunchTemplate
11. Update the API version roundtrip tests for v1alpha4<->v1beta1 conversions of AWSManagedMachinePool
12. Update the EKS e2e tests to add an additional test step where we create an additional managed machine pool using AWSLaunchTemplate.
13. Update any relevant documentation
14. Release note must mention that "action is required" in the future, as fields are being deprecated.
15. As part of the implementation, we should investigate how in the f

User Stories

Story 1

AS a CAPA user
I want to create a managed machine pool using a launch template
So that I can use functionality from the AWS launch template

Story 2

As a CAPA user
I want to have consistency between managed and unmanaged machine pools
So that I can choose which to use based on whether I want managed and not based on missing functionality

Story 3

As a CAPA user
I want to ensure launch templates are versioned
So that I can see a history of the changes in the console

Story 4

As a CAPA user
I want to be able to set the maximum number of versions to keep for launch templates
So that I don't have to worry about cleaning up old versions.

Story 5

As a CAPA user
I want to be able to use the output of a bootstrap provider in my launch template
So that i can bootstrap Kubernetes on the nodes

Requirements

Functional Requirements

FR1: CAPA MUST support using launch templates with non-managed ASG based machine pools (i.e. AWSMachinePool).

FR2: CAPA MUST support using launch templates with EKS managed nodegroup based machine pools (i.e. AWSManagedMachinePool).

FR3: CAPA MUST provide a consistent declarative API to expose Launch Template configuration to the machine pool implementations.

FR4: CAPA MUST manage the lifecycle of a launch template in AWS based on its declaration.

FR5: CAPA MUST version launch templates in AWS.

FR6: CAPA MUST allow keeping a configurable number of previous versions of launch templates.

FR7: CAPA MUST validate the declarations for AWSLaunchTemplate

Non-Functional Requirements

NFR1: CAPA MUST provide logging and tracing to expose the progress of reconciliation of AWSLaunhcTemplate.

NFR2: CAPA MUST raise events at important milestones during reconciliation.

NFR3: CAPA MUST requeue where possible and not wait during reconciliation so as to free up the reconciliation loop

NFR4: CAPA must have e2e tests that cover usage of launch templates with BOTH variants of machine pools.

Implementation Details/Notes/Constraints

The code in reconcileLaunchTemplate must be refactored into a package that can be use by the AWSManagedMachinePool controller as well. We could think about shifting more of this functionality into the "ec2" service.

Cleaning up old versions of launch templates is currently handled by PruneLaunchTemplateVersions which is sufficient for this change. We may want to make the minimum number of versions to keep configurable in the future but this can be covered by a different change.

Security Model

There are no changes required to the security model. Access to the required CRDs is already declared for the controllers and as we are not adding any new kinds this doesn't need to change.

No change is required to the AWS permissions the controller requires for reconciliation.

Risks and Mitigations

The risk is that we are being constrained by the existing API definition used in unmanaged machine pools. This may raise unforeseen issues.

Alternatives

New AWSLaunchTemplate CRD & Controller

The idea is that a AWSLaunchTemplate CRD would be created with an associated controller. The controller would then be responsible for reconciling the definition and managing the lifecycle of launch templates on AWS.

Benefits

• Single point of reconciliation and lifecycle management of launch templates in AWS.
• Separate lifecycle per launch template. So, we can change the number of previous instances to keep etc.

Downsides

• Additional complexity of orchestrating the creation of the launch template with the bootstrap data. The machine pool reconcilers would need to wait for the bootstrap data and the launch template.
• Would require deprecation of fields in 2 CRDs (i.e both machine pool varieties).

Decision

As AWSMachinePool already managed launch templates, it was felt that we should follow the same approach for consistency and it would be a smaller change.

We can revisit the idea of a separate launch template kind in the future. The proposed change in this proposal will not preclude implementing this alternative in the future.

Upgrade Strategy

The changes we are making to AWSManagedMachinePool are optional. Therefore, current users do not have to use the new AWSLaunchTemplate field. On upgrading there will be a new log entry written that informs the user that certain fields will be deprecated in the future.

Additional Details

Test Plan

• There are currently no controller unit tests for the machine pools in CAPA. We do need to add tests, but this can be done as part of a separate change.
• The EKS e2e tests will need to be updated so that a managed machine pool is created with a launch template specified.

Graduation Criteria

With this proposal, we are planning to deprecate a number of fields on AWSManagedMachinePool and as this is a beta level API this means:

• We must support the beta API for 9 months or 3 releases (whichever is longer). See rule 4a

The machine pools feature is also marked as experimental in Cluster API and as such this has to be explicitly enabled via a feature flag. Even though it is experimental, we should adhere to the Kubernetes rules on API deprecation.

So based on the change, we will need to support the deprecated fields until December 2022 or CAPA v1.5...whichever is longer. So probably December 2022