EKS: Kubectl Lambda Function Doesn't Support Regional STS Endpoints

[hakenmt]

Describe the bug

When deploying a private EKS cluster using an STS VPC endpoint, the Kubetctl function access to STS fails because it attempts to use the global endpoint. This is either from using a V1 SDK, the endpoint is explicitly set, or the environment variable is set

Expected Behavior

All SDKs and functions not under user control should default to using regional STS endpoints.

Current Behavior

The function uses the global endpoint.

Reproduction Steps

Deploy a private EKS cluster with an STS VPC endpoint.

Possible Solution

Set the environment variable in the Lambda function definition. This is my current workaround:

        FixUpLambdaFunctions("@aws-cdk--aws-eks.KubectlProvider");

        private void FixUpLambdaFunctions(string name)
        {
            IConstruct resourceProviderNestedStack = this.Node.TryFindChild(name);

            if (resourceProviderNestedStack != null)
            {
                NestedStack nestedStack = resourceProviderNestedStack as NestedStack;

                if (nestedStack != null)
                {
                    Function lambda = nestedStack.Node.TryFindChild("Handler") as Function;

                    if (lambda != null)
                    {
                        lambda.AddEnvironment("AWS_STS_REGIONAL_ENDPOINTS", "regional");
                    }

                    IConstruct provider = nestedStack.Node.TryFindChild("Provider");
                    Function onEvent = provider.Node.TryFindChild("framework-onEvent") as Function;

                    if (onEvent != null)
                    {
                        onEvent.AddEnvironment("AWS_STS_REGIONAL_ENDPOINTS", "regional");
                    }
                }
            }
        }

Additional Information/Context

No response

CDK CLI Version

2.138.0

Framework Version

No response

Node.js Version

v20.9.0

OS

darwin

Language

.NET

Language Version

No response

Other information

No response

[pahud]

I saw this from the doc:

All new SDK major versions releasing after July 2022 will default to regional. New SDK major versions might remove this setting and use regional behavior. To reduce future impact regarding this change, we recommend you start using regional in your application when possible.

I think we need to find out what is the SDK version of the kubectl provider you use in the lambda. Are you able to find out that?

According to this, the runtime should be python3.10. And the SDK version for python3.10 lambda runtime and boto3 version at this moment in us-east-1 is 1.34.42, which was released in Feb 2024 and I think it should include that already? My questions:

1. which region are you deploying to?
2. can you verify your kubectl lambda provider is using python3.10 runtime?

Thanks.

[hakenmt]

1/Deploying in us-east-1.
2/Yes, the function provided in the construct is using python3.10 as the runtime.

My understanding is that all new MAJOR versions should be defaulting to using regional endpoints per the documentation, but I'm not sure how that's interpreted for boto3 since it's still v1.x.x.

[pahud]

Have discussed with @tim-finnigan about boto/botocore#2577 (comment)

Setting it to regional would still be required and recommended. I guess we should figure out how to do that from custom resource framework.

[pahud]

OK we do have this in cluster resource provider

aws-cdk/packages/aws-cdk-lib/aws-eks/lib/cluster-resource-provider.ts

Lines 69 to 71 in b196b13

	environment: {
	AWS_STS_REGIONAL_ENDPOINTS: 'regional',
	...props.environment,

But not added here
https://github.com/aws/aws-cdk/blob/b196b13b0b8a54dcacadf87fdbe744772a6e6c4d/packages/aws-cdk-lib/aws-eks/lib/kubectl-provider.ts#L139C28-L139C46

I'll submit a PR for that.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[aws-cdk-automation]

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.