AWS-EKS: Access Denied when installing a helm chart using a KubernetesManifest or HelmChart or EKS Blueprints

[srispl]

Describe the bug

I have created an EKS cluster using AWS CDK and wanted to install few tools.

Note: The EKS cluster is created in separate CloudFormation stack created through CDK and I am trying to install the tool in separate stack by importing the cluster created in another stack.

I tried to install the tool using EksBlueprint Addons , KubernetesManifest, HelmChart, but all 3 are ending up with Access Denied error.

If I move the same code to the stack where the EKS cluster is created or run the output YAML through kubectl, it is working as expected. It looks like the problem is due to the role created by the Constructs cannot assume the role of kubectl to apply the yaml on EKS cluster.

Expected Behavior

CDK should apply the resources to the EKS cluster.

Current Behavior

An error is thrown and the stack fails to deploy.

❌ Deployment failed: Error: The stack named BuildkitStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: Error: b'\nAn error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::XXXXXXXXXX:assumed-role/XXXXXX (This is the lambda role) is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::XXXXXXXXXX:role/XXXX (This is Kubectl role) \nUnable to connect to the server: getting credentials: exec: executable aws failed with exit code 255\n'

Reproduction Steps

1. Create an EKS cluster stack either with Blueprints or Helm in Stack1
2. Get the cluster name, endpoint, kubectlArn, version, clusterSecurityGroupId, openIdConnectProvider created in Stack1 and import the EKS cluster using ImportClusterProvider or aws_eks.Cluster.fromClusterAttributes construct in Stack2.
3. Create a simple configmap as Manifest file.
4. Pass the manifest file created in Step#3 as property to KubernetesManifest construct.
5. Build and deploy stack2.

Note: You can reproduce above steps by replacing step#3 with helmchart using HelmChart construct or any EKS blueprint addon using blueprints.addons construct.

Possible Solution

Solution#1:
The lambda role created should be added to the kubectl role as trust relationship, so it can assume kubectl role to apply the manifest.
Solution#2:
Output the Lambda role Arn from the construct with standard naming convention, so, the arn can be added in Stack#2.

Additional Information/Context

No response

CDK CLI Version

2.61.0

Framework Version

No response

Node.js Version

18.19.0

OS

Mac OS 14.2.1

Language

TypeScript

Language Version

TypeScript (5.0.4)

Other information

No response

[pahud]

Can you provide a minimal reproducible code snippet? Please note for all EKS BluePrints issue, you will need to report to https://github.com/aws-quickstart/cdk-eks-blueprints/issues instead.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[srispl]

Here is what the sample code.

Stack 1:

const cluster = new eks.Cluster(this, 'hello-eks', {
  version: eks.KubernetesVersion.V1_28,
  kubectlLayer: new KubectlV28Layer(this, 'kubectl'),
});

Stack 2:

const eksCluster = aws_eks.Cluster.fromClusterAttributes(this, `eks-cluster`, {
      clusterName: Fn.importValue('ClusterName_From_Stack1'),
      clusterSecurityGroupId: Fn.importValue('ClusterSecurityGroup_From_Stack1'),
      kubectlRoleArn: Fn.importValue('ClusterKubectlArn_From_Stack1'),
      clusterEndpoint: Fn.importValue('ClusterEndpoint_From_Stack1'),
    });

const installKSM = readYamlDocument(__dirname + '/serviceaccount-ksm.yaml');
//Link for Yaml: https://github.com/kubernetes/kube-state-metrics/blob/main/examples/standard/service-account.yaml

 const manifests = new Map<string, string>([['installKSM', [installKSM]]);

manifests.forEach((yamlDoc: string, itemName: string) => {
      const manifests = yamlDoc.split('---').map((e) => loadYaml(e));
      new KubernetesManifest(this, `${itemName}-manifests`, {
        cluster: eksCluster,
        manifest: manifests,
        overwrite: true,
      });
    });

[pahud]

const eksCluster = aws_eks.Cluster.fromClusterAttributes(this, `eks-cluster`, {
      clusterName: Fn.importValue('ClusterName_From_Stack1'),
      clusterSecurityGroupId: Fn.importValue('ClusterSecurityGroup_From_Stack1'),
      kubectlRoleArn: Fn.importValue('ClusterKubectlArn_From_Stack1'),
      clusterEndpoint: Fn.importValue('ClusterEndpoint_From_Stack1'),
    });

Looks like you are installing helm charts on an imported/existing eks cluster.

What if you create a new eks cluster in the cdk stack and install the charts on it. Will it work?

[srispl]

I tried new and existing cluster. It failed. It works when I run everything on same stack. But, I want to do in 2 different stacks.

[pahud]

OK I think you will need to reuse the kubectlLambdaRole created from the cluster stack.

check out this sample:

export class EksClusterStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props)

    const cluster = new eks.Cluster(this, 'demo-eks-cluster', {
      vpc: getDefaultVpc(this),
      defaultCapacity: 1,
      version: eks.KubernetesVersion.V1_28,
      kubectlLayer: new KubectlV28Layer(this, 'kubectlLayer'),
    });

    const kubectlProvider = cluster.stack.node.tryFindChild('@aws-cdk--aws-eks.KubectlProvider') as eks.KubectlProvider;

    new CfnOutput(this, 'ClusterName', { value: cluster.clusterName, exportName: 'ClusterName' });
    new CfnOutput(this, 'ClusterSecurityGroupId', { value: cluster.clusterSecurityGroupId, exportName: 'ClusterSecurityGroup' });
    new CfnOutput(this, 'KubectlRoleArn', { value: cluster.kubectlRole?.roleArn!, exportName: 'ClusterKubectlArn' });
    new CfnOutput(this, 'ClusterEndpoint', { value: cluster.clusterEndpoint, exportName: 'ClusterEndpoint' });
    new CfnOutput(this, 'KubectlProviderHandlerRole', { value: kubectlProvider.handlerRole.roleArn, exportName: 'KubectlProviderHandlerRoleArn' });
  }
}

export class EksAddOnStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props)

    const eksCluster = eks.Cluster.fromClusterAttributes(this, `eks-cluster`, {
      clusterName: Fn.importValue('ClusterName'),
      clusterSecurityGroupId: Fn.importValue('ClusterSecurityGroup'),
      kubectlRoleArn: Fn.importValue('ClusterKubectlArn'),
      clusterEndpoint: Fn.importValue('ClusterEndpoint'),
      kubectlLambdaRole: iam.Role.fromRoleArn(this, 'kubectlLambdaRole', Fn.importValue('KubectlProviderHandlerRoleArn')),
    });

    //Link for Yaml: https://github.com/kubernetes/kube-state-metrics/blob/main/examples/standard/service-account.yaml
    const parsedYaml = yaml.load(fs.readFileSync(path.join(__dirname + '/serviceaccount-ksm.yaml'), 'utf8')) as Map<string, any>;
  

    new eks.KubernetesManifest(this, 'installKSM', {
      cluster: eksCluster,
      manifest: [parsedYaml],
      overwrite: true,
      skipValidation: true,
    });

  }
}

This works for me.

Read #25835 (comment) for more details.

[trondhindenes]

afaik it's important to provide the kubectl layer in the imported cluster as well

[srispl]

@trondhindenes , Its already added since the lambdarole is pre-created while creating cluster.

@pahud ,Tried as suggested and the solution is working. Thanks for quick turn around! Good to close.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[bhatrohit]

OK I think you will need to reuse the kubectlLambdaRole created from the cluster stack.

check out this sample:

export class EksClusterStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props)

    const cluster = new eks.Cluster(this, 'demo-eks-cluster', {
      vpc: getDefaultVpc(this),
      defaultCapacity: 1,
      version: eks.KubernetesVersion.V1_28,
      kubectlLayer: new KubectlV28Layer(this, 'kubectlLayer'),
    });

    const kubectlProvider = cluster.stack.node.tryFindChild('@aws-cdk--aws-eks.KubectlProvider') as eks.KubectlProvider;

    new CfnOutput(this, 'ClusterName', { value: cluster.clusterName, exportName: 'ClusterName' });
    new CfnOutput(this, 'ClusterSecurityGroupId', { value: cluster.clusterSecurityGroupId, exportName: 'ClusterSecurityGroup' });
    new CfnOutput(this, 'KubectlRoleArn', { value: cluster.kubectlRole?.roleArn!, exportName: 'ClusterKubectlArn' });
    new CfnOutput(this, 'ClusterEndpoint', { value: cluster.clusterEndpoint, exportName: 'ClusterEndpoint' });
    new CfnOutput(this, 'KubectlProviderHandlerRole', { value: kubectlProvider.handlerRole.roleArn, exportName: 'KubectlProviderHandlerRoleArn' });
  }
}

export class EksAddOnStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props)

    const eksCluster = eks.Cluster.fromClusterAttributes(this, `eks-cluster`, {
      clusterName: Fn.importValue('ClusterName'),
      clusterSecurityGroupId: Fn.importValue('ClusterSecurityGroup'),
      kubectlRoleArn: Fn.importValue('ClusterKubectlArn'),
      clusterEndpoint: Fn.importValue('ClusterEndpoint'),
      kubectlLambdaRole: iam.Role.fromRoleArn(this, 'kubectlLambdaRole', Fn.importValue('KubectlProviderHandlerRoleArn')),
    });

    //Link for Yaml: https://github.com/kubernetes/kube-state-metrics/blob/main/examples/standard/service-account.yaml
    const parsedYaml = yaml.load(fs.readFileSync(path.join(__dirname + '/serviceaccount-ksm.yaml'), 'utf8')) as Map<string, any>;
  

    new eks.KubernetesManifest(this, 'installKSM', {
      cluster: eksCluster,
      manifest: [parsedYaml],
      overwrite: true,
      skipValidation: true,
    });

  }
}

This works for me.

Read #25835 (comment) for more details.

Tried this but still it threw same error. @pahud @srispl