codepipeline: Cross region pipelines don't work after updating to 2.80.0 or newer

[makemefeelgr8]

Describe the bug

A pipeline fails to deploy a stack into another region after updating to 2.80.0 (or newer).

Expected Behavior

I expect the deployment to succeed.

Current Behavior

There's a pipeline located in us-west-2 that deploys a stack into a different region (eu-central-1).

import * as cdk from 'aws-cdk-lib';

const myPipeline = new cdk.pipelines.CodePipeline(..., {
    //...
    cliVersion: '2.80.0',
    selfMutation: true
});
const myStage = new MyStage()
myPipeline.addStage(myStage);

// ...

class MyStage extends Stage {
...
    const myStack = new MyStack(..., { env: { ..., region: 'eu-central-1'}} );
}

It works great on aws cdk v2.79.1. Here's the log:

MyStack (MyStack): deploying... [5/7]
Retrieved account ID 12345 from disk cache
Assuming role 'arn:aws:iam::12345:role/cdk-xxxxxxxx-deploy-role-12345-eu-central-1'.

After updating to v2.80.0, the pipeline fails with the folowing error:

MyStack (MyStack): deploying... [6/7]
Retrieved account ID 12345 from disk cache
Call failed: getParameter({"Name":"/cdk-bootstrap/xxxxxxxx/version"}) => User: arn:aws:sts::12345:assumed-role/cdk-xxxxxxxx-deploy-role-12345-us-west-2/aws-cdk-root is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:eu-central-1:12345:parameter/cdk-bootstrap/xxxxxxxx/version because no identity-based policy allows the ssm:GetParameter action (code=AccessDeniedException)

at Deployments.validateBootstrapStackVersion (/usr/local/lib/node_modules/aws-cdk/lib/index.js:415:12432)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Deployments.deployStack (/usr/local/lib/node_modules/aws-cdk/lib/index.js:415:5836)
at async Object.deployStack2 [as deployStack] (/usr/local/lib/node_modules/aws-cdk/lib/index.js:415:153172)
at async /usr/local/lib/node_modules/aws-cdk/lib/index.js:415:136968

As you can see, in 2.79.1, the pipeline automatically assumed the role which is required to access SSM in a different region. But, starting from 2.80.0, the pipeline does not do that anymore. Instead, it tries to access resources in eu-central-1 with its default us-west-2 role, which obviously fails.

Reproduction Steps

Provide a self-contained, concise snippet of code that can be used to reproduce the issue.
For more complex issues provide a repo with the smallest sample that reproduces the bug.
Avoid including business logic or unrelated code, it makes diagnosis more difficult.
The code sample should be an SSCCE. See http://sscce.org/ for details. In short, please provide a code sample that we can copy/paste, run and reproduce.

A repo? A SSCEE standard? Do you really expect me to spend half of my day to create a repo and make sure the code fits requirements? You're the ones who broke pipelines- go fix them yourselves!
Seriously, though, the issue is as easy to reproduce as it gets- just make a pipeline that deploys a stack into a different region.

Possible Solution

Take a look at what happened in 2.80.0 and rollback features 1 by 1 to get to the point when it works.

Additional Information/Context

No response

CDK CLI Version

2.80.0 (build bbdb16a)

Framework Version

No response

Node.js Version

16.17.0

OS

Windows 11; Amazon Linux 2 x86_64 version 4

Language

Typescript

Language Version

No response

Other information

No response

[peterwoodworth]

I'm not able to reproduce this @makemefeelgr8, this setup is working for me post 2.80.0. We adjusted the bootstrap template and increased the bootstrap template version, I wonder if somehow there's a discrepancy when attempting to fetch the version? Make sure your regions are bootstrapped on an appropriate version, and make sure your specified cliVersion follows the guidance seen in its docstring

If this doesn't help, I will need an SSCCE. It helps the team save time when we can work on fixing bugs rather than spending more time reproducing issues to find the root cause.

[makemefeelgr8]

this setup is working for me post 2.80.0

@peterwoodworth Thanks for trying it out! I really appreciate that. Let me do my best to help you with reproducing the issue. While this example looks like something one shouldn't actually do, it's the easiest way to reproduce the issue. Here are the steps:

1. Create a github repo. You'll need to use it as CodePipelineSource, so do whatever necessary on the github side to enable the integration with AWS and make it accessible for the pipeline.
2. Copy the following example into the empty folder. You might also need to grab a default package.json with a few libraries. Specify "aws-cdk": "2.80.0" there, to make sure it pulls the exact version we're interested in.
3. Replace "your account id" with account id, and fill in the "replace with your own connection" part too.
4. Commit, and push into your empty repo
5. Now that we have the repo ready, do the npm i to get libraries and deploy the app by using npx aws-cdk@2.80.0 deploy --all.
6. Open the developer console, navigate to the "Code pipeline" page and wait until the pipeline finishes running.
7. (optional) run it once again by pressing "Release change" button. It'll run just fine (as there are no real changes to deploy, it'll just skip everything)
8. At this point, the DependencyStack is already broken. If it was an actual stack with some constructs there, we could have changed one of the constructs, commit and push changes, trigger the pipeline and see it fail in the "UpdatePipeline" step. But I'm too lazy for that, so instead let us reproduce the issue locally. Run npx aws-cdk@2.80.0 deploy --all --force. It's the same command we used to deploy the app, but with an extra --force parameter to make sure it does not skip stuff.
9. See the error. At this point, it fails with User blah-blah is not authorized to perform: ssm:GetParameter blah-blah.

And here's the code:

import { App, Stack, StackProps, Stage, StageProps } from 'aws-cdk-lib';
import { LinuxBuildImage } from 'aws-cdk-lib/aws-codebuild';
import { Pipeline } from 'aws-cdk-lib/aws-codepipeline';
import { CodeBuildStep, CodePipeline, CodePipelineSource } from 'aws-cdk-lib/pipelines';
import { Construct } from 'constructs';

const account = '1234567890'; // your account id

// a stage with 1 empty stack
class MyTestStage extends Stage {
  constructor(scope: Construct, id: string, props: StageProps) {
    super(scope, id, props);

    new Stack(this, 'MyTestStackFromStage', {
      env: {
        account: account,
        region: 'eu-central-1'
      }
    });
  }
}

class MyTestPipelineStack extends Stack {
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);

    const myPipeline = new CodePipeline(this, 'MyTestCodePipeline', {
      cliVersion: '2.80.0',
      selfMutation: true,
      synth: new CodeBuildStep('MyTestCodeBuildStep', {
        buildEnvironment: {
          buildImage: LinuxBuildImage.AMAZON_LINUX_2_4,
          privileged: true
        },
        primaryOutputDirectory: 'cdk.out',
        installCommands: ['n 16.17.1'],
        commands: ['npm i', 'npx aws-cdk@2.80.0 synth'],
        input: CodePipelineSource.connection(
          // replace with your own connection
          'your_repo/your_repo',
          'your_branch',
          {
            connectionArn: 'your_connection_arn',
            triggerOnPush: false
          }
        )
      }),
      codePipeline: new Pipeline(this, 'MyTestInnerCodePipeline', {
        crossAccountKeys: false
      }),
    });
    myPipeline.addStage(new MyTestStage(this, 'MyTestStage', props));
  }
}

// and now deploy all of the above
const myTestApp = new App();
const myPipeline = new MyTestPipelineStack(myTestApp, 'MyTestPipelineStack', {
  env: {
    account: account,
    region: 'us-west-2'
  }
});
myPipeline.addDependency(
  new Stack(myTestApp, 'MyTestDependencyStackIn-eu-central-1', {
    env: {
      account: account,
      region: 'eu-central-1'
    }
  })
);

[makemefeelgr8]

@peterwoodworth is this one not going to be addressed? Should I just close this issue and open a new one?

[pahud]

Hi @makemefeelgr8 I am helping you to investigate this issue and sorry for the delayed response. As it's been a while, does this issue still exist in the latest CDK 2.111.0 ? If yes, please let me know and I'll try my best to help.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.